Repository navigation
Clarify --sql behavior in Dumpling CSV export docs - #24514
Open
doc-claw-bot wants to merge 3 commits into
Open
doc-claw-bot wants to merge 3 commits into
doc-claw-bot wants to merge 3 commits into
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
ti-chi-bot
Bot
added
missing-translation-status
2 of 12 tasks
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the
Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info
📝 Walkthrough
Merge Risk: ⚪ Minimal · up to
|
| Check name | Status | Explanation |
|---|---|---|
| Title check |  |
The title clearly and concisely describes the documentation change to clarify --sql behavior in Dumpling CSV exports. |
| Description check | |
The description includes the required change summary, affected version, reference links, AI involvement, and change-category checklist. The optional first-time contributor checklist is omitted appropr… |
| Docstring Coverage | |
No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0… |
| Linked Issues check | |
Check skipped because no linked issues were found for this pull request. |
| Out of Scope Changes check | |
Check skipped because no linked issues were found for this pull request. |
Full details: Description check
-
Explanation
The description includes the required change summary, affected version, reference links, AI involvement, and change-category checklist. The optional first-time contributor checklist is omitted appropriately.
Full details: Docstring Coverage
-
Explanation
No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0 files. (1 skipped: 1 unsupported.)
✨ Finishing Touches
-
🧪 Generate unit tests (beta)
-
- Create a new PR
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Comment @coderabbitai help to get the list of available commands.
doc-claw-bot
added
translation/done
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
ℹ️ Review info
⚙️ Run configuration
- Configuration used: Repository: pingcap/docs/.coderabbit.yaml
- Review profile: ASSERTIVE
- Plan: Advanced
- Run ID:
24677ac9-1ae9-4c0d-af5b-ed50131ae2f1
📒 Files selected for processing: 1
dumpling-overview.md
Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 3 remain after this review.
| In the command above: | ||
|
|
||
| - The `--sql` option can be used only for exporting to CSV files. The command above executes the `SELECT * FROM <table-name> WHERE id <100` statement on all tables to be exported. If a table does not have the specified field, the export fails. | ||
| - The `--sql` option can be used only for exporting to CSV files. Dumpling executes the SQL statement that you specify directly instead of generating a query for each table. In this example, Dumpling executes ``SELECT * FROM `test`.`sbtest1` WHERE id < 100``. If the SQL statement references a column that does not exist, the export fails. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🔎 Supported by static analysis
🏁 Script executed:
set -eu
printf '%s\n' '== revisions =='
git cat-file -t a25e533677fe8b69a129b8081d65a07011d40255
git cat-file -t 4c57396c4b8cab80c84786ad8f25e96e585dfaff
printf '%s\n' '== exact-head documentation =='
git show a25e533677fe8b69a129b8081d65a07011d40255:dumpling-overview.md | nl -ba | sed -n '120,140p'
printf '%s\n' '== documentation diff =='
git diff --no-ext-diff --unified=8 4c57396c4b8cab80c84786ad8f25e96e585dfaff a25e533677fe8b69a129b8081d65a07011d40255 -- dumpling-overview.md
printf '%s\n' '== exact-head tree candidates =='
git ls-tree -r --name-only a25e533677fe8b69a129b8081d65a07011d40255 | rg '(^|/)(dumpling|tidb)' || trueRepository: pingcap/docs
Length of output: 34556
🏁 Script executed:
set -eu
printf '%s\n' '== exact-head metadata =='
git show -s --format='commit=%H%nparents=%P%nauthor=%an <%ae>%nauthor-date=%aI%ncommitter=%cn <%ce>%ncommitter-date=%cI%nsubject=%s%nbody=%b' a25e533677fe8b69a129b8081d65a07011d40255
printf '%s\n' '== relevant repository references =='
rg -n -F --glob '*.md' -e 'github.com/pingcap/tidb' -e 'Dumpling' -e '--filetype parquet' -e 'parquet' dumpling-overview.md | sed -n '1,120p'
printf '%s\n' '== commit ancestry around exact head =='
git log --format='%H %aI %s' --ancestry-path 4c57396c4b8cab80c84786ad8f25e96e585dfaff..a25e533677fe8b69a129b8081d65a07011d40255Repository: pingcap/docs
Length of output: 11529
🌐 Web query:
"unsupported config.FileType when we specify --sql" pingcap
💡 Result:
<source_evidence>
<source kind="open_page">
<title>Dumpling Overview | TiDB Docs</title>
<location>https://docs.pingcap.com/tidb/stable/dumpling-overview/</location>
<excerpt>citeturn1view1 [wordlim: 200] Crawled: today; Content type: text/html; Source: open({"ref_id":"turn0search5","lineno":null}); Total lines: 547
L108: * If Dumpling can access the PD address and the 【27†`INFORMATION_SCHEMA.CLUSTER_INFO`】 table of the TiDB cluster, Dumpling supports automatically adjusting the 【28†GC】 safe point time to block GC for TiDB v4.0.0 and later versions.
L109: * Use TiDB's hidden column `_tidb_rowid` to optimize the performance of concurrent data export from a single table.
L111: Note
L112:
L113: Dumpling cannot connect to PD in the following scenarios:
L114:
L115: * The TiDB cluster is running on Kubernetes (unless Dumpling itself is run inside the Kubernetes environment).
L116: * The TiDB cluster is running on TiDB Cloud.
L117:
L118: In such cases, you need to manually 【30†adjust the TiDB GC time】 to avoid export failure.
L119: ## Export data from TiDB or MySQL
L120: ### Required privileges
L121: * PROCESS: Required to query the cluster information to obtain the PD address and then control GC via the PD.
L122: * SELECT: Required when exporting tables.
L123: * RELOAD: Required when the level of `consistency` is `flush`. When the upstream is an RDS database or a managed service, you can ignore this privilege.
L124: * LOCK TABLES: Required when the level of `consistency` is `lock`. This privilege must be granted for all the databases and tables to be exported.
L125: * REPLICATION CLIENT: Required when exporting metadata to record data snapshot. This privilege is optional and you can ignore it if you do not need to export metadata.
L126: * SHOW VIEW: Required to collect view metadata for export.
L127: ### Export to SQL files
L128:
L129: This document assumes that there is a TiDB instance on the 127.0.0.1:4000 host and that this TiDB instance has a root user without a password.
L130:
L131: Dumpling exports data to SQL files by default. You can also export data to SQL files by adding the `--filetype sql` flag:
L132:
L133: `
L134:
L135: tiup dumpling -u root -P 4000 -h 127.0.0.1 --filetype sql -t 8 -o /tmp/test -r 200000 -F 256MiB
L136:
L137:
L138: `
L139:
L140: In the command above:
L141: * The `-h`, `-P`, and `-u` option respectively mean the address, the port, and the user. If a password is required for authentication, you can use `-p $YOUR_SECRET_PASSWORD` to pass the password to Dumpling.
L142:
L143: * The `-o` (or `--output`) option specifies the export directory of the storage, which supports an absolute local file path or an 【31†external storage URI】.
L144: * The `-t` option specifies the number of threads for the export. Increasing the number of threads improves the concurrency of Dumpling and the export speed, and also increases the database's memory consumption. Therefore, it is not recommended to set the number too large. Usually, it's less than 64.
L145:
L146: * The `-r` option enables in-table concurrency to speed up the export. It is disabled by default (value `0`). When enabled with a value greater than `0`, the behavior depends on the source database.
L147: * For TiDB, Dumpling uses region information for splitting, which also reduces memory usage. The specified `-r` value does not affect the split algorithm.
L148: * For MySQL, this option is supported when the primary key (or the first column of a composite primary key) is of an `INT` or `STRING` type.
L149: * The `-F` option is used to specify the maximum size of a single file (the unit here is `MiB`; inputs like `5GiB` or `8KB` are also acceptable). It is recommended to keep its value to 256 MiB or less if you plan to use TiDB Lightning to load this file into a TiDB instance.
L150: Note
L151:
L152: If the size of a single exported table exceeds 10 GB, it is strongly recommended to use the `-r` and `-F` options.
L153: #### URI formats of the storage services
L154:
L155: This section describes the URI formats of the s…[truncated]</excerpt>
</source>
<source kind="open_page">
<title>dumpling/docs/cn/user-guide.md at master · pingcap/dumpling · GitHub</title>
<location>https://github.com/pingcap/dumpling/blob/master/docs/cn/user-guide.md</location>
<excerpt>citeturn1view2 [wordlim: 200] Crawled: 2 weeks ago; Content type: text/html; Source: open({"ref_id":"turn0search0","lineno":null}); Total lines: 275
L119: ## Repository navigation
L120:
L121: * * 【62†Code】
L122: * 【63†Issues23 (23)】
L123: * 【64†Pull requests4 (4)】
L124: * 【65†Actions】
L125: * 【66†Projects】
L126: * 【67†Security and quality】
L127: * 【68†Insights】
L128:
L129: More items
L130:
L131: ## Files Expand file tree
L132:
L133: master
L134:
L135: ## Breadcrumbs
L136:
L137: 1. 【69†dumpling】
L138: 2. /【70†docs】
L139: 3. /【71†cn】
L140:
L141: /
L142:
L143: # user-guide.md
L144:
L145: Copy path
L146:
L147: [Input: Go to file]
L148:
L149: Blame More file actions
L150:
L151: Blame More file actions
L152:
L153: ## Latest commit
L154:
L155:
L156:
L157: ## History
L158:
L159: History
L160:
L161: master
L162: ## Breadcrumbs
L163:
L164: 1. 【69†dumpling】
L165: 2. /【70†docs】
L166: 3. /【71†cn】
L167:
L168: /
L169:
L170: # user-guide.md
L171:
L172: Copy path
L173:
L174: Top
L175:
L176: ## File metadata and controls
L177:
L178: * Preview
L179:
L180: * Code
L181:
L182: * Blame
L183:
L184: 82 lines (62 loc) · 4.49 KB
L185:
L186: 【72†Raw】
L187:
L188: Copy raw file
L189:
L190: Download raw file
L191:
L192: Outline
L193:
L194: Edit and raw actions
L195: # Dumpling 使用手册
L196:
L197: 【62†Dumpling】 是支持以 SQL 文本或者 CSV 格式将 MySQL/TiDB 数据导出的工具。
L198:
L199: 设计初衷是为了替代 【73†Mydumper】, 所以基本用法可以参考 Mydumper, 当然在实现中没有完全照搬 Mydumper, 因此存在与 Mydumper 不同的用法。
L200:
L201: 下表罗列了一些主要参数
L202: 主要参数 |
L203: --- | ---
L204: -B 或 --database | 导出指定数据库
L205: -T 或 --tables-list | 导出指定数据表
L206: -f 或 --filter | 导出能匹配模式的表，语法可参考 【74†table-filter】（只有英文版）
L207: --case-sensitive | table-filter 是否大小写敏感，默认为 false 不敏感
L208: -h 或 --host | 链接节点地址(默认 "127.0.0.1")
L209: -t 或 --threads | 备份并发线程数
L210: -r 或 --rows | 将 table 划分成 row 行数据，一般针对大表操作并发生成多个文件。
L211: --loglevel | 日志级别 {debug,info,warn,error,dpanic,panic,fatal} (默认 "info")
L212: -d 或 --no-data | 不导出数据, 适用于只导出 schema 场景
L213: --no-header | 导出 table csv 数据，不生成 header
L214: -W 或 --no-views | 不导出 view, 默认 true
L215: -m 或 --no-schemas | 不导出 schema , 只导出数据
L216: -s 或--statement-size | 控制 Insert Statement 的大小，单位 bytes
L217: -F 或 --filesize | 将 table 数据划分出来的文件大小, 需指明单位 (如 `128B`, `64KiB`, `32MiB`, `1.5GiB`)
L218: --filetype | 导出文件类型 csv/sql (默认 sql)
L219: -o 或 --output | 设置导出文件路径
L220: --output-filename-template | 设置导出文件名模版，详情见下
L221: -S 或 --sql | 根据指定的 sql 导出数据，该指令不支持并发导出
L222: --consistency | flush: dump 前用 FTWRL
L223: snapshot: 通过 tso 指定 dump 位置
L224: lock: 对需要 dump 的所有表执行 lock tables read
L225: none: 不加锁 dump，无法保证一致性
L226: auto: MySQL flush, TiDB snapshot
L227: --snapshot | snapshot tso, 只在 consistency=snapshot 下生效
L228: --where | 对备份的数据表通过 where 条件指定范围
L229: -p 或 --password | 链接密码
L230: -P 或 --port | 链接端口，默认 4000
L231: -u 或 --user | 默认 root
L232: 更多具体用法可以使用 -h, --help 进行查看。
L233: ## Mydumper 相关参考
L234:
L235: 【75†Mydumper usage】
L236:
L237: 【76†TiDB Mydumper 使用文档†pingcap.com】
L238:
L239: ## Dumpling 下载链接
L240:
L241: 【77†nightly†download.pingcap.org】
L242: ## 导出文件名模版
L243:
L244: `--output-filename-template` 参数指定了所有文件的命名方式（不含扩展名）。它使用 【78†Go 的 `text/template` 语法†golang.org】。
L245:
L246: 模板可使用以下字段：
L247:
L248: * `.DB` — 库名
L249: * `.Table` — 表名、物件名称。
L250: * `.Index` — 由 0 开始的序列号，代表当前导出的表中的哪一份文件
L251:
L252: 库和表名中可能包含 `/` 之类的特殊字符，而这些字符不能用在文件系统中。因此，Dumpling 提供了一个 `fn` 函数来对这些特殊字符进行百分号编码。它们是：
L253:
L254: * U+0000 到 U+001F (控制字符)
L255: * `/`、`\`、`<`、`>`、`:`、`"`、`*`、`?` (无效的 Windows 路径字符)
L256: * `.` (库/表名分隔符)
L257: * `-`，当出现在 `-schema` 字串里
L258: 例如，使用 `--output-filename-template '{{fn .Table}}.{{printf "%09d" .Index}}'` 后，Dumpling 会把表 `"db"."tbl:normal"` 导出到 `tbl%3Anormal.000000000.sql`、`tbl%3Anormal.000000001.sql` 等文件。
L259:
L260: 除数据文件外，Dumpling 还支持透过子模版自定义命名表结构文件的名称。默认的配置是：
L261: 模版名 | 默认内容
L262: --- | ---
L263: dat…[truncated]</excerpt>
</source>
<source kind="search">
<title>Dumpling Overview | TiDB Docs</title>
<location>https://docs.pingcap.com/tidb/stable/dumpling-overview/</location>
<excerpt>citeturn0search5 [wordlim: 200] Crawled: 4 days ago; tiup dumpling -u root -P 4000 -h 127.0.0.1 -o /tmp/test --filetype csv --sql 'select * from `test`.`sbtest1` where id < 100' -F 100MiB --output-filename-template 'test.sbtest1.{{.
</excerpt>
</source>
<source kind="search">
<title>TiDB: dumpling/export/config.go | Fossies</title>
<location>https://fossies.org/linux/tidb/dumpling/export/config.go</location>
<excerpt>citeturn0search7 [wordlim: 200] Published: last month; Crawled: last month; 334 flags.String(flagFiletype, "", "The type of export file (sql/csv)") ... 904 return errors.Errorf("unsupported config.FileType '%s' when we specify --sql, please unset --filetype or set it to 'csv'", conf.FileType)
</excerpt>
</source>
<source kind="search">
<title>dumpling/docs/cn/user-guide.md at master · pingcap/dumpling · GitHub</title>
<location>https://github.com/pingcap/dumpling/blob/master/docs/cn/user-guide.md</location>
<excerpt>citeturn0search0 [wordlim: 200] Crawled: 4 weeks ago; --filetype | 导出文件类型 csv/sql (默认 sql) ... Index}}'` 后，Dumpling 会把表 `"db"."tbl:normal"` 导出到 `tbl%3Anormal.000000000.sql`、`tbl%3Anormal.000000001.sql` 等文件。
-W 或 --no-views | 不导出 view, 默认 true
-m 或 --no-schemas | 不导出 schema , 只导出数据
-s 或--statement-size | 控制 Insert Statement 的大小，单位 bytes
-F 或 --filesize | 将 table 数据划分出来的文件大小, 需指明单位 (如 `128B`, `64KiB`, `32MiB`, `1.5GiB`)
--filetype | 导出文件类型 csv/sql (默认 sql)
-o 或 --output | 设置导出文件路径
--output-filename-template | 设置导出文件名模版，详情见下
-S 或 --sql | 根据指定的 sql 导出数据，该指令不支持并发导出
--consistency | flush: dump 前用 FTWRL
snapshot: 通过 tso 指定 dump 位置
lock: 对需要 dump 的所有表执行 lock tables read
none: 不加锁 dump，无法保证一致性
auto: MySQL flush, TiDB snapshot
--snapshot | snapshot tso, 只在 consistency=snapshot 下生效
--where | 对备份的数据表通过 where 条件指定范围
-p 或 --password | 链接密码
-P 或 --port | 链接端口，默认 4000
-u 或 --user | 默认 root
更多具体用法可以使用 -h, --help 进行查看。
</excerpt>
</source>
<omitted_sources count="13">Additional source records were omitted from this bounded result.</omitted_sources>
</source_evidence>
Citations:
- 1: https://docs.pingcap.com/tidb/stable/dumpling-overview/
- 2: https://fossies.org/linux/tidb/dumpling/export/config.go?utm_source=openai
🌐 Web query:
"FileFormatParquetString" "adjustFileFormat" github pingcap tidb
💡 Result:
<source_evidence>
<source kind="search">
<title>IMPORT INTO - dev | TiDB Docs</title>
<location>https://docs.pingcap.com/tidb/dev/sql-statement-import-into/</location>
<excerpt>citeturn1search4 [wordlim: 200] Crawled: 4 days ago; Ensure that the specified path and files exist on the TiDB node to which your session is connected, and verify that you have the required `FILE` privilege. ... If the `FORMAT` clause is omitted, TiDB automatically detects the format based on the file extension (`.csv`, `.sql`, `.parquet`).
`
ImportIntoStmt ::=
'IMPORT' 'INTO' TableName ColumnNameOrUserVarList? SetClause? FROM fileLocation Format? WithOptions?
|
'IMPORT' 'INTO' TableName ColumnNameList? FROM SelectStatement WithOptions?
ColumnNameOrUserVarList ::=
'(' ColumnNameOrUserVar (',' ColumnNameOrUserVar)* ')'
ColumnNameList ::=
'(' ColumnName (',' ColumnName)* ')'
SetClause ::=
'SET' SetItem (',' SetItem)*
SetItem ::=
ColumnName '=' Expr
Format ::=
'FORMAT' ('"CSV"' | '"SQL"' | '"PARQUET"')
WithOptions ::=
'WITH' OptionItem (',' OptionItem)*
OptionItem ::=
optionName '=' optionVal | optionName
`
## Parameter description
### ColumnNameOrUserVarList
It specifies how each field in the data file corresponds to the columns in the target table. You can also use it to map fields to variables to skip certain fields for the import, or use it in `SetClause`.
* If this parameter is not specified, the number of fields in each row of the data file must match the number of columns in the target table, and the fields will be imported to the corresponding columns in order.
* If this parameter is specified, the number of specified columns or variables must match the number of fields in each row of the data file.
### Format
The `IMPORT INTO` statement supports three data file formats: `CSV`, `SQL`, and `PARQUET`. If the `FORMAT` clause is omitted, TiDB automatically detects the format based on the file extension (`.csv`, `.sql`, `.parquet`). Compressed files are supported, and the compression suffix (`.gz`, `.gzip`, `.zstd`, `.zst`, `.snappy`) is ignored during format detection. If a file does not have an extension, TiDB assumes that the file format is `CSV`.
### WithOptions
You can use `WithOptions` to specify import options and control the data import process. For example, to execute the import of data files asynchronously in the backend, you can enable the `DETACHED` mode for the import by adding the `WITH DETACHED` option to the `IMPORT INTO` statement.
The supported options are described as follows:
Option name | Supported data sources and formats | Description
--- | --- | ---
`CHARACTER_SET='<string>'` | CSV | Specifies the character set of the data file. The default character set is `utf8mb4`. The supported character sets include `binary`, `utf8`, `utf8mb4`, `gb18030`, `gbk`, `latin1`, and `ascii`.
</excerpt>
</source>
<source kind="search">
<title>GitHub - pingcap/tidb: TiDB is built for agentic workloads that grow unpredictably, with ACID guarantees and native support for transactions, analytics, and vector search. No data silos. No noisy neighbors. No infrastructure ceiling. · GitHub</title>
<location>https://github.com/pingcap/tidb</location>
<excerpt>citeturn1search0 [wordlim: 200] Crawled: 3 days ago; 1. pingcap ... TiDB is built for agentic workloads that grow unpredictably, with ACID guarantees and native support for transactions, analytics, and vector search. ... [Input: Go to file] ... .github | .github | |
1. pingcap
2. tidb
Image: pingcap
tidb
Public
* NotificationsYou must be signed in to change notification settings
* Fork6.3k (6.3k)
* Star40.6k (40.6k)You must be signed in to star a repository
## About
TiDB is built for agentic workloads that grow unpredictably, with ACID guarantees and native support for transactions, analytics, and vector search. No data silos. No noisy neighbors. No infrastructure ceiling.
www.tidb.io/
### Topics
agentagent-contextagent-memoryagenticaicloud-nativedatabasedistributed-databasedistributed-transactionsgohacktoberfesthtapmemorymysqlmysql-compatibilityscaleserverlesssqltidb
### Resources
Readme
Apache-2.0 license
### Code of conduct
Code of conduct
### Contributing
Contributing
### Security policy
Security policy
Activity
### Stars
40.6k stars
### Watchers
1.2k watching
### Forks
6.3k forks
master
BranchesTags
[Input: Go to file]
Go to file
Code
Open more actions menu
## Latest commit
## History
28,742 Commits
## Folders and files
Name | Name | Last commit message | Last commit date
--- | --- | --- | ---
.agents/skills | .agents/skills | |
.claude/skills | .claude/skills | |
.github | .github | |
LICENSES | LICENSES | |
br | br | |
build | build | |
cmd | cmd | |
docs | docs | |
dumpling | dumpling | |
hooks | hooks | |
</excerpt>
</source>
<source kind="search">
<title>Unloading Parquet File | TiDB Docs</title>
<location>https://docs.pingcap.com/tidbcloudlake/unload-parquet-file/</location>
<excerpt>citeturn1search1 [wordlim: 200] Crawled: 3 months ago; * More Parquet options refer to Parquet File Format Options ... │ data_a3760513-78a8-4a89-8f92-b1a17e0a61b6_0000_00000000.parquet │ 445 │ 100 │
# Unloading Parquet File
Contribute
* * *
Copy as Markdown
* * *
View as Markdown
## Unloading Parquet File
Syntax:
`
COPY INTO {internalStage | externalStage | externalLocation}
FROM { [<database_name>.]<table_name> | ( <query> ) }
FILE_FORMAT = (TYPE = PARQUET)
[MAX_FILE_SIZE = <num>]
[DETAILED_OUTPUT = true | false]
`
* More Parquet options refer to Parquet File Format Options
* Unloading into multiple files use the MAX_FILE_SIZE Copy Option
* More details about the syntax can be found in COPY INTO location
## Tutorial
### Step 1. Create an External Stage
`
CREATE STAGE parquet_unload_stage
URL = 's3://unload/parquet/'
CONNECTION = (
ACCESS_KEY_ID = '<your-access-key-id>'
SECRET_ACCESS_KEY = '<your-secret-access-key>'
);
`
### Step 2. Create Custom Parquet File Format
`
CREATE FILE FORMAT parquet_unload_format
TYPE = PARQUET
;
</excerpt>
</source>
<source kind="search">
<title>Pull requests · pingcap/tidb · GitHub</title>
<location>https://github.com/pingcap/tidb/pulls</location>
<excerpt>citeturn1search2 [wordlim: 200] Crawled: last month; pingcap / tidb Public ... # Pull requests: pingcap/tidb
pingcap / tidb Public
* Notifications You must be signed in to change notification settings
* Fork 6.2k
* Star
* Code
* Issues 5k+
* Pull requests 1.7k
* Discussions
* Actions
* Projects
* Wiki
* Security and quality 1
* Insights
Additional navigation options
# Pull requests: pingcap/tidb
[Input: Search all issues]
Labels Milestones
New
1,722 Open 44,884 Closed
Author Label Projects Milestones Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee Sort
## Pull requests list
ingestor: clean up Pebble FS after failed open do-not-merge/needs-triage-completed release-note Denotes a PR that will be considered when it comes time to generate release notes. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
</excerpt>
</source>
<source kind="search">
<title>tidb-lightning need to support fields starting with `_` when the import file type is parquet · Issue #36980 · pingcap/tidb · GitHub</title>
<location>GitHub issue 36980 in pingcap/tidb (link omitted to avoid creating a cross-reference)</location>
<excerpt>citeturn1search3 [wordlim: 200] Published: 4.2 years ago; Crawled: last week; # tidb-lightning need to support fields starting with `_` when the import file type is parquet #36980 ... https://github.com/xitongsys/parquet-go/blob/7a9be66324ca29f6ce14c66b239ba79c4896bf25/common/common.go#L603
# tidb-lightning need to support fields starting with `_` when the import file type is parquet #36980
New issue
Copy link
New issue
Copy link
Closed
#36985
Closed
tidb-lightning need to support fields starting with `_` when the import file type is parquet#36980
#36985
Copy link
Labels
affects-5.4This bug affects the 5.4.x(LTS) versions.This bug affects the 5.4.x(LTS) versions.affects-6.1This bug affects the 6.1.x(LTS) versions.This bug affects the 6.1.x(LTS) versions.affects-6.2component/lightningThis issue is related to Lightning of TiDB.This issue is related to Lightning of TiDB.severity/minortype/bugThe issue is confirmed as a bug.The issue is confirmed as a bug.
## Description
glkappe
opened on Aug 9, 2022
Issue body actions
## Bug Report
Please answer these questions before submitting your issue. Thanks!
tidb-lightning need to support fields starting with `_` when the import file type is parquet
### 1. Minimal reproduce step (Required)
1. fields starting with `_` of parquet
2. use tidb-lightning import
### 2. What did you expect to see? (Required)
import success
### 3. What did you see instead (Required)
`Root cause: PreCheckFailed: TiDB schema `test`.`tbl` doesn't have column pargo_prefix__abc, please check table schema
`
### 4. What is your TiDB version? (Required)
tidb、tidb-lightning ：v6.1.0
2. ### D3Hunter commented on Aug 9, 2022
D3Hunter
on Aug 9, 2022
Contributor
More actions
duplicate with #36981
3. D3Hunter
added
severity/minor
affects-5.4This bug affects the 5.4.x(LTS) versions.This bug affects the 5.4.x(LTS) versions.
affects-6.1This bug affects the 6.1.x(LTS) versions.This bug affects the 6.1.x(LTS) versions.
affects-6.2
on Aug 9, 2022
4. ### glkappe commented on Aug 9, 2022
glkappe
on Aug 9, 2022
Author
More actions
ok i wiil close this issue
</excerpt>
</source>
<omitted_sources count="21">Additional source records were omitted from this bounded result.</omitted_sources>
</source_evidence>
Citations:
- 1: https://docs.pingcap.com/tidb/dev/sql-statement-import-into/?utm_source=openai
🌐 Web query:
"WriteInsertInParquet" "newTableData" pingcap tidb
💡 Result:
<source_evidence>
<source kind="search">
<title>TiDB: dumpling/export/ir_impl.go | Fossies</title>
<location>https://fossies.org/linux/tidb/dumpling/export/ir_impl.go</location>
<excerpt>citeturn1search0 [wordlim: 200] Published: last month; Crawled: last month; 10 "github.com/pingcap/tidb/br/pkg/version" 11 tcontext "github.com/pingcap/tidb/dumpling/context" ... 196 func newTableData(query string, colLength int, needColTypes bool) *tableData {
## Image "Fossies" - the Free Open Source Software Archive Image
### Member "tidb-8.5.8/dumpling/export/ir_impl.go" (27 Aug 2026, 8296 Bytes) of package /linux/misc/tidb-8.5.8.tar.gz:
* * *
As a special service "Fossies" has tried to format the requested source page into HTML format using (guessed) Go source code syntax highlighting (style: standard) with prefixed line numbers and code folding option. Alternatively you can here view or download the uninterpreted source code file.
1 // Copyright 2020 PingCAP, Inc. Licensed under Apache-2.0.
2
3 package export
4
5 import (
6 "database/sql"
7 "strings"
8
9 "github.com/pingcap/errors"
10 "github.com/pingcap/tidb/br/pkg/version"
11 tcontext "github.com/pingcap/tidb/dumpling/context"
12 "go.uber.org/zap"
13 )
14
15 // rowIter implements the SQLRowIter interface.
16 // Note: To create a rowIter, please use `newRowIter()` instead of struct literal.
17 type rowIter struct {
18 rows *sql.Rows
19 hasNext bool
20 args []any
21 }
22
23 func newRowIter(rows *sql.Rows, argLen int) *rowIter {
24 r := &rowIter{
25 rows: rows,
26 hasNext: false,
27 args: make([]any, argLen),
28 }
29 r.hasNext = r.rows.Next()
30 return r
31 }
32
33 func (iter *rowIter) Close() error {
34 return iter.rows.Close()
35 }
36
37 func (iter *rowIter) Decode(row RowReceiver) error {
150 if iter.err == nil {
151 iter.hasNext = iter.rows.Next()
152 if !iter.hasNext {
153 iter.nextRows()
154 }
155 }
156 }
157
158 func (iter *multiQueriesChunkIter) HasNext() bool {
159 return iter.hasNext
160 }
161
162 type stringIter struct {
163 idx int
164 ss []string
165 }
166
167 func newStringIter(ss ...string) StringIter {
168 return &stringIter{
169 idx: 0,
170 ss: ss,
171 }
172 }
173
174 func (m *stringIter) Next() string {
175 if m.idx >= len(m.ss) {
176 return ""
177 }
178 ret := m.ss[m.idx]
179 m.idx++
180 return ret
181 }
182
183 func (m *stringIter) HasNext() bool {
184 return m.idx < len(m.ss)
185 }
186
187 type tableData struct {
188 query string
189 rows *sql.Rows
190 colLen int
191 needColTypes bool
192 colTypes []string
193 SQLRowIter
194 }
195
196 func newTableData(query string, colLength int, needColTypes bool) *tableData {
197 return &tableData{
198 query: query,
199 colLen: colLength,
200 needColTypes: needColTypes,
201 }
202 }
203
204 func (td *tableData) Start(tctx *tcontext.Context, conn *sql.Conn) error {
205 tctx.L().Debug("try to start tableData", zap.String("query", td.query))
206 rows, err := conn.QueryContext(tctx, td.query)
207 if err != nil {
208 return errors.Annotatef(err, "sql: %s", td.query)
</excerpt>
</source>
<source kind="search">
<title>tidb ir_impl 源码</title>
<location>https://www.seaxiang.com/blog/6c5a55f05fbe47be836f4fa33e836a8e</location>
<excerpt>citeturn1search1 [wordlim: 200] Published: 4.1 years ago; Crawled: 3 months ago; tcontext "github.com/pingcap/tidb/dumpling/context" ... func newTableData(query string, colLength int, needColTypes bool) *tableData {
iter.hasNext = iter.rows.Next()
if !iter.hasNext {
iter.nextRows()
}
}
}
func (iter *multiQueriesChunkIter) HasNext() bool {
return iter.hasNext
}
type stringIter struct {
idx int
ss []string
}
func newStringIter(ss ...string) StringIter {
return &stringIter{
idx: 0,
ss: ss,
}
}
func (m *stringIter) Next() string {
if m.idx >= len(m.ss) {
return ""
}
ret := m.ss[m.idx]
m.idx++
return ret
}
func (m *stringIter) HasNext() bool {
return m.idx < len(m.ss)
}
type tableData struct {
query string
rows *sql.Rows
colLen int
needColTypes bool
colTypes []string
SQLRowIter
}
func newTableData(query string, colLength int, needColTypes bool) *tableData {
return &tableData{
query: query,
colLen: colLength,
needColTypes: needColTypes,
}
}
func (td *tableData) Start(tctx *tcontext.Context, conn *sql.Conn) error {
tctx.L().Debug("try to start tableData", zap.String("query", td.query))
rows, err := conn.QueryContext(tctx, td.query)
if err != nil {
return errors.Annotatef(err, "sql: %s", td.query)
}
if err = rows.Err(); err != nil {
return errors.Annotatef(err, "sql: %s", td.query)
}
td.SQLRowIter = nil
td.rows = rows
if td.needColTypes {
ns, err := rows.Columns()
if err != nil {
return errors.Trace(err)
}
td.colLen = len(ns)
td.colTypes = make([]string, 0, td.colLen)
colTps, err := rows.ColumnTypes()
if err != nil {
return errors.Trace(err)
}
for _, c := range colTps {
td.colTypes = append(td.colTypes, c.DatabaseTypeName())
</excerpt>
</source>
<source kind="search">
<title>TiDB: dumpling/export/dump.go | Fossies</title>
<location>https://fossies.org/linux/tidb/dumpling/export/dump.go</location>
<excerpt>citeturn1search2 [wordlim: 200] Published: last month; Crawled: last month; 27 "github.com/pingcap/tidb/br/pkg/storage" ... 1616 // reference: https://github.com/pingcap/tidb/blob/c497d5c/dumpling/export/dump.go#L775 ... 1707 func (d *Dumper) newTaskTableData(meta TableMeta, data TableDataIR, currentChunk, totalChunks int) *TaskTableData {
940 if len(handleVals) == 0 {
941 if partition == "" {
942 // return error to make outside function try using rows method to dump data
943 return errors.Annotatef(errEmptyHandleVals, "table: `%s`.`%s`", escapeString(db), escapeString(tbl))
944 }
945 return d.dumpWholeTableDirectly(tctx, meta, taskChan, partition, buildOrderByClauseString(handleColNames), startChunkIdx, totalChunk)
946 }
947 conf := d.conf
948 selectField, selectLen := meta.SelectedField(), meta.SelectedLen()
949 where := buildWhereClauses(handleColNames, handleVals)
950 orderByClause := buildOrderByClauseString(handleColNames)
951
952 for i, w := range where {
953 query := buildSelectQuery(db, tbl, selectField, partition, buildWhereCondition(conf, w), orderByClause)
954 task := d.newTaskTableData(meta, newTableData(query, selectLen, false), i+startChunkIdx, totalChunk)
955 ctxDone := d.sendTaskToChan(tctx, task, taskChan)
956 if ctxDone {
957 return tctx.Err()
958 }
959 }
960 return nil
961 }
962
963 // L returns real logger
964 func (d *Dumper) L() log.Logger {
965 return d.tctx.L()
966 }
967
968 func selectTiDBTableSample(tctx *tcontext.Context, conn *BaseConn, meta TableMeta) (pkFields []string, pkVals [][]string, err error) {
969 pkFields, pkColTypes, err := selectTiDBRowKeyFields(tctx, conn, meta, nil)
970 if err != nil {
971 return nil, nil, errors.Trace(err)
972 }
973
974 query := buildTiDBTableSampleQuery(pkFields, meta.DatabaseName(), meta.TableName())
1257
1258 createTableSQL, err := ShowCreateTable(tctx, conn, db, tbl)
1259 if err != nil {
1260 return nil, err
1261 }
1262 meta.showCreateTable = createTableSQL
1263 return meta, nil
1264 }
1265
1266 func (d *Dumper) dumpSQL(tctx *tcontext.Context, metaConn *BaseConn, taskChan chan<- Task) {
1267 conf := d.conf
1268 meta := &tableMeta{}
1269 data := newTableData(conf.SQL, 0, true)
1270 task := d.newTaskTableData(meta, data, 0, 1)
1271 c := detectEstimateRows(tctx, metaConn, fmt.Sprintf("EXPLAIN %s", conf.SQL), []string{"rows", "estRows", "count"})
1272 AddCounter(d.metrics.estimateTotalRowsCounter, float64(c))
1273 atomic.StoreInt64(&d.totalTables, int64(1))
1274 d.sendTaskToChan(tctx, task, taskChan)
1275 }
1276
1277 func canRebuildConn(consistency string, trxConsistencyOnly bool) bool {
1278 switch consistency {
1279 case ConsistencyTypeLock, ConsistencyTypeFlush:
1280 return !trxConsistencyOnly
1281 case ConsistencyTypeSnapshot, ConsistencyTypeNone:
1282 return true
1283 default:
1284 return false
1285 }
1286 }
1287
1288 // Close closes a Dumper and stop dumping immediately
1289 func (d *Dumper) Close() error {
1290 d.cancelCtx()
1291 d.metrics.unregisterFrom(d.conf.PromRegistry)
</excerpt>
</source>
<source kind="search">
<title>TiDB: pkg/executor/update.go | Fossies</title>
<location>https://fossies.org/linux/tidb/pkg/executor/update.go</location>
<excerpt>citeturn1search3 [wordlim: 200] Published: last month; Crawled: last month; 25 "github.com/pingcap/tidb/pkg/expression" ... 33 "github.com/pingcap/tidb/pkg/table" ... 164 newTableData := newData[content.Start:content.End]
### Member "tidb-8.5.8/pkg/executor/update.go" (27 Aug 2026, 23088 Bytes) of package /linux/misc/tidb-8.5.8.tar.gz:
* * *
As a special service "Fossies" has tried to format the requested source page into HTML format using (guessed) Go source code syntax highlighting (style: standard) with prefixed line numbers and code folding option. Alternatively you can here view or download the uninterpreted source code file. See also the latest Fossies "Diffs" side-by-side code changes report for "update.go": 8.5.7_vs_8.5.8.
1 // Copyright 2018 PingCAP, Inc.
2 //
3 // Licensed under the Apache License, Version 2.0 (the "License");
4 // you may not use this file except in compliance with the License.
5 // You may obtain a copy of the License at
6 //
7 // http://www.apache.org/licenses/LICENSE-2.0
8 //
9 // Unless required by applicable law or agreed to in writing, software
10 // distributed under the License is distributed on an "AS IS" BASIS,
11 // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12 // See the License for the specific language governing permissions and
13 // limitations under the License.
14
15 package executor
16
17 import (
18 "bytes"
19 "context"
20 "fmt"
21 "runtime/trace"
22
23 "github.com/pingcap/errors"
24 "github.com/pingcap/tidb/pkg/executor/internal/exec"
25 "github.com/pingcap/tidb/pkg/expression"
26 "github.com/pingcap/tidb/pkg/kv"
27 "github.com/pingcap/tidb/pkg/meta/autoid"
28 mmodel "github.com/pingcap/tidb/pkg/meta/model"
29 "github.com/pingcap/tidb/pkg/parser/model"
30 "github.com/pingcap/tidb/pkg/parser/mysql"
31 plannercore "github.com/pingcap/tidb/pkg/planner/core"
32 "github.com/pingcap/tidb/pkg/sessionctx"
33 "github.com/pingcap/tidb/pkg/table"
34 "github.com/pingcap/tidb/pkg/types"
35 "github.com/pingcap/tidb/pkg/util/chunk"
36 "github.com/pingcap/tidb/pkg/util/execdetails"
37 "github.com/pingcap/tidb/pkg/util/memory"
38 "github.com/tikv/client-go/v2/txnkv/txnsnapshot"
39 )
40
41 // UpdateExec represents a new update executor.
42 type UpdateExec struct {
43 exec.BaseExecutor
44
45 OrderedList []*expression.Assignment
46 assignmentsPerTable map[int][]*expression.Assignment
47
48 // updatedRowKeys is a map for unique (TableAlias, handle) pair.
49 // The value is true if the row is changed, or false otherwise
50 updatedRowKeys map[int]*kv.MemAwareHandleMap[bool]
51 tblID2table map[int64]table.Table
52 // mergedRowData is a map for unique (Table, handle) pair.
218 }
219 tbl := e.tblID2table[content.TblID]
220 oldData := row[content.Start:content.End]
221 newTableData := newData[content.Start:content.End]
222
223 // We don't check the second return value here because we have already called mergeNonGenerated() before.
224 mergedData, _ = e.mergedRowData[content.TblID].Get(handle)
225 for i, flag := range flags {
226 if !tbl.WritableCols()[i].…[truncated]</excerpt>
</source>
<source kind="search">
<title>automa/src/newtab/pages/storage/Tables.vue at main · AutomaApp/automa · GitHub</title>
<location>https://github.com/AutomaApp/automa/blob/main/src/newtab/pages/storage/Tables.vue</location>
<excerpt>citeturn1search4 [wordlim: 200] Crawled: 2 weeks ago; <script setup> ... dbStorage.tablesData.where('tableId').equals(tableId).first() ... const newTableData = []; ... newTableData.push({ ...item });
);
const table = shallowRef({
body: [],
header: [],
});
const state = shallowReactive({
query: '',
});
const editState = shallowReactive({
name: '',
columns: [],
show: false,
});
function editTable() {
editState.name = tableDetail.value.name;
editState.columns = tableDetail.value.columns;
editState.show = true;
}
function additionalHeaders(headers) {
headers.unshift({ value: '$$id', text: '', sortable: false });
headers.push({
value: 'action',
text: '',
sortable: false,
align: 'right',
attrs: {
width: '100px',
},
});
return headers;
}
function exportData(type) {
dataExporter(
tableData.value.items,
{ name: tableDetail.value.name, type },
true
);
}
async function saveEditedTable({ columns, name, changes }) {
const columnsChanges = Object.values(changes);
try {
await dbStorage.tablesItems.update(tableId, {
name,
columns,
});
const headers = [];
const newTableData = [];
const newColumnsIndex = {};
const { columnsIndex } = tableData.value;
columns.forEach(({ name: columnName, id, type }) => {
const index = columnsIndex[id]?.index || 0;
newColumnsIndex[id] = {
type,
index,
name: columnName,
};
headers.push({
text: columnName,
value: columnName,
filterable: ['string', 'any'].includes(type),
});
});
if (columnsIndex.column) {
newColumnsIndex.column = toRaw(columnsIndex.column);
}
table.value.header = additionalHeaders(headers);
table.value.body = table.value.body.map((item, index) => {
columnsChanges.forEach(
({ type, oldValue, newValue, name: columnName }) => {
if (type === 'rename' && objectHasKey(item, oldValue)) {
item[newValue] = item[oldValue];
delete item[oldValue];
} else if (type === 'delete') {
delete item[columnName];
}
}
);
delete item.$$id;
newTableData.push({ ...item });
item.$$id = index + 1;
return item;
});
await dbStorage.tablesData.where('tableId').equals(tableId).modify({
items: newTableData,
columnsIndex: newColumnsIndex,
});
editState.show = false;
} catch (error) {
console.error(error);
}
}
function deleteRow(item) {
const rowIndex = table.value.body.findIndex(({ $$id }) => $$id === item.$$id);
</excerpt>
</source>
<omitted_sources count="9">Additional source records were omitted from this bounded result.</omitted_sources>
</source_evidence>
Citations:
- 1: https://fossies.org/linux/tidb/dumpling/export/ir_impl.go?utm_source=openai
🌐 Web query:
site:fossies.org/linux/tidb/dumpling "FileFormatParquetString" OR "WriteInsertInParquet" OR "case FileFormatParquet"
💡 Result:
I found no results on Fossies for those exact Dumpling symbols. The search does not establish whether they exist in the source; it only indicates that the indexed Fossies pages didn’t return a match.
🏁 Script executed:
set -eu
python3 - <<'PY'
import json
import urllib.parse
import urllib.request
base = "https://api.github.com/repos/pingcap/tidb/commits"
params = urllib.parse.urlencode({
"path": "dumpling/export/config.go",
"until": "2026-10-21T02:00:00Z",
"per_page": "10",
})
req = urllib.request.Request(
base + "?" + params,
headers={"Accept": "application/vnd.github+json", "User-Agent": "review-inspection"},
)
with urllib.request.urlopen(req, timeout=20) as response:
commits = json.load(response)
for item in commits:
print(item["sha"], item["commit"]["committer"]["date"], item["commit"]["message"].splitlines()[0])
PYRepository: pingcap/docs
Length of output: 1471
🏁 Script executed:
set -eu
python3 - <<'PY'
from urllib.request import Request, urlopen
sha = "03148cd4adf3f26f5c8f529b89a30c1dba690713"
files = [
"dumpling/export/config.go",
"dumpling/export/dump.go",
"dumpling/export/ir_impl.go",
"dumpling/export/writer.go",
"dumpling/export/writer_util.go",
]
needles = {
"dumpling/export/config.go": ["func (conf *Config) adjustFileFormat", "FileFormatParquetString", "unsupported config.FileType"],
"dumpling/export/dump.go": ["func (d *Dumper) dumpSQL", "newTableData(conf.SQL"],
"dumpling/export/ir_impl.go": ["func newTableData", "func (td *tableData) Start"],
"dumpling/export/writer.go": ["FileFormatParquetString", "func (w *Writer) WriteTableData", "WriteInsert"],
"dumpling/export/writer_util.go": ["WriteInsertInParquet", "FileFormatParquetString"],
}
for path in files:
url = f"https://raw.githubusercontent.com/pingcap/tidb/{sha}/{path}"
req = Request(url, headers={"User-Agent": "review-inspection"})
with urlopen(req, timeout=20) as response:
lines = response.read().decode().splitlines()
print(f"== {path} ({len(lines)} lines) ==")
starts = []
for needle in needles[path]:
for i, line in enumerate(lines):
if needle in line:
starts.append(i)
for start in sorted(set(starts)):
lo = max(0, start - 8)
hi = min(len(lines), start + 18)
print(f"-- lines {lo + 1}-{hi} --")
for i in range(lo, hi):
print(f"{i + 1}: {lines[i]}")
PYRepository: pingcap/docs
Length of output: 14598
🏁 Script executed:
set -eu
python3 - <<'PY'
from urllib.request import Request, urlopen
sha = "03148cd4adf3f26f5c8f529b89a30c1dba690713"
path = "dumpling/export/writer_util.go"
url = f"https://raw.githubusercontent.com/pingcap/tidb/{sha}/{path}"
req = Request(url, headers={"User-Agent": "review-inspection"})
with urlopen(req, timeout=20) as response:
lines = response.read().decode().splitlines()
for lo, hi in [(503, 590), (652, 671)]:
print(f"-- {path}:{lo}-{hi} --")
for i in range(lo - 1, min(hi, len(lines))):
print(f"{i + 1}: {lines[i]}")
PYRepository: pingcap/docs
Length of output: 4265
🏁 Script executed:
set -eu
python3 - <<'PY'
from urllib.request import Request, urlopen
sha = "03148cd4adf3f26f5c8f529b89a30c1dba690713"
files = {
"dumpling/export/dump.go": [
"type tableMeta struct",
"func (d *Dumper) newTaskTableData",
"func (d *Dumper) dumpSQL",
],
"dumpling/export/writer_util.go": [
"type TableMeta interface",
"func (t *tableMeta) ColumnInfos",
"func (t *tableMeta) ColumnTypes",
],
"dumpling/export/ir.go": [
"type TableDataIR interface",
],
}
for path, needles in files.items():
url = f"https://raw.githubusercontent.com/pingcap/tidb/{sha}/{path}"
req = Request(url, headers={"User-Agent": "review-inspection"})
try:
with urlopen(req, timeout=20) as response:
lines = response.read().decode().splitlines()
except Exception as exc:
print(f"== {path}: unavailable: {exc} ==")
continue
print(f"== {path} ({len(lines)} lines) ==")
starts = []
for needle in needles:
for i, line in enumerate(lines):
if needle in line:
starts.append(i)
for start in sorted(set(starts)):
lo = max(0, start - 12)
hi = min(len(lines), start + 45)
print(f"-- lines {lo + 1}-{hi} --")
for i in range(lo, hi):
print(f"{i + 1}: {lines[i]}")
PYRepository: pingcap/docs
Length of output: 4238
🏁 Script executed:
set -eu
python3 - <<'PY'
import json
from urllib.request import Request, urlopen
sha = "03148cd4adf3f26f5c8f529b89a30c1dba690713"
api = f"https://api.github.com/repos/pingcap/tidb/git/trees/{sha}?recursive=1"
req = Request(api, headers={"Accept": "application/vnd.github+json", "User-Agent": "review-inspection"})
with urlopen(req, timeout=30) as response:
tree = json.load(response)["tree"]
paths = [
item["path"] for item in tree
if item["path"].startswith("dumpling/export/") and item["path"].endswith(".go")
]
print("== export files ==")
for path in paths:
print(path)
print("== metadata matches ==")
for path in paths:
raw = f"https://raw.githubusercontent.com/pingcap/tidb/{sha}/{path}"
req = Request(raw, headers={"User-Agent": "review-inspection"})
with urlopen(req, timeout=20) as response:
text = response.read().decode()
if "type tableMeta" in text or "ColumnInfos()" in text or "func (.*tableMeta" in text:
print(f"-- {path} --")
lines = text.splitlines()
for i, line in enumerate(lines):
if "type tableMeta" in line or "ColumnInfos()" in line or "ColumnTypes()" in line or "SelectedField()" in line:
lo = max(0, i - 8)
hi = min(len(lines), i + 22)
for j in range(lo, hi):
print(f"{j + 1}: {lines[j]}")
PYRepository: pingcap/docs
Length of output: 25759
🏁 Script executed:
set -eu
python3 - <<'PY'
from urllib.request import Request, urlopen
sha = "03148cd4adf3f26f5c8f529b89a30c1dba690713"
paths = ["dumpling/export/writer.go", "dumpling/export/ir.go", "dumpling/export/dump.go"]
for path in paths:
url = f"https://raw.githubusercontent.com/pingcap/tidb/{sha}/{path}"
req = Request(url, headers={"User-Agent": "review-inspection"})
with urlopen(req, timeout=20) as response:
lines = response.read().decode().splitlines()
print(f"== {path} ==")
for i, line in enumerate(lines):
if "setTableMetaFromRows" in line or ("func (w *Writer) WriteTableData" in line) or "ColumnInfos()" in line:
lo = max(0, i - 12)
hi = min(len(lines), i + 80)
print(f"-- lines {lo + 1}-{hi} --")
for j in range(lo, hi):
print(f"{j + 1}: {lines[j]}")
PYRepository: pingcap/docs
Length of output: 10011
Document Parquet support for --sql.
--sql --filetype parquet is accepted. Dumpling derives metadata from the SQL result and writes it through the Parquet writer. Replace “only for exporting to CSV files” with “for exporting to CSV or Parquet files.”
Suggested fix
-- The `--sql` option can be used only for exporting to CSV files.
+- The `--sql` option can be used for exporting to CSV or Parquet files.
qiancai
reviewed
Oct 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is changed, added, or deleted? (Required)
Clarify that
--sqlruns the SQL statement you provide directly when exporting CSV files. This removes the misleading wording that implied Dumpling executes the example statement on every table to be exported.Which TiDB version(s) do your changes apply to? (Required)
Tips for choosing the affected version(s):
By default, CHOOSE MASTER ONLY so your changes will be applied to the next TiDB major or minor releases. If your PR involves a product feature behavior change or a compatibility change, CHOOSE THE AFFECTED RELEASE BRANCH(ES) AND MASTER.
For details, see tips for choosing the affected versions.
What is the related PR or file link(s)?
AI agent involvement
Do your changes match any of the following descriptions?
Summary by CodeRabbit
--sqloption is limited to CSV exports.SELECTstatement and exports its results to CSV files.