-
Notifications
You must be signed in to change notification settings - Fork 692
br: disable restore checksum #20847
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Tristan1900
wants to merge
3
commits into
pingcap:master
Choose a base branch
from
Tristan1900:disable-restore
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+48
−3
Open
br: disable restore checksum #20847
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -21,6 +21,7 @@ This document describes the commands of TiDB snapshot backup and restore accordi | |||||
- [Restore multiple tables with table filter](#restore-multiple-tables-with-table-filter) | ||||||
- [Restore execution plan bindings from the `mysql` schema](#restore-execution-plan-bindings-from-the-mysql-schema) | ||||||
- [Restore encrypted snapshots](#restore-encrypted-snapshots) | ||||||
- [Checksum](#checksum) | ||||||
|
||||||
For more information about snapshot backup and restore, refer to: | ||||||
|
||||||
|
@@ -48,7 +49,6 @@ In the preceding command: | |||||
|
||||||
> **Note:** | ||||||
> | ||||||
> - Starting from v8.5.0, the BR tool disables the table-level checksum calculation during full backups by default (`--checksum=false`) to improve backup performance. | ||||||
> - The BR tool already supports self-adapting to GC. It automatically registers `backupTS` (the latest PD timestamp by default) to PD's `safePoint` to ensure that TiDB's GC Safe Point does not move forward during the backup, thus avoiding manually setting GC configurations. | ||||||
|
||||||
During backup, a progress bar is displayed in the terminal, as shown below. When the progress bar advances to 100%, the backup is complete. | ||||||
|
@@ -178,7 +178,7 @@ In the preceding command: | |||||
- `--ratelimit`: The maximum speed **per TiKV** performing restore tasks. The unit is in MiB/s. | ||||||
- `--log-file`: The target file where the `br` log is written. | ||||||
|
||||||
During restore, a progress bar is displayed in the terminal as shown below. When the progress bar advances to 100%, the restore task is completed. Then `br` will verify the restored data to ensure data security. | ||||||
During restore, a progress bar is displayed in the terminal as shown below. When the progress bar advances to 100%, the restore task is completed. After the restoration is complete, if table-level checksum is enabled (see [Checksum](#checksum)), the BR tool performs table data verification to ensure the logical integrity of the data. File-level checksums are always performed to ensure the basic integrity of the restored files. | ||||||
|
||||||
```shell | ||||||
Split&Scatter Region <--------------------------------------------------------------------> 100.00% | ||||||
|
@@ -287,3 +287,46 @@ tiup br restore full\ | |||||
--crypter.method aes128-ctr \ | ||||||
--crypter.key 0123456789abcdef0123456789abcdef | ||||||
``` | ||||||
|
||||||
## Checksum | ||||||
|
||||||
Checksum is a method used by the BR tool to verify the integrity of backup and restore data. BR supports two levels of checksums: | ||||||
|
||||||
1. **File-level checksum**: Verifies the backup files themselves to ensure integrity during storage and transmission. This checksum is always enabled and cannot be disabled. | ||||||
2. **Table-level checksum**: Verifies the integrity of table data content and confirms the business logic consistency of the data. This checksum is disabled by default but can be enabled through parameters. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
||||||
Balancing performance and security considerations, BR handles table-level checksums as follows: | ||||||
|
||||||
### Backup Checksum | ||||||
|
||||||
Starting from v8.5.0, when performing full backups, the BR tool does not calculate table-level checksums by default (`--checksum=false`) to improve backup performance. If you need to calculate table-level checksums during backup, you can explicitly specify `--checksum=true`. File-level checksums will always be calculated to ensure the integrity of backup files. | ||||||
|
||||||
Calculating table-level checksums can verify data integrity during backup but increases backup time. In most cases, it's safe to use the default setting (no table-level checksum) to improve backup speed. | ||||||
|
||||||
### Restore Checksum | ||||||
|
||||||
Starting from v9.0.0, the BR tool does not perform table-level checksum verification (`--checksum=false`) by default during restore operations to improve restore performance. If you need to perform table-level checksum verification, you can explicitly specify `--checksum=true`. File-level checksum verification is always performed to ensure the basic integrity of restored data. | ||||||
|
||||||
After restoration, data validation is usually performed to ensure data security. When table-level checksums are disabled, the comprehensive validation step for table data is skipped, thereby accelerating the restore process. For scenarios with strict data integrity requirements, you may choose to enable table-level checksums. | ||||||
|
||||||
### Checksum Configuration Examples | ||||||
|
||||||
Enable table-level checksums during backup: | ||||||
|
||||||
```shell | ||||||
tiup br backup full \ | ||||||
--pd "${PD_IP}:2379" \ | ||||||
--storage "s3://${backup_collection_addr}/snapshot-${date}?access-key=${access-key}&secret-access-key=${secret-access-key}" \ | ||||||
--checksum=true \ | ||||||
--log-file backupfull.log | ||||||
``` | ||||||
|
||||||
Enable table-level checksums during restore: | ||||||
|
||||||
```shell | ||||||
tiup br restore full \ | ||||||
--pd "${PD_IP}:2379" \ | ||||||
--storage "s3://${backup_collection_addr}/snapshot-${date}?access-key=${access-key}&secret-access-key=${secret-access-key}" \ | ||||||
--checksum=true \ | ||||||
--log-file restorefull.log | ||||||
``` |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It might be helpful to add a brief introductory sentence before defining checksum to provide context. Make sure the documentation is easy to understand for TiDB users.1
Style Guide References
Footnotes
Make sure the documentation is easy to understand for TiDB users. (link) ↩