Skip to content

Commit 76a41a7

Browse files
committed
docs: improve troubleshooting and handle integrity check failures
Previously, the utility aborted immediately when an integrity check failed after restoring the database. It now prints a complete report along with the specific error that triggered the failure, making it easier to diagnose issues before exiting.
1 parent 7234256 commit 76a41a7

File tree

2 files changed

+64
-1
lines changed

2 files changed

+64
-1
lines changed

README.md

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,67 @@ blobcheck s3 --uri 's3://mybucket/cluster1_backup?AWS_ACCESS_KEY_ID=..&AWS_SECRE
7878
└──────┴────────────┴─────────────┴────────┘
7979
```
8080

81+
## Troubleshooting
82+
83+
When issues arise, you can use verbosity flags to understand what’s happening under the hood.
84+
85+
### Enable Debug Output
86+
87+
Running with `-v` enables debug logging. This shows all parameter combinations that `blobcheck` tries when connecting to the storage provider.
88+
89+
For example, connecting to a MinIO server with default settings may fail if virtual host–style requests are used (where the bucket name is treated as part of the hostname):
90+
91+
```text
92+
2025/09/29 14:32:54 DEBUG Trying params env="map[AWS_ACCESS_KEY_ID:cockroach AWS_ENDPOINT:http://localhost:29000 AWS_REGION:aws-global AWS_SECRET_ACCESS_KEY:******]"
93+
2025/09/29 14:32:54 DEBUG Failed to list objects error="operation error S3: ListObjectsV2, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get \"http://test.localhost:29000/?list-type=2\": dial tcp: lookup test.localhost: no such host" env="map
94+
```
95+
96+
In this case, blobcheck will continue trying alternative combinations until it finds one that works. The first successful combination is then used for backup/restore validation.
97+
98+
### Enable AWS SDK Tracing
99+
100+
Adding a second -v flag provides even deeper insight by enabling AWS SDK trace logs. These include full request/response details exchanged with the storage provider.
101+
102+
Using the same failing MinIO example, the output now shows the full request signature, headers, and why the request failed:
103+
104+
105+
```text
106+
2025/09/29 14:33:51 DEBUG Trying params env="map[AWS_ACCESS_KEY_ID:cockroach AWS_ENDPOINT:http://localhost:29000 AWS_REGION:aws-global AWS_SECRET_ACCESS_KEY:******]"
107+
SDK 2025/09/29 14:33:51 DEBUG Request Signature:
108+
---[ CANONICAL STRING ]-----------------------------
109+
GET
110+
/
111+
list-type=2
112+
accept-encoding:identity
113+
amz-sdk-invocation-id:4fad79e8-9ae8-43ba-b2a1-7036342f9295
114+
amz-sdk-request:attempt=1; max=1
115+
host:test.localhost:29000
116+
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
117+
x-amz-date:20250929T183351Z
118+
119+
accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date
120+
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
121+
---[ STRING TO SIGN ]--------------------------------
122+
AWS4-HMAC-SHA256
123+
20250929T183351Z
124+
20250929/us-east-1/s3/aws4_request
125+
7bbccaeea2b0816cf48e8b7e3e916fcb6d641044f7d9fd743c42eca6bcfc018b
126+
-----------------------------------------------------
127+
SDK 2025/09/29 14:33:51 DEBUG Request
128+
GET /?list-type=2 HTTP/1.1
129+
Host: test.localhost:29000
130+
User-Agent: aws-sdk-go-v2/1.39.0 ua/2.1 os/macos lang/go#1.24.2 md/GOOS#darwin md/GOARCH#arm64 api/s3#1.88.1 m/g
131+
Accept-Encoding: identity
132+
Amz-Sdk-Invocation-Id: 4fad79e8-9ae8-43ba-b2a1-7036342f9295
133+
Amz-Sdk-Request: attempt=1; max=1
134+
Authorization: AWS4-HMAC-SHA256 Credential=cockroach/20250929/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=e6ed31368624571de7b3b0bb01d658ada05001bb33a20217478c5f09aaaeee55
135+
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
136+
X-Amz-Date: 20250929T183351Z
137+
138+
SDK 2025/09/29 14:33:51 DEBUG request failed with unretryable error https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://test.localhost:29000/?list-type=2": dial tcp: lookup test.localhost: no such host
139+
2025/09/29 14:33:51 DEBUG Failed to list objects error="operation error S3: ListObjectsV2, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get \"http://test.localhost:29000/?list-type=2\": dial tcp: lookup test.localhost: no such host" env="map[AWS_ACCESS_KEY_ID:cockroach AWS_ENDPOINT:http://localhost:29000 AWS_REGION:aws-global AWS_SECRET_ACCESS_KEY:******]"
140+
```
141+
81142
---
82143

83144
## High-Level Architecture

internal/validate/validate.go

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -170,7 +170,9 @@ func (v *Validator) Validate(ctx *stopper.Context) (*Report, error) {
170170
}
171171

172172
if err := v.verifyIntegrity(ctx, conn); err != nil {
173-
return nil, err
173+
// If we fail to verify the integrity, just log the error, but
174+
// still provide a complete report
175+
slog.Error("failed to verify integrity", slog.Any("error", err))
174176
}
175177

176178
return &Report{

0 commit comments

Comments
 (0)