Skip to content

docs: improve troubleshooting and handle integrity check failures #19

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Oct 1, 2025
Merged

docs: improve troubleshooting and handle integrity check failures #19

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 61 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,67 @@ blobcheck s3 --uri 's3://mybucket/cluster1_backup?AWS_ACCESS_KEY_ID=..&AWS_SECRE
└──────┴────────────┴─────────────┴────────┘
```

## Troubleshooting

When issues arise, you can use verbosity flags to understand what’s happening under the hood.

### Enable Debug Output

Running with `-v` enables debug logging. This shows all parameter combinations that `blobcheck` tries when connecting to the storage provider.

For example, connecting to a MinIO server with default settings may fail if virtual host–style requests are used (where the bucket name is treated as part of the hostname):

```text
2025/09/29 14:32:54 DEBUG Trying params env="map[AWS_ACCESS_KEY_ID:cockroach AWS_ENDPOINT:http://localhost:29000 AWS_REGION:aws-global AWS_SECRET_ACCESS_KEY:******]"
2025/09/29 14:32:54 DEBUG Failed to list objects error="operation error S3: ListObjectsV2, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get \"http://test.localhost:29000/?list-type=2\": dial tcp: lookup test.localhost: no such host" env="map
```

In this case, blobcheck will continue trying alternative combinations until it finds one that works. The first successful combination is then used for backup/restore validation.

### Enable AWS SDK Tracing

Adding a second -v flag provides even deeper insight by enabling AWS SDK trace logs. These include full request/response details exchanged with the storage provider.

Using the same failing MinIO example, the output now shows the full request signature, headers, and why the request failed:


```text
2025/09/29 14:33:51 DEBUG Trying params env="map[AWS_ACCESS_KEY_ID:cockroach AWS_ENDPOINT:http://localhost:29000 AWS_REGION:aws-global AWS_SECRET_ACCESS_KEY:******]"
SDK 2025/09/29 14:33:51 DEBUG Request Signature:
---[ CANONICAL STRING ]-----------------------------
GET
/
list-type=2
accept-encoding:identity
amz-sdk-invocation-id:4fad79e8-9ae8-43ba-b2a1-7036342f9295
amz-sdk-request:attempt=1; max=1
host:test.localhost:29000
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20250929T183351Z

accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
---[ STRING TO SIGN ]--------------------------------
AWS4-HMAC-SHA256
20250929T183351Z
20250929/us-east-1/s3/aws4_request
7bbccaeea2b0816cf48e8b7e3e916fcb6d641044f7d9fd743c42eca6bcfc018b
-----------------------------------------------------
SDK 2025/09/29 14:33:51 DEBUG Request
GET /?list-type=2 HTTP/1.1
Host: test.localhost:29000
User-Agent: aws-sdk-go-v2/1.39.0 ua/2.1 os/macos lang/go#1.24.2 md/GOOS#darwin md/GOARCH#arm64 api/s3#1.88.1 m/g
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 4fad79e8-9ae8-43ba-b2a1-7036342f9295
Amz-Sdk-Request: attempt=1; max=1
Authorization: AWS4-HMAC-SHA256 Credential=cockroach/20250929/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=e6ed31368624571de7b3b0bb01d658ada05001bb33a20217478c5f09aaaeee55
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20250929T183351Z

SDK 2025/09/29 14:33:51 DEBUG request failed with unretryable error https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "http://test.localhost:29000/?list-type=2": dial tcp: lookup test.localhost: no such host
2025/09/29 14:33:51 DEBUG Failed to list objects error="operation error S3: ListObjectsV2, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get \"http://test.localhost:29000/?list-type=2\": dial tcp: lookup test.localhost: no such host" env="map[AWS_ACCESS_KEY_ID:cockroach AWS_ENDPOINT:http://localhost:29000 AWS_REGION:aws-global AWS_SECRET_ACCESS_KEY:******]"
```

---

## High-Level Architecture
Expand Down
4 changes: 3 additions & 1 deletion internal/validate/validate.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -170,7 +170,9 @@ func (v *Validator) Validate(ctx *stopper.Context) (*Report, error) {
}

if err := v.verifyIntegrity(ctx, conn); err != nil {
return nil, err
// If we fail to verify the integrity, just log the error, but
// still provide a complete report
slog.Error("failed to verify integrity", slog.Any("error", err))
}

return &Report{
Expand Down