sql/inspect: add hash-based precheck to index consistency validation

[spilchen]

Before performing the expensive full join to compare primary and secondary indexes during the INSPECT index consistency check, this change adds a fast, low-cost precheck step.

We now compute and compare row counts and hash values from both indexes. If the counts and hashes match, we can confidently skip the full check. This optimization should improves performance for healthy indexes.

Closes: #150927
Epic: CRDB-55075
Release note: None

[cockroach-teamcity]

This change is 

[spilchen]

Here are the results of the performance tests when I ran this in roachtest:

---

1. Admission Control Test (25M rows)

Fast version:

• Calibration INSPECT (no workload): 3m58.8s (3,272 rows/s/cpu)
• INSPECT under load (AC enabled): 3m21.7s (3,874 rows/s/cpu)
• INSPECT under load (AC disabled): 42.9s (18,223 rows/s/cpu)

Slow version:

• Calibration INSPECT (no workload): 11m6.6s (1,172 rows/s/cpu)
• INSPECT under load (AC enabled): 24m55.5s (522 rows/s/cpu)
• INSPECT under load (AC disabled): 2m24.8s (5,396 rows/s/cpu)

Performance improvement (fast vs slow):

• Calibration: 2.8x faster (11m6s → 3m59s)
• Under load with AC: 7.4x faster (24m55s → 3m22s)
• Under load without AC: 3.4x faster (2m25s → 43s)

---

2. Throughput Test - 500M rows, 1 index

Fast version:

• INSPECT execution: 32m0.4s

Slow version:

• INSPECT execution: 58m53.9s

Performance improvement:

• 1.8x faster (58m54s → 32m0s)

---

3. Throughput Test - 1B rows, 2 indexes

Fast version:

• INSPECT with 1 check: 1h53m0.5s
• INSPECT with 2 checks: 2h29m45.4s

Slow version:

• INSPECT with 1 check: 2h25m15.8s
• INSPECT with 2 checks: 3h51m30.7s

Performance improvement:

• 1 check: 1.3x faster (2h25m16s → 1h53m1s)
• 2 checks: 1.6x faster (3h51m31s → 2h29m45s)

[rafiss]

this is a great improvement, nice work! lgtm -- my comments are just about error wrapping and word choice, so feel free to merge after responding.

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @spilchen)

---

pkg/sql/inspect/index_consistency_check.go line 242 at r1 (raw file):

			// be ignored.
			if isQueryConstructionError(hashErr) {
				return hashErr

shouldn't we never expect a query construction error? if so, then this could be wrapped with an assertion failure.

---

pkg/sql/inspect/index_consistency_check.go line 244 at r1 (raw file):

				return hashErr
			}
			log.Dev.Infof(ctx, "hash precheck for index consistency failed; falling back to full check: %v", hashErr)

super nit: instead of "failed" could we use language closer to "did not match"? i'm thinking that might make this log message more clear, since although the check did "fail" someone who sees this might think that CRDB itself encountered a failure.

[spilchen]

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @rafiss)

---

pkg/sql/inspect/index_consistency_check.go line 242 at r1 (raw file):

Previously, rafiss (Rafi Shamim) wrote…

shouldn't we never expect a query construction error? if so, then this could be wrapped with an assertion failure.

Yes, makes sense. We should never expect this since it's an internally generated query.

---

pkg/sql/inspect/index_consistency_check.go line 244 at r1 (raw file):

Previously, rafiss (Rafi Shamim) wrote…

super nit: instead of "failed" could we use language closer to "did not match"? i'm thinking that might make this log message more clear, since although the check did "fail" someone who sees this might think that CRDB itself encountered a failure.

Done.

[spilchen]

TFTR!

bors r+

[craig]

Build succeeded:

• acceptance
• linux_amd64_fips_build
• linux_amd64_build
• check_generated_code
• examples_orms
• docker_image_amd64
• local_roachtest
• local_roachtest_fips
• lint
• unit_tests