metamorphic: TestMetaTwoInstance doesn't test what it should

[RaduBerinde]

See #4760 (comment)

Jira issue: PEBBLE-455

[RaduBerinde]

I think the root cause is that we're never testing shared storage without value separation. We never disable value separation when we randomize options, and it wouldn't matter because we never write out disable_value_separation=true when it is disabled.

[RaduBerinde]

Ah, no, value separation doesn't disable shared storage or shared storage replication, it's the other way around.

[RaduBerinde]

Ah, yes, value separation does disable shared ingest in the metamorphic test:

pebble/metamorphic/ops.go

Line 2137 in bd64861

!t.testOpts.Opts.Experimental.ValueSeparationPolicy().Enabled

[RaduBerinde]

Fixed in #4776. Will leave open until this remaining TODO is addressed as well:

pebble/metamorphic/ops.go

Line 2137 in 5dd6908

// TODO(radu): allow shared ingest with value separation. We automatically

[jbowens]

Possibly relatedly, got a failure in CI:

https://github.com/cockroachdb/pebble/actions/runs/15333122628/job/43144408271

Fails on master with:

 go test -run TestMetaTwoInstance -seed 1748551294968475000 -exec 'stress -p 1'

--- FAIL: TestMetaTwoInstance (7.67s)
    --- FAIL: TestMetaTwoInstance/execution (0.00s)
        --- FAIL: TestMetaTwoInstance/execution/random-025 (2.43s)
            meta.go:244: error running /var/folders/pd/06dx48hs4y71k15rr2mw4sdm0000gq/T/go-build492671555/b001/metamorphic.test -keep=false -run-dir=_meta/250529-164705.6841215609899/random-025 -test.run=TestMetaTwoInstance$ --op-timeout=2m0s --key-format=testkeys --num-instances=2
                ===== SEED =====
                1748551294968475000
                ===== ERR =====
                exit status 2
                ===== OUT =====
                panic: element has outstanding references [recovered]
                	panic: pebble: evicting in-use file 001470(blob): element has outstanding references
                
                goroutine 81 [running]:
                github.com/cockroachdb/pebble.(*fileCacheHandle).Evict.func1()
                	/Users/jackson/go/src/github.com/cockroachdb/pebble/file_cache.go:279 +0xcc
                panic({0x100da5400?, 0x100ec82b0?})
                	/usr/local/go/src/runtime/panic.go:785 +0x124
                github.com/cockroachdb/pebble/internal/genericcache.(*shard[...]).Evict(0x100edf560, {0x14000354000, 0x5be, 0x7})
                	/Users/jackson/go/src/github.com/cockroachdb/pebble/internal/genericcache/shard.go:322 +0x120
                github.com/cockroachdb/pebble/internal/genericcache.(*Cache[...]).Evict(0x100edcf80, {0x14000354000, 0x100014000600808?, 0x5d5?})
                	/Users/jackson/go/src/github.com/cockroachdb/pebble/internal/genericcache/cache.go:120 +0x84
                github.com/cockroachdb/pebble.(*fileCacheHandle).Evict(0x14000354000, 0x5be, 0x100dd0e01?)
                	/Users/jackson/go/src/github.com/cockroachdb/pebble/file_cache.go:282 +0x74
                github.com/cockroachdb/pebble.(*DB).deleteObsoleteFiles(0x14000283208, 0x2d4)
                	/Users/jackson/go/src/github.com/cockroachdb/pebble/obsolete_files.go:562 +0x778
                github.com/cockroachdb/pebble.(*DB).maybeScheduleObsoleteObjectDeletion(0x140013b3c08?)
                	/Users/jackson/go/src/github.com/cockroachdb/pebble/obsolete_files.go:578 +0xd4
                github.com/cockroachdb/pebble.(*readState).unref(0x14000c3cd80)
                	/Users/jackson/go/src/github.com/cockroachdb/pebble/read_state.go:48 +0xd0
                github.com/cockroachdb/pebble.(*scanInternalIterator).close(0x140013fae08)
                	/Users/jackson/go/src/github.com/cockroachdb/pebble/scan_internal.go:1131 +0x54
                github.com/cockroachdb/pebble.(*DB).ScanInternal(0x12?, {0x100ed2870, 0x1013800e0}, 0x0?, {0x140004cb848, 0x6, 0x8}, {0x140004cb860, 0xc, 0x10}, ...)
                	/Users/jackson/go/src/github.com/cockroachdb/pebble/db.go:1294 +0x138
                github.com/cockroachdb/pebble/metamorphic.(*replicateOp).runSharedReplicate(0x140004cc200, 0x1400011a120, {0x14000119860?, 0x3c?, 0x0?}, 0x14000283208, 0x14000238488, 0x140005d6460, {0x14000e75bd0, 0x48})
                	/Users/jackson/go/src/github.com/cockroachdb/pebble/metamorphic/ops.go:2004 +0x1ac
                github.com/cockroachdb/pebble/metamorphic.(*replicateOp).run(0x140004cc200, 0x1400011a120, {0x14000119860?, 0x14000c81bc0?, 0x0?})
                	/Users/jackson/go/src/github.com/cockroachdb/pebble/metamorphic/ops.go:2158 +0x858
                github.com/cockroachdb/pebble/metamorphic.(*Test).runOp(0x1400011a120?, 0x1400094bed0?, {0x14000119860?, 0x0?, 0x0?})
                	/Users/jackson/go/src/github.com/cockroachdb/pebble/metamorphic/test.go:440 +0x1a8
                github.com/cockroachdb/pebble/metamorphic.Execute.func1()
                	/Users/jackson/go/src/github.com/cockroachdb/pebble/metamorphic/meta.go:632 +0x114
                golang.org/x/sync/errgroup.(*Group).Go.func1()
                	/Users/jackson/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x54
                created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 8
                	/Users/jackson/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x98
 

[RaduBerinde]

Thanks, I think we need to close our blob value fetcher before releasing the read state.