[firestore-emulator: datastore-mode] No concurrent transaction manages to write successfully

[artoale]

[REQUIRED] Environment info

firebase-tools: Using the gcloud cli version 511.0.0, firestore emulator: 1.19.9
Platform: macOS (arm), python client library

[REQUIRED] Test case

import os
import time
import itertools

from google.cloud import datastore
import concurrent.futures


def get_new_client():
    return datastore.Client(
        project=os.environ.get("GCLOUDC_PROJECT_ID", "test"),
        namespace=None,
        _http=None,
    )


def sdk_increment_integer_entity(integer_pk):
    client = get_new_client()
    key = client.key("Integer", integer_pk)
    with client.transaction():
        entity = client.get(key=key)
        entity["value"] += 1
        client.put(entity)


def run_test():
    PORT = 10901
    os.environ["DATASTORE_EMULATOR_HOST"] = "127.0.0.1:%s" % PORT
    os.environ["DATASTORE_PROJECT_ID"] = "test"

    initial_value = 0
    concurrent_writes = 10
    futures = []

    local_client = get_new_client()

    key = local_client.key("Integer", 1)
    entity = datastore.Entity(key)
    entity["value"] = initial_value
    local_client.put(entity)


    with concurrent.futures.ThreadPoolExecutor(max_workers=concurrent_writes) as executor:
        for _ in range(concurrent_writes):
            futures.append(executor.submit(sdk_increment_integer_entity, 1))

    concurrent.futures.wait(futures)

    futures_exceptions = [future.exception() for future in futures]
    number_successful_writes = len(list(itertools.filterfalse(None, futures_exceptions)))
    if number_successful_writes == 0:
        print(futures_exceptions)
    else:
        print(f"All good, successful writes: {number_successful_writes}")

    from_db = local_client.get(key)

    assert number_successful_writes != 0
    assert from_db["value"] == number_successful_writes


def main():
    for _ in range(1, 50):
        run_test()
        time.sleep(2)


if __name__ == "__main__":
    main()

[REQUIRED] Steps to reproduce

1. Start the emulator with gcloud emulators firestore start --host-port=127.0.0.1:10901 --database-mode=datastore-mode
2. Install the python deps: pip install google-cloud-datastore==2.19.0
3. Run the above python script

[REQUIRED] Expected behavior

Runs successfully, without errors, every time

[REQUIRED] Actual behavior

Command fails with

[Aborted('Transaction lock timeout.'), Aborted('Transaction lock timeout.'), Aborted('Transaction lock timeout.'), Aborted('Transaction lock timeout.'), Aborted('Transaction lock timeout.'), Aborted('Transaction lock timeout.'), Aborted('Transaction lock timeout.'), Aborted('Transaction lock timeout.'), Aborted('Transaction lock timeout.'), Aborted('Transaction lock timeout.')]
Traceback (most recent call last):
  File "/Users/artoale/dev/gcloudc/./example.py", line 69, in <module>
    main()
  File "/Users/artoale/dev/gcloudc/./example.py", line 64, in main
    run_test()
  File "/Users/artoale/dev/gcloudc/./example.py", line 58, in run_test
    assert number_successful_writes != 0

I've tested the same script against two real firestore in datastore mode instances (one configured with pessimistic concurrency, one with optimistic) - running the script multiple time.

Transaction failing because of lock timeout make complete sense, but I would expect at least one transaction (typically the first to acquire the lock with a pessimistic model) to pass - but the test shows that they do not

[aalej]

Hey @artoale, thanks for creating a detailed report. I really appreciate it! I’ll also raise this to our engineering so they can take a look. Sharing the mcve here.