Proposal: multi-request transaction support for Fuseki, built on TDB2

[Jamie-SA]

Version

latest

Feature

Proposal: multi-request transaction support for Fuseki, built on TDB2

I've hoped for this for a long time, now it might be within grasp.
Yes, AI wrote this. I haven't dug into the Jena/Fuseki code, but I did review the proposal (and made minor edits).

Motivation

SPARQL 1.1/1.2 Protocol only defines atomic, single-shot query and update operations, there's no notion of a transaction spanning more than one HTTP request. This has come up repeatedly outside this project too: w3c/sparql-dev#83 is a long-running discussion of exactly this gap, and @afs already sketched an approach there:

An alternative is an addition query string parameter and use the usual endpoint for query/update etc. Given that security may apply on some endpoints based on URL (e.g. different URLs for query and update), using the same endpoints for operations with transactions and single operations would be helpful. It is also helpful to client libraries as the sequence "begin"-any existing code-"commit" works. /transactions/<id> is for transaction control.

This issue proposes building that out for Fuseki, on top of TDB2's existing internal ACID transactions (Transactional/TxnType, Serializable isolation). The TDB2 storage engine already does the hard part, what's missing is a way for a client to hold one of those transactions open across several HTTP requests.

Other triple stores have shipped support: RDF4J's REST API has a dedicated transaction resource, Blazegraph had a Transaction-Location header proposal over its REST Transaction API), Allegrograph supports transactions with session ports.

Proposed API

Following @afs's sketch:

• /ds/query and /ds/update stay exactly as they are for ordinary, non-transactional requests — zero behavior change for existing clients.
• New control endpoint, /ds/transactions (a new Operation, dispatched the same way Query/Update/GSP_* are today via OperationRegistry/DataService):
  • POST /ds/transactions?type={read|write|read-promote|read-committed-promote} — begins a transaction (TxnType maps directly onto the existing enum in org.apache.jena.query.TxnType). Response: Location: /ds/transactions/{txnId}, txnId a server-generated opaque token (UUID).
  • POST /ds/transactions/{txnId}?action=commit — commit.
  • POST /ds/transactions/{txnId}?action=abort — abort/rollback.
  • GET /ds/transactions/{txnId} — status (active/idle-time remaining), mainly for diagnostics.
• A transaction-scoped request is just an ordinary /ds/query or /ds/update call carrying an SPARQL-Transaction: {txnId} request header. No other change to the query/update request shape.

Mapping onto the existing Jena/Fuseki internals

(Filed against current main; class/line references may drift.)

• HttpAction (jena-fuseki2/jena-fuseki-core/.../servlets/HttpAction.java) currently assumes one begin/end pair per request, on one thread (isInActionTxn, activeDSG are set up in begin(TxnType) and torn down in endInternal() within the same servlet call — see SPARQLQueryProcessor.execute() and SPARQL_Update.execute()). A held, cross-request transaction can't just reuse this path unmodified, since a transaction begun on one request's thread must be resumed on a different request's thread.
• TDB2 already has the primitive this needs: DatasetGraphTDB's TransactionalSystem (org.apache.jena.dboe.transaction.txn.TransactionalBase, in jena-dboe-transaction) implements

  TransactionCoordinatorState detach();
  void attach(TransactionCoordinatorState coordinatorState);

  which suspends a transaction off its current thread and resumes it on another. It's just not exposed above DatasetGraphTDB today — reaching it means unwrapping to the TDB2-specific type. Proposal: expose a narrow SPI (e.g. DetachableTransactional with default no-op detach()/attach(), implemented for real by TDB2's DatasetGraphTDB) so Fuseki can ask a DatasetGraph "can you hand a transaction across threads?" generically, and reject/501 the new endpoint for datasets that answer no.
• TDB1 has no equivalent. DatasetGraphTransaction binds via a plain ThreadLocal<DatasetGraphTxn> with no suspend/resume API. Same for in-memory and other non-TDB2 DatasetGraphs. v1 of this proposal is TDB2-only; other backends would report "not detachable" and a POST /ds/transactions against them would fail cleanly rather than half-working.
• A new server-side registry (per DataService, alongside its existing activeTxn/totalTxn counters in DataService.java) holds {txnId -> HeldTransaction}, where HeldTransaction wraps the detached TransactionCoordinatorState, the TxnType, and a last-touched timestamp. On a query/update request carrying SPARQL-Transaction, Fuseki looks up the entry, attach()s it for the duration of that one request, executes normally, then detach()s again in finally — explicit commit()/abort() only happen via the control endpoint.
• Abandoned-transaction handling is not optional. A client that opens a write transaction and disappears would otherwise block every other writer indefinitely, and a leaked read transaction on the fallback lock path (TransactionalLock/LockMRSW, for non-TDB2 datasets, though those are out of scope here) can walk into the ReentrantReadWriteLock "maximum lock count exceeded" failure Jena has hit before (e.g. java error Maximum lock count exceeded #1499, java.lang.Error: Maximum lock count exceeded #2584). Proposal: an idle-timeout reaper thread per dataset that force-aborts held transactions past a configurable deadline, plus a configurable cap on concurrently-held transactions per dataset.
• Multi-instance deployments are explicitly out of scope for v1. A held transaction is in-process state on one Fuseki instance; it has no meaning across a farm of Fuseki instances behind a load balancer without a shared coordinator. This should be documented as a hard limitation, not silently broken — an operator running Fuseki behind a non-sticky LB needs to know this feature won't work as expected there.
• Suggest packaging this as an optional FusekiModule (jena-fuseki-main's pluggable module mechanism) rather than baking it into fuseki-core/fuseki-main unconditionally, so operators who don't want the operational risk of server-held locks (timeouts, cap tuning, the multi-instance caveat above) can simply not load it, and the core query/update path is untouched either way.

Non-goals

• Distributed/XA transactions spanning multiple datasets or multiple Fuseki instances.
• Any change to the semantics or wire format of a plain, non-transactional /query or /update request.
• TDB1 or non-TDB2 backend support (v1 rejects cleanly; could be revisited later if there's demand).

Open questions for maintainers

1. Endpoint/header naming above is a starting sketch, not a bikeshed I'm attached to — SPARQL-Transaction vs. a query parameter (both were discussed in Multi-request transaction support in the SPARQL protocol w3c-cg/sparql-dev#83) is worth a decision before implementation starts.
2. Does packaging as a FusekiModule (opt-in) vs. a built-in fuseki-core feature match how you'd want this maintained?
3. Is TDB2-only for v1 acceptable, or is TDB1 parity a hard requirement before this would be considered?

Would the maintainers be open to a PR along these lines? Happy to adjust the design based on feedback before investing in an implementation — in particular I'd rather settle the API shape and TDB1-scope questions above first.

• Are you interested in contributing a solution yourself? Yes, pending agreement on the approach above.

Are you interested in contributing a solution yourself?

Yes

[m3l0n-sq]

This can be achieved with a custom Fuseki module in the meantime. I've developed it for use in our application following RDF4J's specs (since we migrated from a RDF4J based triple store) and has been stable so far.

[afs]

Interesting! Is it available publicly anywhere?

[Jamie-SA]

@m3l0n-sq thanks for the suggestion. I've gone ahead and had Claude create a module for me.

@afs I can post my module as a separate repo, or I can fork Jena and add it and send a PR request if you think you would accept it.

Also, Claude found (what it thinks is) a bug in TDB2 while working on it. See link/mention above this comment for the new github issue.

[afs]

@Jamie-SA Thank you for this. It would be good to have

AI wrote this.

It will take a while before I can do a deep review of this proposal.
The key part is the architecture.

Overall, as with all big AI-generated PRs, we need to consider the how the project provides long-term support, both for the code and for user questions.

FusekiModule

Later. First, let's discuss what it is and how it does it.
Modifying request dispatch is not module-territory.

ATM I am more in favor of "doing it right", than being constrained by the module system and assumptions.

Following @afs's sketch:

Which sketch? (there have been several :-)

This issue proposes building that out for Fuseki, on top of TDB2's existing internal ACID transactions

I believe we have to include in-memory as well (there is an ACID implementation "TIM").

I do not see TDB1 as necessarily in-scope. It would be harder to adapt and it can also become memory intensive in ways that TDB2 addresses - that destabilizes the whole server.

 

There is no need to go into the internals of TDB2 parking transactions.

One approach for Fuseki is to have a dedicated thread for the multi-request transaction, and route requests to that thread during Fuseki dispatch. This is much less intrusive. It covers all database types.

[m3l0n-sq]

Our module took quite a bumpy road to reach a level of stability, and its particularly with the aborts. TDB2 aborts, especially for larger transactions, takes a while to complete unlike RDF4J which is instantaneous. It's not open sourced, and while we would like to create a PR to contribute back, we remain hesitant as it's mostly AI generated (though with human in the loop of course).

And about the threads, yeap we went with the approach @afs stated, routing requests to the thread that started the transaction. Doesn't require any changes to the tdb2 internals.

[Jamie-SA]

It will take a while before I can do a deep review of this proposal. The key part is the architecture.

No problem.

I'm moving forward on a module solution for myself and happy to keep working on something that you would be satisfied with. I've fed your reply into the AI and it has made changes based on it (the module now supports TIM) and it has a revised design not based on modules. But I don't want to throw a bunch of AI stuff at you without getting more direction from you.

Overall, as with all big AI-generated PRs, we need to consider the how the project provides long-term support, both for the code and for user questions.

I'm not sure if I know all of your specific concerns. If something gets merged, I am open to trying to help with anything related to the code.

Right now the module solution is about 700 lines (as reported by wc, includes headers etc), not including tests. I hope to try and get it to run with our tools this week and verify that it really works.

FusekiModule

Later. First, let's discuss what it is and how it does it. Modifying request dispatch is not module-territory.

ATM I am more in favor of "doing it right", than being constrained by the module system and assumptions.

Yes, definitely.

I have no experience with Jena/Fuseki internals (which is why I didn't attempt this before) so I probably am not a lot of help on the "doing it right" aspect other than general code quality. So I am open to any feedback from you that you want to give.

Following @afs's sketch:

Which sketch? (there have been several :-)

I think this one: w3c-cg/sparql-dev#83 (comment)

There didn't seem much consensus in that thread on mechanism. There hasn't been much progress on SPARQL 1.2 but this an issue I'd like to see addressed consistently someday.

This issue proposes building that out for Fuseki, on top of TDB2's existing internal ACID transactions

I believe we have to include in-memory as well (there is an ACID implementation "TIM").

Yes, that seems to be an obvious miss on the proposal.

One approach for Fuseki is to have a dedicated thread for the multi-request transaction, and route requests to that thread during Fuseki dispatch. This is much less intrusive. It covers all database types.

Makes sense. After the issue that the AI found, I think it switched to and implemented a thread per transaction solution.

[Jamie-SA]

If anyone wants to look at the module so far: https://github.com/Jamie-SA/fuseki-mod-transactions

[Jamie-SA]

Integration testing with that module has shown some of the limitations of using a module to get this functionality.

[afs]

Integration testing with that module has shown some of the limitations of using a module to get this functionality.

Please explain.

[afs]

Ask yourself the following questions:

• Would you deploy it to an SA customer?
• How does it get maintained in 2 years, 5 years time?

Jena's bottleneck is reviewing - that is the bottleneck for many OSS projects once they get the first complete version out.

-- DISPATCH-LEVEL-DESIGN.md --

Why this can't be done with today's extension points

That statement is wrong. The logic given is just awful. Wring in big bold letters

The tests in https://github.com/Jamie-SA/fuseki-mod-transactions are no help. All they are is testing is the wrong design.

Having AI write the code and write the tests does not add anything.

Transaction code is deep inside Fuseki. If there is a bug in released code, everyone is affected, and that includes possibly loosing the database.

To give over architecture to an AI is a one-way decision for the project.

[Jamie-SA]

Integration testing with that module has shown some of the limitations of using a module to get this functionality.

Please explain.

From my understanding there are two ways to implement the module and they both have some downsides. The way the current module was implemented seem to require a lot of extra (somewhat redundant code like to get content negotiation) to get back standard functionality (this is what my comment was initially about). Found another way that seems better, like what jena-fuseki-access does, but that seems to be non-stackable and touches internals of objects that maybe it shouldn't.

Of course, this comes without a deep understanding of all the code and could be wrong.