CASSJAVA-97: Let users inject an ID for each request and write to the custom payload #2037

SiyaoIsHiding · 2025-04-16T00:11:05Z

No description provided.

SiyaoIsHiding · 2025-04-16T00:18:35Z

I did integration testing with C* OSS 5.0.2. @lukasz-antoniak helped me add a LoggingQueryHandler and set it as the cassandra.custom_query_handler_class.
I developed a client app using this Java driver with the following config

datastax-java-driver.advanced = {
  distributed-tracing.id-generator.class = W3CContextDistributedTraceIdGenerator
  distributed-tracing.custom-payload-with-key = "traceparent"
}

Running this client app, I got

17:03:20.860 [s0-io-5] TRACE InFlightHandler - [s0|id: 0xeefaab65, L:/127.0.0.2:52389 - R:/127.0.0.2:9042] Writing 00-d51ed2012c1f31b4434f409e50294da9-25da248d0a23981c-00 on stream id 0
17:03:20.863 [s0-io-5] TRACE CqlRequestHandler$NodeResponseCallback - [00-d51ed2012c1f31b4434f409e50294da9-25da248d0a23981c-00] Request sent on [id: 0xeefaab65, L:/127.0.0.2:52389 - R:/127.0.0.2:9042]
17:03:20.864 [s0-io-5] TRACE CqlRequestHandler$NodeResponseCallback - [00-d51ed2012c1f31b4434f409e50294da9-25da248d0a23981c-00] Speculative execution policy returned -1, no next execution
17:03:20.877 [s0-io-5] DEBUG InFlightHandler - [s0|id: 0xeefaab65, L:/127.0.0.2:52389 - R:/127.0.0.2:9042] Got last response on in-flight stream id 0, completing and releasing
17:03:20.877 [s0-io-5] TRACE InFlightHandler - [s0|id: 0xeefaab65, L:/127.0.0.2:52389 - R:/127.0.0.2:9042] Releasing stream id 0
17:03:20.877 [s0-io-5] TRACE CqlRequestHandler$NodeResponseCallback - [00-d51ed2012c1f31b4434f409e50294da9-25da248d0a23981c-00] Got result, completing

And the debug.log at server side got

DEBUG [Native-Transport-Requests-1] 2025-04-15 17:03:20,870 LoggingQueryHandler.java:44 - Processing CQL statement SelectStatement[aggregationSpecFactory=,bindVariables=[],isReversed=false,limit=,orderingComparator=,parameters=org.apache.cassandra.cql3.statements.SelectStatement$Parameters@5d46ec82,perPartitionLimit=,restrictions=StatementRestrictions[clusteringColumnsRestrictions=ClusteringColumnRestrictions[allowFiltering=false,comparator=comparator(),restrictions=RestrictionSet[hasAnn=false,hasContains=false,hasIn=false,hasMultiColumnRestrictions=false,hasOnlyEqualityRestrictions=true,hasSlice=false,restrictions={}]],filterRestrictions=IndexRestrictions[customExpressions=[],regularRestrictions=[]],hasRegularColumnsRestrictions=false,isKeyRange=true,nonPrimaryKeyRestrictions=RestrictionSet[hasAnn=false,hasContains=false,hasIn=false,hasMultiColumnRestrictions=false,hasOnlyEqualityRestrictions=true,hasSlice=false,restrictions={}],notNullColumns=[],partitionKeyRestrictions=PartitionKeySingleRestrictionSet[comparator=comparator(org.apache.cassandra.db.marshal.UTF8Type),restrictions=RestrictionSet[hasAnn=false,hasContains=false,hasIn=false,hasMultiColumnRestrictions=false,hasOnlyEqualityRestrictions=true,hasSlice=false,restrictions={}]],table=system.local,type=SELECT,usesSecondaryIndexing=false],selection=SimpleSelection{columns=[key, bootstrapped, broadcast_address, broadcast_port, cluster_name, cql_version, data_center, gossip_generation, host_id, listen_address, listen_port, native_protocol_version, partitioner, rack, release_version, rpc_address, rpc_port, schema_version, tokens, truncated_at], columnMapping={ Columns:[key, bootstrapped, broadcast_address, broadcast_port, cluster_name, cql_version, data_center, gossip_generation, host_id, listen_address, listen_port, native_protocol_version, partitioner, rack, release_version, rpc_address, rpc_port, schema_version, tokens, truncated_at], Mappings:{rack:[rack], cql_version:[cql_version], listen_address:[listen_address], release_version:[release_version], data_center:[data_center], broadcast_port:[broadcast_port], broadcast_address:[broadcast_address], partitioner:[partitioner], host_id:[host_id], gossip_generation:[gossip_generation], listen_port:[listen_port], rpc_address:[rpc_address], schema_version:[schema_version], rpc_port:[rpc_port], truncated_at:[truncated_at], cluster_name:[cluster_name], native_protocol_version:[native_protocol_version], tokens:[tokens], key:[key], bootstrapped:[bootstrapped]} }, metadata=[key(system, local), org.apache.cassandra.db.marshal.UTF8Type][bootstrapped(system, local), org.apache.cassandra.db.marshal.UTF8Type][broadcast_address(system, local), org.apache.cassandra.db.marshal.InetAddressType][broadcast_port(system, local), org.apache.cassandra.db.marshal.Int32Type][cluster_name(system, local), org.apache.cassandra.db.marshal.UTF8Type][cql_version(system, local), org.apache.cassandra.db.marshal.UTF8Type][data_center(system, local), org.apache.cassandra.db.marshal.UTF8Type][gossip_generation(system, local), org.apache.cassandra.db.marshal.Int32Type][host_id(system, local), org.apache.cassandra.db.marshal.UUIDType][listen_address(system, local), org.apache.cassandra.db.marshal.InetAddressType][listen_port(system, local), org.apache.cassandra.db.marshal.Int32Type][native_protocol_version(system, local), org.apache.cassandra.db.marshal.UTF8Type][partitioner(system, local), org.apache.cassandra.db.marshal.UTF8Type][rack(system, local), org.apache.cassandra.db.marshal.UTF8Type][release_version(system, local), org.apache.cassandra.db.marshal.UTF8Type][rpc_address(system, local), org.apache.cassandra.db.marshal.InetAddressType][rpc_port(system, local), org.apache.cassandra.db.marshal.Int32Type][schema_version(system, local), org.apache.cassandra.db.marshal.UUIDType][tokens(system, local), org.apache.cassandra.db.marshal.SetType(org.apache.cassandra.db.marshal.UTF8Type)][truncated_at(system, local), org.apache.cassandra.db.marshal.MapType(org.apache.cassandra.db.marshal.UUIDType,org.apache.cassandra.db.marshal.BytesType)]},table=system.local] with custom payload {traceparent=30302d64353165643230313263316633316234343334663430396535303239346461392d323564613234386430613233393831632d3030}

The value 30302d64353165643230313263316633316234343334663430396535303239346461392d323564613234386430613233393831632d3030 is the hex of the id 00-d51ed2012c1f31b4434f409e50294da9-25da248d0a23981c-00. This shows the capability of tracing a request across client and server.

core/revapi.json

...ava/com/datastax/oss/driver/internal/core/tracker/W3CContextDistributedTraceIdGenerator.java

core/src/main/java/com/datastax/oss/driver/internal/core/cql/CqlRequestHandler.java

lukasz-antoniak · 2025-04-16T05:51:50Z

core/src/main/java/com/datastax/oss/driver/internal/core/cql/CqlRequestHandler.java

+      // We cannot do statement.getCustomPayload().put() because the default empty map is abstract
+      // But this will create new Statement instance for every request. We might want to optimize
+      // this
+      Map<String, ByteBuffer> existingMap = new HashMap<>(statement.getCustomPayload());


Statement is by design immutable. Maybe a nicer way would be to create method StatementBuilder.from(Statement) where you could create builder again based on statement. The code would look like: StatementBuilder.from(statement).addCustomPayload(...).build().

I think you can copy just the payload, not the whole statement:

Map<String, ByteBuffer> customPayload = statement.getCustomPayload(); if (!this.customPayloadKey.isEmpty()) { customPayload = NullAllowingImmutableMap.<String, ByteBuffer>builder() .putAll(customPayload) .put( this.customPayloadKey, ByteBuffer.wrap(nodeRequestId.getBytes(StandardCharsets.UTF_8))) .build(); }

Then modify line 307 like so:

channel - .write(message, statement.isTracing(), statement.getCustomPayload(), nodeResponseCallback) + .write(message, statement.isTracing(), customPayload, nodeResponseCallback) .addListener(nodeResponseCallback);

This solves the concurrency problem, but it also means the subsequent setFinalError(statement...), NodeResponseCallback(statement,...), and RequestTracker invocations do not have the statement with the actual custom payload.

lukasz-antoniak · 2025-04-16T05:55:31Z

core/src/main/java/com/datastax/oss/driver/internal/core/cql/CqlRequestHandler.java

+      Map<String, ByteBuffer> existingMap = new HashMap<>(statement.getCustomPayload());
+      existingMap.put(
+          this.customPayloadKey, ByteBuffer.wrap(nodeRequestId.getBytes(StandardCharsets.UTF_8)));
+      statement = statement.setCustomPayload(existingMap);


Overriding custom payload here is not thread-safe. If client application executes the same statement instance multiple times concurrently (not a good use-case, but still possible), we do not guarantee how this map will be changed. Maybe indeed, there is no other way than make a shallow copy of the statement. Will think about it.

/** * Sets the custom payload to use for execution. * * <p>All the driver's built-in statement implementations are immutable, and return a new instance * from this method. However custom implementations may choose to be mutable and return the same * instance. * * <p>Note that it's your responsibility to provide a thread-safe map. This can be achieved with a * concurrent or immutable implementation, or by making it effectively immutable (meaning that * it's never modified after being set on the statement). */ @NonNull @CheckReturnValue SelfT setCustomPayload(@NonNull Map<String, ByteBuffer> newCustomPayload);

core/src/main/java/com/datastax/oss/driver/api/core/session/SessionBuilder.java

...main/java/com/datastax/oss/driver/internal/core/tracker/UuidDistributedTraceIdGenerator.java

core/src/main/java/com/datastax/oss/driver/api/core/tracker/DistributedTraceIdGenerator.java

...test/java/com/datastax/oss/driver/internal/core/tracker/DistributedTraceIdGeneratorTest.java

Yuqi-Du · 2025-05-19T15:35:57Z

...-tests/src/test/java/com/datastax/oss/driver/core/tracker/DistributedTraceIdGeneratorIT.java

+    try (CqlSession session = SessionUtils.newSession(ccmRule, loader)) {
+      String query = "SELECT * FROM system.local";
+      ResultSet rs = session.execute(query);
+      ByteBuffer id = rs.getExecutionInfo().getRequest().getCustomPayload().get("trace_key");


how do you inject for individual CQL request though?
Did i miss this kind of test?

I added a test "should_use_customized_request_id_generator". Do you think it answers your question?

absurdfarce · 2025-05-20T15:22:55Z

core/src/main/java/com/datastax/oss/driver/api/core/tracker/RequestIdGenerator.java

+   * @param hashCode the hashcode of the CqlRequestHandler
+   * @return a unique identifier for the session request
+   */
+  String getSessionRequestId(@NonNull Request statement, @NonNull String sessionName, int hashCode);


This interface seems a bit too connected to the default impl of RequestIdGenerator. It makes sense to pass the hash code of the relevant CqlRequestHandler given that implementation but is that parameter going to be generally usable?

I'd almost prefer to see the complete CqlRequestHandler passed here rather than just a hash code. That way if other implementers want to pull other values out of the handler (or even provider their own custom handlers with additional info available) they have an easy way to do so.

This getSessionRequestId is invoked in CqlRequestHandler's constructor. If we pass the CqlRequestHandler in, the object will not be initialized yet.

Maybe we can rename the parameter to salt (similar to cryptography, an integer that just provides uniqueness of IDs)?

absurdfarce · 2025-05-20T15:24:02Z

core/src/main/java/com/datastax/oss/driver/api/core/tracker/RequestIdGenerator.java

+   * @return a unique identifier for the node request
+   */
+  String getNodeRequestId(
+      @NonNull Request statement, @NonNull String sessionRequestId, int executionCount);


Same thing here I guess; execution count feels very tied to how the default request ID generator works. Is there a way we can generalize this a bit?

I think that this parameter makes sense. Within one session, we can retry sending the same request due to retry policy.

Sure, but that doesn't mean execution count is relevant to all implementations. It also begs the question of whether other things can/should be included for all implementations.

More generally, I'd argue it's inclusion here is primarily a function of the necessity of implementing the current log prefix as a request ID generator... which I'm not sure is a good idea (more on that elsewhere).

absurdfarce · 2025-05-20T15:27:04Z

manual/core/request_id/README.md

+
+Usage:
+* Inject ID generator: set the desired `RequestIdGenerator` in `advanced.request-id.generator.class`. 
+  The default implementation generates the session request ID as `{session_name}|{hash_code}`, and node request ID as `{session_name}|{hash_code}|{execution_count}`.


We don't really explain what {hash_code} or {execution_count} mean here

aratno · 2025-05-20T20:47:28Z

core/src/main/resources/reference.conf

+    # add the request id to the custom payload with the given key
+    # if empty, the request id will not be added to the custom payload
+    custom-payload-with-key = ""
+  }


Style nit - elsewhere we have a space before the opening brace {

aratno · 2025-05-20T20:54:45Z

manual/core/request_id/README.md

+- Session request ID: an identifier for an entire session.execute() call
+- Node request ID: an identifier for the execution of a CQL statement against a particular node. There can be one or more node requests for a single session request, due to retries or speculative executions.


Retries and speculative executions are often against different nodes than the original request, might prefer another name here like "Request Attempt ID".

Currently the server has no way to know whether a given request is a retry, this feature could help us provide a metric for original requests vs. retries on the server, which would be pretty cool.

I took the name "node request" v.s. "session request" from c# opentelemetry feature.

db.operation.name The type name of the operation being executed. Session_Request({RequestType}) for session level calls and Node_Request({RequestType}) for node level calls

And Lukasz's outstanding request tracker interface PR.
Do you think we should align with their naming?

aratno · 2025-05-20T22:19:33Z

core/src/main/java/com/datastax/oss/driver/api/core/config/DefaultDriverOption.java

+   *
+   * <p>Value-type: {@link String}
+   */
+  REQUEST_ID_CUSTOM_PAYLOAD_KEY("advanced.request-id.custom-payload-with-key");


(nit) naming: I find "custom-payload-key" clearer, you're already using that naming elsewhere

Do you mean the enum name, or the typesafe config path?

I believe @aratno is referring to the TypeSafe name @SiyaoIsHiding ... "custom-payload-key" rather than "custom-payload-with-key". Assuming that's correct I think he's on to something there.

manual/core/request_id/README.md

aratno · 2025-05-20T22:38:49Z

core/src/main/resources/reference.conf

+    }
+    # add the request id to the custom payload with the given key
+    # if empty, the request id will not be added to the custom payload
+    custom-payload-with-key = ""


Is there a way to disable Request IDs altogether? Seems like at least three possible states are needed:

Disabled Request IDs, no behavior changes on upgrade

Request IDs in driver logs only, not propagated to the server

Request IDs in driver logs and propagated to the server

The existing driver has a built-in logic to generate the log prefix, which is the same logic as the DefaultRequestIdGenerator. So your no.1 state is the same no.2 state, where the DefaultRequestIdGenerator is used.

I'd actually collapse your second and third cases into one @aratno. I'd also specify the rule a bit differently:

If the client has configured a request ID generator we'll use that to generate a consistent request ID via the log prefix on the client side and the custom payload params delivered to the server. Otherwise we'll preserve the current log prefix on the client side and add nothing to the custom payload.

I'm on board with 2 + 3 being a single case, especially in the near-term, but 1 is different

aratno · 2025-05-20T22:41:33Z

core/src/main/resources/reference.conf

+    }
+    # add the request id to the custom payload with the given key
+    # if empty, the request id will not be added to the custom payload
+    custom-payload-with-key = ""


How would a user know what to set this to? Can we come up with a reasonable default that's more likely to be interoperable between C* protocol implementations (C*, Scylla, DSE / Astra, etc)?

If we really need to choose one to recommend, I think we can recommend traceparent, as it's specified in W3C context propagation protocol.

Presumably this will vary with the implementation, right @aratno? Individaul C* request handlers might want to map this value to some name that makes sense for them. So I guess this would be very implementation-dependent... ?

Side note: it does raise an interesting question for Astra actually. We'd want to automatically set a request ID generator if the user is using Astra... but that's only half the problem. In addition to generating IDs in the expected format we'd also want to make sure the custom payload is being added at the right key for Astra. Hmmm... that's an interesting problem.

lukasz-antoniak · 2025-05-22T11:08:05Z

core/src/main/java/com/datastax/oss/driver/internal/core/cql/CqlRequestHandler.java

    }
+    String nodeRequestId =
+        this.requestIdGenerator.getNodeRequestId(statement, logPrefix, currentExecutionIndex);
+    if (!this.customPayloadKey.isEmpty()) {


We are not missing else block here?

absurdfarce

Apologies all, I had to retreat to a cave for a bit and ponder some of the questions under consideration here as well as what was nagging me about the original API. I think I landed on a reasonable compromise that can be extended to address most (all?) of the outstanding concerns... but I'm not completely convinced of that yet. Comments welcomed/encouraged.

absurdfarce · 2025-05-22T17:25:21Z

core/src/main/java/com/datastax/oss/driver/api/core/context/DriverContext.java


+  /** @return The driver's request ID generator; never {@code null}. */
+  @NonNull
+  RequestIdGenerator getRequestIdGenerator();


I'm going to argue this should actually return Optional<RequestIdGenerator>. I think part of the confusion for various other aspects of this ticket come down to (a) an impl which requires the driver to always have a request ID generator and (b) a confusion between a log prefix in the driver and what we're sending as a request ID.

absurdfarce · 2025-05-22T17:29:53Z

core/src/main/java/com/datastax/oss/driver/api/core/tracker/RequestIdGenerator.java

+   * @return a unique identifier for the node request
+   */
+  String getNodeRequestId(
+      @NonNull Request statement, @NonNull String sessionRequestId, int executionCount);


Sure, but that doesn't mean execution count is relevant to all implementations. It also begs the question of whether other things can/should be included for all implementations.

More generally, I'd argue it's inclusion here is primarily a function of the necessity of implementing the current log prefix as a request ID generator... which I'm not sure is a good idea (more on that elsewhere).

absurdfarce · 2025-05-22T17:30:45Z

core/src/main/java/com/datastax/oss/driver/api/core/tracker/RequestIdGenerator.java

+   * @return a unique identifier for the node request
+   */
+  String getNodeRequestId(
+      @NonNull Request statement, @NonNull String sessionRequestId, int executionCount);


In related news: how do we not include the node in question when we're generating a node request ID? Requests/Statements can have a node set as state but that's an optional thing a user can set in order to target a specific node; that's not automatically set for every request.

absurdfarce · 2025-05-22T17:32:58Z

core/src/main/java/com/datastax/oss/driver/internal/core/cql/CqlRequestHandler.java

-    this.logPrefix = sessionLogPrefix + "|" + this.hashCode();
+    this.requestIdGenerator = context.getRequestIdGenerator();
+    this.logPrefix =
+        this.requestIdGenerator.getSessionRequestId(statement, sessionLogPrefix, this.hashCode());


I think this is the root cause of my problem with the API. I think we need to clearly distinguish between a log prefix and a request ID. If a user doesn't configure a request ID generator that's totally fine... that means:

Nothing is added to custom payload AND

The old logic for generating a logPrefix is employed

That means our request ID generator API doesn't have to be retrofitted to support the existing log prefix syntax. It also resolve the issue @aratno has raised elsewhere, specifically "how do we shut this off if we don't want it?"

This current implementation of request ID relies on the old log prefix implementation to propagate to other classes, like RequestLogger and InFlightHandler. If we separate request ID with log prefix, how do we propagate request ID?

absurdfarce · 2025-05-22T17:34:23Z

core/src/main/java/com/datastax/oss/driver/internal/core/cql/CqlRequestHandler.java

+              .build();
+      // TODO: we are creating a new statement object for every request. We should optimize this.
+      statement = statement.setCustomPayload(customPayload);
+    }


This is the wrong place to do this. In most cases we haven't even selected the node yet; note that this happens immediately below where we poll the query plan if no node is explicitly set in the request. Assuming we update the request ID generation logic to correctly account for the target node the setting of custom payload fields should happen after we determine which node we're actually sending to.

absurdfarce · 2025-05-22T17:36:29Z

core/src/main/resources/reference.conf

+    }
+    # add the request id to the custom payload with the given key
+    # if empty, the request id will not be added to the custom payload
+    custom-payload-with-key = ""


I'd actually collapse your second and third cases into one @aratno. I'd also specify the rule a bit differently:

If the client has configured a request ID generator we'll use that to generate a consistent request ID via the log prefix on the client side and the custom payload params delivered to the server. Otherwise we'll preserve the current log prefix on the client side and add nothing to the custom payload.

absurdfarce · 2025-05-22T17:39:56Z

core/src/main/resources/reference.conf

+    }
+    # add the request id to the custom payload with the given key
+    # if empty, the request id will not be added to the custom payload
+    custom-payload-with-key = ""


Presumably this will vary with the implementation, right @aratno? Individaul C* request handlers might want to map this value to some name that makes sense for them. So I guess this would be very implementation-dependent... ?

Side note: it does raise an interesting question for Astra actually. We'd want to automatically set a request ID generator if the user is using Astra... but that's only half the problem. In addition to generating IDs in the expected format we'd also want to make sure the custom payload is being added at the right key for Astra. Hmmm... that's an interesting problem.

absurdfarce · 2025-05-22T17:41:28Z

core/src/main/java/com/datastax/oss/driver/api/core/config/DefaultDriverOption.java

+   *
+   * <p>Value-type: {@link String}
+   */
+  REQUEST_ID_CUSTOM_PAYLOAD_KEY("advanced.request-id.custom-payload-with-key");


I believe @aratno is referring to the TypeSafe name @SiyaoIsHiding ... "custom-payload-key" rather than "custom-payload-with-key". Assuming that's correct I think he's on to something there.

SiyaoIsHiding · 2025-05-29T08:15:36Z

Updated to use Optional and delete DefaultRequestIdGenerator

…ing logprefix behavior

…ution info does not have actual custom payload

core/src/main/java/com/datastax/oss/driver/internal/core/cql/CqlRequestHandler.java

SiyaoIsHiding · 2025-09-02T16:33:29Z

core/src/main/java/com/datastax/oss/driver/internal/core/cql/CqlRequestHandler.java

        setFinalError(statement, AllNodesFailedException.fromErrors(this.errors), null, -1);
      }
    } else {
+      String nodeLogPrefix;


Bret proposed: revert

Revert to which version?

SiyaoIsHiding · 2025-09-02T16:34:25Z

core/src/main/java/com/datastax/oss/driver/internal/core/cql/CqlRequestHandler.java

-    this.logPrefix = sessionLogPrefix + "|" + this.hashCode();
+    this.requestIdGenerator = context.getRequestIdGenerator();
+    this.logPrefix =
+        this.requestIdGenerator.isPresent()


Bret proposed:
this.logPrefix = sessionLogPrefix + "|" + (this.reqeustIdGenerator.isPresent() ? getSessionRequestID() : this.hashCode);

The chief goal here is to represent the generated request ID as matching up to the CqlRequestHandler hash code but nothing else. Basically we want to use request ID to identify the handler only since (a) it's bound at that scale anyway and (b) the user can define their own session values.

core/src/main/java/com/datastax/oss/driver/api/core/tracker/RequestTracker.java

lukasz-antoniak · 2025-09-03T12:34:28Z

core/src/main/java/com/datastax/oss/driver/internal/core/cql/CqlRequestHandler.java


    this.startTimeNanos = System.nanoTime();
-    this.logPrefix = sessionLogPrefix + "|" + this.hashCode();
+    this.requestIdGenerator = context.getRequestIdGenerator();


If we provide default request ID generator implementation (that matches current log prefix), we will not need to do if-else statements to check if value is present.

this.requestIdGenerator = context.getRequestIdGenerator().orElse(new RequestIdGenerator() { @Override public String getSessionRequestId(@NonNull Request statement) { return sessionLogPrefix + "|" + CqlRequestHandler.this.hashCode(); } @Override public String getNodeRequestId(@NonNull Request statement, @NonNull String sessionRequestId, int executionCount) { return sessionRequestId; } });

cc @absurdfarce, it seems like Bret doesn't like the idea of the default implementation because people would wonder how to turn off this feature

To rephrase the answer from @SiyaoIsHiding just a bit... this gets to the point I was making about distinguishing between log prefix and the request ID generator. They aren't the same thing. They don't do the same thing. We always need a log prefix. But we aren't required to have a request ID generator. Trying to model log prefix as a request ID generator in all cases confuses those two roles.

I think this conversation is largely resolved now. This code pretty clearly demonstrates that the simple use of the Optional type allows us to handle the default case in a more concise way. If anybody disagrees let's discuss further but given the current contents of the PR my inclination is to resolve this conversation as well.

absurdfarce · 2025-09-08T04:24:43Z

In order to make this more concrete I've created a PR which clearly states my chief concerns for the current impl (as of this coment) and proposes a concrete implementation which I believe addresses them. This PR is made against a local version of the "request-traceability" branch from @SiyaoIsHiding 's original PR... it seemed like the easiest way to represent the proposed changes.

I also include logging output from a small sample app using this code.... hopefully it clearly demonstrates the goal of these changes.

… distinct entity (which keeps support for session names configured in the config) and (2) makes the request ID scoped by CqlRequestHandler.

absurdfarce · 2025-09-10T20:45:56Z

After discussing the PR containing my proposed changes (referenced above) with @SiyaoIsHiding there seemed to be broad agreement on the structural changes (essentially decoupling the request ID from the session name and treating it as a replacement for the CqlRequestHandler hash code only) but some disagreement on method parameters and a few naming questions. In order to avoid fragmenting the discussion I've merged those changes into this PR (see this commit). We'll continue the discussion on these remaining points here.

I'll be closing the alternate PR with a similar note shortly.

absurdfarce · 2025-09-10T21:08:04Z

core/src/main/java/com/datastax/oss/driver/api/core/tracker/RequestIdGenerator.java

+   * @param parentId the session request identifier
+   * @return a unique identifier for the node request
+   */
+  String getRequestId(@NonNull Request statement, @NonNull String parentId);


Perhaps the single largest outstanding issue from the other PR: the naming of these methods. @SiyaoIsHiding has argued for "getSessionRequestId" and "getNodeRequestId" to maintain consistency with the C# interface (discussed in more detail in her comment elsewhere). I do not like either of these names, in no small part because the "session request ID" (a) isn't bound to the Session instance (and in fact the Session can have a completely different name) and (b) it's not actually used as an identifier for an outgoing message.

I'd argue that the names "getRequestId" and "getMessageId" are perhaps a bit clearer since they relate to what's actually going on. The request ID is associated with a single Request (which may in fact require multiple messages based on retries, speculative execution etc.) while the message ID is very clearly associated with a single message to a single node.

My intent with the names "getParentId" and "getRequestId" was to reinforce the parent-child relationship of these IDs but if we were really going to do that in a generic way this should probably have been "getParentId" and "getChildId"... and I agree that's kinda dumb. In the end I wound up going halfway with the parent-child thing with "getParentId" and the request/message split with "getRequestId" and the result is... just a mess.

I'm not sure what @lukasz-antoniak thinks. :)

I think I'm still arguing for getRequestId() and getMessageId(). I completely agree with @SiyaoIsHiding that we don't want to diverge from the public API for no reason but we're talking here about the API for request ID generators. I'm not expecting a huge number of new impls of this functionality, and even if we are the fact is the drivers represent discrete projects... and sometimes it may make more sense for the projects to diverge in naming if it's more sane for their impl.

Bret covered my point above about aligning with the C# driver. An additional aspect is that our existing RequestTracker interface is already using onSuccess and onNodeSuccss to differentiate the parent request v.s. the child. So using sessionRequest and nodeRequest aligns with the existing driver more.

Also, when we explain the 2 kinds of requests to the users, we say "one kind is the session.execute() call, another kind is the request that is sent to a node. One session.execute() can send multiple requests to the nodes due to retry and speculative executions." The words "session" and "node" have been in our explanation, that's where the naming "session request" and "node request" come from. Even if we use the wording request v.s. message, we still need to mention session.execute() and "sent to a node" when explaining to our users.

I think the distinction "request" v.s. "message" is vague, because a lot of people also use the word "request" to describe the requests sent to a node. For example, when our DB team engineers talk about requests, they for sure mean the requests sent to the nodes instead of a session.execute() call.

Also ping @aratno because Abe was also asking about the naming.

cc @joao-r-reis because I think Joao came up with the naming "session request" and "node request" first

I am with @SiyaoIsHiding on the method naming. Points from Bret are understandable and valid, but I think that we gain more consistency with "session request" and "node request". From what I see, Bret is mostly concerned about getSessionRequestId but if you read it as "get session-level request ID" I think it makes sense. RequestTracker interface already introduces the naming of "node request", so the parent to that could be "session request".

I think Session Request and Node Request are easier to explain to users in the context of the request tracker API but I do admit there wasn't a whole lot of discussion on these names when we went for these on the C# driver implementation. The java driver request tracker API docs refer to "node level" requests so making that leap to "Node Request" shouldn't be hard. I'd argue that "Message" is less clear since we don't have that name anywhere in the request tracker documentation AFAIK.

SessionRequest is not an ideal name but I didn't want it to just be "Request" since the "node level" request is called "Node Request". I wanted an additional qualifier so I went with "Session" but it could have been something different like "Parent Request" Idk

absurdfarce · 2025-09-10T21:11:18Z

core/src/main/java/com/datastax/oss/driver/api/core/tracker/RequestIdGenerator.java

+   *
+   * @return a unique identifier for the session request
+   */
+  String getParentId();


A holdover question from the other PR: should we include a non-null Request here as well to keep the method calls consistent? I didn't do so since none of the current impls require such a parameter and it isn't immediately obvious to me that it's something that future impls might want to leverage. I don't want to add parameters just because somebody might need them someday. If we have a reasonable use case to suggest that there's something in the request other implementors might be interested in we can talk about that... but without a pretty clear idea there my default is to keep it out.

The other thing worth pointing out: if we're wrong and we start seeing lots of RequestIdGenerator impls in the wild we can always ask the community for feedback and/or expand the API. But I'd rather start with the minimal set of params we know we need right now and expand that set if there's a case to do so.

I can agree to above statement. In any case we can add overloaded method signatures like in RequestTracker.

SiyaoIsHiding · 2025-09-17T16:21:53Z

core/src/main/java/com/datastax/oss/driver/api/core/session/SessionBuilder.java

+        if (programmaticArguments.getRequestIdGenerator() == null) {
+          programmaticArgumentsBuilder.withRequestIdGenerator(new W3CContextRequestIdGenerator());
+          LOG.debug(
+              "A secure connect bundle is provided, using W3CContextRequestIdGenerator as request ID generator.");


@absurdfarce the meeting today agreed that Astra will use traceparent as the key. We should change the default key for Astra to traceparent. It is request-id right now.

Agreed we should update to match the requirement for the Astra case @SiyaoIsHiding but I don't think we need to change the default in RequestIdGenerator. We should be able to address that by updating W3CContextRequestIdGenerator constructors:

new W3CContextRequestIdGenerator() == use the default key
new W3CContextRequestIdGenerator(String key) == use the provided key

Here we're in the Astra case so we'd clearly want to provide an arg.

Remember, W3CContextRequestidGenerator != AstraRequestIdGenerator. Just because the Astra requirements change doesn't mean we change the defaults.

Bret and Andy and Jane agreed in the Sep 22 meeting that we will change the default key to traceparent and allow override

lukasz-antoniak reviewed Apr 16, 2025

View reviewed changes

SiyaoIsHiding marked this pull request as ready for review April 17, 2025 08:01

olim7t reviewed Apr 28, 2025

View reviewed changes

absurdfarce changed the title ~~CASSJAVA97: Let users inject an ID for each request and write to the custom payload~~ CASSJAVA-97: Let users inject an ID for each request and write to the custom payload Apr 29, 2025

SiyaoIsHiding requested review from olim7t and lukasz-antoniak May 2, 2025 15:03

Yuqi-Du reviewed May 19, 2025

View reviewed changes

...test/java/com/datastax/oss/driver/internal/core/tracker/DistributedTraceIdGeneratorTest.java Outdated Show resolved Hide resolved

Yuqi-Du reviewed May 19, 2025

View reviewed changes

absurdfarce reviewed May 20, 2025

View reviewed changes

aratno reviewed May 20, 2025

View reviewed changes

lukasz-antoniak reviewed May 22, 2025

View reviewed changes

absurdfarce requested changes May 22, 2025

View reviewed changes

SiyaoIsHiding requested review from absurdfarce, Yuqi-Du, aratno and lukasz-antoniak May 29, 2025 08:15

SiyaoIsHiding added 12 commits August 20, 2025 19:33

DistributedTraceIdGenerator

a8b9420

CustomPayloadKey and W3CContextDistributedTraceIdGenerator

cd6ddfd

Change Noop to DefaultDistributedTraceIdGenerator, preserve the exist…

ec9f4a1

…ing logprefix behavior

fix tests

efec67d

add tests, add doc

70f2832

Use ByteBuffer.remaining()

8b20588

copy the custom payload, add doc. Integration tests fail because exec…

f702e65

…ution info does not have actual custom payload

rename to Request Id

b4112c4

Address PR review from Yuqi DU

8c45639

add doc

a623e67

remove default request id generator

309713c

Optional of request id generator

c12ffd8

SiyaoIsHiding and others added 5 commits August 20, 2025 19:37

update manual

2709cda

use W3C generator when Astra

c57b0eb

revapi

4059728

empty

752b958

resolve rebase conflict

96ddaaa

SiyaoIsHiding force-pushed the request-traceability branch from c86e099 to 96ddaaa Compare August 21, 2025 02:39

SiyaoIsHiding commented Sep 2, 2025

View reviewed changes

core/src/main/java/com/datastax/oss/driver/internal/core/cql/CqlRequestHandler.java Show resolved Hide resolved

SiyaoIsHiding commented Sep 2, 2025

View reviewed changes

core/src/main/java/com/datastax/oss/driver/api/core/tracker/RequestTracker.java Show resolved Hide resolved

lukasz-antoniak reviewed Sep 3, 2025

View reviewed changes

core/src/main/java/com/datastax/oss/driver/api/core/tracker/RequestTracker.java Show resolved Hide resolved

lukasz-antoniak reviewed Sep 3, 2025

View reviewed changes

absurdfarce mentioned this pull request Sep 5, 2025

Log prefix updates for CASSJAVA-97 (request traceability) absurdfarce/cassandra-java-driver#3

Closed

Implement new log prefix formatting which (1) preserves session ID as…

0120f03

… distinct entity (which keeps support for session names configured in the config) and (2) makes the request ID scoped by CqlRequestHandler.

absurdfarce reviewed Sep 10, 2025

View reviewed changes

SiyaoIsHiding commented Sep 17, 2025

View reviewed changes

		- Session request ID: an identifier for an entire session.execute() call
		- Node request ID: an identifier for the execution of a CQL statement against a particular node. There can be one or more node requests for a single session request, due to retries or speculative executions.

CASSJAVA-97: Let users inject an ID for each request and write to the custom payload #2037

Are you sure you want to change the base?

CASSJAVA-97: Let users inject an ID for each request and write to the custom payload #2037

Conversation

SiyaoIsHiding commented Apr 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SiyaoIsHiding commented Apr 16, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

absurdfarce left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

SiyaoIsHiding commented Apr 16, 2025 •

edited

Loading