ES|QL: Add TBUCKET function #131449

leontyevdv · 2025-07-17T14:07:10Z

Introduce the function TBUCKET() which applies grouping on the @timestamp field, truncating its value to the specified granularity:

TBUCKET(1h) is equivalent to BUCKET(1 hour, @timestamp) TBUCKET(7d) is equivalent to BUCKET(7 days, @timestamp)

Closes #131068

@timestamp

Introduce the function TBUCKET(<time interval>) which applies grouping on the @timestamp field, truncating its value to the specified granularity: TBUCKET(1h) is equivalent to BUCKET(1 hour, @timestamp) TBUCKET(7d) is equivalent to BUCKET(7 days, @timestamp) Closes elastic#131068

@timestamp

Introduce the function TBUCKET(<time interval>) which applies grouping on the @timestamp field, truncating its value to the specified granularity: TBUCKET(1h) is equivalent to BUCKET(1 hour, @timestamp) TBUCKET(7d) is equivalent to BUCKET(7 days, @timestamp) Closes elastic#131068

github-actions · 2025-07-22T14:12:56Z

🔍 Preview links for changed docs

...in/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/grouping/TBucket.java

x-pack/plugin/esql/qa/testFixtures/src/main/resources/tbucket.csv-spec

Replace evaluation by a surrogate. Closes elastic#131068

Fix tests Closes elastic#131068

elasticsearchmachine · 2025-07-24T13:54:19Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

elasticsearchmachine · 2025-07-24T13:54:19Z

Pinging @elastic/es-storage-engine (Team:StorageEngine)

alex-spies

Heya, I'm only chiming in with regard to the proposed change of the optimizer rules; didn't consider the rest of the PR.

I'd like to look into how we can avoid another copy of SubstituteSurrogateExpressions as additional rules add complexity to the optimizer and are difficult to refactor later.

alex-spies · 2025-08-21T13:57:08Z

...k/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/LogicalPlanOptimizer.java

@@ -142,6 +142,7 @@ protected static Batch<LogicalPlan> substitutions() {
            new ReplaceAggregateAggExpressionWithEval(),
            // lastly replace surrogate functions
            new SubstituteSurrogateAggregations(),
+            new SubstituteSurrogateExpressions(),


Heya, if possible, I'd like to avoid adding another copy of SubstituteSurrogateExpressions just for TBucket. It very much looks like SubstituteSurrogateAggregations should be dealing with this.

I think SubstituteSurrogateAggregations may currently not substitute the surrogate in the grouping because groupings work a little differently from other aggregates. Can we investigate if this can be amended before adding a new rule to the substitution batch?

Scratch that, I'm looking at the optimizer sequence right now and will get back with a suggestion that does not not make sense, I hope.

- Remove SubstituteSurrogateExpressions rule from LogicalPlanOptimizer - Add TBucket translation to TranslateTimeSeriesAggregate

leontyevdv · 2025-08-22T14:17:54Z

.../java/org/elasticsearch/xpack/esql/optimizer/rules/logical/TranslateTimeSeriesAggregate.java

@@ -225,6 +226,12 @@ LogicalPlan translate(TimeSeriesAggregate aggregate) {
                        throw new IllegalArgumentException("expected at most one time bucket");
                    }
                    timeBucketRef.set(e);
+                } else if (child instanceof TBucket tbucket && tbucket.field().equals(timestamp.get())) {


@alex-spies , @fang-xing-esql I've removed the duplicating rule from LogicalPlanOptimizer in favor of this piece of code. Please, take a look. Thank you!

Looks more localized now, thank you.

You could attempt the substitution before checking if the substitution result is a Bucket, but the current version should work, too.

alex-spies · 2025-08-22T15:00:14Z

.../java/org/elasticsearch/xpack/esql/optimizer/rules/logical/TranslateTimeSeriesAggregate.java

@@ -225,6 +226,12 @@ LogicalPlan translate(TimeSeriesAggregate aggregate) {
                        throw new IllegalArgumentException("expected at most one time bucket");
                    }
                    timeBucketRef.set(e);
+                } else if (child instanceof TBucket tbucket && tbucket.field().equals(timestamp.get())) {


Looks more localized now, thank you.

You could attempt the substitution before checking if the substitution result is a Bucket, but the current version should work, too.

alex-spies · 2025-08-22T15:40:02Z

x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/session/FieldNameUtilsTests.java

+    public void testImplicitFieldNames() {
+        assertFieldNames("""
+            FROM sample_data
+            | STATS x = 1 year + TBUCKET(1 day) BY b1d = TBUCKET(1 day)""", Set.of("@timestamp", "@timestamp.*"));


Hey, ideally let's add a bunch of other queries like this.
Interesting examples use e.g. KEEP @timestamp before the STATS, or a KEEP @* or KEEP *stamp*.

Is it valid to have another STATS later if @timestampsurvives? LikeSTATS ... BY TBUCKET(1 day), @timestamp | WHERE ... | STATS BY TBUCKET(1 hour)`?

Also, what happens if theres an eval FROM sample_data | EVAL @timestamp = "2024-01-01"::date | STATS ... BY TBUCKET(2 days)?

For these tests to be robust, we want to be as creative as possible :)

Hey Alex, I added all the tests that you mentioned to both FieldNameUtilsTests and CsvTests. Thanks!

alex-spies · 2025-08-22T15:40:43Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/FieldNameUtils.java

@@ -166,6 +171,13 @@ public static PreAnalysisResult resolveFieldNames(LogicalPlan parsed, EnrichReso
                    // METRICS aggs generally rely on @timestamp without the user having to mention it.
                    referencesBuilder.get().add(new UnresolvedAttribute(ur.source(), MetadataAttribute.TIMESTAMP_FIELD));
                }
+
+                p.forEachExpression(UnresolvedFunction.class, uf -> {
+                    if (FUNCTIONS_REQUIRING_TIMESTAMP.contains(uf.name().toLowerCase(Locale.ROOT))) {


This change looks correct to me, but I'd like to solicit a review by @astefan just for this specific part of the PR as this is delicate when done wrong.

I am looking now at this. Thank you for the ping @alex-spies

- Add more tests for corner cases

- Fix IT by adding SORT

fang-xing-esql · 2025-08-25T22:34:38Z

...c/main/java/org/elasticsearch/xpack/esql/expression/function/grouping/GroupingWritables.java

@@ -14,6 +14,6 @@
 public class GroupingWritables {

    public static List<NamedWriteableRegistry.Entry> getNamedWriteables() {
-        return List.of(Bucket.ENTRY, Categorize.ENTRY);
+        return List.of(Bucket.ENTRY, Categorize.ENTRY, TBucket.ENTRY);


Is this needed if tbucket is on coordinator node only?

It's not needed. Removed it from here and replaced code in the methods writeTo and getWriteableName to throw exceptions similarly to ToIp. Thank you!

# Conflicts: # x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/analysis/VerifierTests.java

fang-xing-esql · 2025-08-26T14:53:04Z

...in/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/grouping/TBucket.java

+    private TBucket(StreamInput in) throws IOException {
+        this(Source.readFrom((PlanStreamInput) in), in.readNamedWriteable(Expression.class), in.readNamedWriteable(Expression.class));
+    }


This can be removed if serialization is not needed.

not-napoleon

Great work, sorry it's been more complicated than expected.

not-napoleon · 2025-08-26T13:52:55Z

x-pack/plugin/esql/qa/testFixtures/src/main/resources/tbucket.csv-spec

@@ -0,0 +1,343 @@
+// TBUCKET-specific tests
+
+tbucketByTenSecondsDuration


For future reference, it's possible to include spaces in the test names for CSV tests

not-napoleon · 2025-08-26T14:12:29Z

x-pack/plugin/esql/qa/testFixtures/src/main/resources/tbucket.csv-spec

+
+FROM sample_data
+| KEEP @timestamp, event_duration, message
+| EVAL t = @timestamp


I'm curious what happens if an eval actually changes the timestamp, something like | EVAL @timestamp = @timestamp + 3 hours. Does TBUCKET pick up the original or modified timestamp value?

This could be tested in a follow up PR, doesn't have to block this from merging.

not-napoleon · 2025-08-26T14:55:11Z

x-pack/plugin/esql/qa/testFixtures/src/main/resources/tbucket.csv-spec

I think it would be good to also include a test or two for TBUCKET in an eval; something like | EVAL key = TBUCKET(1 hour) | STATS minimum = MIN(whatever) BY key

fang-xing-esql

Thank you @leontyevdv ! I added one last comment related to serialization of tbucket, the rest LGTM.

astefan

I am wondering why do we consider such a scenario an acceptable one:

from test | stats max(emp_no) by tbucket(1 hour)`

generating an error like Unknown column [@timestamp].
Do we have another case in ESQL where the user is supposed to know that @timestamp is a field that must be present somewhere even though the user didn't actually type in the query @timestamp?

Imho, this would be an acceptable use case if the query would be TS test | stats ...... by tbucket(1 hour) meaning the user is aware that by using TS source command, it is expected to be in the area of "timeseries" indices and queries and some things (like the @timestamp field presence) are somewhat expected to happen.

Also, if I run this query from employees | stats min(salary) by tbucket(birth_date) I get back an error message Unknown column [@timestamp].

It is ok when running ..... by tbucket(birth_date,1month) to get back ql_illegal_argument_exception expects exactly one argument but, as an user who is exploring tbucket, if I remove 1month and keep birth_date (which is a date field) to get back something that has nothing to do with my query, it is unexpected.

@martijnvg @kkrik-es @dnhatn thoughts?

kkrik-es · 2025-08-27T08:06:41Z

@astefan we expect tbucket to apply to all data streams that implicitly define @timestamp. This is not limited to metrics, should be applicable to logs and more. I'm not sure if we have such a precedent in ESQL, but we should try to simplify the syntax for such heavily used applications imho.

astefan

LGTM. My earlier comment is unrelated to what, technically, the PR is doing. Please, regard that comment as an observation and something to discuss post-merge, if my observation is valid.

kkrik-es · 2025-08-27T08:39:22Z

Please, regard that comment as an observation and something to discuss post-merge, if my observation is valid.

Thanks Andrei, makes sense. I think there may be a pattern here, let's see how we can better accommodate this paradigm in the language.

leontyevdv · 2025-08-27T09:07:50Z

Thank you all for your feedback, folks! I will definitely address all the suggestions for improvements in the following PR since this one is getting harder to maintain and to follow because of its size and the number of discussions.

leontyevdv requested review from not-napoleon, gmarouli and dnhatn July 17, 2025 14:07

leontyevdv self-assigned this Jul 17, 2025

elasticsearchmachine added the v9.2.0 label Jul 17, 2025

github-actions bot deployed to docs-preview July 22, 2025 14:11 View deployment

Merge branch 'main' into feature/esql-tbucket-function

a5fe0fa

github-actions bot deployed to docs-preview July 22, 2025 14:16 View deployment

leontyevdv requested a review from kkrik-es July 22, 2025 14:20

not-napoleon reviewed Jul 22, 2025

View reviewed changes

...in/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/grouping/TBucket.java Outdated Show resolved Hide resolved

...in/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/grouping/TBucket.java Outdated Show resolved Hide resolved

fang-xing-esql reviewed Jul 22, 2025

View reviewed changes

...in/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/grouping/TBucket.java Outdated Show resolved Hide resolved

fang-xing-esql reviewed Jul 22, 2025

View reviewed changes

x-pack/plugin/esql/qa/testFixtures/src/main/resources/tbucket.csv-spec Outdated Show resolved Hide resolved

gmarouli added :StorageEngine/TSDB You know, for Metrics :Analytics/ES|QL AKA ESQL >enhancement labels Jul 23, 2025

leontyevdv added 2 commits July 24, 2025 14:54

ES|QL: Add TBUCKET function

45be0fe

Replace evaluation by a surrogate. Closes elastic#131068

ES|QL: Add TBUCKET function

e72467f

Replace evaluation by a surrogate. Closes elastic#131068

github-actions bot had a problem deploying to docs-preview July 24, 2025 12:55 Failure

github-actions bot deployed to docs-preview July 24, 2025 12:56 View deployment

ES|QL: Add TBUCKET function

5bd6f36

Fix tests Closes elastic#131068

github-actions bot deployed to docs-preview July 24, 2025 13:36 View deployment

leontyevdv marked this pull request as ready for review July 24, 2025 13:53

Merge branch 'main' into feature/esql-tbucket-function

e4a2e04

elasticsearchmachine added Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) Team:StorageEngine labels Jul 24, 2025

leontyevdv requested a review from fang-xing-esql August 20, 2025 16:27

leontyevdv added 2 commits August 20, 2025 19:11

Merge branch 'main' into feature/esql-tbucket-function

91d75a0

Merge branch 'main' into feature/esql-tbucket-function

86b8d06

alex-spies reviewed Aug 21, 2025

View reviewed changes

ES|QL: Refactor tests to

83bfd2a

- Remove SubstituteSurrogateExpressions rule from LogicalPlanOptimizer - Add TBucket translation to TranslateTimeSeriesAggregate

leontyevdv commented Aug 22, 2025

View reviewed changes

leontyevdv requested a review from alex-spies August 22, 2025 14:36

Merge branch 'main' into feature/esql-tbucket-function

ed92d11

alex-spies reviewed Aug 22, 2025

View reviewed changes

leontyevdv added 2 commits August 25, 2025 10:09

Merge branch 'main' into feature/esql-tbucket-function

92b02d1

ES|QL: Improve tests

2bf9111

- Add more tests for corner cases

leontyevdv requested a review from alex-spies August 25, 2025 12:09

ES|QL: Improve tests

24e8870

- Fix IT by adding SORT

fang-xing-esql reviewed Aug 25, 2025

View reviewed changes

leontyevdv added 2 commits August 26, 2025 09:37

ES|QL: Improve TBucket

2e2ea88

Merge branch 'main' into feature/esql-tbucket-function

ee5bfc0

# Conflicts: # x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/analysis/VerifierTests.java

leontyevdv requested a review from fang-xing-esql August 26, 2025 07:45

leontyevdv added 2 commits August 26, 2025 15:23

Merge branch 'main' into feature/esql-tbucket-function

3d52fbb

ES|QL: Improve tests

5603fae

fang-xing-esql reviewed Aug 26, 2025

View reviewed changes

not-napoleon approved these changes Aug 26, 2025

View reviewed changes

fang-xing-esql approved these changes Aug 26, 2025

View reviewed changes

ES|QL: Improve tests

e74f6ac

astefan reviewed Aug 27, 2025

View reviewed changes

astefan self-requested a review August 27, 2025 08:33

astefan approved these changes Aug 27, 2025

View reviewed changes

leontyevdv merged commit f2b364c into elastic:main Aug 27, 2025
33 checks passed

		@@ -0,0 +1,343 @@
		// TBUCKET-specific tests

		tbucketByTenSecondsDuration

ES|QL: Add TBUCKET function #131449

ES|QL: Add TBUCKET function #131449

Uh oh!

Conversation

leontyevdv commented Jul 17, 2025

Uh oh!

github-actions bot commented Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔍 Preview links for changed docs

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elasticsearchmachine commented Jul 24, 2025

Uh oh!

elasticsearchmachine commented Jul 24, 2025

Uh oh!

alex-spies left a comment

Choose a reason for hiding this comment

Uh oh!

alex-spies Aug 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fang-xing-esql Aug 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

not-napoleon left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fang-xing-esql left a comment

Choose a reason for hiding this comment

Uh oh!

astefan left a comment

Choose a reason for hiding this comment

Uh oh!

kkrik-es commented Aug 27, 2025

Uh oh!

astefan left a comment

Choose a reason for hiding this comment

Uh oh!

kkrik-es commented Aug 27, 2025

Uh oh!

leontyevdv commented Aug 27, 2025

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jul 22, 2025 •

edited

Loading

alex-spies Aug 21, 2025 •

edited

Loading

fang-xing-esql Aug 26, 2025 •

edited

Loading