AWS CloudWatch Event Sink Implementation #1965

adnanhemani · 2025-06-27T02:21:21Z

As per discussion at this morning's Polaris Community Sync, I am introducing this event listener that will allow users to sink events to AWS CloudWatch Logs - where they can search through events for auditing purposes, if they wish. I do intend to introduce similar changes for Azure and GCP once this is merged so that we have parity across all CSPs - but this will help nail down the exact pattern to make this work.

Below is a screenshot of a sample event sent to CloudWatch:

snazy · 2025-06-27T08:38:21Z

Didn't look into the whole PR, but two things I noticed:

The PR's doing more than mentioned in the summary & description
There's no (integration) testing for the feature (IIRC there are solutions to test cloudwatch locally)

adnanhemani · 2025-06-27T09:43:09Z

Hi @snazy,

The PR's doing more than mentioned in the summary & description

Yes, I'm assuming you're referring to the creation of one new event (onAfterCatalogCreated). It is much faster to use that to rapidly test new event listeners. If it is a sticking point, I'm happy to break that into a separate, few-line PR. WDYT?

There's no (integration) testing for the feature (IIRC there are solutions to test cloudwatch locally)

I didn't want to sink (no pun intended) more time trying to setup LocalStack for this repository if this approach was wildly off-the-mark. If you think the general approach is reasonable and the main things that will need change are implementation details, I will be glad to add that in the next revision. IIRC, this is also something that we only get with AWS - both Azure and GCP equivalents of CloudWatch do not have equivalent testing solutions (this could be outdated information but I can't find anything in a quick Google search). If having parity across CSPs on testing is important, maybe we settle for unit tests instead? Let me know your thoughts.

…ent_listener

adnanhemani · 2025-07-12T12:57:02Z

I've added a test using LocalStack, which should help emulate CloudWatch. This PR is ready for a full review.

runtime/defaults/src/main/resources/application.properties

service/common/src/main/java/org/apache/polaris/service/events/EventListenerConfiguration.java

service/common/src/main/java/org/apache/polaris/service/events/AwsCloudWatchEventListener.java

service/common/src/main/java/org/apache/polaris/service/events/PolarisEventListener.java

...e/common/src/test/java/org/apache/polaris/service/events/AwsCloudWatchEventListenerTest.java

...java/org/apache/polaris/service/quarkus/events/QuarkusPolarisEventListenerConfiguration.java

service/common/src/main/java/org/apache/polaris/service/events/AwsCloudWatchEventListener.java

snazy

There are some thread-safety/concurrency issues and memory leaks in this change, which have to be addressed.

The implementation can also be simplified.

I'd also recommend to move this to its own module to streamline CI and testing in general.

service/common/src/main/java/org/apache/polaris/service/events/AwsCloudWatchEventListener.java

...e/common/src/test/java/org/apache/polaris/service/events/AwsCloudWatchEventListenerTest.java

.../polaris/service/events/jsonEventListener/aws/cloudwatch/AwsCloudWatchEventListenerTest.java

@snazy

Hey @snazy, can you take another pass over this one? It's changed a lot since we initially reviewed.

RussellSpitzer · 2025-08-28T15:38:01Z

...ice/src/main/java/org/apache/polaris/service/events/jsonEventListener/JsonEventListener.java

+ * #transformAndSendEvent(HashMap)} method to define how the JSON event data should be transmitted
+ * or stored.
+ */
+public abstract class JsonEventListener extends PolarisEventListener {


A note, that I really like having this as a public class because if someone does want to write a plugin that doesn't interact with any of the Iceberg or other based libraries, they can always utilize this instead and only depend on Polaris.

RussellSpitzer · 2025-08-28T16:04:38Z

.../polaris/service/events/jsonEventListener/aws/cloudwatch/AwsCloudWatchEventListenerTest.java

+  }
+
+  @Test
+  void shouldCreateLogGroupAndStream() {


We are missing the negative test here, we also ned to test that it doesn't fail if the log group and stream already exists. You could just start up again after the the first asserts?

RussellSpitzer

I think we are just missing one test of the "already exists" branch but other than that I think this is ready to go

...ice/src/main/java/org/apache/polaris/service/events/jsonEventListener/JsonEventListener.java

eric-maynard · 2025-08-29T16:12:28Z

...ache/polaris/service/events/jsonEventListener/aws/cloudwatch/AwsCloudWatchEventListener.java

+    properties.put("realm", callContext.getRealmContext().getRealmIdentifier());
+    properties.put("principal", securityContext.getUserPrincipal().getName());


nit but I would recommend putting these into constants somewhere, e.g. AwsCloudWatchEventEventListener.KEY_REALM

Putting them as class-wide constants isn't a great idea IMO. These variables are RequestScoped, while the class as a whole is ApplicationScoped. The best we can probably do is to keep them as variables within the transformAndSendEvent function (which will run per request) - but I'm not sure what that helps to clarify.

To be clear, I'm talking about the string literals like "realm" here. Having a constant will be useful if someone else wants to extract the value from the event, e.g. they won't have to hard-code properties.get("realm")

eric-maynard · 2025-08-29T16:12:49Z

...rvice/quarkus/events/jsonEventListener/aws/cloudwatch/QuarkusAwsCloudWatchConfiguration.java

+  @WithName("log-group")
+  @WithDefault("polaris-cloudwatch-default-group")
+  @Override
+  String awsCloudwatchlogGroup();


Cloudwatch or CloudWatch? We are using a mix of both casings

Good point, changed.

...ache/polaris/service/events/jsonEventListener/aws/cloudwatch/AwsCloudWatchEventListener.java

...rvice/quarkus/events/jsonEventListener/aws/cloudwatch/QuarkusAwsCloudWatchConfiguration.java

...ache/polaris/service/events/jsonEventListener/aws/cloudwatch/AwsCloudWatchEventListener.java

site/content/in-dev/unreleased/configuration.md

singhpk234 · 2025-08-29T16:24:10Z

site/content/in-dev/unreleased/configuration.md

- | `polaris.config.rollback.compaction.on-conflicts.enabled`                              | `false`              | When set to true Polaris will apply the deconfliction by rollbacking those REPLACE operations snapshots which have the property of `polaris.internal.rollback.compaction.on-conflict` in their snapshot summary set to `rollback`, to resolve conflicts at the server end.                                                                                    |
+ | `polaris.config.rollback.compaction.on-conflicts.enabled`                              | `false`               | When set to true Polaris will apply the deconfliction by rollbacking those REPLACE operations snapshots which have the property of `polaris.internal.rollback.compaction.on-conflict` in their snapshot summary set to `rollback`, to resolve conflicts at the server end.                                                                                    |
+| `polaris.event-listener.type`                                                          | `no-op`               | Define the Polaris event listener type. Supported values are `no-op`, `aws-cloudwatch`.                                                                                                                                                                                                                                                                      |
+| `polaris.event-listener.aws-cloudwatch.log-group`                                      |                       | Define the AWS CloudWatch log group name for the event listener.                                                                                                                                                                                                                                                                                              |


we would IAM that polaris is running with have permissions to write to this log group right ? i don't know where to best write it ? thoughts ?

Added it to this page, let me know your thoughts!

singhpk234

LGTM, Thanks a lot @adnanhemani for all the work !

I have added some suggestions, which are optional to address and are good to have !

singhpk234 · 2025-09-02T16:18:29Z

...ache/polaris/service/events/jsonEventListener/aws/cloudwatch/AwsCloudWatchEventListener.java

+      String resourceType,
+      String resourceName) {
+    if (existsCheck.get()) {
+      LOGGER.debug("Log {} [{}] already exists", resourceType, resourceName);


Suggested change

LOGGER.debug("Log {} [{}] already exists", resourceType, resourceName);

LOGGER.debug("Resource {} [{}] already exists", resourceType, resourceName);

No - the resourceType here will be "stream" or "group" to make a final log line stating "Log stream [xyz] already exists"

...ache/polaris/service/events/jsonEventListener/aws/cloudwatch/AwsCloudWatchEventListener.java

singhpk234 · 2025-09-02T16:26:32Z

site/content/in-dev/unreleased/configuration.md

+| `polaris.config.rollback.compaction.on-conflicts.enabled`                              | `false`                            | When set to true Polaris will apply the deconfliction by rollbacking those REPLACE operations snapshots which have the property of `polaris.internal.rollback.compaction.on-conflict` in their snapshot summary set to `rollback`, to resolve conflicts at the server end.                                                                                    |
+| `polaris.event-listener.type`                                                          | `no-op`                            | Define the Polaris event listener type. Supported values are `no-op`, `aws-cloudwatch`.                                                                                                                                                                                                                                                                       |
+| `polaris.event-listener.aws-cloudwatch.log-group`                                      | `polaris-cloudwatch-default-group` | Define the AWS CloudWatch log group name for the event listener.                                                                                                                                                                                                                                                                                              |
+| `polaris.event-listener.aws-cloudwatch.log-stream`                                     | `polaris-cloudwatch-default-stream`| Define the AWS CloudWatch log stream name for the event listener. Ensure that Polaris' IAM credentials have the following actions: "PutLogEvents", "DescribeLogStreams", and "DescribeLogGroups" on the specified log stream/group. If the specified log stream/group does not exist, then "CreateLogStream" and "CreateLogGroup" will also be required.      |


[not a blocker] IMHO adding IAM requirement might not be right place, is there a place where IAM with Polaris runs with requirements are there ? let me check too.

I don't see one other than the Getting Started - and I don't think that page is relevant here.

singhpk234 · 2025-09-02T16:30:49Z

.../main/java/org/apache/polaris/service/events/jsonEventListener/PropertyMapEventListener.java

+  public void onAfterTableRefreshed(AfterTableRefreshedEvent event) {
+    HashMap<String, Object> properties = new HashMap<>();
+    properties.put("event_type", event.getClass().getSimpleName());
+    properties.put("table_identifier", event.tableIdentifier().toString());


[optional] wondering if we need catalog name too in this ? lets say i have a same namespace and table in the different catalogs in the realm ?

I agree with this but we should put this as part of the instrumentation of the event itself. I will look into this for #2480.

adnanhemani · 2025-09-03T18:45:28Z

Working on rebase for this - seems like it is now failing in main

adnanhemani added 2 commits June 26, 2025 19:14

Add AWS CloudWatch integration through Event Listener

2e07dde

cleanup

06eaca4

github-project-automation bot added this to Basic Kanban Board Jun 27, 2025

github-project-automation bot moved this to PRs In Progress in Basic Kanban Board Jun 27, 2025

spotlessapply

1515fbb

adnanhemani added 5 commits July 12, 2025 04:55

Added unit test with LocalStack

854501b

typo

c1c94b2

spotlessapply

ab3c5f9

Merge remote-tracking branch 'origin/main' into ahemani/cloudwatch_ev…

ab9ccbe

…ent_listener

recompile from main

a641136

eric-maynard reviewed Jul 15, 2025

View reviewed changes

adnanhemani added 4 commits July 15, 2025 21:57

first revision change, based on review from @eric-maynard

5a355d1

merge from origin/main

bab4439

spotlessapply

04c310a

Merge branch 'main' into ahemani/cloudwatch_event_listener

d4b44ff

eric-maynard reviewed Jul 16, 2025

View reviewed changes

service/common/src/main/java/org/apache/polaris/service/events/AwsCloudWatchEventListener.java Outdated Show resolved Hide resolved

eric-maynard reviewed Jul 16, 2025

View reviewed changes

service/common/src/main/java/org/apache/polaris/service/events/AwsCloudWatchEventListener.java Outdated Show resolved Hide resolved

eric-maynard reviewed Jul 16, 2025

View reviewed changes

service/common/src/main/java/org/apache/polaris/service/events/AwsCloudWatchEventListener.java Outdated Show resolved Hide resolved

snazy requested changes Jul 17, 2025

View reviewed changes

adnanhemani added 7 commits July 21, 2025 00:18

injected securitycontext and callcontext

4d0554a

todo

cc715ad

modify test

518aaaa

first draft of revision

8758255

resolve comments from @eric-maynard and @snazy

f3f62a0

refactor into separate package

d21dabc

typo

9054511

adnanhemani requested a review from snazy July 21, 2025 07:39

adnanhemani added 5 commits August 22, 2025 01:03

Address comments from @RussellSpitzer and merge from main

4c91c57

typo

b0c6160

spotlessapply

44fad9a

fix docstrings

e8447a9

refactor

688ec97

RussellSpitzer reviewed Aug 25, 2025

View reviewed changes

.../polaris/service/events/jsonEventListener/aws/cloudwatch/AwsCloudWatchEventListenerTest.java Outdated Show resolved Hide resolved

use awaitility

360cb91

RussellSpitzer reviewed Aug 28, 2025

View reviewed changes

RussellSpitzer previously approved these changes Aug 28, 2025

View reviewed changes

github-project-automation bot moved this from PRs In Progress to Ready to merge in Basic Kanban Board Aug 28, 2025

Add negative case testing

7c09d9b

adnanhemani dismissed RussellSpitzer’s stale review via 7c09d9b August 28, 2025 20:29

spotlessapply

5e34884

RussellSpitzer previously approved these changes Aug 28, 2025

View reviewed changes

eric-maynard reviewed Aug 29, 2025

View reviewed changes

...ice/src/main/java/org/apache/polaris/service/events/jsonEventListener/JsonEventListener.java Outdated Show resolved Hide resolved

eric-maynard reviewed Aug 29, 2025

View reviewed changes

singhpk234 reviewed Aug 29, 2025

View reviewed changes

Revision based on comments from @eric-maynard and @singhpk234

e2ed743

adnanhemani dismissed RussellSpitzer’s stale review via e2ed743 September 1, 2025 01:59

singhpk234 previously approved these changes Sep 2, 2025

View reviewed changes

Addressing comments from @singhpk234

a870aea

adnanhemani dismissed singhpk234’s stale review via a870aea September 3, 2025 01:45

singhpk234 approved these changes Sep 3, 2025

View reviewed changes

eric-maynard approved these changes Sep 3, 2025

View reviewed changes

singhpk234 merged commit 20753ed into apache:main Sep 3, 2025
12 checks passed

github-project-automation bot moved this from Ready to merge to Done in Basic Kanban Board Sep 3, 2025

		properties.put("realm", callContext.getRealmContext().getRealmIdentifier());
		properties.put("principal", securityContext.getUserPrincipal().getName());

	LOGGER.debug("Log {} [{}] already exists", resourceType, resourceName);
	LOGGER.debug("Resource {} [{}] already exists", resourceType, resourceName);

AWS CloudWatch Event Sink Implementation #1965

AWS CloudWatch Event Sink Implementation #1965

Uh oh!

Conversation

adnanhemani commented Jun 27, 2025

Uh oh!

snazy commented Jun 27, 2025

Uh oh!

adnanhemani commented Jun 27, 2025

Uh oh!

adnanhemani commented Jul 12, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

snazy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RussellSpitzer Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RussellSpitzer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

singhpk234 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RussellSpitzer Aug 28, 2025 •

edited

Loading

singhpk234 left a comment •

edited

Loading