Feature request: set correlation ID in Logger

[dreamorosi]

Use case

When working with logs in distributed systems like AWS Lambda, customers want to be able to set arbitrary correlation IDs for their workloads.

Today this is possible using existing Logger features (see "Alternative solutions" section), however the experience is somewhat verbose and could be improved.

Powertools for AWS Lambda (Python) has a feature that allows customers to include a correlation_id key to each log. This feature request proposes to implement the same feature in this version of Powertools for AWS.

This would be a first, low hanging, effort for allowing customers to more easily work with correlation IDs with Powertools for AWS (see #129).

Important

If you're interested in this feature, please consider leaving a 👍 under this post, or leave a comment - measuring demand and having customer names helps us prioritize features in our backlog.

Solution/User Experience

The new feature is primarily aimed at customers using the injectLambdaContext class method decorator and Middy.js middleware.

Given an event with this type:

type MyEvent = {
  headers: {
    my_request_id_header: string
  }
}

When using decorators, the experience would look like this:

import { Logger } from '@aws-lambda-powertools/logger';
import type { Context } from 'aws-lambda';
import type { LambdaInterface } from '@aws-lambda-powertools/commons/types';

const logger = new Logger();

class Lambda extends LambdaInterface {
  @logger.injectLambdaContext({ correlationIdPath: 'headers.my_request_id_header' })
  public async handler(event: MyEvent, context: Context) {
    // ...
  }
}

When using the Middy.js middleware, the experience would instead look like this:

import { Logger } from '@aws-lambda-powertools/logger';
import { injectLambdaContext } from '@aws-lambda-powertools/logger/middleware';
import middy from '@middy/core';

const logger = new Logger();

export const handler = middy(async (event: MyEvent) => {
  // ...
}).use(injectLambdaContext(logger, {
  correlationIdPath: 'headers.my_request_id_header'
});

For the manual instrumentation the method is less compelling, but also a very low hanging fruit as it would just be an alias for logger.appendKeys({ correlation_id: ... }) (see docs):

import { Logger } from '@aws-lambda-powertools/logger';

const logger = new Logger();

export const handler = async (event: MyEvent) => {
  logger.setCorrelationId(event.headers?.headers.my_request_id_heeader);
  
  // ...
}

Each one of these snippets would generate a log entry similar to this:

{
    "level": "INFO",
    "message": "Collecting payment",
    "timestamp": "2021-05-03 11:47:12,494+0000",
    "service": "payment",
    "cold_start": true,
    "function_name": "test",
    "function_memory_size": 128,
    "function_arn": "arn:aws:lambda:eu-west-1:12345678910:function:test",
    "function_request_id": "52fdfc07-2182-154f-163f-5f0f9a621d72",
    "correlation_id": "correlation_id_value" // <- here
}

The feature would also come with a handful built-in correlation ID expressions, for example:

• API_GATEWAY_REST => "requestContext.requestId"
• APPSYNC_RESOLVER => 'request.headers."x-amzn-trace-id"'
• full list here

These would be used like this:

import { Logger } from '@aws-lambda-powertools/logger';
import { API_GATEWAY_REST } from '@aws-lambda-powertools/logger/correlation-paths';
import type { Context } from 'aws-lambda';
import type { LambdaInterface } from '@aws-lambda-powertools/commons/types';

const logger = new Logger();

class Lambda extends LambdaInterface {
  @logger.injectLambdaContext({ correlationIdPath: API_GATEWAY_REST })
  public async handler(event: MyEvent, context: Context) {
    // ...
  }
}

Open questions

Note

The points below should be addressed before the feature is ready to be implemented, if you want to contribute please help us instead of opening a PR.

The current implementation in Powertools for AWS Lambda (Python) relies on JMESPath expressions for extracting the correlation ID from the payload. This choice was sensible for them both because of their ecosystem but also because they already have the jmespath module in their dependency tree because of other features.

For us this is still a decision point to be addressed. Taking a dependency on @aws-lambda-powertools/jmespath that we released for the Idempotency utility would be easy in terms of complexity, however before moving forward with this I’d like us to explore 1/ whether it’s possible to set the dependency as optional peerDependency and load it only when using the feature, and if not, 2/ whether the impact on the bundle is acceptable.

I am not particularly inclined towards other JSON-query languages at this stage, but I am open to consider proposals if any. However implementing a function-based query feature (i.e. correlationId: (event) => (event.headers.my_request_id_header)) is out of question because of the reasons described in #1644 (unless anyone has compelling arguments).

Alternative solutions

import { Logger } from '@aws-lambda-powertools/logger';

const logger = new Logger();

export const handler = async (event: MyEvent) => {
  logger.setCorrelationId(event.headers?.headers.my_request_id_heeader);
  
  // ...
}

Acknowledgment

• This feature request meets Powertools for AWS Lambda (TypeScript) Tenets
• Should this be considered in other Powertools for AWS Lambda languages? i.e. Python, Java, and .NET

Future readers

Please react with 👍 and your use case to help us understand customer demand.

[thiagomeireless]

I'm sad to see this feature having only 1 👍🏻

I'm sure this is something multiple people would (should?) be using, especially in event source contexts. We have a lot of services that works like API Gateway entry point -> Lambda -> SNS/SQS -> Lambda and it's such a mess to debug things without setting up all the correlationId context, getters/setters, etc.

It would be great to have this functionality baked in PowerTools so we could just do correlationIdPath: 'headers.my_request_id_header'.

Hopefully this gets more traction so we can get feature parity with the Python version 🙏🏻

[dreamorosi]

Hi @thiagomeireless - thank you for the interest in the feature.

My main concern in adding the feature is having an extra dependency to Logger, which is widely used.

I'd like to find a way to make it optional unless you're using the feature, but I don't know how yet.