Skip to content

Improve CW Logs pagination #2

@aidansteele

Description

@aidansteele

Right now we have the following code:

gossm/pkg/gossm/client.go

Lines 234 to 241 in d236fcd

var lastTimestamp int64 = math.MaxInt64
for _, ts := range lastTimestamps {
if ts < lastTimestamp {
lastTimestamp = ts
}
}
lastTimestamp++
input.StartTime = &lastTimestamp

This will paginate from the most recent timestamp for the instance that least recently posted to CW Logs. We need to do it this way to avoid missing logs from laggards, but it means keeping a cache of "seen" messages to avoid printing duplicates.

This is suboptimal because:

  • Right now the cache of seen messages grows unbounded
  • If one instance in a group finishes command execution long before others, we'll always be paginating from its last message timestamp. This could be minutes, resulting in a lot of duplicated requests. We should remove instances from the lastTimestamps map a few seconds after their command invocation completion. (A few seconds to account for delayed CW Logs appearance)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions