As an AWS user, I want to analyze AWS Config logs so that I can reason about my account's resource footprint/evolution

[sopel]

This story has been extracted from epic #1: While the Config log data is still constrained to the most important core AWS resource types right now, it has recently been expanded to all nine public regions, thus allows to deduce a VPC and EC2 instance configuration snapshot across an entire account, thereby providing valuable data points for correlating metrics with infrastructure changes. It can be facilitated for many scenarios accordingly, notably from the operations realm, though business (cost) can also benefit.

[sopel]

Scenarios

There is an open ended number of interesting scenarios conceivable, so this is just to list some obvious ones to get started:

Operations

• auditing - resource tracking for reporting and e.g environment diagrams
• debugging - correlation of resource lifecycle events with incidents
• monitoring - correlation of resource lifecycle events with metrics

Business (Cost)

• capacity planning and cost optimization - visualize/analyze trends in resource usage across regions and deduce e.g. reserved instance purchases or consolidation recommendations

[sopel]

Implementation Notes

This can basically reuse the current log ingestion architecture implemented by @dpb587 for #2, possibly adjusted for the different JSON payloads at hand (in absence of explicit type mappings, the resulting data will suffer from the same analysis constraints outlined in #5). The only complication might be the three different payloads in use:

• Configuration Item
• Configuration Snapshot
• Configuration History

Accordingly, those should be analyzed regarding overlap and priority.

• ❔ Maybe we can implement one after the other, or subsets thereof, and drop everything else until available?

[sopel]

ℹ️ this story is on hold until #5 is going to be resolved.

[dpb587]

I don't use AWS Config, so I'm not planning on implementing this. @sopel (or anyone else), feel free to do so if you're interested; a summary of steps is at https://github.com/logsearch/logsearch-for-aws#implementing-a-new-log-type and now there are a couple other log type examples to reference.