Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

As a Logsearch user, I want to analyze all my AWS logs so that I can get on top of my account #1

Open
sopel opened this issue Jan 26, 2015 · 3 comments
Open

As a Logsearch user, I want to analyze all my AWS logs so that I can get on top of my account #1

sopel opened this issue Jan 26, 2015 · 3 comments
Labels
epic

Comments

@sopel
Copy link
Member

sopel commented Jan 26, 2015

This epic basically resembles the raison d'être for this project/repository. It will be broken down into separate stories per specific use case resp. AWS log type and mainly serves as a wrapper for gathering information until more appropriate contexts are available.
@sopel sopel added the epic label Jan 26, 2015
@sopel
Copy link
Member Author

sopel commented Jan 26, 2015

Use cases

The following use cases have been identified so far:

Logs

Log data usually needs to be read from S3 buckets, either by polling, or preferably after receiving a log delivery push notification (meanwhile available as a generic S3 object creation notification).

Events

Events generate push notifications, but the common and preferred pattern for consuming them reliably is subscribing an Amazon SQS queue to a SNS topic and consuming SQS messages in turn, see Queues/Streams below

  • As an AWS user, I want to monitor Amazon SNS notifications so that I can correlate arbitrary events with operational logs and metrics
  • As an AWS user, I want to analyze Auto Scaling notifications so that I can correlate resource lifecycle events with operational metrics
  • As an AWS user, I want to analyze AWS CloudFormation notifications so that I can correlate resource lifecycle events with operational metrics
  • As an AWS user, I want to analyze Amazon S3 notifications so that I can correlate resource lifecycle events with operational metrics

Queues/Streams

Queues/Streams usually need to be polled by workers that are auto scaling based on queue/stream metrics.

  • As an AWS user, I want to ingest Amazon SQS messages from SQS queues so that I can analyze arbitrary messages
  • As an AWS user, I want to ingest Amazon Kinesis events from Kinesis streams so that I can analyze arbitrary events
  • As an AWS user, I want to ingest Amazon DynamoDB events from DynamoDB streams so that I can correlate resource lifecycle events with operational metrics
    • ❗ This is going to be pretty much the same as ingesting Kinesis event streams, insofar AWS has sufficiently aligned the resp. formats to be handled by a single library/solution.
    • ❗ Also worth noting that AWS has stated the goal of converging to only a few schemata for this purpose.

@sopel
Copy link
Member Author

sopel commented Jan 26, 2015

Log formats

The following log formats have been identified so far:

JSON

Most logs and events are facilitating JSON these days, and AWS has stated that they aim to converge to only a few schemata for this purpose.

CSV

Some logs like e.g. Amazon CloudFront and Amazon S3 are facilitating the common structured web server log formats. The AWS Account Billing Data is logged in CSV.

@sopel sopel mentioned this issue Jan 26, 2015
Closed
@sopel sopel mentioned this issue Apr 22, 2015
Open
@dpb587
Copy link
Contributor

dpb587 commented Aug 13, 2015

Billing Logs started in 8c96bb7

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
epic
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dpb587 @sopel