You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: tutorials/kafka-live-viewer-profiles/conclusion.md
+25-25Lines changed: 25 additions & 25 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -3,39 +3,39 @@ position: 4
3
3
title: Conclusion
4
4
---
5
5
6
-
# Conclusion
6
+
In this tutorial, you've explored the **live viewer profiles** solution accelerator for video streaming, gaining practical insights into building, deploying, and extending real-time, event-driven architectures using Snowplow and Kafka.
7
7
8
-
In this tutorial, we have explored the **live viewer profiles** solution accelerator for video streaming, gaining practical insights into building, deploying, and extending real-time, event-driven architectures using Snowplow and Kafka.
8
+
## Understanding the process
9
9
10
-
## Key Takeaways
11
-
12
-
### Understanding the Process
13
-
We successfully have built a real time system for processing event data including:
14
-
-**Web Tracking Application** for collecting media events.
10
+
You have successfully built a real time system for processing event data including:
11
+
-**Web tracking application** for collecting media events
15
12

16
-
-**Snowplow Collector and Snowbridge** for event processing and forwarding.
17
-
-**Live Viewer Backend** for managing real-time data with Kafka and DynamoDB.
18
-
-**Live Viewer Frontend** for visualizing real-time user activity on the web tracking application.
13
+
-**Snowplow Collector and Snowbridge** for event processing and forwarding
14
+
-**Live Viewer back-end** for managing real-time data with Kafka and DynamoDB
15
+
-**Live Viewer front-end** for visualizing real-time user activity on the web tracking application
19
16

20
17
21
18
This architecture highlights how real-time insights can be achieved using event-driven systems in a streaming context.
22
19
23
-
### What We Achieved
24
-
We explored how to:
25
-
1. Use LocalStack to emulate AWS services for local development and testing.
26
-
2. Launch and interact with the system components, such as the Kafka UI and LocalStack UI.
27
-
3. View and verify the real-time event data from the browser using Snowplow's media tracking capabilities.
28
-
4. Deploy the solution in an AWS environment using Terraform.
20
+
## What you achieved
21
+
22
+
You explored how to:
23
+
1. Use LocalStack to emulate AWS services for local development and testing
24
+
2. Launch and interact with the system components, such as the Kafka UI and LocalStack UI
25
+
3. View and verify the real-time event data from the browser using Snowplow's media tracking capabilities
26
+
4. Deploy the solution in an AWS environment using Terraform
27
+
28
+
## Practical applications
29
29
30
-
### Practical Applications
31
30
This tutorial can be extended to utilize Snowplow event data for other real-time use cases, such as:
32
-
- Web Engagement analytics.
33
-
- Personalized recommendations.
34
-
- Ad performance tracking.
31
+
- Web Engagement analytics
32
+
- Personalized recommendations
33
+
- Ad performance tracking
34
+
35
+
## Next steps
35
36
36
-
## Next Steps
37
-
-**Extend tracking:** Extend the solution to track more granular user interactions or track on a new platform such as mobile.
38
-
-**Extend dashboard:** Extend the Live Viewer to include information on the media being watched and the user.
39
-
-**Replace the state store:** Replace Amazon DynamoDB with an alternative to be cloud agnostic, e.g. Google Bigtable or MongoDB.
37
+
-**Extend tracking:** extend the solution to track more granular user interactions or track on a new platform such as mobile
38
+
-**Extend dashboard:** extend the Live Viewer to include information on the media being watched and the user
39
+
-**Replace the state store:** replace Amazon DynamoDB with an alternative to be cloud agnostic, e.g. Google Bigtable or MongoDB
40
40
41
-
By completing this tutorial, you are equipped to harness the power of event-driven systems and Snowplow’s analytics framework to build dynamic, real-time solutions tailored to your streaming and analytics needs.
41
+
By completing this tutorial, you're equipped to harness the power of event-driven systems and Snowplow's analytics framework to build dynamic, real-time solutions tailored to your streaming and analytics needs.
Copy file name to clipboardExpand all lines: tutorials/kafka-live-viewer-profiles/deploy-aws-terraform.md
+22-28Lines changed: 22 additions & 28 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -3,19 +3,13 @@ position: 3
3
3
title: Deploy on AWS with Terraform
4
4
---
5
5
6
-
# Deploy on AWS with Terraform
6
+
The following steps will deploy the solution accelerator to AWS using Terraform. This is an alternative to the Localstack method. There is no need to manually install [Terraform](https://www.terraform.io/). It is executed via [Docker](https://www.docker.com/) using the `terraform.sh` script.
7
7
8
-
## Introduction
8
+
## Step 0: Prerequisites
9
9
10
-
The following [Steps](#steps) will deploy the solution accelerator to AWS using Terraform. There is no need to manually install [Terraform](https://www.terraform.io/). It is executed via [Docker](https://www.docker.com/) using the `terraform.sh` script.
11
-
12
-
## Steps to Deploy the Application
13
-
14
-
### Step 0: Prerequisites
15
-
16
-
1. Open a terminal.
17
-
2. Install **Docker** and **Docker Compose**.
18
-
3.[Clone the project](https://github.com/snowplow-industry-solutions/kafka-live-viewer-profiles) and navigate to its directory.
10
+
1. Open a terminal
11
+
2. Install **Docker** and **Docker Compose**
12
+
3.[Clone the project](https://github.com/snowplow-industry-solutions/kafka-live-viewer-profiles) and navigate to its directory
Review the [LocalStack guide](/tutorials/kafka-live-viewer-profiles/quickstart-localstack) for the default configuration for each component. Open public access to the two frontend applications and the Snowplow Collector using a HTTP load balancer so that anyone can watch the video, submit events to the pipeline, and see information on concurrent users.
54
+
Review the [LocalStack guide](/tutorials/kafka-live-viewer-profiles/quickstart-localstack) for the default configuration for each component. Open public access to the two front-end applications and the Snowplow Collector using a HTTP load balancer so that anyone can watch the video, submit events to the pipeline, and see information on concurrent users.
62
55
63
56
The applications listen for HTTP traffic on the following ports
64
-
- Web tracker front end - 3000
65
-
- Live viewer front end - 8280
66
-
- Snowplow collector - 9090
57
+
- Web tracker front-end - 3000
58
+
- Live viewer front-end - 8280
59
+
- Snowplow Collector - 9090
60
+
61
+
## Next steps
67
62
68
-
## Next Steps
69
63
- You can implement Snowplow media tracking on any [HTML5](/docs/sources/trackers/javascript-trackers/web-tracker/tracking-events/media/html5/) or [YouTube](/docs/sources/trackers/javascript-trackers/web-tracker/tracking-events/media/youtube/) media of your choice
70
-
- Look into the output from Kafka and extend the Live Viewer to include information on the media being watched and the user.
71
-
- Replace Amazon DynamoDB with an alternative to be cloud agnostic, e.g. Google Bigtable or MongoDB.
64
+
- Look into the output from Kafka and extend the Live Viewer to include information on the media being watched and the user
65
+
- Replace Amazon DynamoDB with an alternative to be cloud agnostic, e.g. Google Bigtable or MongoDB
Copy file name to clipboardExpand all lines: tutorials/kafka-live-viewer-profiles/introduction.md
+18-20Lines changed: 18 additions & 20 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -3,55 +3,53 @@ position: 1
3
3
title: Introduction
4
4
---
5
5
6
-
## About This Accelerator
7
-
Welcome to the **live viewer profiles** solution accelerator for video streaming!
6
+
Welcome to the **live viewer profiles** solution accelerator for video streaming.
8
7
9
8
This accelerator demonstrates how to build a real-time use case leveraging **Snowplow event data** to create live viewer profiles for a video streaming site. By combining Snowplow's streaming pipeline with **Apache Kafka**, a **Java application** and **AWS DynamoDB**, the solution processes live streaming events to visualize user interactions with video content and advertisements.
10
9
11
-
On the left side of the image below we have someone watching a video. Their events are sent through a Snowplow pipeline to Kafka where they are consumed and processed by an application. The result of this processing is displayed in the right window. This shows the number of active users and their current state.
10
+
On the left side of the image below, someone is watching a video. Their events are sent through a Snowplow pipeline to Kafka where they are consumed and processed by an application. The result of this processing is displayed in the right window. This shows the number of active users and their current state.
12
11
13
12

14
13
15
-
Through this hands-on guide, you’ll learn how to build, deploy, and extend real-time, event-driven architectures using Snowplow and Kafka, enabling personalized recommendations, real-time insights, and dynamic analytics for streaming platforms. The framework is inspired by common challenges in video streaming, including tracking user behavior, ad engagement, and session activities, with the goal of maintaining up-to-date viewer profiles in DynamoDB.
14
+
Through this hands-on guide, you'll learn how to build, deploy, and extend real-time, event-driven architectures using Snowplow and Kafka, enabling personalized recommendations, real-time insights, and dynamic analytics for streaming platforms. The framework is inspired by common challenges in video streaming, including tracking user behavior, ad engagement, and session activities, with the goal of maintaining up-to-date viewer profiles in DynamoDB.
16
15
17
16
This accelerator is open source and can serve as the foundation to build practical applications like real-time viewer insights, engagement analytics, ad performance tracking, and personalized recommendations. Whether you're optimizing ad placements or enhancing viewer satisfaction, this guide equips you to unlock the full potential of Snowplow event data.
18
17
19
18
Please start by reviewing how the application works in the next page on Localstack, even if you're planning to deploy with Terraform.
20
19
21
-
---
22
-
23
-
## Solution Accelerator Code
24
-
[**The code for this infrastructure is available on here on GitHub.**](https://github.com/snowplow-industry-solutions/kafka-live-viewer-profiles)
20
+
## Solution Accelerator code
25
21
26
-
---
22
+
The code for this infrastructure is available [here on GitHub](https://github.com/snowplow-industry-solutions/kafka-live-viewer-profiles).
27
23
28
24
## Architecture
29
25
30
26
The solution comprises several interconnected components:
31
27
32
-
-**Web Tracking Application**:
33
-
- A React application with a video to watch. Snowplow's media tracking has been configured to send events (e.g., play, pause, ad skipped) to the [Snowplow Collector](/docs/fundamentals).
28
+
-**Web tracking application**:
29
+
- A React application with a video to watch
30
+
- Snowplow's media tracking has been configured to send events (e.g., play, pause, ad skipped) to the [Snowplow Collector](/docs/fundamentals)
34
31
- Code available in [tracker-frontend](https://github.com/snowplow-industry-solutions/kafka-live-viewer-profiles/tree/main/tracker-frontend) folder in GitHub
35
32
36
33
-**Snowplow Collector**:
37
-
- Collects and forwards events via [Stream Enrich](/docs/fundamentals) and Kinesis to [Snowbridge](/docs/destinations/forwarding-events/snowbridge).
34
+
- Collects and forwards events via [Stream Enrich](/docs/fundamentals) and Kinesis to [Snowbridge](/docs/destinations/forwarding-events/snowbridge)
38
35
39
36
-**Snowplow Snowbridge**:
40
-
- Publishes events to Kafka for the Live Viewer Backend to consume
37
+
- Publishes events to Kafka for the Live Viewer back-end to consume
41
38
42
-
-**Live Viewer Backend**:
43
-
- A Java application which processes events from Kafka, stores the data in DynamoDB, and generates JSON state data for the Live Viewer Frontend
39
+
-**Live Viewer back-end**:
40
+
- A Java application which processes events from Kafka, stores the data in DynamoDB, and generates JSON state data for the Live Viewer front-end
44
41
- Code available in [live-viewer-backend](https://github.com/snowplow-industry-solutions/kafka-live-viewer-profiles/tree/main/live-viewer-backend) folder in GitHub
45
42
46
-
-**Live Viewer Frontend**:
47
-
- A HTML website which displays the state of users currently watching the video.
43
+
-**Live Viewer front-end**:
44
+
- A HTML website which displays the state of users currently watching the video
48
45
- Code available in [live-viewer-frontend](https://github.com/snowplow-industry-solutions/kafka-live-viewer-profiles/tree/main/live-viewer-frontend) folder in GitHub
49
46
50
-
The following diagram maps out where each component sits in the end to end communication flow.
47
+
The following diagram maps out where each component sits in the end-to-end communication flow.
51
48

52
49
53
-
### Components & Configuration
54
-
The following files in the [GitHub repository](https://github.com/snowplow-industry-solutions/kafka-live-viewer-profiles) can be used to configure the project's components.
50
+
### Components and configuration
51
+
52
+
The following files in the [GitHub repository](https://github.com/snowplow-industry-solutions/kafka-live-viewer-profiles) can be used to configure the project's components:
Run the following command to download and run everything in Docker:
30
29
31
30
```bash
32
31
./docker/up.sh
33
32
```
34
-
Details on everything that is installed can be found in [architecture](/tutorials/kafka-live-viewer-profiles/introduction#architecture)
33
+
Details on everything that's installed can be found in the [architecture](/tutorials/kafka-live-viewer-profiles/introduction#architecture) section on the previous page.
35
34
36
35
**Tips:**
37
-
- Use `Ctrl+C` to stop services but keep containers running.
38
-
- Pass service-specific options to `./docker/up.sh` (e.g., `./docker/up.sh kafka-services`).
36
+
- Use `Ctrl+C` to stop services but keep containers running
37
+
- Pass service-specific options to `./docker/up.sh` (e.g., `./docker/up.sh kafka-services`)
39
38
40
-
###Step 2: Open the Web Tracking Frontend
39
+
## Step 2: Open the web tracking front-end
41
40
42
41
Visit [http://localhost:3000](http://localhost:3000) to configure the Stream Collector endpoint and start tracking events. Enter the Collector URL: `localhost:9090` and click `Create tracker`.
43
42
@@ -51,38 +50,39 @@ You must keep this window open with the video playing because everything here is
51
50
52
51

53
52
54
-
###Step 3: Open the Live Viewer Frontend
53
+
## Step 3: Open the Live Viewer front-end
55
54
56
55
Open [http://localhost:8280](http://localhost:8280) in a separate window. This will display the active users and their current state (e.g. watching video, watching advertisement, paused).
57
56
58
57

59
58
60
59
Congratulations! You have successfully run the accelerator to stream web behavior through Snowplow and Kafka to a real-time dashboard.
61
60
62
-
## Next Steps
61
+
## Next steps
62
+
63
63
- You can implement Snowplow media tracking on any [HTML5](/docs/sources/trackers/javascript-trackers/web-tracker/tracking-events/media/html5/) or [YouTube](/docs/sources/trackers/javascript-trackers/web-tracker/tracking-events/media/youtube/) media of your choice
64
-
- Look into the output from Kafka and extend the Live Viewer to include information on the media being watched and the user.
65
-
- Use our supplied Terraform in the next section to run this on AWS and make it publicly available.
64
+
- Look into the output from Kafka and extend the Live Viewer to include information on the media being watched and the user
65
+
- Use the supplied Terraform in the next section to run this on AWS and make it publicly available
66
66
67
-
## Other Things You Can Do
67
+
## Other things you can do
68
68
69
69
### View Events in Kafka UI
70
70
71
71
Access [http://localhost:8080](http://localhost:8080) to review events within the Kafka UI.
72
72
73
-
### Manage Containers with LazyDocker
73
+
### Manage containers with LazyDocker
74
74
75
75
Run the following command to manage containers visually:
76
76
77
77
```bash
78
78
sudo ./docker/lazy.sh
79
79
```
80
80
81
-
### Inspect Infrastructure with LocalStack UI
81
+
### Inspect infrastructure with LocalStack UI
82
82
83
83
Visit the [LocalStack UI](https://app.localstack.cloud/) to inspect infrastructure components such as Kinesis and DynamoDB. Please note that a LocalStack account is required to view this.
84
84
85
-
## Cleaning Up
85
+
## Cleaning up
86
86
87
87
### Stop the containers
88
88
@@ -92,12 +92,12 @@ Shut down all running containers:
92
92
./docker/down.sh
93
93
```
94
94
95
-
### Clean Up and Delete
95
+
### Clean up and delete
96
96
97
97
To remove all containers and images, use:
98
98
99
99
```bash
100
100
./docker/clean.sh
101
101
```
102
102
103
-
**Warning**: This command will delete all generated data.
0 commit comments