Skip to content

#2673: Microservice pattern: Polling publisher #3243 #3292

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 30 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
faf0a12
added new microservice pattern polling-publisher, created folder stru…
quantdevv Feb 27, 2025
c4e9276
added two service polling service & publisher service to handle two d…
quantdevv Mar 12, 2025
0051d3f
polling-publisher service implementation
quantdevv Mar 21, 2025
a2b8743
fixed checkstyle errors
quantdevv Mar 21, 2025
4988d2e
subscriber-service implementation
quantdevv Mar 21, 2025
bb065d3
added entry point for subscriber service & unit test-case
quantdevv Mar 21, 2025
931ab84
added unit test-case for DataSourceService class
quantdevv Mar 21, 2025
78f1ea9
resolved dependency & structured classes
quantdevv Apr 3, 2025
a7497c4
implemented Data repository, DataSource Service & added unittest-case…
quantdevv Apr 3, 2025
a7cc2c3
implemented subscriber microservice using kafkaConsumer
quantdevv Apr 3, 2025
46abbfb
Fixed Kafka bug & added a controller to send message using API
quantdevv Apr 3, 2025
1b3d933
Fixed subscriber-service & updated pom.xml
quantdevv Apr 3, 2025
a59362c
Fixed topic name
quantdevv Apr 3, 2025
5aa46ac
added new listener for other topic
quantdevv Apr 3, 2025
4eeb011
updated all README.md
quantdevv Apr 3, 2025
24b87bb
added description on both application.yaml file
quantdevv Apr 6, 2025
5e3c951
synced with remote branch
quantdevv Apr 6, 2025
84d08f0
added scope before class definition in Test classes & removed duplica…
quantdevv Apr 6, 2025
059f292
added polling-publisher module in pom.xml
quantdevv Apr 6, 2025
8c50536
Merge branch 'master' into issue-2673
quantdevv Apr 6, 2025
f799e6f
Merge branch 'iluwatar:master' into issue-2673
quantdevv Jun 1, 2025
589fe40
merged changes from origin/master
quantdevv Jun 1, 2025
c2848da
Updated README.md file based on new template
quantdevv Jun 1, 2025
ee8c228
improved the README.md file & fixed it's format
quantdevv Jun 1, 2025
6b1fefd
deleted unnecessary README.md files
quantdevv Jun 1, 2025
f16fb4f
fixed the printing statements & used log function
quantdevv Jun 1, 2025
ff872e7
used logger to print statement
quantdevv Jun 1, 2025
256d81b
addressed review comment: removed unnecessary dependency
quantdevv Jun 1, 2025
e0dbd48
Merge branch 'issue-2673' of https://github.com/quantdevv/java-design…
quantdevv Jun 1, 2025
e49025f
Merge branch 'master' into issue-2673
quantdevv Jun 1, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
119 changes: 119 additions & 0 deletions polling-publisher/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
---
title: "Polling Publisher-Subscriber Pattern in Java: Mastering Asynchronous Messaging Elegantly"
shortTitle: Polling Pub/Sub
description: "Learn how to implement a Polling Publisher-Subscriber system in Java using Spring Boot and Kafka. Explore the architecture, real-world analogies, and benefits of asynchronous communication with clean code examples."
category: Architectural
language: en
tag:
- Spring Boot
- Kafka
- Microservices
- Asynchronous Messaging
- Decoupling
---

## Also known as

* Event-Driven Architecture
* Asynchronous Pub/Sub Pattern
* Message Queue-Based Polling System

## Intent of Polling Publisher-Subscriber Pattern

The Polling Publisher-Subscriber pattern decouples data producers from consumers by enabling asynchronous, message-driven communication. A service polls a data source and publishes messages to a message broker (e.g., Kafka), which are then consumed by one or more subscriber services.

## Detailed Explanation of the Pattern with Real-World Examples

### Real-world analogy

> A news agency constantly polls for the latest news updates. Once it receives new information, it publishes them to different news outlets (TV, newspapers, apps). Each outlet consumes and displays the updates independently.

### In plain words

> One service regularly checks for updates (polls) and sends messages to Kafka. Another service listens to Kafka and processes the messages asynchronously.

### Wikipedia says

> This pattern closely resembles the [Publish–subscribe model](https://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern), where messages are sent by publishers and received by subscribers without them knowing each other.

### Architecture Flow

```
+------------+ +--------+ +-------------+
| Publisher | ---> | Kafka | ---> | Subscriber |
+------------+ +--------+ +-------------+
```

## Programmatic Example (Spring Boot + Kafka)

### Publisher Service

- Uses Spring's `@Scheduled` to poll data periodically.
- Publishes data to a Kafka topic.
- Optionally exposes a REST API for manual data publishing.

```java
@Scheduled(fixedRate = 5000)
public void pollAndPublish() {
String data = pollingService.getLatestData();
kafkaTemplate.send("updates-topic", data);
}
```

### Subscriber Service

- Listens to Kafka topic using `@KafkaListener`.
- Processes messages asynchronously.

```java
@KafkaListener(topics = "updates-topic")
public void processUpdate(String message) {
log.info("Received update: {}", message);
updateProcessor.handle(message);
}
```

## When to Use the Polling Publisher-Subscriber Pattern

Use this pattern when:

* Real-time push from the producer is not possible.
* Loose coupling between producers and consumers is desired.
* You need asynchronous, scalable event processing.
* You are building an event-driven microservices architecture.

## Real-World Applications

* Real-time reporting dashboards
* Health check aggregators for distributed systems
* IoT telemetry processing
* Notification and alerting systems

## Benefits and Trade-offs of Polling Pub/Sub Pattern

### Benefits

* Loose coupling between services
* Asynchronous and scalable architecture
* Fault-tolerant with message persistence in Kafka
* Easy to extend with new consumers or publishers

### Trade-Offs

* Polling introduces latency between data generation and consumption
* Requires managing and configuring Kafka (or other brokers)
* Slightly more complex deployment and infrastructure setup

## Related Java Design Patterns

* [Observer Pattern](https://java-design-patterns.com/patterns/observer/)
* [Mediator Pattern](https://java-design-patterns.com/patterns/mediator/)
* [Message Queue Pattern](https://java-design-patterns.com/patterns/event-queue/)

## References and Credits

* [Apache Kafka Documentation](https://kafka.apache.org/documentation/)
* [Spring Kafka Documentation](https://docs.spring.io/spring-kafka)
* [Spring Scheduled Tasks](https://www.baeldung.com/spring-scheduled-tasks)
* [Spring Kafka Tutorial – Baeldung](https://www.baeldung.com/spring-kafka)
* Inspired by: [iluwatar/java-design-patterns](https://github.com/iluwatar/java-design-patterns)
2 changes: 2 additions & 0 deletions polling-publisher/etc/polling-publisher.urm.puml
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
@startuml
@enduml
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
@startuml
@enduml
59 changes: 59 additions & 0 deletions polling-publisher/polling-service/pom.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--

This project is licensed under the MIT license. Module model-view-viewmodel is using ZK framework licensed under LGPL (see lgpl-3.0.txt).

The MIT License
Copyright © 2014-2022 Ilkka Seppälä

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.iluwatar</groupId>
<artifactId>polling-publisher</artifactId>
<version>1.26.0-SNAPSHOT</version>
</parent>

<artifactId>polling-service</artifactId>
<packaging>jar</packaging>

<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<executions>
<execution>
<configuration>
<archive>
<manifest>
<mainClass>com.iluwatar.polling-service.App</mainClass>
</manifest>
</archive>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>

</project>
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
/*
* This project is licensed under the MIT license. Module model-view-viewmodel is using ZK framework licensed under LGPL (see lgpl-3.0.txt).
*
* The MIT License
* Copyright © 2014-2022 Ilkka Seppälä
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to deal
* in the Software without restriction, including without limitation the rights
* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is
* furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*/

package com.iluwatar.polling;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.scheduling.annotation.EnableScheduling;

/** Polling-Publisher pattern paradigm. */
@SpringBootApplication
@EnableScheduling
public class App {

/**
* Program entry point.
*
* @param args command line args.
*/
public static void main(String[] args) {
SpringApplication.run(App.class, args);
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
/*
* This project is licensed under the MIT license. Module model-view-viewmodel is using ZK framework licensed under LGPL (see lgpl-3.0.txt).
*
* The MIT License
* Copyright © 2014-2022 Ilkka Seppälä
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to deal
* in the Software without restriction, including without limitation the rights
* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is
* furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*/

package com.iluwatar.polling;

import java.util.HashMap;
import java.util.Map;
import javax.annotation.PostConstruct;
import org.springframework.stereotype.Repository;

/** Data repository to keep or store data. */
@Repository
public class DataRepository {

private final Map<Integer, String> dataStorage = new HashMap<>();

/** init after map creation ... to put dummy data. */
@PostConstruct
public void init() {
// Injecting dummy data at startup
dataStorage.put(2, "Initial Dummy Data - two - 2");
dataStorage.put(3, "Initial Dummy Data - three - 3");
dataStorage.put(4, "Initial Dummy Data - four - 4");
}

/** Save data to the repository. */
public void save(int id, String value) {
dataStorage.put(id, value);
}

/** Retrieve data by ID. */
public String findById(int id) {
return dataStorage.getOrDefault(id, "Data not found");
}

/** Delete data by ID. */
public void delete(int id) {
dataStorage.remove(id);
}

/** Get all data. */
public Map<Integer, String> findAll() {
return dataStorage;
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
/*
* This project is licensed under the MIT license. Module model-view-viewmodel is using ZK framework licensed under LGPL (see lgpl-3.0.txt).
*
* The MIT License
* Copyright © 2014-2022 Ilkka Seppälä
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to deal
* in the Software without restriction, including without limitation the rights
* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is
* furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*/

package com.iluwatar.polling;

import java.util.Map;
import java.util.Random;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Service;

/** This class is responsible for keeping the events. */
@Service
public class DataSourceService {

private static final Logger log = LoggerFactory.getLogger(DataSourceService.class);

private final DataRepository repository;
private final ScheduledExecutorService scheduler = Executors.newSingleThreadScheduledExecutor();

/** Constructor & Scheduler to push random data. */
public DataSourceService(DataRepository repository) {
this.repository = repository;
scheduleDataGeneration();
}

private void scheduleDataGeneration() {
Random random = new Random();
scheduler.scheduleAtFixedRate(
() -> {
int id = random.nextInt(100); // Random ID
String value = "Auto-Data-" + id;
this.addData(id, value);
log.info("🔵 Data Added: {} -> {}", id, value);
},
0,
3,
TimeUnit.SECONDS);
}

public void addData(int id, String value) {
repository.save(id, value);
}

public String getData(int id) {
return repository.findById(id);
}

public void removeData(int id) {
repository.delete(id);
}

public Map<Integer, String> getAllData() {
return repository.findAll();
}
}
Loading
Loading