Skip to content

Commit 723ca46

Browse files
zhufuyizhufuyi
committed
add example eshop
1 parent 08f868f commit 723ca46

File tree

2,010 files changed

+317088
-0
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

2,010 files changed

+317088
-0
lines changed

‎_14_eshop/README.md

+299
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,299 @@
1+
2+
## Eshop Microservices Example with Sponge
3+
4+
This article demonstrates how to use the Sponge framework to quickly build a simplified high-performance e-commerce system, implementing flash sale and order functionality, while ensuring data consistency through the Distributed Transaction Manager (DTM). The architecture of the e-commerce system is shown below:
5+
6+
![flash-sale-order-cache](https://raw.githubusercontent.com/zhufuyi/sponge_examples/main/assets/flash-sale-order-cache.png)
7+
8+
<br>
9+
10+
This includes [two source code examples](https://github.com/zhufuyi/sponge_examples/blob/main/_14_eshop). The two examples are identical except for slight differences in code structure. The main purpose is to demonstrate that Sponge supports creating microservice projects with different repository patterns. `example-1-multi-repo` is suitable for microservices with multiple repositories, while `example-2-mono-repo` is suitable for microservices with a single repository.
11+
12+
<br>
13+
14+
### Environment Setup
15+
16+
To build this e-commerce system, you need the following tools and dependencies:
17+
18+
- **[Sponge](https://github.com/zhufuyi/sponge)**: A tool for rapidly generating service and module code for different systems. Follow the [Sponge installation guide](https://github.com/zhufuyi/sponge/blob/main/assets/install-cn.md) for more information.
19+
20+
Additionally, the following services are required:
21+
22+
- **[DTM](https://github.com/dtm-labs/dtm)**: A distributed transaction manager for ensuring data consistency across multiple service calls.
23+
- **Redis**: Used in conjunction with DTM to handle flash sale caching.
24+
- **Kafka**: A message queue system for asynchronous order processing.
25+
- **MySQL**: Database for storing service data.
26+
27+
All services will be running on a virtual machine with the IP address `192.168.3.37`.
28+
29+
<br>
30+
31+
### Launching DTM Services
32+
33+
[DTM](https://github.com/dtm-labs/dtm) is a core component of this system, responsible for managing distributed transactions in the flash sale and order processes. Two DTM service instances are required—one for MySQL and one for Redis storage.
34+
35+
| Service Name | Port Configuration |
36+
|----------------|-------------------------------|
37+
| DTM-MySQL | HTTP: 36789, gRPC: 36790 |
38+
| DTM-Redis | HTTP: 35789, gRPC: 35790 |
39+
40+
<br>
41+
42+
#### 1. Launching DTM-MySQL Service
43+
44+
- Import the required table structure into MySQL:
45+
- [dtmcli.barrier.mysql.sql](https://github.com/dtm-labs/dtm/blob/main/sqls/dtmcli.barrier.mysql.sql)
46+
- [dtmsvr.storage.mysql.sql](https://github.com/dtm-labs/dtm/blob/main/sqls/dtmsvr.storage.mysql.sql)
47+
48+
- Modify the DTM configuration file ([Sample Configuration](https://github.com/dtm-labs/dtm/blob/main/conf.sample.yml)):
49+
50+
```yaml
51+
Store: # specify which engine to store trans status
52+
Driver: 'mysql'
53+
Host: '192.168.3.37'
54+
User: 'root'
55+
Password: '123456'
56+
Port: 3306
57+
Db: 'dtm'
58+
```
59+
60+
- Start the DTM service:
61+
62+
```bash
63+
./dtm -c conf.yml
64+
```
65+
66+
<br>
67+
68+
#### 2. Launching DTM-Redis Service
69+
70+
- Modify the DTM configuration file ([Sample Configuration](https://github.com/dtm-labs/dtm/blob/main/conf.sample.yml)):
71+
72+
```yaml
73+
Store: # specify which engine to store trans status
74+
Driver: 'redis'
75+
Host: '192.168.3.37'
76+
User: 'default'
77+
Password: '123456'
78+
Port: 6379
79+
```
80+
81+
- Start the DTM service:
82+
83+
```bash
84+
./dtm -c conf.yml
85+
```
86+
87+
<br>
88+
89+
### Rapid Development of an E-Commerce System Using Sponge
90+
91+
The simplified e-commerce system consists of the following eight microservices:
92+
93+
- **eshop_gw**: API gateway service
94+
- **user**: User service
95+
- **product**: Product service
96+
- **order**: Order service
97+
- **stock**: Stock service
98+
- **coupon**: Coupon service
99+
- **pay**: Payment service
100+
- **flashSale**: Flash sale service
101+
102+
<br>
103+
104+
#### 1. Prepare MySQL Databases and Tables for Each Service
105+
106+
Import the corresponding database tables for each service into MySQL:
107+
108+
- [User service schema](https://github.com/zhufuyi/sponge_examples/blob/main/_14_eshop/test/sql/user.sql)
109+
- [Product service schema](https://github.com/zhufuyi/sponge_examples/blob/main/_14_eshop/test/sql/product.sql)
110+
- [Order service schema](https://github.com/zhufuyi/sponge_examples/blob/main/_14_eshop/test/sql/order_record.sql)
111+
- [Stock service schema](https://github.com/zhufuyi/sponge_examples/blob/main/_14_eshop/test/sql/stock.sql)
112+
- [Coupon service schema](https://github.com/zhufuyi/sponge_examples/blob/main/_14_eshop/test/sql/coupon.sql)
113+
- [Payment service schema](https://github.com/zhufuyi/sponge_examples/blob/main/_14_eshop/test/sql/pay.sql)
114+
115+
<br>
116+
117+
#### 2. Prepare Protobuf Files for Each Service
118+
119+
These protobuf files allow Sponge to quickly create services:
120+
121+
- [User service protobuf](https://github.com/zhufuyi/sponge_examples/tree/main/_14_eshop/test/protobuf/user.proto)
122+
- [Product service protobuf](https://github.com/zhufuyi/sponge_examples/tree/main/_14_eshop/test/protobuf/product.proto)
123+
- [Order service protobuf](https://github.com/zhufuyi/sponge_examples/tree/main/_14_eshop/test/protobuf/order.proto)
124+
- [Stock service protobuf](https://github.com/zhufuyi/sponge_examples/tree/main/_14_eshop/test/protobuf/stock.proto)
125+
- [Coupon service protobuf](https://github.com/zhufuyi/sponge_examples/tree/main/_14_eshop/test/protobuf/coupon.proto)
126+
- [Payment service protobuf](https://github.com/zhufuyi/sponge_examples/tree/main/_14_eshop/test/protobuf/pay.proto)
127+
- [Flash sale service protobuf](https://github.com/zhufuyi/sponge_examples/tree/main/_14_eshop/test/protobuf/flashSale.proto)
128+
- [API Gateway protobuf](https://github.com/zhufuyi/sponge_examples/tree/main/_14_eshop/test/protobuf/eshop_gw.proto)
129+
130+
<br>
131+
132+
#### 3. Generate gRPC+HTTP Hybrid Service Code Based on Protobuf
133+
134+
Open the sponge UI page, switch to the menu `Protobuf` --> `Create grpc+http service`, fill in the parameters, and generate 7 hybrid service codes that support both grpc and http: user, product, order, stock, coupon, pay, flashSale, as shown below:
135+
136+
![eshop-grpc-http-pb](https://raw.githubusercontent.com/zhufuyi/sponge_examples/main/assets/en_eshop-grpc-http-pb.png)
137+
138+
After downloading the code, unzip each service code into the eshop directory.
139+
140+
> Note: If the large repository option is enabled on the code generation page, it means that the created service is suitable for a microservice mono-repo mode.
141+
142+
<br>
143+
144+
#### 4. Generate CRUD Code Based on MySQL Tables
145+
146+
Open the sponge UI page, switch to the menu `Public` --> `Generate service+handler CRUD code`, fill in the parameters, and generate CRUD codes for the user, product, order, stock, coupon, and pay services, as shown below:
147+
148+
![eshop-service-handler](https://raw.githubusercontent.com/zhufuyi/sponge_examples/main/assets/en_eshop-service-handler.png)
149+
150+
After downloading the code, unzip the CRUD code and move the CRUD code (the `api` and `internal` directories) into the corresponding service code (if prompted with duplicate proto files, just ignore it).
151+
152+
> Note: If the large repository option is enabled on the code generation page, it means that the created service is suitable for a microservice mono-repo mode.
153+
154+
<br>
155+
156+
#### 5. Generate API Gateway Service Code Based on Protobuf
157+
158+
Open the sponge UI page, switch to the menu `Protobuf` --> `Create grpc gateway service`, fill in the parameters, and generate the API gateway service code for eshop_gw, as shown below:
159+
160+
![eshop-grpc-gw-pb](https://raw.githubusercontent.com/zhufuyi/sponge_examples/main/assets/en_eshop-grpc-gw-pb.png)
161+
162+
After downloading the code, unzip the service code into the eshop directory.
163+
164+
To allow the eshop_gw service to connect to the various services, you need to generate the connection code. Open the sponge UI page, switch to the menu `Public` --> `Generate grpc connection code`, fill in the parameters, and generate the connection code for eshop_gw to connect to various grpc services, as shown below:
165+
166+
![eshop-grpc-conn](https://raw.githubusercontent.com/zhufuyi/sponge_examples/main/assets/en_eshop-grpc-conn.png)
167+
168+
After downloading the code, unzip it, and move the connection code (the `internal` directory) into the eshop_gw service code.
169+
170+
> Note: If the large repository option is enabled on the code generation page, it means that the created service is suitable for a microservice mono-repo mode.
171+
172+
<br>
173+
174+
#### 6. Write Business Logic Code
175+
176+
At this point, the service framework is basically set up. Next, write the actual business logic code in the `internal/service` directory of each service.
177+
178+
<br>
179+
180+
#### 7. Start the Services
181+
182+
Before starting the services, modify the configuration files of each service, including the port numbers, database connections, etc. The default HTTP port for each service is 8080, and the grpc port is 8282. Since they are running locally on the same machine (local test IP is 192.168.3.90), to avoid port conflicts, the ports of each service have been modified (ports can be found and modified in the `configs/xxx.yml` directory and `api/xxx/v1/xxx.proto`). Below are the modified ports:
183+
184+
| Service | Protocol | HTTP Port | gRPC Port |
185+
|------------------|--------------|-----------|-----------|
186+
| eshop_gw | HTTP | 8080 | - |
187+
| user | HTTP, gRPC | 30080 | 30082 |
188+
| product | HTTP, gRPC | 30180 | 30182 |
189+
| order | HTTP, gRPC | 30280 | 30282 |
190+
| stock | HTTP, gRPC | 30380 | 30382 |
191+
| coupon | HTTP, gRPC | 30480 | 30482 |
192+
| pay | HTTP, gRPC | 30580 | 30582 |
193+
| flashSale | HTTP, gRPC | 30680 | 30682 |
194+
195+
> Note: If running in containers or on different machines, you don’t need to modify the default ports, just change the mapped ports.
196+
197+
198+
<br>
199+
200+
### Testing and Verification
201+
202+
#### Single Service Testing
203+
204+
After all services have successfully started, verify that each service is functioning properly by testing the APIs of the following 7 services: user, product, order, stock, coupon, pay, and flashSale.
205+
206+
Open your browser and navigate to `http://localhost:<service_port>/apis/swagger/index.html` to check if the APIs of each service are working correctly. Besides testing the APIs on the Swagger page, you can also run gRPC API tests by filling in parameters in the `internal/service/xxx_client_test.go` file under each service.
207+
208+
<br>
209+
210+
#### Integration Testing
211+
212+
Once individual services pass the tests, use the API gateway service of `eshop_gw` to test the entire system. In your browser, visit the Swagger page of the API gateway service at [http://localhost:8080/apis/swagger/index.html](http://localhost:8080/apis/swagger/index.html), as shown in the figure below:
213+
214+
![eshop-gw-swagger](https://raw.githubusercontent.com/zhufuyi/sponge_examples/main/assets/eshop-gw-swagger.png)
215+
216+
<br>
217+
218+
**Testing the Submit Order API**
219+
220+
- The submit order process uses DTM's distributed transaction model `saga`, mainly to verify the consistency of creating orders, deducting stock, creating payment orders, and coupon data.
221+
222+
- To avoid order failure due to insufficient stock, set the stock before testing. In the Swagger page, find the API for setting product stock, and fill in the parameters. For example, set product ID to 1 and stock to 10:
223+
224+
```json
225+
{
226+
"productID": 1,
227+
"stock": 10
228+
}
229+
```
230+
231+
- Test the submit order API by requesting both the non-buffered queue and buffered queue APIs. In the Swagger page, find the respective API and fill in the parameters, such as user ID = 1, product ID = 1, product quantity = 1, and order amount = 100:
232+
233+
```json
234+
{
235+
"userID": 1,
236+
"productID": 1,
237+
"productCount": 1,
238+
"amount": 100,
239+
"couponID": 0
240+
}
241+
```
242+
243+
> Note: If `couponID` is not set to 0, it means a coupon will be used. If the coupon is invalid or expired, the order will fail. To ensure the order succeeds, find the API for creating a new coupon on the Swagger page, create a new coupon, get the coupon ID, and fill it into the `couponID` field of the submit order API.
244+
245+
<br>
246+
247+
**Testing the Flash Sale API**
248+
249+
- The flash sale process uses Kafka's message queue, DTM+Redis's two-phase message, and DTM+MySQL's `saga` distributed transaction models to verify consistency in flash sales, stock deduction, order creation, and payment order creation.
250+
251+
- To avoid order failure due to insufficient stock, set the stock before testing. In the Swagger page, find the API for setting product stock and fill in the parameters. For example, set product ID to 1 and stock to 10:
252+
253+
```json
254+
{
255+
"productID": 1,
256+
"stock": 10
257+
}
258+
```
259+
260+
- Test the flash sale API to verify data consistency. In the Swagger page, find the flash sale API and fill in the parameters, such as user ID = 1 and product ID = 1:
261+
262+
```json
263+
{
264+
"userID": 1,
265+
"productID": 1,
266+
"amount": 100
267+
}
268+
```
269+
270+
<br>
271+
272+
### Stress Testing
273+
274+
To perform stress testing on the `eshop_gw` API gateway service and verify the system's performance under high concurrency, use the stress testing tool [k6](https://github.com/grafana/k6). Before conducting the stress test, ensure that enough stock is set to avoid order failure.
275+
276+
1. For stress testing the Submit Order API scenario, use the k6 script [submit_order.js](https://github.com/zhufuyi/sponge_examples/blob/main/_14_eshop/test/submit_order.js) and run the following command:
277+
278+
```bash
279+
# 1000 virtual users, running for 10 seconds
280+
k6 run --vus 1000 --duration 10s test/k6/submitOrder.js
281+
282+
# Alternatively, specify the number of virtual users and the number of request iterations, for example, 1000 virtual users performing 100,000 request iterations
283+
k6 run -u 1000 -i 100000 submit_order.js
284+
```
285+
286+
2. For stress testing the Flash Sale API scenario, use the k6 script [flash_sale.js](https://github.com/zhufuyi/sponge_examples/blob/main/_14_eshop/test/flash_sale.js) and run the following command:
287+
288+
```bash
289+
# 10,000 virtual users, running for 1 second
290+
k6 run --vus 10000 --duration 1s test/k6/flashSale.js
291+
```
292+
293+
> Note: The results of stress testing depend on factors such as machine configuration, network environment, and database setup. Adjust accordingly based on actual conditions.
294+
295+
<br>
296+
297+
### Conclusion
298+
299+
By using the **Sponge** framework and the **DTM** distributed transaction manager, this example demonstrates how to quickly build a high-performance e-commerce system. The system architecture is divided into multiple services, including user, product, order, stock, and payment. Each service can implement CRUD operations through automatically generated code and business logic, ensuring data consistency in high-concurrency flash sale scenarios. By integrating Redis and Kafka, the system also supports efficient caching and message queuing, improving overall performance and scalability.

‎_14_eshop/example-1-multi-repo/coupon/.gitignore

+26
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
# Binaries for programs and plugins
2+
*.exe
3+
*.exe~
4+
*.dll
5+
*.so
6+
*.dylib
7+
*.log
8+
9+
# Test binary, built with `go test -c`
10+
*.test
11+
12+
# Output of the go coverage tool, specifically when used with LiteIDE
13+
*.out
14+
15+
# Dependency directories (remove the comment below to include it)
16+
vendor/
17+
dist/
18+
19+
# idea
20+
.idea
21+
*.iml
22+
*.ipr
23+
*.iws
24+
25+
cmd/coupon/coupon
26+

0 commit comments

Comments
 (0)