Skip to content

Commit

Permalink
initial import!
Browse files Browse the repository at this point in the history
  • Loading branch information
buishglc committed Jun 21, 2020
0 parents commit dbd050a
Show file tree
Hide file tree
Showing 239 changed files with 19,541 additions and 0 deletions.
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
.idea
/*.iml
target
22 changes: 22 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# 大数据平台工程原型（Big Data Platform Project Prototype）


2008年Hadoop成为Apache的顶级项目，以此为开端，大数据技术迎来了十多年的持续发展，其间随着Spark的异军突起，整个大数据生态圈又经历了一次“装备升级”，变得更加完善和强大。今天，很多企业已经完成了早期对大数据技术的尝试和探索转而进入到应用阶段，但不得不说的是，大数据平台的架构体系非常庞大，技术堆栈特别深，从事大数据开发的同学对此应该都深有体会。

而在很多细分领域（例如实时计算、作业调度）也没有像样的工程模板, 这一现状与Java社区使用Spring Boot信手拈来地搭建Web工程原型形成了鲜明了对比。这导致很多团队在启动大数据平台建设时往往感到无所侍从，也使得希望深入学习大数据技术的开发者由于缺少工程级的示例参考而感到迷茫。

该原型项目就是以此为命题创建的，它基于过往项目的最佳实践提炼而来，可以帮助团队快速启动开发，上手就写业务代码。

本项目同时是《[大数据平台架构与原型实现：数据中台建设实战](https://item.jd.com/12677623.html)》一书的配套示例代码。该书已由知名IT图书品牌电子工业出版社博文视点出版发行，在京东和当当有售。
关于如何部署和运行该项目，书中做了非常细致的讲解。

京东购书链接：[https://item.jd.com/12677623.html](https://item.jd.com/12677623.html)

当当购书链接：[http://product.dangdang.com/28974965.html](http://product.dangdang.com/28974965.html)


点击[《重磅推荐：建大数据平台太难了！给我发个工程原型吧！》](https://laurence.blog.csdn.net/article/details/106851739)了解图书详情，扫码进入京东手机购书页面！

![在这里插入图片描述](https://img-blog.csdnimg.cn/2020061912530549.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JsdWlzaGdsYw==,size_16,color_FFFFFF,t_70#pic_center)


3 changes: 3 additions & 0 deletions bdp-collect/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
.idea
/*.iml
target
1 change: 1 addition & 0 deletions bdp-collect/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
关于本子项目的部署、运行与代码细节，请参考《大数据平台架构与原型实现：数据中台建设实战》一书第5章以及第4章4.5节
68 changes: 68 additions & 0 deletions bdp-collect/build.bat
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
@echo off
rem A batch script to build -> deploy -> restart
rem -- Laurence Geng
if [%1]==[] (
echo.
echo Usage: %0 [-delta] maven-profile-1 maven-profile-2 ...
echo.
echo Option: -delta: only deploy modified part, i.e. project artifact, used for development deploy.
goto end
)

set deltaDeploy=0
if "%~1"=="-delta" (
set deltaDeploy=1
shift
)

set profiles=%~1

:loopProfiles
shift
if "%~1"=="" (
goto build
) else (
set profiles=%profiles%,%~1
goto loopProfiles
)

:build
echo.
echo ***************************************************************************************
echo BUILD...
echo ***************************************************************************************
echo.

if "%profiles%"=="" (
call mvn clean install -DskipTests=true
) else (
call mvn clean install -DskipTests=true -P%profiles%
)

if "%errorlevel%"=="1" goto :buildfailed

if "%deltaDeploy%"=="1" (
call target\classes\deploy.bat -delta
) else (
call target\classes\deploy.bat
)

goto buildsuccess

:buildsuccess
echo.
echo.
echo ***************************************************************************************
echo BUILD SUCCESS!!
echo ***************************************************************************************
goto end

:buildfailed
echo.
echo.
echo ***************************************************************************************
echo BUILD FAILED!!
echo ***************************************************************************************
goto end

:end
199 changes: 199 additions & 0 deletions bdp-collect/pom.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,199 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>

<groupId>com.github</groupId>
<artifactId>bdp-collect</artifactId>
<version>1.0</version>
<packaging>jar</packaging>

<name>bdp-collect</name>

<parent>
<groupId>com.github</groupId>
<artifactId>bdp-parent</artifactId>
<version>1.0</version>
<relativePath>../bdp-parent/pom.xml</relativePath>
</parent>

<repositories>
<repository>
<id>maven.lib.umd.edu</id>
<url>http://maven.lib.umd.edu/maven2/</url>
</repository>
</repositories>

<dependencies>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-kafka</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-stream</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-sql</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-jackson</artifactId>
</dependency>
<dependency>
<groupId>com.zaxxer</groupId>
<artifactId>HikariCP</artifactId>
</dependency>
<dependency>
<groupId>joda-time</groupId>
<artifactId>joda-time</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-core</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-spring</artifactId>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-jdbc</artifactId>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-expression</artifactId>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-context</artifactId>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-http4</artifactId>
</dependency>
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.11</artifactId>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
<exclusion>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
</dependency>
<dependency>
<groupId>com.typesafe</groupId>
<artifactId>config</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-aws</artifactId>
</dependency>
</dependencies>

<profiles>
<profile>
<id>standalone</id>
<activation>
<activeByDefault>true</activeByDefault>
</activation>
<properties>
<profile.id>standalone</profile.id>
</properties>
</profile>
<profile>
<id>cluster</id>
<activation>
<activeByDefault>true</activeByDefault>
</activation>
<properties>
<profile.id>cluster</profile.id>
</properties>
</profile>
</profiles>

<build>
<filters>
<filter>src/main/profiles/${profile.id}.properties</filter>
</filters>
<resources>
<resource>
<directory>src/main/resources</directory>
<filtering>true</filtering>
</resource>
</resources>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>${jdk.version}</source>
<target>${jdk.version}</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<!-- disable test when building. tests will run by external testng cmd! -->
<configuration>
<skip>true</skip>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<configuration>
<excludes>
<exclude>bin/*.bat</exclude>
<exclude>bin/*.sh</exclude>
<exclude>conf/*.xml</exclude>
<exclude>conf/*.conf</exclude>
<exclude>conf/*.properties</exclude>
</excludes>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<inherited>false</inherited>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
<configuration>
<descriptors>
<descriptor>src/main/assembly/bin.xml</descriptor>
<descriptor>src/main/assembly/bin-delta.xml</descriptor>
</descriptors>
</configuration>
</plugin>
</plugins>
</build>

</project>
36 changes: 36 additions & 0 deletions bdp-collect/src/main/assembly/bin-delta.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
<assembly xmlns="http://maven.apache.org/ASSEMBLY/2.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/ASSEMBLY/2.0.0 http://maven.apache.org/xsd/assembly-2.0.0.xsd">
<id>bin-delta</id>
<formats>
<format>zip</format>
</formats>
<includeBaseDirectory>true</includeBaseDirectory>
<fileSets>
<fileSet>
<directory>target/classes/bin</directory>
<outputDirectory>./bin</outputDirectory>
<includes>
<include>*.sh</include>
</includes>
<fileMode>755</fileMode>
</fileSet>
<fileSet>
<directory>target/classes/conf</directory>
<outputDirectory>./conf</outputDirectory>
<includes>
<include>*.xml</include>
<include>*.conf</include>
<include>*.properties</include>
</includes>
</fileSet>
</fileSets>
<dependencySets>
<dependencySet>
<outputDirectory>./lib</outputDirectory>
<includes>
<include>com.github:bdp-collect</include>
</includes>
</dependencySet>
</dependencySets>
</assembly>
40 changes: 40 additions & 0 deletions bdp-collect/src/main/assembly/bin.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
<assembly xmlns="http://maven.apache.org/ASSEMBLY/2.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/ASSEMBLY/2.0.0 http://maven.apache.org/xsd/assembly-2.0.0.xsd">
<id>bin</id>
<formats>
<format>zip</format>
</formats>
<includeBaseDirectory>true</includeBaseDirectory>
<fileSets>
<fileSet>
<directory>target</directory>
<outputDirectory>./lib</outputDirectory>
<includes>
<include>*.jar</include>
</includes>
</fileSet>
<fileSet>
<directory>target/classes/bin</directory>
<outputDirectory>./bin</outputDirectory>
<includes>
<include>*.sh</include>
</includes>
<fileMode>755</fileMode>
</fileSet>
<fileSet>
<directory>target/classes/conf</directory>
<outputDirectory>./conf</outputDirectory>
<includes>
<include>*.xml</include>
<include>*.conf</include>
<include>*.properties</include>
</includes>
</fileSet>
</fileSets>
<dependencySets>
<dependencySet>
<outputDirectory>./lib</outputDirectory>
</dependencySet>
</dependencySets>
</assembly>
Loading

0 comments on commit dbd050a

Please sign in to comment.