Skip to content

Commit ecf90fe

Browse files
sargenttisargentti
authored
Merge pull request #410 from WeBankFinTech/dev-1.0.0
[release] Prepare for release DSS-1.0.0
2 parents 35aab10 + 40fc666 commit ecf90fe

File tree

2,707 files changed

+186571
-86904
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

2,707 files changed

+186571
-86904
lines changed

‎README-ZH.md‎

Lines changed: 73 additions & 111 deletions
Large diffs are not rendered by default.

‎README.md‎

Lines changed: 19 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -33,75 +33,77 @@ Please be patient, it will take some time to load gif.
3333

3434
       c. [Qualitis](https://github.com/WeBankFinTech/Qualitis) - Data Quality Management Tool
3535

36-
       d. [Azkaban](https://azkaban.github.io/) - Batch workflow job scheduler
36+
       d. [Schedulis](https://github.com/WeBankFinTech/Schedulis) - Batch workflow job scheduler
37+
38+
       f. [Exchangis](https://github.com/WeBankFinTech/Exchangis) - Data Exchange Tool
3739

3840
![DSS one-stop video](images/en_US/readme/onestop.gif)
3941

40-
### 2. AppJoint, based on Linkis，defines a unique design concept
42+
### 2. AppConn, based on Linkis，defines a unique design concept
4143

42-
       AppJoint——application joint, defining unified front-end and back-end
44+
       AppConn——application connector, defining unified front-end and back-end
4345
integration specifications, can quickly and easily integrate with external data application systems,
4446
making them as part of DSS data application development.
4547

46-
       DSS arranges multiple AppJoints in series to form a workflow that supports real-time execution and scheduled execution. Users can complete the entire process development of data applications with simple drag and drop operations.
48+
       DSS arranges multiple AppConns in series to form a workflow that supports real-time execution and scheduled execution. Users can complete the entire process development of data applications with simple drag and drop operations.
4749

48-
       Since AppJoint is integrated with Linkis, the external data application system shares the capabilities of resource management, concurrent limiting, and high performance. AppJoint also allows sharable context across system level and completely gets away from application silos.
50+
       Since AppConn is integrated with Linkis, the external data application system shares the capabilities of resource management, concurrent limiting, and high performance. AppConn also allows sharable context across system level and completely gets away from application silos.
4951

5052
### 3. Project, as the management unit
5153

5254
       With Project as the management unit, DSS organizes and manages the business applications of each data application system, and defines a set of common standards for collaborative development of projects across data application systems.
5355

5456
### 4. Integrated data application components
5557

56-
      a. Azkaban AppJoint —— Batch workflow job scheduler
58+
      a. Schedulis AppConn —— Batch workflow job scheduler
5759

5860
         Many data applications developed by users usually require periodic scheduling capability.
5961

6062
         At present, the open source scheduling system in the community is pretty unfriendly to integrate with other data application systems.
6163

62-
         DSS implements Azkaban AppJoint, which allows users to publish DSS workflows to Azkaban for regular scheduling.
64+
         DSS implements Schedulis AppConn, which allows users to publish DSS workflows to Azkaban for regular scheduling.
6365

6466
         DSS also defines standard and generic workflow parsing and publishing specifications for scheduling systems, allowing other scheduling systems to easily achieve low-cost integration with DSS.
6567

6668
![Azkaban](images/en_US/readme/Azkaban_AppJoint.gif)
6769

68-
      b. Scriptis AppJoint —— Data Development IDE Tool
70+
      b. Scriptis AppConn —— Data Development IDE Tool
6971

7072
         What is [Scriptis](https://github.com/WeBankFinTech/Scriptis)?
7173

7274
         Scriptis is for interactive data analysis with script development(SQL, Pyspark, HiveQL), task submission(Spark, Hive), UDF, function, resource management and intelligent diagnosis.
7375

74-
         Scriptis AppJoint integrates the data development capabilities of Scriptis to DSS, and allows various script types of Scriptis to serve as nodes in the DSS workflow to participate in the application development process.
76+
         Scriptis AppConn integrates the data development capabilities of Scriptis to DSS, and allows various script types of Scriptis to serve as nodes in the DSS workflow to participate in the application development process.
7577

7678
         Currently supports HiveSQL, SparkSQL, Pyspark, Scala and other script node types.
7779

7880
![Scriptis](images/en_US/readme/Scriptis_AppJoint.gif)
7981

80-
      c. Visualis AppJoint —— Data Visualization Tool
82+
      c. Visualis AppConn —— Data Visualization Tool
8183

8284
         What is [Visualis](https://github.com/WeBankFinTech/Visualis)?
8385

8486
         Visualis is a BI tool for data visualization. It provides financial-grade data visualization capabilities on the basis of data security and permissions, based on the open source project Davinci contributed by CreditEase.
8587

86-
         Visualis AppJoint integrates data visualization capabilities to DSS, and allows displays and dashboards, as nodes of DSS workflows, to be associated with upstream data market.
88+
         Visualis AppConn integrates data visualization capabilities to DSS, and allows displays and dashboards, as nodes of DSS workflows, to be associated with upstream data market.
8789

8890
![Visualis](images/en_US/readme/Visualis_AppJoint.gif)
8991

90-
      d. Qualitis AppJoint —— Data quality management Tool
92+
      d. Qualitis AppConn —— Data quality management Tool
9193

92-
         Qualitis AppJoint integrates data quality verification capabilities for DSS, allows Qualitis as a node in DSS workflow
94+
         Qualitis AppConn integrates data quality verification capabilities for DSS, allows Qualitis as a node in DSS workflow
9395

9496
![Qualitis](images/en_US/readme/Qualitis_AppJoint.gif)
9597

96-
      e. Data Sender——Sender AppJoint
98+
      e. Data Sender——Sender AppConn
9799

98-
         Sender AppJoint provides data delivery capability for DSS. Currently it supports the SendEmail node type, and the result sets of all other nodes can be sent via email.
100+
         Sender AppConn provides data delivery capability for DSS. Currently it supports the SendEmail node type, and the result sets of all other nodes can be sent via email.
99101

100102
         For example, the SendEmail node can directly send the screen shot of a display as an email.
101103

102-
      f. Signal AppJoint —— Signal Nodes
104+
      f. Signal AppConn —— Signal Nodes
103105

104-
         Signal AppJoint is used to strengthen the correlation between business and process while keeping them decoupled.
106+
         Signal AppConn is used to strengthen the correlation between business and process while keeping them decoupled.
105107

106108
         DataChecker Node：Checks whether a table or partition exists.
107109

‎assembly/bin/appconn-install.sh‎

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
#!/bin/sh
2+
#Actively load user env
3+
source ~/.bashrc
4+
shellDir=`dirname $0`
5+
workDir=`cd ${shellDir}/..;pwd`
6+
7+
SOURCE_ROOT=${workDir}
8+
9+
#load config
10+
source ${SOURCE_ROOT}/conf/config.sh
11+
source ${SOURCE_ROOT}/conf/db.sh
12+
13+
APPCONN_NAME=''
14+
APPCONN_INSTALL_IP=127.0.0.1
15+
APPCONN_INSTALL_PORT=8088
16+
17+
#echo "Current path of init sql is ${DB_DML_PATH}"
18+
LOCAL_IP="`ifconfig | grep 'inet' | grep -v '127.0.0.1' | cut -d: -f2 | awk '{ print $2}'`"
19+
20+
function isSuccess(){
21+
if [ $? -ne 0 ]; then
22+
echo "Failed to " + $1
23+
exit 1
24+
else
25+
echo "Succeed to" + $1
26+
fi
27+
}
28+
29+
PROC_NAME=DSSProjectServerApplication
30+
ProcNumber=`ps -ef |grep -w $PROC_NAME|grep -v grep|wc -l`
31+
if [ $ProcNumber -le 0 ];then
32+
echo "${PROC_NAME} is not running,Please check whether DSS is installed"
33+
exit 1000
34+
else
35+
echo "Begine to install appconn"
36+
fi
37+
38+
##choose install mysql mode
39+
function initInstallAppConn() {
40+
echo "Please select the type of installation component?"
41+
echo " 1: schedulis"
42+
echo " 2: visualis"
43+
echo " 3：Your AppConn Name"
44+
echo " 4：exit"
45+
read -p "Please input the choice:" idx
46+
if [[ '1' = "$idx" ]];then
47+
APPCONN_NAME="schedulis"
48+
elif [[ '2' = "$idx" ]];then
49+
APPCONN_NAME="visualis"
50+
elif [[ '4' = "$idx" ]];then
51+
echo "no choice,exit!"
52+
exit 1
53+
else
54+
APPCONN_NAME=$idx
55+
fi
56+
echo "Current installation component is ${APPCONN_NAME}"
57+
58+
echo ""
59+
echo "If this machine(127.0.0.1) is installed, enter 1"
60+
echo "For others, you need to enter a complete IP address."
61+
read -p "Please enter the ip of appconn: " ip
62+
APPCONN_INSTALL_IP=$ip
63+
if [[ '1' = "$ip" ]];then
64+
APPCONN_INSTALL_IP="127.0.0.1"
65+
fi
66+
echo "You input ip is ${APPCONN_INSTALL_IP}"
67+
68+
echo ""
69+
read -p "Please enter the port of appconn:" port
70+
APPCONN_INSTALL_PORT=$port
71+
echo "You input ip is ${APPCONN_INSTALL_PORT}"
72+
}
73+
74+
function replaceCommonIp() {
75+
if [[ $APPCONN_INSTALL_IP == "127.0.0.1" ]] || [[ $APPCONN_INSTALL_IP == "0.0.0.0" ]];then
76+
echo "APPCONN_INSTALL_IP is equals $APPCONN_INSTALL_IP, we will change it to ip address"
77+
APPCONN_INSTALL_IP=$LOCAL_IP
78+
fi
79+
}
80+
81+
##choose execute mysql mode
82+
function executeSQL() {
83+
TEMP_DB_DML_PATH=${SOURCE_ROOT}/dss-appconns/${APPCONN_NAME}/db
84+
DB_DML_PATH=$TEMP_DB_DML_PATH/init_real.sql
85+
cp -rf $TEMP_DB_DML_PATH/init.sql $DB_DML_PATH
86+
sed -i "s/APPCONN_INSTALL_IP/$APPCONN_INSTALL_IP/g" $DB_DML_PATH
87+
sed -i "s/APPCONN_INSTALL_PORT/$APPCONN_INSTALL_PORT/g" $DB_DML_PATH
88+
sed -i "s#DSS_INSTALL_HOME_VAL#$DSS_INSTALL_HOME#g" $DB_DML_PATH
89+
mysql -h$MYSQL_HOST -P$MYSQL_PORT -u$MYSQL_USER -p$MYSQL_PASSWORD -D$MYSQL_DB --default-character-set=utf8 -e "source $DB_DML_PATH"
90+
isSuccess "source $DB_DML_PATH"
91+
echo "the table update finished"
92+
}
93+
94+
echo ""
95+
echo "step1:Initialize installation settings"
96+
initInstallAppConn
97+
echo ""
98+
99+
echo "step2:replaceIp"
100+
replaceCommonIp
101+
echo ""
102+
103+
echo "step3:update database"
104+
executeSQL
105+
echo ""
106+
107+
echo "step4:refresh appconn load"
108+
curl -H "Token-Code:BML-AUTH" -H "Token-User:hadoop" -X GET http://${GATEWAY_INSTALL_IP}:${GATEWAY_PORT}/api/rest_j/v1/dss/framework/project/appconn/${APPCONN_NAME}/load
109+
echo ""

bin/checkEnv.sh renamed to assembly/bin/checkEnv.sh

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -36,8 +36,6 @@ echo "<-----start to check used cmd---->"
3636
need_cmd yum
3737
need_cmd java
3838
need_cmd mysql
39-
need_cmd unzip
40-
need_cmd expect
4139
need_cmd telnet
4240
need_cmd tar
4341
need_cmd sed

‎assembly/bin/excecuteSQL.sh‎

Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
#!/bin/sh
2+
3+
function checkExternalServer(){
4+
echo "telnet check for your $SERVER_NAME, if you wait for a long time,may be your $SERVER_NAME does not prepared"
5+
result=`echo -e "\n" | telnet $EXTERNAL_SERVER_IP $EXTERNAL_SERVER_PORT 2>/dev/null | grep Connected | wc -l`
6+
if [ $result -eq 1 ]; then
7+
echo "$SERVER_NAME is OK."
8+
else
9+
echo "$SERVER_NAME is Bad. You need to prepare the' $SERVER_NAME ' environment in advance"
10+
exit 1
11+
fi
12+
}
13+
14+
## choose install mode
15+
function chooseInstallMode() {
16+
echo "Simple installation mode"
17+
#check for Java
18+
checkJava
19+
#check for mysql
20+
SERVER_NAME=MYSQL
21+
EXTERNAL_SERVER_IP=$MYSQL_HOST
22+
EXTERNAL_SERVER_PORT=$MYSQL_PORT
23+
checkExternalServer
24+
}
25+
26+
##choose install mysql mode
27+
function chooseInstallMySQLMode() {
28+
echo "Do you want to clear Dss table information in the database?"
29+
echo " 1: Do not execute table-building statements"
30+
echo " 2: Dangerous! Clear all data and rebuild the tables."
31+
echo ""
32+
MYSQL_INSTALL_MODE=1
33+
read -p "Please input the choice:" idx
34+
if [[ '2' = "$idx" ]];then
35+
MYSQL_INSTALL_MODE=2
36+
echo "You chose Rebuild the table"
37+
elif [[ '1' = "$idx" ]];then
38+
MYSQL_INSTALL_MODE=1
39+
echo "You chose not execute table-building statements"
40+
else
41+
echo "no choice,exit!"
42+
exit 1
43+
fi
44+
45+
##init db
46+
if [[ '2' = "$MYSQL_INSTALL_MODE" ]];then
47+
ENV_FLAG="dev"
48+
DB_CONF_PATH=${workDir}/db
49+
DB_DML_PATH=$DB_CONF_PATH/dss_dml_real.sql
50+
replaceAppConnInstanceSQL
51+
executeSQL
52+
fi
53+
}
54+
55+
##choose execute mysql mode
56+
function executeSQL() {
57+
chooseInstallMode
58+
59+
sed -i "s/GATEWAY_INSTALL_IP/$GATEWAY_INSTALL_IP/g" $DB_DML_PATH
60+
sed -i "s/GATEWAY_PORT/$GATEWAY_PORT/g" $DB_DML_PATH
61+
62+
sed -i "s#DSS_INSTALL_HOME_VAL#$DSS_INSTALL_HOME#g" $DB_DML_PATH
63+
64+
mysql -h$MYSQL_HOST -P$MYSQL_PORT -u$MYSQL_USER -p$MYSQL_PASSWORD -D$MYSQL_DB --default-character-set=utf8 -e "source $DB_CONF_PATH/dss_ddl.sql"
65+
isSuccess "source dss_ddl.sql"
66+
mysql -h$MYSQL_HOST -P$MYSQL_PORT -u$MYSQL_USER -p$MYSQL_PASSWORD -D$MYSQL_DB --default-character-set=utf8 -e "source $DB_DML_PATH"
67+
isSuccess "source dss_dml_real.sql"
68+
echo "Rebuild the table"
69+
}
70+
71+
function replaceAppConnInstanceSQL() {
72+
DB_DML_PATH=$DB_CONF_PATH/dss_dml_real.sql
73+
cp -rf $DB_CONF_PATH/dss_dml.sql $DB_DML_PATH
74+
sed -i "s#ORCHESTRATOR_IP#$DSS_FRAMEWORK_ORCHESTRATOR_SERVER_INSTALL_IP#g" $DB_DML_PATH
75+
sed -i "s#ORCHESTRATOR_PORT#$DSS_FRAMEWORK_ORCHESTRATOR_SERVER_PORT#g" $DB_DML_PATH
76+
77+
sed -i "s#WORKFLOW_IP#$DSS_WORKFLOW_SERVER_INSTALL_IP#g" $DB_DML_PATH
78+
sed -i "s#WORKFLOW_PORT#$DSS_WORKFLOW_SERVER_PORT#g" $DB_DML_PATH
79+
80+
sed -i "s#EVENTCHECKER_JDBC_URL#$EVENTCHECKER_JDBC_URL#g" $DB_DML_PATH
81+
sed -i "s#EVENTCHECKER_JDBC_USERNAME#$EVENTCHECKER_JDBC_USERNAME#g" $DB_DML_PATH
82+
sed -i "s#EVENTCHECKER_JDBC_PASSWORD#$EVENTCHECKER_JDBC_PASSWORD#g" $DB_DML_PATH
83+
84+
sed -i "s#DATACHECKER_JOB_JDBC_URL#$DATACHECKER_JOB_JDBC_URL#g" $DB_DML_PATH
85+
sed -i "s#DATACHECKER_JOB_JDBC_USERNAME#$DATACHECKER_JOB_JDBC_USERNAME#g" $DB_DML_PATH
86+
sed -i "s#DATACHECKER_JOB_JDBC_PASSWORD#$DATACHECKER_JOB_JDBC_PASSWORD#g" $DB_DML_PATH
87+
88+
sed -i "s#DATACHECKER_BDP_JDBC_URL#$DATACHECKER_BDP_JDBC_URL#g" $DB_DML_PATH
89+
sed -i "s#DATACHECKER_BDP_JDBC_USERNAME#$DATACHECKER_BDP_JDBC_USERNAME#g" $DB_DML_PATH
90+
sed -i "s#DATACHECKER_BDP_JDBC_PASSWORD#$DATACHECKER_BDP_JDBC_PASSWORD#g" $DB_DML_PATH
91+
92+
sed -i "s#BDP_MASK_IP#127.0.0.1#g" $DB_DML_PATH
93+
sed -i "s#BDP_MASK_PORT#8087#g" $DB_DML_PATH
94+
95+
sed -i "s#EMAIL_HOST#${EMAIL_HOST}#g" $DB_DML_PATH
96+
sed -i "s#EMAIL_PORT#${EMAIL_PORT}#g" $DB_DML_PATH
97+
sed -i "s#EMAIL_USERNAME#${EMAIL_USERNAME}#g" $DB_DML_PATH
98+
sed -i "s#EMAIL_PASSWORD#${EMAIL_PASSWORD}#g" $DB_DML_PATH
99+
sed -i "s#EMAIL_PROTOCOL#${EMAIL_PROTOCOL}#g" $DB_DML_PATH
100+
}
101+
102+
chooseInstallMySQLMode

0 commit comments

Comments
 (0)