|
| 1 | +--- |
| 2 | +layout: post |
| 3 | +title: "HOW TO USE OOIZE" |
| 4 | +subtitle: "大数据调度框架 ooize 的基本使用以及图解" |
| 5 | +date: 2010-08-25 |
| 6 | +author: "LSG" |
| 7 | +header-img: "img/ooize.png" |
| 8 | +catalog: true |
| 9 | +tags: |
| 10 | + - hadoop |
| 11 | + - ooize |
| 12 | + - spark |
| 13 | + - 调度框架 |
| 14 | +--- |
| 15 | + |
| 16 | +> **Oozie是大数据四大协作框架之一——任务调度框架，另外三个分别为数据转换工具Sqoop,文件收集库框架Flume,大数据WEB工具Hue。** |
| 17 | +
|
| 18 | +### 一. 安装和配置 |
| 19 | +配置邮件: |
| 20 | + |
| 21 | + |
| 22 | +### 二 .使用和常用命令 |
| 23 | +#### 2.1 验证正确性 |
| 24 | + |
| 25 | +1. 验证wokflow.xml : |
| 26 | +`oozie validate /appcom/apps/hduser0401/mbl_webtrends/workflow.xml` |
| 27 | + |
| 28 | +1. 查看oozie服务状态： |
| 29 | +`oozie admin -oozie http://localhost:11000/oozie -status` |
| 30 | + |
| 31 | +#### 2.2 执行,准备,提交,直接运行任务 |
| 32 | + |
| 33 | +1. 提交作业，作业进入PREP状态 submit |
| 34 | +`oozie job -oozie http://localhost:11000/oozie -config job.properties -submit` |
| 35 | +命令行出现: `job: jobID` |
| 36 | + |
| 37 | +1. 执行已提交的作业 start |
| 38 | +`oozie job -ooziehttp://localhost:11000/oozie -start jobID` |
| 39 | + |
| 40 | +1. 直接运行作业 run ->一般使用该方式 |
| 41 | +`oozie job -oozie http://localhost:11000/oozie -config job.properties -run` |
| 42 | + |
| 43 | +1. 挂起和恢复 |
| 44 | +挂起->`oozie job -suspend 0000004-180119141609585-oozie-hado-C` |
| 45 | +恢复->`oozie job -resume 0000004-180119141609585-oozie-hado-C` |
| 46 | + |
| 47 | +1. 杀死作业 |
| 48 | +`oozie job -oozie http://localhost:11000/oozie -kill 14-20090525161321-oozie-joe` |
| 49 | + |
| 50 | +1. 改变作业参数，不能修改killed状态的作业 |
| 51 | +`oozie job -oozie http://localhost:11000/oozie -change 14-20090525161321-oozie-joe -value endtime=2011-12-01T05:00Z;concurrency=100;2011-10-01T05:00Z` |
| 52 | + |
| 53 | +1. 重新运行作业 |
| 54 | +`oozie job -rerun 0006360-160701092146726-oozie-hado-C -refresh -action 477-479` |
| 55 | +`oozie job -rerun 0006360-160701092146726-oozie-hado-C -D oozie.wf.rerun.failnodes=false` |
| 56 | + |
| 57 | +1. 检查作业状态 |
| 58 | +`oozie job -oozie http://localhost:11000/oozie -info 14-20090525161321-oozie-joe` |
| 59 | + |
| 60 | +1. 查看日志 |
| 61 | +`oozie job -oozie http://localhost:11000/oozie -log 14-20090525161321-oozie-joe` |
| 62 | + |
| 63 | +1. 提交pig作业 |
| 64 | +`oozie pig -oozie http://localhost:11000/oozie -file pigScriptFile -config job.properties -X -param_file params` |
| 65 | + |
| 66 | +1. 提交MR作业 |
| 67 | +`oozie mapreduce -oozie http://localhost:11000/oozie -config job.properties` |
| 68 | + |
| 69 | +1. 查看共享库 : |
| 70 | +`oozie admin -shareliblist sqoop` |
| 71 | + |
| 72 | +1. 动态更新共享库 : |
| 73 | +`oozie admin -sharelibupdate` |
| 74 | + |
| 75 | +### 三. wookflow语法 |
| 76 | + |
| 77 | +```xml |
| 78 | + |
| 79 | +<workflow-app xmlns="uri:oozie:workflow:0.2" name="Test"> |
| 80 | + <start to="GoodsNameGroup"/> |
| 81 | + <action name="GoodsNameGroup"> |
| 82 | + <spark xmlns="uri:oozie:spark-action:0.2"> |
| 83 | + <job-tracker>${jobTracker}</job-tracker> |
| 84 | + <name-node>${nameNode}</name-node> |
| 85 | + <job-xml>${workflowAppUri}/hive-site.xml</job-xml> |
| 86 | + <configuration> |
| 87 | + <property> |
| 88 | + <name>oozie.action.sharelib.for.spark</name> |
| 89 | + <value>spark2 </value> |
| 90 | + </property> |
| 91 | + <property> |
| 92 | + <name>oozie.use.system.libpath</name> |
| 93 | + <value>true</value> |
| 94 | + </property> |
| 95 | + <property> |
| 96 | + <name>mapred.job.queue.name</name> |
| 97 | + <value>${queueName}</value> |
| 98 | + </property> |
| 99 | + <property> |
| 100 | + <name>tez.queue.name</name> |
| 101 | + <value>${queueName}</value> |
| 102 | + </property> |
| 103 | + |
| 104 | + </configuration> |
| 105 | + <master>yarn-cluster</master> |
| 106 | + <mode>cluster</mode> |
| 107 | + <name>GoodsNameGroup</name> |
| 108 | + <class>com.wangdian.spark.tasks.main.clear.test.GoodsNameGroup</class> |
| 109 | + <jar>${jarPath}</jar> |
| 110 | + <spark-opts>--executor-cores 1 --executor-memory 3584M --num-executors 20 --queue ${queueName}</spark-opts> |
| 111 | + </spark> |
| 112 | + <ok to="end"/> |
| 113 | + <error to="an-email"/> |
| 114 | + </action> |
| 115 | + <action name="an-email"> |
| 116 | + <email xmlns="uri:oozie:email-action:0.1"> |
| 117 | + |
| 118 | + <subject>Email notifications for ${wf:id()} spark-test</subject> |
| 119 | + <body>The wf ${wf:id()} failed completed.</body> |
| 120 | + </email> |
| 121 | + <ok to="kill"/> |
| 122 | + <error to="kill"/> |
| 123 | + </action> |
| 124 | + <kill name="kill"> |
| 125 | + <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> |
| 126 | + </kill> |
| 127 | + <end name="end"/> |
| 128 | +</workflow-app> |
| 129 | +``` |
| 130 | + |
| 131 | +### 四. coordinator语法: |
| 132 | +#### 4.1 coordinator.xml |
| 133 | +```xml |
| 134 | +<coordinator-app name="clear-stat-coordinator" frequency="*/10 0-17,20-23 * * *" start="${start}" end="${end}" timezone="Asia/Shanghai" |
| 135 | + xmlns="uri:oozie:coordinator:0.2"> |
| 136 | + <action> |
| 137 | + <workflow> |
| 138 | + <app-path>${workflowAppUri}</app-path> |
| 139 | + <configuration> |
| 140 | + <property> |
| 141 | + <name>jobTracker</name> |
| 142 | + <value>${jobTracker}</value> |
| 143 | + </property> |
| 144 | + <property> |
| 145 | + <name>nameNode</name> |
| 146 | + <value>${nameNode}</value> |
| 147 | + </property> |
| 148 | + <property> |
| 149 | + <name>queueName</name> |
| 150 | + <value>${queueName}</value> |
| 151 | + </property> |
| 152 | + </configuration> |
| 153 | + </workflow> |
| 154 | + </action> |
| 155 | +</coordinator-app> |
| 156 | + |
| 157 | +``` |
| 158 | +#### 4.2 job.properties |
| 159 | +```typescript |
| 160 | +nameNode=hdfs://cluster |
| 161 | +jobTracker=master |
| 162 | +examplesRoot=oozie-apps |
| 163 | +queueName=task |
| 164 | + |
| 165 | +# oozie.wf.application.path 标识这是一个workflow任务。 |
| 166 | +# oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/clear-stat-coordinator/workflow.xml |
| 167 | +oozie.coord.application.path=${nameNode}/user/${user.name}/${examplesRoot}/clear-stat-coordinator/coordinator.xml |
| 168 | +sparkopts=--executor-cores 1 --executor-memory 1g --num-executors 8 --queue ${queueName} --conf spark.yarn.maxAppAttempts=1 |
| 169 | +workflowAppUri=${nameNode}/user/${user.name}/${examplesRoot}/clear-stat-coordinator |
| 170 | +jarPath=lib/SparkTasks.jar |
| 171 | +statClass=com.wangdian.spark.tasks.main.stat.ExecuteSparkSqlTask |
| 172 | +clearClass=com.wangdian.spark.tasks.main.clear.StatDateClearApiTrade |
| 173 | +distinctClass=com.wangdian.spark.tasks.main.clear.HiveGoodsDistinctAndCategoryStatistics |
| 174 | +start=2020-01-09T15:00+0800 |
| 175 | +end=2020-01-11T17:45+0800 |
| 176 | + |
| 177 | +``` |
| 178 | + |
| 179 | +### 五. 使用脚本来节省时间: |
| 180 | +##### 由于每次提交任务都要上传到集群上,所以将这些命令自动化 |
| 181 | +任务提交: |
| 182 | +`./start.sh 文件包名` |
| 183 | + |
| 184 | +```bash |
| 185 | +jarNumStr=`ls -ll /data/bin/oozie-apps/spark-jar-warehouse | grep SparkTasks.jar* | wc -l` |
| 186 | +jarNum=${#jarNumStr} |
| 187 | +min=1 |
| 188 | +if [ $jarNum -gt $min ]; then |
| 189 | + echo "SparkTasks.jar类似文件存在多个,请保证一个最新版本" |
| 190 | + exit 1 |
| 191 | +fi |
| 192 | +hadoop fs -rm -r /user/hdfs/oozie-apps/$1 |
| 193 | +hadoop fs -put $1 /user/hdfs/oozie-apps/ |
| 194 | +oozie job -oozie http://hadoop03:11000/oozie -config $1/job.properties -run |
| 195 | +``` |
| 196 | + |
| 197 | +### 六. 常见实例图解: |
| 198 | +#### 6.1 执行开始: |
| 199 | + |
| 200 | +#### 6.2 执行完毕: |
| 201 | + |
| 202 | +### 七.ooize执行,长时间处于prep的一次优化 |
| 203 | +#### 7.1 ooize prep状态 |
| 204 | + |
| 205 | +#### 7.2发现原因是Yarn 中application太多了,导致两个问题:1.页面访问变慢2.ooize提交任务变慢 |
| 206 | + |
| 207 | +#### 7.3 解决办法: |
| 208 | + |
| 209 | +1. 删除log日志10000多个appl,省出大概800G空间, (自己集群设置的hdfs的位置) |
| 210 | +1. 删除zk上applacation任务, (默认位置: `zookeeper中: rmr /rmstore/ZKRMStateRoot/RMAppRoot` ) |
| 211 | +1. 重启ooize |
| 212 | + |
| 213 | + |
| 214 | + |
| 215 | + |
| 216 | +- 删除前 |
| 217 | + |
| 218 | + |
| 219 | + |
| 220 | +- 删除后: |
| 221 | + |
| 222 | + |
| 223 | +**最后问题解决!** |
| 224 | + |
| 225 | +### Reference |
| 226 | + |
| 227 | +* [ooize官网](http://oozie.apache.org/) |
| 228 | +* [什么是Oozie——大数据任务调度框架](https://blog.csdn.net/TNTZS666/article/details/81915820) |
| 229 | +* [ooize学习](https://www.cnblogs.com/cac2020/p/10509950.html) |
0 commit comments