Skip to content

Commit 2594e22

Browse files
tangzhankuntangzhankun
committed
SUBMARINE-85. Submarine 0.2.0 release submarine site update. Contributed by Xun Liu, Zhankun Tang, Wangda Tan.
1 parent 3cc8533 commit 2594e22

26 files changed

+2523
-19
lines changed

‎.gitignore

+1
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,3 @@
11
hugo.exe
2+
.idea
23
.DS_Store

‎submarine/config.toml

+1
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ footnotereturnlinkcontents = "<sup>^</sup>"
1919
uglyurls = false
2020
contentDir = "src"
2121
publishDir = "../content/submarine"
22+
disablePathToLower = true
2223

2324
[[menu.main]]
2425
name = "Download"

‎submarine/layouts/custompage/downloads.html

+27-7
Original file line numberDiff line numberDiff line change
@@ -30,26 +30,46 @@ <h1>Download</h1>
3030
<th>Binary download</th>
3131
<th>Release notes</th>
3232
</tr>
33-
</thead>
33+
</thead>
34+
<!--
3435
{{range first 5 (where (where .Site.Pages "Section" "release") ".Params.linked" true) }}
3536
<tr>
3637
<td>{{.File.BaseFileName }}</td>
3738
<td>{{dateFormat "2006 Jan 2 " .Date}}</td>
3839
<td>
39-
<a href="https://www.apache.org/dyn/closer.cgi/hadoop/ozone/ozone-{{.File.BaseFileName }}/hadoop-ozone-{{.File.BaseFileName }}-src.tar.gz">source</a>
40-
(<a href="https://dist.apache.org/repos/dist/release/hadoop/ozone/ozone-{{.File.BaseFileName }}/hadoop-ozone-{{.File.BaseFileName }}-src.tar.gz.mds">checksum</a>
41-
<a href="https://dist.apache.org/repos/dist/release/hadoop/ozone/ozone-{{.File.BaseFileName }}/hadoop-ozone-{{.File.BaseFileName }}-src.tar.gz.asc">signature</a>)
40+
<a href="https://www.apache.org/dyn/closer.cgi/hadoop/submarine/submarine-{{.BaseFileName }}/hadoop-submarine-{{.BaseFileName }}-src.tar.gz">source</a>
41+
(<a href="https://dist.apache.org/repos/dist/release/hadoop/submarine/submarine-{{.BaseFileName }}/hadoop-submarine-{{.BaseFileName }}-src.tar.gz.mds">checksum</a>
42+
<a href="https://dist.apache.org/repos/dist/release/hadoop/submarine/submarine-{{.BaseFileName }}/hadoop-submarine-{{.BaseFileName }}-src.tar.gz.asc">signature</a>)
4243
</td>
4344
<td>
44-
<a href="https://www.apache.org/dyn/closer.cgi/hadoop/ozone/ozone-{{.File.BaseFileName }}/hadoop-ozone-{{.File.BaseFileName }}.tar.gz">binary</a>
45-
(<a href="https://dist.apache.org/repos/dist/release/hadoop/ozone/ozone-{{.File.BaseFileName }}/hadoop-ozone-{{.File.BaseFileName }}.tar.gz.mds">checksum</a>
46-
<a href="https://dist.apache.org/repos/dist/release/hadoop/ozone/ozone-{{.File.BaseFileName }}/hadoop-ozone-{{.File.BaseFileName }}.tar.gz.asc">signature</a>)
45+
<a href="https://www.apache.org/dyn/closer.cgi/hadoop/submarine/submarine-{{.BaseFileName }}/hadoop-submarine-{{.BaseFileName }}.tar.gz">binary</a>
46+
(<a href="https://dist.apache.org/repos/dist/release/hadoop/submarine/submarine-{{.BaseFileName }}/hadoop-submarine-{{.BaseFileName }}.tar.gz.mds">checksum</a>
47+
<a href="https://dist.apache.org/repos/dist/release/hadoop/submarine/submarine-{{.BaseFileName }}/hadoop-submarine-{{.BaseFileName }}.tar.gz.asc">signature</a>)
4748
</td>
4849
<td>
4950
<a href="release/{{.File.BaseFileName }}/">Announcement</a>
5051
</td>
5152
</tr>
5253
{{end}}
54+
-->
55+
<tr>
56+
<td>0.1.0 (Apache Hadoop 3.2.0)</td>
57+
<td>2019 Jan 16</td>
58+
<td>
59+
<a href="https://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-3.2.0/hadoop-3.2.0-src.tar.gz">source</a>
60+
(<a href="https://www.apache.org/dist/hadoop/common/hadoop-3.2.0/hadoop-3.2.0-src.tar.gz.mds">checksum</a>
61+
<a href="https://www.apache.org/dist/hadoop/common/hadoop-3.2.0/hadoop-3.2.0-src.tar.gz.asc">signature</a>)
62+
</td>
63+
<td>
64+
<a href="https://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-3.2.0/hadoop-3.2.0.tar.gz">binary</a>
65+
(<a href="https://www.apache.org/dist/hadoop/common/hadoop-3.2.0/hadoop-3.2.0.tar.gz.mds">checksum</a>
66+
<a href="https://www.apache.org/dist/hadoop/common/hadoop-3.2.0/hadoop-3.2.0.tar.gz.asc">signature</a>)
67+
</td>
68+
<td>
69+
<a href="https://hadoop.apache.org/docs/r3.2.0/index.html">Announcement</a>
70+
</td>
71+
</tr>
72+
5373
</table>
5474
</p>
5575

‎submarine/layouts/partials/header.html

+2-2
Original file line numberDiff line numberDiff line change
@@ -73,8 +73,8 @@
7373
<div id="ratis-menu" class="collapse navbar-collapse">
7474
<ul class="nav navbar-nav navbar-right">
7575
<!-- <li><a href="downloads">Download</a></li> -->
76-
<li><a href="https://hadoop.apache.org/docs/r3.2.0/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/Index.html">Documentation</a></li>
77-
<li><a href="https://github.com/apache/hadoop">Source</a></li>
76+
<li><a href="docs/Index">Documentation</a></li>
77+
<li><a href="downloads">Download</a></li>
7878
<li><a href="ecosystem">Ecosystem</a></li>
7979
<li><a href="#qa">Q&A</a></li>
8080
<li><a href="https://cwiki.apache.org/confluence/display/HADOOP/Submarine+Contributor+Guide">Wiki</a></li>

‎submarine/src/activity.md

+13
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,19 @@ type: custompage
88

99
------
1010

11+
## Qcon 2019, Beijing
12+
> MAY 06–08, 2019
13+
14+
**Apache Hadoop machine learning engine Submarine and ecology.**
15+
16+
Submarine is a machine learning platform jointly developed by the Hadoop and Zeppelin communities. It supports Tensorflow, and machine learning frameworks such as Pytorch run in Kubernetes and YARN in a stand-alone or distributed manner.
17+
18+
Now you can use the `Submarine-installer` to easily install and deploy `NVIDIA-Docker`, `ETCD`, `Calico` and other machines to learn the running environment. In `Zeppelin`, you can visualize the interactive notebook before the Spark machine learning data processing. Develop and validate Tensorflow's Pythone algorithm, complete data processing of machine learning jobs and model training in Zeppelin's Workflow for full-link, and periodically perform offline model training in `Kubernetes`/`Hadoop`.
19+
20+
More infomation: https://2019.qconbeijing.com/presentation/1440
21+
22+
------
23+
1124
## DataWorks Summit in Barcelona, Spain
1225
> MARCH 18–21, 2019
1326

‎submarine/src/docs.md

-7
This file was deleted.

‎submarine/src/docs/0.2.0/BuildFromCode.md

+40
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
---
2+
title: "Apache Hadoop Submarine Documentation"
3+
layout: default
4+
type: custompage
5+
---
6+
7+
<!--
8+
Licensed to the Apache Software Foundation (ASF) under one or more
9+
contributor license agreements. See the NOTICE file distributed with
10+
this work for additional information regarding copyright ownership.
11+
The ASF licenses this file to You under the Apache License, Version 2.0
12+
(the "License"); you may not use this file except in compliance with
13+
the License. You may obtain a copy of the License at
14+
http://www.apache.org/licenses/LICENSE-2.0
15+
Unless required by applicable law or agreed to in writing, software
16+
distributed under the License is distributed on an "AS IS" BASIS,
17+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
18+
See the License for the specific language governing permissions and
19+
limitations under the License.
20+
-->
21+
22+
## Build Submarine From Source Code
23+
24+
- Run 'mvn install -DskipTests' from Hadoop source top level once.
25+
26+
- Navigate to hadoop-submarine folder and run 'mvn clean package'.
27+
28+
- By default, hadoop-submarine is built based on hadoop 3.1.2 dependencies.
29+
Both yarn service runtime and tony runtime are built.
30+
You can also add a parameter of "-Phadoop-3.2" to specify the dependencies
31+
to hadoop 3.2.0.
32+
33+
- Hadoop-submarine can support hadoop 2.9.2 and hadoop 2.7.4 as well.
34+
You can add "-Phadoop-2.9" to build submarine based on hadoop 2.9.2.
35+
For example:
36+
```
37+
mvn clean package -Phadoop-2.9
38+
```
39+
As yarn service is based on hadoop 3.*, so only tony runtime is built
40+
in this case.

‎submarine/src/docs/0.2.0/Examples.md

+27
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
---
2+
title: "Apache Hadoop Submarine Documentation"
3+
layout: default
4+
type: custompage
5+
---
6+
7+
<!---
8+
Licensed under the Apache License, Version 2.0 (the "License");
9+
you may not use this file except in compliance with the License.
10+
You may obtain a copy of the License at
11+
12+
http://www.apache.org/licenses/LICENSE-2.0
13+
14+
Unless required by applicable law or agreed to in writing, software
15+
distributed under the License is distributed on an "AS IS" BASIS,
16+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
17+
See the License for the specific language governing permissions and
18+
limitations under the License. See accompanying LICENSE file.
19+
-->
20+
21+
# Examples
22+
23+
Here're some examples about Submarine usage.
24+
25+
[Running Distributed CIFAR 10 Tensorflow Job](docs/0.2.0/RunningDistributedCifar10TFJobs)
26+
27+
[Running Standalone CIFAR 10 PyTorch Job](docs/0.2.0/RunningSingleNodeCifar10PTJobs)

‎submarine/src/docs/0.2.0/HowToInstall.md

+42
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
---
2+
title: "Apache Hadoop Submarine Documentation"
3+
layout: default
4+
type: custompage
5+
---
6+
<!--
7+
Licensed to the Apache Software Foundation (ASF) under one or more
8+
contributor license agreements. See the NOTICE file distributed with
9+
this work for additional information regarding copyright ownership.
10+
The ASF licenses this file to You under the Apache License, Version 2.0
11+
(the "License"); you may not use this file except in compliance with
12+
the License. You may obtain a copy of the License at
13+
http://www.apache.org/licenses/LICENSE-2.0
14+
Unless required by applicable law or agreed to in writing, software
15+
distributed under the License is distributed on an "AS IS" BASIS,
16+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
17+
See the License for the specific language governing permissions and
18+
limitations under the License.
19+
-->
20+
21+
# How to Install Dependencies
22+
23+
Submarine project may uses YARN Service (When Submarine YARN service runtime is being used, see [QuickStart](docs/0.2.0/QuickStart)), Docker container, and GPU (when GPU hardware available and properly configured).
24+
25+
That means as an admin, you may have to properly setup YARN Service related dependencies, including:
26+
- YARN Registry DNS
27+
28+
Docker related dependencies, including:
29+
- Docker binary with expected versions.
30+
- Docker network which allows Docker container can talk to each other across different nodes.
31+
32+
And when GPU plan to be used:
33+
- GPU Driver.
34+
- Nvidia-docker.
35+
36+
For your convenience, we provided installation documents to help you to setup your environment. You can always choose to have them installed in your own way.
37+
38+
Use Submarine installer to install dependencies: [EN](https://github.com/hadoopsubmarine/hadoop-submarine-ecosystem/tree/master/submarine-installer) [CN](https://github.com/hadoopsubmarine/submarine-installer/blob/master/README-CN.md)
39+
40+
Alternatively, you can follow manual install dependencies: [EN](docs/0.2.0/InstallationGuide) [CN](docs/0.2.0//InstallationGuideChineseVersion)
41+
42+
Once you have installed dependencies, please follow following guide to [TestAndTroubleshooting](docs/0.2.0/TestAndTroubleshooting).

‎submarine/src/docs/0.2.0/Index.md

+54
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
---
2+
title: "Apache Hadoop Submarine Documentation"
3+
layout: default
4+
type: custompage
5+
---
6+
7+
<!--
8+
Licensed to the Apache Software Foundation (ASF) under one or more
9+
contributor license agreements. See the NOTICE file distributed with
10+
this work for additional information regarding copyright ownership.
11+
The ASF licenses this file to You under the Apache License, Version 2.0
12+
(the "License"); you may not use this file except in compliance with
13+
the License. You may obtain a copy of the License at
14+
http://www.apache.org/licenses/LICENSE-2.0
15+
Unless required by applicable law or agreed to in writing, software
16+
distributed under the License is distributed on an "AS IS" BASIS,
17+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
18+
See the License for the specific language governing permissions and
19+
limitations under the License.
20+
-->
21+
22+
Submarine is a project which allows infra engineer / data scientist to run
23+
*unmodified* Tensorflow or PyTorch programs on YARN or Kubernetes.
24+
25+
Goals of Submarine:
26+
27+
- It allows jobs for easy access to data/models in HDFS and other storages.
28+
29+
- Can launch services to serve Tensorflow/MXNet models.
30+
31+
- Support run distributed Tensorflow jobs with simple configs.
32+
33+
- Support run standalone PyTorch jobs with simple configs.
34+
35+
- Support run user-specified Docker images.
36+
37+
- Support specify GPU and other resources.
38+
39+
- Support launch tensorboard for training jobs if user specified.
40+
41+
- Support customized DNS name for roles (like tensorboard.$user.$domain:6006)
42+
43+
44+
Click below contents if you want to understand more.
45+
46+
- [QuickStart Guide](docs/0.2.0/QuickStart)
47+
48+
- [Examples](docs/0.2.0/Examples)
49+
50+
- [How to write Dockerfile for Submarine TensorFlow jobs](docs/0.2.0/WriteDockerfileTF)
51+
52+
- [How to write Dockerfile for Submarine PyTorch jobs](docs/0.2.0/WriteDockerfilePT)
53+
54+
- [Installation guides](docs/0.2.0/HowToInstall)

0 commit comments

Comments
 (0)