Skip to content
This repository was archived by the owner on Dec 4, 2024. It is now read-only.

Commit 8fb5493

Browse files
ssk2ssk2
committed
New, improved Python script
1 parent 2aabffc commit 8fb5493

32 files changed

+1238
-207
lines changed

‎Dockerfile‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,2 @@
1-
FROM mprasil/dokuwiki:latest
2-
ADD start.txt /dokuwiki/data/pages/start.txt
3-
ENTRYPOINT chown www-data:www-data /dokuwiki/data/pages/start.txt && /usr/sbin/lighttpd -D -f /etc/lighttpd/lighttpd.conf
1+
FROM jekyll/jekyll
2+
ADD site /srv/jekyll

‎README.md‎

Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,126 @@
11
# cd-demo
22
A continuous delivery demo using Jenkins on DCOS.
3+
4+
This demo is a Python script that performs the following sequence of actions when run with the `install` command:
5+
6+
1. Installs Jenkins if it isn't already available.
7+
2. Sets up a series of build jobs, the necessary credentials and a [Build Pipeline](https://wiki.jenkins-ci.org/display/JENKINS/Build+Pipeline+Plugin) view to demonstrate a basic continuous delivery pipeline. These jobs:
8+
+ Build a Docker container based off the [Jekyll Docker image](https://hub.docker.com/r/jekyll/jekyll/) that includes the content stored in [/site](/site).
9+
+ Run the newly created container and a [Linkchecker container](https://github.com/mesosphere/docker-containers/blob/master/utils/linkchecker/Dockerfile) that runs a basic integration test against the container, checking that the webserver comes up correctly and that all links being served are valid (i.e. no 404s).
10+
+ Deploys the newly created container to the DCOS base Marathon instance.
11+
3. Creates 50 build jobs that take a random amount of time between 1 and 2 minutes. These jobs will randomly fail.
12+
13+
When run with the `uninstall` command, it will:
14+
15+
1. Remove any persisted credentials, build job and view configurations.
16+
2. Uninstall Jenkins.
17+
18+
`bin/demo.py --help` will show you full help text and usage information.
19+
20+
## Basic Usage
21+
22+
### Set Up
23+
24+
1. Clone this repository!
25+
26+
```
27+
git clone https://github.com/mesosphere/cd-demo.git
28+
```
29+
2. [Set up the DCOS CLI](https://docs.mesosphere.com/administration/introcli/cli/) locally.
30+
3. Ensure you have a DCOS cluster available. 1 node will work but more than 1 node is preferable to demonstrate build parallelism.
31+
32+
### Running Demo
33+
34+
1. Run the demo script. You will need to replace the password here with the password for the `cddemo` user with permission to push to `mesosphere/cd-demo-app`:
35+
36+
```
37+
bin/demo.py install --branch=my-demo-branch --password=mypass123 http://my.dcos.cluster/
38+
```
39+
2. The script will install Jenkins and pause. Check that the Jenkins UI is running before hitting enter to proceed.
40+
3. The script will now use the Jenkins HTTP API to install jobs, necessary credentials and a view. It will automatically trigger the initial build before pausing.
41+
4. Navigate to the Jenkins UI to see the builds in progress. After a few seconds, you should see a build executor spinning up on Mesos. If you navigate to the configured view, you'll see the pipeline in progress.
42+
5. Once the tests have completed successfully, you will need to manually deploy the build using the button in the bottom right of the "deploy" box on the view.
43+
![deploy](/img/manual-deploy.png)
44+
6. The deploy will happen almost instantaneously. After a few seconds, you should be able to load the application by navigating to your public slave's IP address in your browser.
45+
![deployed-app](/img/deployed-jekyll-app.png)
46+
7. Hit Enter to proceed to the next step of the demo. It will create 50 jobs that will randomly fail.
47+
8. Navigate back to the Jenkins and/or DCOS UI to show build slaves spinning up manually.
48+
9. Hit enter to complete the demo.
49+
50+
### Uninstalling
51+
52+
1. Simply run the uninstall command to remove any persisted configuration and to uninstall the DCOS service itself. This will allow you to run multiple demos on the same cluster but you should recycle clusters if the version of the Jenkins package has changed (to ensure plugins are upgraded):
53+
bin/demo.py uninstall http://my.dcos.cluster/
54+
55+
## Advanced Usage
56+
57+
### Using a Custom Docker Hub Organisation
58+
59+
By default, this script assumes you will be pushing to the [mesosphere/cd-demo-app repository on Docker Hub](https://hub.docker.com/r/mesosphere/cd-demo-app/).
60+
61+
1. Create a public repo called `cd-demo-app` under your example organisation.
62+
2. Create a Docker Hub user that has credentials to push to this repository.
63+
3. Run the demo script, passing in the credentials:
64+
65+
```
66+
bin/demo.py install --branch=my-demo-branch --org=myorg --username=myuser --password=mypass123 http://my.dcos.cluster/
67+
```
68+
69+
### Build on Commit
70+
71+
If you'd like to demonstrate the build running automatically with every commit:
72+
73+
1. By default this build operates off the `demo` branch. However, it's recommended that you create your own branches for demo purposes to avoid collisions. Create a new branch in this repository and push it up to origin:
74+
75+
```
76+
git checkout -b my-demo-branch
77+
git push origin my-demo-branch
78+
```
79+
2. Run the demo to completion with the `--branch` parameter to monitor your branch. The pipeline will continue to monitor your branch after the script finishes:
80+
81+
```
82+
bin/demo.py install --branch=my-demo-branch --password=mypass123 http://my.dcos.cluster/
83+
```
84+
3. Create a new blog post with today's date, open it up in your text editor and make whatever changes you'd like to:
85+
86+
```
87+
cp site/_posts/2016-02-25-welcome-to-cd-demo.markdown site/_posts/$(date +%Y-%m-%d)-my-test-post.markdown
88+
nano site/_posts/$(date +%Y-%m-%d)-my-test-post.markdown
89+
```
90+
4. Commit your changes and push them up to GitHub:
91+
92+
```
93+
git add site/_posts/$(date +%Y-%m-%d)-my-test-post.markdown
94+
git commit -m "Demo change"
95+
git push origin my-demo-branch
96+
```
97+
5. Jenkins will pick up the change within a minute and kick off the pipeline. If you want to fail the build, simply insert a broken link into your post.
98+
99+
### Demonstrating Multi-tenancy
100+
101+
To demonstrate how you can install multiple Jenkins instances side by side on DCOS, simply give your Jenkins instances unique names using the `--name` argument and run the demo as follows. Note that if you only have one public slave, you will not be able to deploy applications from multiple pipelines (each application requires port 80).
102+
103+
1. Create one instance:
104+
105+
```
106+
bin/demo.py install --name=jenkins-1 --password=mypass123 http://my.dcos.cluster/
107+
```
108+
2. Open a new terminal tab and create a second instance and so on:
109+
110+
```
111+
bin/demo.py install --name=jenkins-2 --password=mypass123 http://my.dcos.cluster/
112+
```
113+
3. You can uninstall these in the same way:
114+
115+
```
116+
bin/demo.py uninstall --name=jenkins-1 http://my.dcos.cluster/
117+
bin/demo.py uninstall --name=jenkins-2 http://my.dcos.cluster/
118+
```
119+
120+
### Skipping Demos
121+
122+
Only want to run one of the demos? Simply specify `--no-pipeline` to skip the continuous delivery demo, or `--no-dynamic-slaves` to skip the dynamic slaves demo.
123+
124+
## TODO
125+
126+
+ This script is currently untested on Windows.

‎bin/demo.py‎

Lines changed: 228 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,228 @@
1+
#!/usr/bin/python
2+
"""demo.py
3+
4+
Usage:
5+
demo.py install [--name=<name>] (--no-pipeline | [--branch=<branch>] [--org=<org>] [--username=<user>] --password=<pass>) [--no-dynamic-slaves] [--builds=<n>] <dcos_url>
6+
demo.py uninstall [--name=<name>] [--builds=<n>] <dcos_url>
7+
8+
Options:
9+
--name=<name> Jenkins instance name to use [default: jenkins-demo].
10+
--no-pipeline Don't run continuous delivery demo.
11+
--branch=<branch> Git branch for continuous delivery demo [default: demo].
12+
--org=<org> Docker Hub organisation where repo lives [default: mesosphere].
13+
--username=<user> Docker Hub username to push image with [default: cddemo].
14+
--password=<pass> Docker Hub password to push image with.
15+
--no-dynamic-slaves Don't run dynamic slaves demo.
16+
--builds=<n> Number of builds to create [default: 50].
17+
18+
This script is used to demonstrate various features of Jenkins on the DCOS.
19+
20+
Pre-requisites:
21+
+ A DCOS CLI is configured and available on the PATH of the host system
22+
+ A running DCOS cluster greater than version 1.4. It currently does not
23+
work with clusters that have authentication enabled.
24+
+ Python dependencies are installed (pip install -r requirements.txt)
25+
26+
The continuous delivery demo will create a build pipeline that will deploy a Docker
27+
container to the DCOS Marathon.
28+
29+
The dynamic slaves demo will create 50 (by default) "freestyle" Jenkins jobs.
30+
Each of these jobs will appear as a separate Jenkins build, and will randomly
31+
pass or fail. The duration of each job will be between 120 and 240 seconds.
32+
"""
33+
34+
from docopt import docopt
35+
import json
36+
import os
37+
import random
38+
import requests
39+
import shutil
40+
from subprocess import call
41+
42+
def log(message):
43+
print "[demo] {}".format(message)
44+
45+
def config_dcos_cli(dcos_url):
46+
if call (["dcos", "config", "set", "core.dcos_url", dcos_url],stdout=open(os.devnull, 'wb')) == 1:
47+
log("Unable to configure DCOS CLI.")
48+
exit(1)
49+
50+
def make_temp_dir():
51+
remove_temp_dir()
52+
os.mkdir("tmp")
53+
54+
def rename(path, jenkins_name):
55+
with open(path, 'r+') as f:
56+
config = json.load(f)
57+
config['jenkins']['framework-name'] = jenkins_name
58+
f.seek(0)
59+
json.dump(config, f, indent=4)
60+
61+
def install(dcos_url, jenkins_name, jenkins_url):
62+
log ("Installing Jenkins with name {}".format(jenkins_name))
63+
shutil.copyfile("conf/jenkins.json", "tmp/jenkins.json")
64+
rename("tmp/jenkins.json", jenkins_name)
65+
command = "dcos package install --yes --options=tmp/jenkins.json jenkins"
66+
print (command)
67+
if call (['dcos', 'package', 'install', '--yes', '--options=tmp/jenkins.json', 'jenkins']) != 0:
68+
log ("Failed to install Jenkins")
69+
exit(1)
70+
print("[demo] Jenkins has been installed! Wait for it to come up before proceeding at: {}".format(jenkins_url))
71+
raw_input("[demo] Press [Enter] to continue, or ^C to cancel...")
72+
os.system('clear')
73+
74+
def verify(jenkins_url):
75+
r = requests.get(jenkins_url)
76+
if r.status_code != 200:
77+
log ("Couldn't find a Jenkins instance running at {}".format(jenkins_url))
78+
return False
79+
log ("Jenkins is up and running! Got Jenkins version {}".format(r.headers['x-jenkins']))
80+
return True
81+
82+
def create_credentials(jenkins_url, id, username, password):
83+
credential = { 'credentials' : { 'scope' : 'GLOBAL', 'id' : id, 'username' : username, 'password' : password, 'description' : id, '$class' : 'com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl'} }
84+
data = {'json' : json.dumps(credential) }
85+
post_url = "{}/credential-store/domain/_/createCredentials".format(jenkins_url)
86+
r = requests.post(post_url, data=data)
87+
88+
def create_job(jenkins_url, job_name, job_config):
89+
log ("Creating job")
90+
headers = {'Content-Type' : 'application/xml' }
91+
post_url = "{}/createItem?name={}".format(jenkins_url, job_name)
92+
r = requests.post(post_url, headers=headers, data=job_config)
93+
if r.status_code != 200:
94+
log ("Failed to create job {} at {}".format(job_name, jenkins_url))
95+
exit(1)
96+
log ("Job {} created successfully".format(job_name))
97+
98+
def create_view(jenkins_url, view_name, view_config):
99+
log ("Creating view")
100+
headers = {'Content-Type' : 'text/xml' }
101+
post_url = "{}/createView?name={}".format(jenkins_url, view_name)
102+
r = requests.post(post_url, headers=headers, data=view_config)
103+
104+
def trigger_build(jenkins_url, job_name, parameter_string = None):
105+
log ("Triggering build {}".format(job_name))
106+
if parameter_string:
107+
post_url = "{}/job/{}/buildWithParameters?{}".format(jenkins_url, job_name, parameter_string)
108+
else:
109+
post_url = "{}/job/{}/build".format(jenkins_url, job_name)
110+
r = requests.post(post_url)
111+
112+
def delete_credentials(jenkins_url, credential_name):
113+
log ("Deleting credentials {}".format(credential_name))
114+
post_url = "{}/credential-store/domain/_/credential/{}/doDelete".format(jenkins_url, credential_name)
115+
r = requests.post(post_url)
116+
117+
def delete_job(jenkins_url, job_name):
118+
log ("Deleting job {}".format(job_name))
119+
post_url = "{}/job/{}/doDelete".format(jenkins_url, job_name)
120+
r = requests.post(post_url)
121+
122+
def delete_view(jenkins_url, view_name):
123+
log ("Deleting view {}".format(view_name))
124+
post_url = "{}/view/{}/doDelete".format(jenkins_url, view_name)
125+
r = requests.post(post_url)
126+
127+
def remove_temp_dir():
128+
shutil.rmtree("tmp", ignore_errors=True)
129+
130+
def demo_pipeline(jenkins_url, dcos_url, name, branch, org, username, password):
131+
log ("Creating demo pipeline")
132+
create_credentials(jenkins_url, 'docker-hub-credentials', username, password)
133+
with open("jobs/build-cd-demo/config.xml") as build_job:
134+
job_config = build_job.read().replace("GIT_BRANCH", branch)
135+
job_config = job_config.replace("DOCKER_HUB_ORG", org)
136+
create_job(jenkins_url, "build-cd-demo", job_config)
137+
with open("jobs/test-cd-demo/config.xml") as test_job:
138+
job_config = test_job.read().replace("GIT_BRANCH", branch)
139+
create_job(jenkins_url, "test-cd-demo", job_config)
140+
with open("jobs/deploy-cd-demo/config.xml") as deploy_job:
141+
job_config = deploy_job.read().replace("GIT_BRANCH", branch)
142+
job_config = job_config.replace("DCOS_URL", dcos_url)
143+
job_config = job_config.replace("JENKINS_NAME", name)
144+
create_job(jenkins_url, "deploy-cd-demo", job_config)
145+
with open("views/cd-demo-pipeline.xml") as pipeline_view:
146+
view_config = pipeline_view.read()
147+
create_view(jenkins_url, "cd-demo-pipeline", view_config)
148+
trigger_build(jenkins_url, "build-cd-demo")
149+
log ("Created demo pipeline")
150+
raw_input("[demo] Press [Enter] to continue, or ^C to cancel...")
151+
os.system('clear')
152+
153+
def demo_dynamic_slaves(jenkins_url, builds):
154+
log ("Creating {} freestyle Jenkins jobs".format(builds))
155+
random.seed()
156+
with open("jobs/demo-job/config.xml") as demo_job:
157+
job_config = demo_job.read()
158+
for i in range(1, builds):
159+
job_name = "demo-job-{0:02d}".format(i)
160+
create_job(jenkins_url, job_name, job_config)
161+
duration = random.randint(120, 240)
162+
result = random.randint(0, 1)
163+
parameter_string = '?DURATION={}&RESULT={}'.format(duration, result)
164+
trigger_build(jenkins_url, job_name, parameter_string)
165+
log ("Job {} created successfully. Duration: {}. Result: {}. Triggering build.".format(job_name, duration, result))
166+
log ("Created {} freestyle Jenkins jobs".format(builds))
167+
raw_input("[demo] Press [Enter] to continue, or ^C to cancel...")
168+
os.system('clear')
169+
170+
def cleanup_pipeline_jobs (jenkins_url):
171+
log ("Cleaning up demo pipeline")
172+
delete_credentials(jenkins_url, "docker-hub-credentials")
173+
delete_view(jenkins_url, "cd-demo-pipeline")
174+
delete_job(jenkins_url, "deploy-cd-demo")
175+
delete_job(jenkins_url, "test-cd-demo")
176+
delete_job(jenkins_url, "build-cd-demo")
177+
178+
def cleanup_dynamic_slaves_jobs(jenkins_url, builds):
179+
log ("Cleaning up {} builds".format(builds))
180+
for i in range(1, builds):
181+
job_name = "demo-job-{0:02d}".format(i)
182+
delete_job(jenkins_url, job_name)
183+
log ("Cleaned up {} builds".format(builds))
184+
185+
def cleanup(jenkins_url, builds):
186+
log ("Cleaning up Jenkins")
187+
cleanup_pipeline_jobs(jenkins_url)
188+
cleanup_dynamic_slaves_jobs(jenkins_url, builds)
189+
190+
def uninstall(dcos_url, jenkins_name):
191+
log ("Uninstalling Jenkins with name {}".format(jenkins_name))
192+
command = "dcos package uninstall --app-id={} jenkins".format(jenkins_name)
193+
print (command)
194+
if call (['dcos','package','uninstall','--app-id={}'.format(jenkins_name), 'jenkins']) != 0:
195+
log ("Failed to uninstall Jenkins")
196+
exit(1)
197+
print("[demo] Jenkins has been uninstalled!")
198+
199+
if __name__ == "__main__":
200+
arguments = docopt(__doc__, version="CD Demo 0.1")
201+
202+
jenkins_name = arguments['--name']
203+
builds = int(arguments['--builds'])
204+
dcos_url = arguments['<dcos_url>']
205+
jenkins_url = '{}service/{}/'.format(dcos_url, jenkins_name)
206+
207+
os.system('clear')
208+
config_dcos_cli(dcos_url)
209+
210+
try:
211+
if arguments['install']:
212+
make_temp_dir()
213+
if not verify(jenkins_url):
214+
install(dcos_url, jenkins_name, jenkins_url)
215+
if not arguments['--no-pipeline']:
216+
branch = arguments['--branch']
217+
org = arguments['--org']
218+
username = arguments['--username']
219+
password = arguments['--password']
220+
demo_pipeline(jenkins_url, dcos_url, jenkins_name, branch, org, username, password)
221+
if not arguments['--no-dynamic-slaves']:
222+
demo_dynamic_slaves(jenkins_url, builds)
223+
remove_temp_dir()
224+
elif arguments['uninstall']:
225+
cleanup(jenkins_url, builds)
226+
uninstall(dcos_url, jenkins_name)
227+
except KeyboardInterrupt:
228+
exit(0)

0 commit comments

Comments
 (0)