Improve the dockerfile #45

bilal-mk · 2018-10-13T07:29:39Z

Signed-off-by: Bilal.M.K [email protected]

Fixes #41

Cleanup apt Cache and reduce the docker layer
sysbench-mongo support without MySQL support

Fixes #42

Use alpine base image

Signed-off-by: Bilal.M.K <[email protected]>

bilal-mk · 2018-10-15T07:11:00Z

@ksatchit Could you please check and approve the pull request

ksatchit · 2018-10-15T08:14:07Z

@bilal-mk , the litmus tests with updated docker images are in progress. @shashank855 will approve the PR once done!

ksatchit · 2018-10-15T09:06:27Z

Just looking at the sysbench documentation, looks like building w/ the --without-mysql flag causes database tests to not work, which incidentally is needed for load generation tests in litmus (we use oltp.lua).

@shashank855 , can you confirm ?

bilal-mk · 2018-10-15T09:16:51Z

Just looking at the sysbench documentation, looks like building w/ the --without-mysql flag causes database tests to not work, which incidentally is needed for load generation tests in litmus (we use oltp.lua).

@shashank855 , can you confirm ?

I think mongo-client does not require mysql support.

shashank855

@bilal-mk If no database drivers are available database-related scripts will not work with --without-mysql.

shashank855 · 2018-10-15T09:14:21Z

mongo-client/Dockerfile


 RUN cp /usr/include/libbson-1.0/* /usr/include/ \
  && cp /usr/include/libmongoc-1.0/* /usr/include/

 ADD sysbench-mongo/sysbench /sysbench
 WORKDIR /sysbench
-RUN ./autogen.sh && ./configure && make
+RUN ./autogen.sh && ./configure --without-mysql && make


@bilal-mk Can you remove --without-mysql becouse its throwing error.

ksatchit · 2018-10-15T10:05:31Z

Just looking at the sysbench documentation, looks like building w/ the --without-mysql flag causes database tests to not work, which incidentally is needed for load generation tests in litmus (we use oltp.lua).
@shashank855 , can you confirm ?

I think mongo-client does not require mysql support.

Agree that the mongo-client is a bit of a misnomer. We named the image so because of the purpose it serves within litmus, but under the hood it is really sysbench. sysbench, as an IO tool is capable of generating synthetic workloads - but can also be built with DB drivers to help generate load on databases. Apparently, it looks like these tests work only when built w/ mysql (this is derived from percona's sysbench fork - https://github.com/Percona-Lab/sysbench/blob/0.5/README.md)

Signed-off-by: Bilal.M.K <[email protected]>

shashank855 · 2018-10-15T10:20:54Z

mysql-client/Dockerfile

+FROM alpine
+
+LABEL maintainer="OpenEBS"
+


@bilal-mk In our liveness script we used bash commands and Alpine docker image doesn't have bash installed by default. You will need to add following commands to get bash:
RUN apk update && apk add bash, Otherwise it will throw an error.

Signed-off-by: Bilal.M.K <[email protected]>

shashank855 · 2018-10-15T14:21:00Z

mongo-client/Dockerfile

-  libssl-dev \
-  libmongoc-dev \
-  libbson-dev 
+RUN apt-get update && apt-get -y --force-yes install --no-install-recommends \


@bilal-mk We figued out that --no-install-recommends does not allows to install some other dependency related to sysbench.

shashank855 · 2018-10-15T14:25:39Z

mongo-client/Dockerfile

-  libmongoc-dev \
-  libbson-dev 
+RUN apt-get update && apt-get -y --force-yes install --no-install-recommends \
+    python3 \ 


@bilal-mk As we are making this docker file optmised pyhon 3 & --upgrade setup tools may not be need. Your changes on these commented lines will highy apreciated.

shashank855 · 2018-10-15T14:28:49Z

mongo-client/Dockerfile

-    python3 \
-    python                                             \
-    python-pip  
+
 RUN pip install --upgrade setuptools                                        


@bilal-mk Can we bring these seperate RUN command into a single RUN ?

Signed-off-by: Bilal.M.K <[email protected]>

shashank855 · 2018-10-18T10:53:07Z

mongo-client/Dockerfile

@@ -1,35 +1,29 @@
 FROM ubuntu:16.04

-RUN apt-get clean \
+RUN apt-get update && apt-get -y --force-yes install \


Thanks for the changes @bilal-mk. This changes seems to be good but we tested this Docker file in our own environment and it doesn't work for the purpose. Can you kindly push the changes with the following Docker spec:

FROM ubuntu:16.04 RUN apt-get update -y || true RUN apt-get install -y --force-yes --no-install-recommends \ python \ python-pip \ make \ automake \ libmysqlclient-dev \ libtool \ libsasl2-dev \ libssl-dev \ libmongoc-dev \ libbson-dev RUN apt-get clean \ && rm -rf /var/lib/apt/lists/* \ && cp /usr/include/libbson-1.0/* /usr/include/ \ && cp /usr/include/libmongoc-1.0/* /usr/include/ # && pip install --upgrade pip RUN pip install --upgrade setuptools RUN python -m pip install pystrich \ && python -m pip install pymongo ADD sysbench-mongo/sysbench /sysbench WORKDIR /sysbench RUN ./autogen.sh && ./configure && make # components for liveness script ADD liveness/server.py ./

Same is this pull request with reduced the few-layer as part of the Dockerfile best practice. Please share how to properly test this or add those test as part of travis integration test.

Agreed @bilal-mk. Have raised #49 & #50 to track this.

Currently travis performs a build check alone for all the images and doesn't look for existence of packages/artifacts specified in the Dockerfile (saw a few commits pushed as "fixes" for those that broke the build), which is what has necessitated certain manual tests. One way of testing this is to look at the python/bash scripts being pushed into the image & verifying the presence of the binaries &/or auxiliary files used in it in the locally built image. Or maybe simpler, compare the existing docker image layers w/ the one built with changes (docker diff, docker export - or even simply entering the fs)

Amongst the multiple docker images maintained in this repo & litmus - hacktoberfest has helped us refactor the images & its CI to the "extent possible", i.e., w/o breaking prod deployments where they are being used as K8s resources. Of course, further optimizations are possible by refactoring the scripts used in them (which came about before containerizing them!) - but these will be taken up based on priority.

The dockerfile actually looks good. However, In this particular case, the following seem to sporadically cause a skipped install of some packages (though this shouldn't ideally impact us) :

the order of execution of apt-get clean/removal of repo refs in apt/lists

combining installation of python pip & pip-install of packages into same command.

setuptools dependency (typically this is needed only if easy_install is used)

Having said this, it may also depend on the build environment where the build is happening (its n/w connections, proxy settings etc.,).

The travis build has passed(https://api.travis-ci.org/v3/job/443697398/log.txt) & I have tested this on a GCP compute instance (you can guarantee a clean build env here!). So, we should be good to go.

We will revisit this for further optimizations

shashank855 · 2018-10-18T10:59:37Z

mysql-client/Dockerfile

+
+LABEL maintainer="OpenEBS"
+
+RUN apk add --no-cache mysql-client && apk add --no-cache jq && apk add --no-cache bash


@bilal-mk I think that adding --no-cache bash will not support all the bash command. What we can do is either installing moreutils package or we can use the base image as ubuntu only. Please accomodate this change it will highy appreciated.

Added moreutils package

@shashank855 moerutils is necessitated on account of usage of ts in the script. --no-cache shouldn't matter IMO.

shashank855

Few comments.
PTAL

Signed-off-by: Bilal.M.K <[email protected]>

ksatchit · 2018-10-20T02:56:30Z

Thanks for your tremendous patience with this @bilal-mk !

ksatchit

lgtm

Bilal.M.K added 2 commits October 13, 2018 12:56

cleanup apt cache | build without mysql

29a9245

Signed-off-by: Bilal.M.K <[email protected]>

Fixed the Dockerfile issue

126edd2

Signed-off-by: Bilal.M.K <[email protected]>

ksatchit requested a review from shashank855 October 14, 2018 00:51

ksatchit added the Hacktoberfest label Oct 14, 2018

Improve the dockerfile for mysql-client image

00ba6da

Signed-off-by: Bilal.M.K <[email protected]>

bilal-mk changed the title ~~cleanup apt cache | build without mysql~~ Improve the dockerfile Oct 14, 2018

shashank855 reviewed Oct 15, 2018

View reviewed changes

Add SysBench with MySQL support

7d79739

Signed-off-by: Bilal.M.K <[email protected]>

shashank855 suggested changes Oct 15, 2018

View reviewed changes

Add bash support

88765ff

Signed-off-by: Bilal.M.K <[email protected]>

shashank855 reviewed Oct 15, 2018

View reviewed changes

Bilal.M.K added 3 commits October 15, 2018 19:59

Remove python3 and --no-install-recommends

3d3bff1

Signed-off-by: Bilal.M.K <[email protected]>

Reduce docker image layer

eae9346

Signed-off-by: Bilal.M.K <[email protected]>

Bug fix

0602944

Signed-off-by: Bilal.M.K <[email protected]>

shashank855 reviewed Oct 18, 2018

View reviewed changes

shashank855 suggested changes Oct 18, 2018

View reviewed changes

Add moreutils package

dde97f8

Signed-off-by: Bilal.M.K <[email protected]>

ksatchit approved these changes Oct 20, 2018

View reviewed changes

ksatchit merged commit 1fee74b into litmuschaos:master Oct 20, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve the dockerfile #45

Improve the dockerfile #45

bilal-mk commented Oct 13, 2018 •

edited

Loading

bilal-mk commented Oct 15, 2018

ksatchit commented Oct 15, 2018

ksatchit commented Oct 15, 2018

bilal-mk commented Oct 15, 2018

shashank855 left a comment

shashank855 Oct 15, 2018

ksatchit commented Oct 15, 2018

shashank855 Oct 15, 2018

shashank855 Oct 15, 2018

shashank855 Oct 15, 2018

shashank855 Oct 15, 2018

shashank855 Oct 18, 2018

bilal-mk Oct 19, 2018

ksatchit Oct 20, 2018 •

edited

Loading

ksatchit Oct 20, 2018 •

edited

Loading

ksatchit Oct 20, 2018 •

edited

Loading

shashank855 Oct 18, 2018

bilal-mk Oct 19, 2018

ksatchit Oct 20, 2018

shashank855 left a comment

ksatchit commented Oct 20, 2018

ksatchit left a comment


		LABEL maintainer="OpenEBS"

		RUN apk add --no-cache mysql-client && apk add --no-cache jq && apk add --no-cache bash

Improve the dockerfile #45

Improve the dockerfile #45

Conversation

bilal-mk commented Oct 13, 2018 • edited Loading

bilal-mk commented Oct 15, 2018

ksatchit commented Oct 15, 2018

ksatchit commented Oct 15, 2018

bilal-mk commented Oct 15, 2018

shashank855 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ksatchit commented Oct 15, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ksatchit Oct 20, 2018 • edited Loading

Choose a reason for hiding this comment

ksatchit Oct 20, 2018 • edited Loading

Choose a reason for hiding this comment

ksatchit Oct 20, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shashank855 left a comment

Choose a reason for hiding this comment

ksatchit commented Oct 20, 2018

ksatchit left a comment

Choose a reason for hiding this comment

bilal-mk commented Oct 13, 2018 •

edited

Loading

ksatchit Oct 20, 2018 •

edited

Loading

ksatchit Oct 20, 2018 •

edited

Loading

ksatchit Oct 20, 2018 •

edited

Loading