Go uses buildr for its build infrastructure except for mac installers which are done through ant.
Go's dependencies are managed through maven. The build scripts are mostly written in maven pom files with some parts being written in ant, rake & buildr.
There are 2 components in Go: Server & Agent.
Config:
- The code related to config, that is required by both the Server & the Agent (like object mapping of config etc.), is in 'config-api' module.
- The code related to config, that is required by the Server alone (like config read, write etc.), is in 'config-server' module.
Plugin infrastructure:
- 'go-plugin-api' contains API's exposed for plugin authors and is required by plugins during compile time.
- 'go-plugin-api-internal'
- 'go-plugin-infra'
- 'go-plugin-activator'
- 'go-plugin-access'
Agent:
- 'agent-bootstrapper' is the code that is available on fresh Agent installation. This is the only part that is not auto-upgraded. Its only responsibility is to launch the launcher.
- 'agent-launcher' is the code that takes care of downloading latest launcher, agent, plugins & restart of agent when it dies / when upgrade is required.
- 'agent' - contains code to fetch jobs, run tasks & communicate results back to server.
- The code that is common to above 3 agent modules is in 'agent-common'.
Server:
- The complete code related to the server is in the 'server' module. Contains code related to database, security, view, etc. The server webapp is created using rake tasks. We generate help docs, minify css, js & inline rails partials as part of this.
- The module 'server-launcher' is a module definition with dependencies specified in pom file, which is used in creation of server launcher using buildr scripts.
Others:
- The code that is common to both the 'server' & 'agent' modules is in 'common' module.
- The code that is common to all the modules is in the 'base' module.
- Code related to TFS implementation is in a separate module 'tfs-impl'. It was made into a separate module to resolve classpath issues.
- The 'test-utils' module contain code that is used in unit & integration tests and which should not be part of main codebase.
Deprecated:
- The 'util' dependency is deprecated. Please refrain from adding code into this module.
Documentation is created through a rake task inside the helper module. The default task generates documentation based on changed files only while another task 'site' generates the whole documetation as it cleans before generating the documentation.
The documentation is written in xml format and all images are placed in resources. The xml files are parsed (parsing is done by ruby files present in builder module of helper) to generate corresponding html files.
-
Version of all Go modules is fixed to 1.0 which means the code for every module assumes that the latest code of its dependencies is available at version 1.0 in the local maven repository (~/.m2/repository).
-
The code does not follow the standard maven structure. We have code in src/, resources is in resources/, test code is in test/ & test resources is in test-resources/ folder. The locations are specified in the pom file to override defaults.
-
maven-compiler-plugin is configured with source & target to be 1.6.
-
All source dependencies are listed followed by test dependencies. In both source & test dependencies the Go modules are listed first followed by external library dependencies. We have tried our best to not redeclare dependencies that are available as transitive dependency from other modules/libraries. We have also tried to exclude transitive dependencies that are of a older/newer version than the ones that are required by Go. We have a pipeline dedicated to watch packaged dependencies & fail whenever there is a change to it. This acts as a manual gate to make sure the dependency change was intended.
- We use onejar-maven-plugin to package 'agent-bootstrapper', 'agent' & 'server-launcher'
- 'agent-launcher' & 'tfs-impl' is just a zip containing its own jar & its dependencies
- 'server' is packaged with standard 'maven-war-plugin'
- The rest of the modules are packaged with standard 'maven-jar-plugin'
5.5.1.5 TLB integration
Modules 'common' & 'server' have a lot of tests. We use TLB to run them in parallel in our builds. To achieve this we use 'maven-antrun-plugin' to run tests through ant-junit, with which TLB integrates, to prune the list of test to run before running them. The classpath is provided by maven to maintain consistency with the 'maven-surefire-plugin'. So any test would run under the same environment when run through maven or through TLB.
All the necessary files are put in a jar through buildr 'package'. This jar known as go.jar is then used by buildr while creating the installers.
Go creates all its installers except mac through buildr tasks in cruise-modules.rb. Mac installers are generated through ant tasks. The module structure of installers in code is as follows:
installers
+
|
+----->agent +----->deb
| +----->osx
| +----->release
| +----->rpm
| +----->sol
| +----->win
|
+----->server +----->deb
| +----->debian
| +----->osx
| +----->release
| +----->rpm
| +----->sol
| +----->win
|
+----->sharedInstaller module contains the Go server/agent installer scripts. The shared module inside installer contains debian and rpm pre/post install scripts shared between server and agent. Buildr packages these scripts while generating installers.
Before the installers are created, buildr packages the war file along with h2 db, db deltas, command repository, internal plugins, agent jar, agent bootstrapper and launcer jar, along with config files and libraries like tfs-impl, go-plugin-activator, etc. The packaging is done through buildr package and the jar created is known as go.jar.
For packaging command repository, the repo is cloned from the public git repo github.com/goteam/go-command-repo.git. Every time an installer is created the command repository version is bumped up. This is done to make sure the users are always on the latest version of command repository. The command repository version is independent of Go Server version.
The plugin jars placed inside tw-go-plugins/dist/ are packaged in Go as bundled plugin.
The buildr task cruise:pkg:zip calls cruise:agent-bootstrapper:dist:zip and cruise:server:dist:zip which creates the agent and server zip installers respectively. The release module inside server and agent modules of installer contains the files that needs to be present in the zip installer along with the License and the go.jar.
The release module provides all the scripts to start and stop server/agent. It has a CMD file which is used by the batch scripts on windows to start server/agent and has shell scripts for linux based systems to do the same. Also, scripts are provided to start/stop Go server/agent through a daemon conforming to Linux Standard Base (LSB) implementation to ensure it works across distros. Apart from these scripts, it also provides the installer with the default configuration to be used while starting Go server/agent. These scripts are also used by other installers!
The buildr task cruise:pkg:redhat calls cruise:agent-bootstrapper:dist:rpm and cruise:server:dist:rpm which creates the agent and server rpm installers respectively.
To prepare the installer, the zip installer is taken, unzipped and the files from it are placed accordingly along with the pre/post install scripts from shared module of installer. The spec file necessary for rpm installers is taken from rpm module inside agent/server modules of installer. rpmbuild with fakeroot is then used to generate the rpm installers.
The shared pre install script is used to create go user within a new go group if not already present while the post install script is used to create necessary directories and set ownership to user go. It also sets defaults path, log4j properties and java home to be used by Go. To set the java for Go, the system variable "JAVA_HOME" is checked if it's set. If not, the command which java is executed and its output is used.
The buildr task cruise:pkg:debian calls cruise:agent-bootstrapper:dist:debian and cruise:server:dist:debian which creates the agent and server rpm installers respectively.
To prepare the installer, the zip installer is taken, unzipped and the files from it are placed accordingly along with the pre/post install scripts from shared module and other necessary files for debian packaging that are taken from deb module inside agent/server modules of installer. dpkg-deb with fakeroot is then used to generate the debian installers.
Besides providing with the pre and post install scripts like rpm, the deb module provides pre and post rm (remove) scripts used while un-installing Go. prerm.go-server might be used during the upgrade flow as well and checks if the Go server version is greater than the existing one. If not, it displays a message accordingly.
The buildr task cruise:pkg:windows calls cruise:agent-bootstrapper:dist:exe and cruise:server:dist:exe which creates the agent and server windows installers respectively.
To prepare the installer, the zip installer is taken, unzipped and the files from it are placed accordingly along with the necessary files for windows packaging that are taken from win module inside agent/server modules of installer.
Nullsoft Scriptable Install System (NSIS) is used to create windows installers. INI files, JavaHome.ini and ServerIP.ini, are used to provide dialog boxes during installation for providing users the choice to use bundled jre or user specified java for Go to use, and to provide IP of server to the agent respectively.
The go-server.nsi and go-agent.nsi files present in the win module are the ones responsible for the installation procedure on windows. These files create the windows registry keys and install the wrapper-server.conf/wrapper-agent.conf to be used by Go server/agent for their start-up configurations respectively. These files also take care of deleting the windows registry keys (created by Go during installation) at the time of un-install. Both these files include windows-installer-base.nsi which is responsible to maintain the main flow of the installation process setting up the installation directory and reading/writing windows registry keys besides taking care of the upgrade flow. Silent installation of agent is also taken care of by these files. The line #include config/wrapper-properties.conf\ in wrapper-server.conf and wrapper-agent.conf provides the ability to override default startup arguments with the values written in the user provided file - wrapper-properties.conf.
The ant task dist.osx generates the mac installers. The information property list file, Info.plist present in osx module of agent/server module of installer provides the necessary configuration information for the bundled Go server/agent executable. The 'JavaApplicationStub64', used by the mac system to read the applications plist and then launch the Java VM with the appropriate properties such as the classpath and main class name, is provided from osx submodule inside build.
The buildr task cruise:pkg:solaris calls cruise:agent-bootstrapper:dist:solaris and cruise:server:dist:solaris which creates the agent and server solaris installers respectively.
To generate solaris installer Go uses the unzipped zip installer like other linux installers along with the necessary files for solaris packaging that are taken from sol module inside agent/server modules of installer.
The sol module provides the scripts for pre/post install and un-insatll solaris installer. pkginfo file contains the necessary information about the package and the svc.xml provides the data required by svcadm and register Go as a service on solaris. Prototype file entries are generated with pkgproto command and given as input to pkgmk command which produces an installable package. The installable package is then taken and translated to datastream with pkgtrans command followed by gzip to generate the installer.