Skip to content

ENT-12600: Document and refactor compile-options #1744

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

ENT-12600: Document and refactor compile-options #1744

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
f48a6cc
compile-options: Fixed shellcheck warnings
larsewi Jun 10, 2025
11a7c78
compile-options: Remove option to specify embedded DB
larsewi Jun 10, 2025
876d5c0
compile-options: Document script
larsewi Jun 10, 2025
e8dab1a
compile-options: Refactor/document dependencies
larsewi Jun 13, 2025
9908f47
compile-options: Consistent case statement format
larsewi Jun 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
214 changes: 144 additions & 70 deletions build-scripts/compile-options
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

# Autodect PROJECT if not set

if [ x"$PROJECT" = x ]
if [ -z "$PROJECT" ]
then
case x"$JOB_NAME" in
*-community-*) PROJECT=community;;
Expand All @@ -18,9 +18,9 @@ fi
# If still not set, then either we are running outside Jenkins, or this
# is not a main "build" type job (it could be the bootstrap job).
# Do directory-based auto-detection.
if [ x"$PROJECT" = x ]
if [ -z "$PROJECT" ]
then
if [ -d $BASEDIR/nova ]
if [ -d "$BASEDIR"/nova ]
then
PROJECT=nova
else
Expand All @@ -34,7 +34,7 @@ export PROJECT
# When running manually, you can just export this variable.
# It's a flag: if it's set to 1 - then we use system OpenSSL.
# Otherwise, we build it.
if [ x"$SYSTEM_SSL" = x ]
if [ -z "$SYSTEM_SSL" ]
then
# We don't bundle OpenSSL on RHEL 8 (and newer in the future)
if [ "$OS" = "rhel" ] && expr "$OS_VERSION" ">=" "8" >/dev/null
Expand Down Expand Up @@ -62,6 +62,13 @@ case "$OS_FAMILY" in
;;
hpux)
LDFLAGS="-L$BUILDPREFIX/lib -Wl,+b$BUILDPREFIX/lib"

# Use ‘gcc’ when building things on HP-UX
#
# HP-UX ships with ‘cc’ which invokes the ‘HP-UX bundled C compiler’ which
# lacks some functionality we need to build things. Among the other things,
# it doesn’t know how to link ‘.so’ files directly and it doesn’t recognize
# them as valid input type.
Comment on lines +68 to +71
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# HP-UX ships with ‘cc’ which invokes the ‘HP-UX bundled C compiler’ which
# lacks some functionality we need to build things. Among the other things,
# it doesn’t know how to link ‘.so’ files directly and it doesn’t recognize
# them as valid input type.
# HP-UX ships with ‘cc’ which invokes the ‘HP-UX bundled C compiler’ which lacks some functionality we need to build things. Among the other things, it doesn’t know how to link ‘.so’ files directly and it doesn’t recognize them as valid input type.

CC=gcc
export CC
;;
Expand All @@ -82,7 +89,7 @@ case "$OS_FAMILY" in
esac

# When we don't bundle OpenSSL, then we need to pull it from /usr/lib64.
if [ x"$SYSTEM_SSL" = x1 ]
if [ "$SYSTEM_SSL" = 1 ]
then
LDFLAGS="$LDFLAGS -L/usr/lib64"
fi
Expand All @@ -92,59 +99,85 @@ export LDFLAGS
DEB_LDFLAGS_APPEND="$LDFLAGS"
export DEB_LDFLAGS_APPEND

# Embedded DB selection
EMBEDDED_DB="lmdb"



############### Fill in build dependencies in DEPS variable ################

DEPS=
[ $OS_FAMILY = mingw ] && var_append DEPS "pthreads-w32 libgnurx"

# libgcc_s.so is needed before we compile any other dependency
# on some platforms!
case "$OS_FAMILY" in
solaris|aix) var_append DEPS "libgcc" ;;
esac

var_append DEPS "$EMBEDDED_DB pcre2"
# Windows specific dependencies
if [ "$OS_FAMILY" = mingw ]; then
# Win32 does not support pthreads natively. The pthreads-w32 project provide
# solution to this problem.
Comment on lines +109 to +110
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# Win32 does not support pthreads natively. The pthreads-w32 project provide
# solution to this problem.
# Win32 does not support pthreads natively. The pthreads-w32 project provide solution to this problem.

var_append DEPS "pthreads-w32"

if ! [ x"$SYSTEM_SSL" = x1 ]
then
# FIXME: Why do we need zlib?
# ANSWER: Openssl uses it optionally, TODO DISABLE
var_append DEPS "zlib openssl"
# A port of the regex functionality from the glibc Library for use on
# Windows platforms
Comment on lines +113 to +114
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# A port of the regex functionality from the glibc Library for use on
# Windows platforms
# A port of the regex functionality from the glibc Library for use on Windows platforms.

var_append DEPS "libgnurx"
fi

# libsasl needed for solaris
# AIX / Solaris specific dependencies
case "$OS_FAMILY" in
solaris|hpux) var_append DEPS "sasl2" ;;
esac
solaris|aix)

# iconv is needed for libxml2 on some platforms
case "$OS_FAMILY" in
aix|solaris) var_append DEPS "libiconv" ;;
# libgcc_s.so is needed before we compile any other dependency
# on some platforms!
Comment on lines +122 to +123
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# libgcc_s.so is needed before we compile any other dependency
# on some platforms!
# libgcc_s.so is needed before we compile any other dependencies on some platforms!

Is it specifically libgcc_s.so or is it the gcc command? Curious.

var_append DEPS "libgcc"

# iconv is needed for libxml2 on some platforms
var_append DEPS "libiconv"
;;
esac

var_append DEPS "libxml2 libyaml"
var_append DEPS "diffutils"
var_append DEPS "librsync"
# We use system bundled SSL on RHEL >= 8
if ! [ "$SYSTEM_SSL" = 1 ]
then
# zlib is a compression library witch is a dependency of OpenSSL. However,
# can we remove it (CFE-4013)?
var_append DEPS "zlib"

# LDAP functions in the agent
# and LDAP authentication functionality in Mission Portal
case "$PROJECT" in
nova) var_append DEPS "openldap"
esac
# Toolkit for TLS
var_append DEPS "openssl"
fi

case "$PROJECT" in
nova) var_append DEPS "leech"
# Solaris / HP-UX specific dependencies
case "$OS_FAMILY" in
solaris|hpux)
# Generic library that implements the Simple Authentication and Security
# Layer (SASL) framework
Comment on lines +145 to +146
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# Generic library that implements the Simple Authentication and Security
# Layer (SASL) framework
# Generic library that implements the Simple Authentication and Security Layer (SASL) framework

var_append DEPS "sasl2"
;;
esac

# libacl & libattr - not for the exotics (linux only?)
# Common dependencies
var_append DEPS "libxml2" # Library for parsing XML
var_append DEPS "libyaml" # Library for parsing YAML
var_append DEPS "diffutils" # Library for comparing files
var_append DEPS "librsync" # Library for synchronization of files
var_append DEPS "lmdb" # Library for key-value store
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
var_append DEPS "lmdb" # Library for key-value store
var_append DEPS "lmdb" # Library for key-value store, used extensively by CFEngine components (cf-agent, etc)

var_append DEPS "pcre2" # Library for compiling/matching regex

# Enterprise only dependencies
if [ "$PROJECT" = nova ]; then
# Used for storing accounts, passwords and group memberships
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# Used for storing accounts, passwords and group memberships
# openldap is used for storing accounts, passwords and group memberships

# - Agent has LDAP related policy functions
# - Mission Portal uses LDAP for authentication functionality
var_append DEPS "openldap"

# Used for efficient synchronization of tabular data
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# Used for efficient synchronization of tabular data
# leech is used for efficient synchronization of tabular data

Where/how is it used? might be helpful to know like I commented on lmdb is used in various components (cf-agent, etc)

var_append DEPS "leech"
fi

# Non-exotics dependencies
case "$OS_FAMILY" in
hpux|aix|solaris|freebsd|mingw) ;;
*) var_append DEPS "libattr libacl" ;;
hpux|aix|solaris|freebsd|mingw)
;;
*)
# Library for managing Extended Attributes (xattrs) on filesystems
var_append DEPS "libattr"

# POSIX Access Control Lists (ACLs) on filesystems
var_append DEPS "libacl"
Comment on lines +175 to +179
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

might be interesting to note if (and I think they are) dependencies of apache?

;;
esac

# ROLE
Expand All @@ -156,25 +189,51 @@ esac
# and build according to the role specified by it.

case "$EXPLICIT_ROLE" in
agent) ROLE=agent;;
hub) ROLE=hub;;
agent)
ROLE=agent
;;
hub)
ROLE=hub
;;
*)
# Not running under Jenkins?
if [ x"$JENKINS_SERVER_COOKIE" = x ]
if [ -z "$JENKINS_SERVER_COOKIE" ]
then
case "$PROJECT-$ARCH-$OS-${OS_VERSION}" in
community-*) ROLE=agent;;
community-*)
ROLE=agent
;;
# We do not support 32 bits hubs anymore
nova-i386-*-*) ROLE=agent;;
nova-s390*-*-*) ROLE=agent;;
nova-*-centos-*) ROLE=hub;;
nova-*-debian-*) ROLE=hub;;
nova-*-opensuse-*) ROLE=hub;;
nova-*-rhel-*) ROLE=hub;;
nova-*-sles-*) ROLE=hub;;
nova-*-ubuntu-*) ROLE=hub;;
nova-*-mingw-*) ROLE=agent;;
nova-*) ROLE=agent;;
nova-i386-*-*)
ROLE=agent
;;
nova-s390*-*-*)
ROLE=agent
;;
nova-*-centos-*)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This section could use a comment near the top. Why do we assume hub for platforms that support it? Interesting.

ROLE=hub
;;
nova-*-debian-*)
ROLE=hub
;;
nova-*-opensuse-*)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we don't support hub on opensuse.

ROLE=hub
;;
nova-*-rhel-*)
ROLE=hub
;;
nova-*-sles-*)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no support for sles hub

ROLE=hub
;;
nova-*-ubuntu-*)
ROLE=hub
;;
nova-*-mingw-*)
ROLE=agent
;;
nova-*)
ROLE=agent
;;
*)
echo "Unknown project: $PROJECT"
exit 42
Expand Down Expand Up @@ -212,32 +271,47 @@ export ROLE
case "$ROLE" in
# HUB-ONLY dependencies
hub)
var_append DEPS "libcurl-hub"
var_append DEPS "nghttp2 libexpat apr apr-util apache git rsync"
var_append DEPS "postgresql php"
# Note that we make a separate curl package for the hub. This is because
# the hub will include the curl binary, but we don't want that for the
# clients.
var_append DEPS "libcurl-hub" # Provides API for performing network requests
var_append DEPS "nghttp2" # Provides implementation of the HTTP/2 protocol.
var_append DEPS "libexpat" # Provides stream-oriented XML parser
var_append DEPS "apr apr-util" # Provides a common interface to underlying OS features (used by Apache)
var_append DEPS "apache" # Provides HTTP server
var_append DEPS "git" # Provides a version control system
var_append DEPS "rsync" # Binary to efficiently synchronize files
var_append DEPS "postgresql" # Relational database
var_append DEPS "php" # Scripting language for web development
;;
# AGENT-ONLY dependencies
agent)
var_append DEPS "libcurl"
var_append DEPS "libcurl" # Provides API for performing network requests
;;
esac

# Make sure init.d script and systemd service is used where needed.
#
# We install in all Linux platforms, because it does no harm on platforms
# without systemd, and it has the advantage of working out of the box on
# platforms that adopt systemd later.
Comment on lines +295 to +297
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# We install in all Linux platforms, because it does no harm on platforms
# without systemd, and it has the advantage of working out of the box on
# platforms that adopt systemd later.
# We install in all Linux platforms, because it does no harm on platforms without systemd, and it has the advantage of working out of the box on platforms that adopt systemd later.

if [ "$OS_FAMILY" = linux ]; then
WITH_SYSTEMD=yes
else
WITH_SYSTEMD=no
fi

# systemd is not a build dependency, but should we generate
# unit files for it?
case "$OS_FAMILY" in
linux) WITH_SYSTEMD=yes;;
*) WITH_SYSTEMD=no;;
esac


# Determine whether or not to run tests
case "$OS_FAMILY" in
mingw|freebsd) TESTS=no ;;
*) TESTS=all ;;
mingw|freebsd)
TESTS=no
;;
*)
TESTS=all
;;
esac
export TESTS


# Don't let existing LD_LIBRARY_PATH variables disturb the build.
# Java sets this when Jenkins launches it.
# PS! Solaris shell returns false if the var is already unset.
Expand Down
6 changes: 1 addition & 5 deletions build-scripts/configure
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,12 +15,8 @@ esac

P=$BUILDPREFIX

ARGS="--prefix=$P --libdir=$P/lib --with-workdir=$P --sysconfdir=/etc --with-openssl=$P --with-pcre2=$P --with-librsync=$P --with-init-script"
ARGS="--prefix=$P --libdir=$P/lib --with-workdir=$P --sysconfdir=/etc --with-lmdb=$P --with-openssl=$P --with-pcre2=$P --with-librsync=$P --with-init-script"

if [ $EMBEDDED_DB = lmdb ]
then
var_append ARGS "--with-lmdb=$P"
fi
case "$DEPS" in
*pthreads-w32*) var_append ARGS "--with-pthreads=$P" ;;
esac
Expand Down
3 changes: 0 additions & 3 deletions scripts/deptool.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -236,8 +236,6 @@ def deps_list(self, ref="master"):
"""Returns a sorted list of dependencies for given ref, for example: `["lcov", "libgnurx", "pthreads-w32"]`.
Assumes the proper ref is checked out by `self.buildscripts_repo`.
"""
# TODO: get value of $EMBEDDED_DB from file
embedded_db = "lmdb"
if ref == "3.7.x":
options_file = self.buildscripts_repo.get_file(
"build-scripts/install-dependencies"
Expand All @@ -264,7 +262,6 @@ def deps_list(self, ref="master"):
# in the middle we also do some clean-ups
only_deps = (
" ".join(only_deps)
.replace("$EMBEDDED_DB", embedded_db)
.replace("libgcc ", "")
.split(" ")
)
Expand Down
Loading