CODA EJFAT-based Frame Builder

A standalone C++ program that receives UDP packets from EJFAT load balancers, reconstructs fragmented frames, and outputs them in EVIO-6 format to files or ET systems. This program links against the installed E2SAR libraries.

Features

UDP Frame Reassembly: Receives segmented UDP packets and reconstructs complete event frames
Multi-threaded Reception: Configurable receiver threads with CPU core binding
EVIO-6 Frame Building: Aggregates frames from multiple data streams with timestamp synchronization
Dual Output Modes: Write to files (with 2GB auto-rollover) and/or ET systems simultaneously
EVIO Payload Validation: Magic number verification, endianness detection, metadata extraction
NUMA Awareness: Optional memory binding to specific NUMA nodes (platform-dependent)
Real-time Statistics: Periodic performance reporting with data rates
Graceful Shutdown: Signal handling with proper control plane deregistration

Architecture

                 UDP Packets from EJFAT
                         |
                    Reassembler
              (multi-threaded reception)
                         |
                   Data Frames
                         |
                   FrameBuilder
           (distributes by timestamp hash)
                         |
        +----------------+----------------+
        |                |                |
   BuilderThread-0  BuilderThread-1  BuilderThread-N
        |                |                |
   [Frame Buffer]   [Frame Buffer]   [Frame Buffer]
        |                |                |
        +----------------+----------------+
                         |
              ET System and/or Files
                  (EVIO-6 format)

Design Decisions:

Lock-free distribution using timestamp hashing for high throughput
Thread-local buffers and statistics to minimize contention
Separate ET attachments per builder thread for parallel publishing
EVIO-6 format compatible with Jefferson Lab CODA data acquisition
Frame builder is always used when ET library is available at build time

Prerequisites

System Requirements

C++17 compatible compiler (GCC 8+, Clang 6+)
E2SAR library installed
Boost libraries (1.83.0 - 1.86.0)
gRPC++ (>= 1.51.1)
Protocol Buffers
GLib 2.0
ET library (required for frame builder functionality)

Install Dependencies

macOS:

brew install boost grpc protobuf glib pkg-config meson ninja

Ubuntu/Debian:

sudo apt install libboost-all-dev libgrpc++-dev libprotobuf-dev libglib2.0-dev \
                 pkg-config meson ninja-build build-essential

RHEL/CentOS/Fedora:

sudo dnf install boost-devel grpc-devel protobuf-devel glib2-devel \
                 pkg-config meson ninja-build gcc-c++

E2SAR Library

Verify E2SAR is installed:

pkg-config --exists e2sar && echo "E2SAR found" || echo "E2SAR not found"

ET Library

For frame builder functionality, set environment variables if ET is in a non-standard location:

export CPATH=/path/to/et/include:$CPATH
export LIBRARY_PATH=/path/to/et/lib:$LIBRARY_PATH
export LD_LIBRARY_PATH=/path/to/et/lib:$LD_LIBRARY_PATH

Building

Using the Build Script (Recommended)

The scripts/build.sh script automates dependency checking, building, and installation:

# Basic release build
./scripts/build.sh

# Clean debug build
./scripts/build.sh --clean --type debug

# Build and install
./scripts/build.sh --install

# Build and install to custom location
./scripts/build.sh --install --prefix /opt/e2sar

Build Script Options

Option	Description	Default
`-h, --help`	Show help message	-
`-t, --type TYPE`	Build type: `debug`, `release`, `debugoptimized`	release
`-c, --clean`	Clean build directory before building	false
`-i, --install`	Install after building	false
`-p, --prefix PREFIX`	Installation prefix	/usr/local

The script automatically:

Checks for required dependencies (meson, ninja, pkg-config, E2SAR)
Reports the E2SAR library version found
Configures and builds the project
Tests the executable by running --help
Handles installation permissions (uses sudo only when necessary)

Manual Build

meson setup builddir
meson compile -C builddir

Build Types

# Debug build (with symbols, no optimization)
meson setup --buildtype=debug builddir

# Release build (optimized)
meson setup --buildtype=release builddir

# Debug with optimization
meson setup --buildtype=debugoptimized builddir

The build system automatically detects:

NUMA support
CPU affinity support
ET library availability (enables frame builder)

Check the build summary for enabled features:

Dependencies
  ET Library: Found (frame builder enabled)

Installation

The build system automatically determines the installation location:

Custom prefix: meson setup --prefix=/custom/path builddir
CODA environment: If $CODA is set, installs to $CODA/Linux-x86_64/bin
Default: Installs to ~/.local/bin (no root required)

meson install -C builddir

Usage

When built with ET library support, the frame builder is always active. At least one output mode (ET or file) must be specified.

Using the Example Run Script

The scripts/example_run.sh script provides a template for running the receiver. Edit the configuration variables at the top of the script:

# Configuration variables in example_run.sh
EJFAT_URI="ejfat://token@ctrl-plane:18347/lb/1?data=192.168.1.100:10000"
RECEIVER_IP="192.168.1.100"
RECEIVER_PORT="10000"
OUTPUT_DIR="/tmp/e2sar_frames"
FILE_PREFIX="frame"
FILE_EXTENSION=".bin"
NUM_THREADS="2"
REPORT_INTERVAL="5000"

Then run:

./scripts/example_run.sh

The script automatically:

Locates the executable (build directory or PATH)
Creates the output directory if needed
Displays the configuration before starting
Passes any additional arguments to the receiver

You can append additional options:

./scripts/example_run.sh --bufsize 33554432 --timeout 100

Note: The example script uses fallback mode (--output-dir). For frame builder mode, use --fb-output-dir and --et-file options directly.

File Output with EVIO-6 Aggregation

Writes aggregated frames to EVIO-6 files with 2GB auto-rollover:

coda_ejfat_fb \
  --uri 'ejfat://token@ctrl-plane:18347/lb/1?data=192.168.1.100:10000' \
  --ip 192.168.1.100 \
  --port 10000 \
  --fb-output-dir /data/frames \
  --fb-output-prefix run1234 \
  --fb-threads 4

ET Output with EVIO-6 Aggregation

Sends aggregated frames to ET system (attaches to GRAND_CENTRAL station):

# Start ET system first
et_start -f /tmp/et_sys_pagg -n 1000 -s 2000000

# Run receiver
coda_ejfat_fb \
  --uri 'ejfat://token@ctrl-plane:18347/lb/1?data=192.168.1.100:10000' \
  --ip 192.168.1.100 \
  --port 10000 \
  --et-file /tmp/et_sys_pagg \
  --fb-threads 4

Dual Output (ET + Files)

Send to ET for real-time processing and archive to files simultaneously:

coda_ejfat_fb \
  --uri 'ejfat://token@ctrl-plane:18347/lb/1?data=192.168.1.100:10000' \
  --ip 192.168.1.100 \
  --port 10000 \
  --et-file /tmp/et_sys_pagg \
  --fb-output-dir /data/backup \
  --fb-output-prefix backup \
  --fb-threads 4

Raw File Output (Fallback Mode)

When built without ET library, only raw file output is available:

coda_ejfat_fb \
  --uri 'ejfat://token@ctrl-plane:18347/lb/1?data=192.168.1.100:10000' \
  --ip 192.168.1.100 \
  --port 10000 \
  --output-dir /data/output \
  --prefix events

Command-Line Options

Required Parameters

Option	Description
`--uri, -u`	EJFAT URI for control plane connection
`--ip` or `--autoip`	IP address for receiving UDP (one required)

Network Parameters

Option	Description	Default
`--ip`	IP address for receiving UDP packets	-
`--port, -p`	Starting UDP port number	10000
`--autoip`	Auto-detect local host IP address	false

Frame Builder Parameters (when ET available)

Option	Description	Default
`--fb-output-dir`	Output directory for EVIO-6 files	-
`--fb-output-prefix`	File name prefix	frames
`--fb-threads`	Number of builder threads	1
`--et-file`	ET system file name	-
`--et-host`	ET host (empty=broadcast)	""
`--et-port`	ET port (0=default)	0
`--et-event-size`	ET event size in bytes	2MB
`--timestamp-slop`	Max timestamp difference (ticks)	100
`--frame-timeout`	Frame timeout (ms)	1000
`--expected-streams`	Expected number of data streams	1

Fallback File Output (without ET)

Option	Description	Default
`--output-dir, -o`	Output directory for raw frames	-
`--prefix`	Filename prefix	events
`--extension, -e`	File extension	.bin

Performance Parameters

Option	Description	Default
`--threads, -t`	Number of receiver threads	1
`--bufsize, -b`	Socket buffer size (bytes)	3MB
`--timeout`	Reassembly timeout (ms)	500
`--cores`	CPU cores for thread binding	-
`--numa`	NUMA node for memory binding	-1
`--report-interval`	Stats reporting interval (ms)	5000

Control Plane Parameters

Option	Description	Default
`--withcp, -c`	Enable control plane interactions	true
`--ipv6, -6`	Prefer IPv6 for control plane	false
`--novalidate, -v`	Don't validate TLS certificates	false

Output Formats

EVIO-6 File Output (with frame builder)

Files are named: {prefix}_thread{N}_file{NNNN}.evio

Example:

/data/frames/run1234_thread0_file0000.evio  (2GB, closed)
/data/frames/run1234_thread0_file0001.evio  (2GB, closed)
/data/frames/run1234_thread0_file0002.evio  (active)

Features:

EVIO-6 aggregated time frame bank format
Automatic 2GB rollover
Per-thread file sequences for parallel I/O

Raw File Output (fallback mode)

Single file: {prefix}{extension} (e.g., events.bin)

Performance Tuning

High Throughput

coda_ejfat_fb \
  --threads 4 \
  --fb-threads 8 \
  --bufsize 33554432 \
  --cores 0 1 2 3 4 5 6 7 8 9 10 11 \
  --numa 0 \
  ...

Low Latency

coda_ejfat_fb \
  --timeout 100 \
  --frame-timeout 500 \
  --fb-threads 2 \
  ...

Troubleshooting

Build Issues

E2SAR not found:

export PKG_CONFIG_PATH=/path/to/e2sar/lib/pkgconfig:$PKG_CONFIG_PATH

ET library not found:

export CPATH=/path/to/et/include:$CPATH
export LIBRARY_PATH=/path/to/et/lib:$LIBRARY_PATH
export LD_LIBRARY_PATH=/path/to/et/lib:$LD_LIBRARY_PATH

Protobuf/Abseil conflicts (RHEL9): The build system automatically searches /usr/local first. Ensure a clean rebuild:

rm -rf builddir && meson setup builddir && meson compile -C builddir

Runtime Issues

"Frame builder requires at least one output mode": Specify either --et-file or --fb-output-dir (or both).

ET connection fails:

et_monitor -f /tmp/et_sys_pagg  # Verify ET is running
telnet et-host 11111            # Test network connectivity

Permission denied on output:

mkdir -p /path/to/frames && chmod 755 /path/to/frames

Statistics

On shutdown (Ctrl+C), statistics are displayed:

======= Final Statistics =======
--- Data Frames (Reassembled from UDP) ---
    Data Frames: 10523
    Data Volume: 1024.50 MB
    Avg Frame Rate: 350.77 frames/sec
    Avg Data Rate: 34.15 MB/sec
--- Build Events (Aggregated/Written) ---
    Build Events: 2631
    Data Volume: 1024.50 MB
    Avg Event Rate: 87.70 events/sec
    Avg Data Rate: 34.15 MB/sec
--- Errors ---
    Write Errors: 0
    Receive Errors: 0
    Payload Validation Errors: 0
    Wrong Endianness Count: 0
--- Runtime ---
    Total Elapsed Time: 30.0 sec
================================

License

This program is provided as an example of using E2SAR libraries. License terms follow those of the E2SAR project.

Name		Name	Last commit message	Last commit date
Latest commit History 2 Commits
scripts		scripts
src		src
LICENSE		LICENSE
README.md		README.md
meson.build		meson.build
meson_options.txt		meson_options.txt

License

JeffersonLab/coda-ejfat-fb

Folders and files

Latest commit

History

Repository files navigation

CODA EJFAT-based Frame Builder

Features

Architecture

Prerequisites

System Requirements

Install Dependencies

E2SAR Library

ET Library

Building

Using the Build Script (Recommended)

Build Script Options

Manual Build

Build Types

Installation

Usage

Using the Example Run Script

File Output with EVIO-6 Aggregation

ET Output with EVIO-6 Aggregation

Dual Output (ET + Files)

Raw File Output (Fallback Mode)

Command-Line Options

Required Parameters

Network Parameters

Frame Builder Parameters (when ET available)

Fallback File Output (without ET)

Performance Parameters

Control Plane Parameters

Output Formats

EVIO-6 File Output (with frame builder)

Raw File Output (fallback mode)

Performance Tuning

High Throughput

Low Latency

Troubleshooting

Build Issues

Runtime Issues

Statistics

License

About

Resources

License

Code of conduct

Contributing

Security policy

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages