Skip to content

chore: add Trivy security scanning and fix non-root container users #3082

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tac0turtle wants to merge 12 commits into
base: main
Choose a base branch
from
Open

chore: add Trivy security scanning and fix non-root container users #3082

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
7a15f82
chore: add Trivy security scanning and fix non-root container users
tac0turtle Feb 18, 2026
f237083
docs: add breaking changelog entry for non-root Docker images
tac0turtle Feb 18, 2026
d4b15fc
docs: add migration guide for non-root Docker containers
tac0turtle Feb 18, 2026
62e68bd
Merge branch 'main' into marko/add-trivy-scan
tac0turtle Mar 4, 2026
8cb45b0
apply suggetions
auricom Mar 5, 2026
517f2f4
Merge branch 'main' into marko/add-trivy-scan
auricom Mar 5, 2026
5e5035d
Merge branch 'main' into marko/add-trivy-scan
alpe Mar 16, 2026
e48a5ef
Harden containers
alpe Mar 16, 2026
6dc069f
Build Docker containers from project root
alpe Mar 16, 2026
ecf19d7
Fix evm dockerfile
alpe Mar 16, 2026
cbb0185
Scan puhed images
alpe Mar 16, 2026
bdd2dd6
Review feedback
alpe Mar 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 12 additions & 2 deletions .github/workflows/docker-build-push.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
# This workflow builds and pushes Docker images to GHCR
name: Build Docker Images
permissions: {}
permissions: { }
"on":
workflow_call:
inputs:
Expand All @@ -12,7 +12,7 @@ permissions: {}
apps:
required: true
type: string
description: 'JSON array of apps to build (e.g., [{"name": "testapp", "dockerfile": "apps/testapp/Dockerfile"}])'
description: 'JSON array of apps (e.g., [{"name": "testapp", "dockerfile": "apps/testapp/Dockerfile"}])'

jobs:
build-images:
Expand Down Expand Up @@ -48,3 +48,13 @@ jobs:
push: true
platforms: linux/amd64,linux/arm64
tags: ghcr.io/${{ github.repository_owner }}/${{ matrix.app.name }}:${{ inputs.image-tag }}

- name: Scan pushed image with Trivy
uses: aquasecurity/trivy-action@57a97c7e7821a5776cebc9bb87c984fa69cba8f1 # v0.35.0
with:
scan-type: image
image-ref: ghcr.io/${{ github.repository_owner }}/${{ matrix.app.name }}:${{ inputs.image-tag }}
scanners: vuln,secret
severity: HIGH,CRITICAL
ignore-unfixed: true
exit-code: "1"
37 changes: 37 additions & 0 deletions .just/security.just
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# Security scanning with Trivy (https://trivy.dev)

trivy_image := "aquasec/trivy:latest"
Copy link
Contributor

@coderabbitai coderabbitai bot Mar 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🌐 Web query:

Trivy container image latest tag best practices CI/CD reproducibility

💡 Result:

Recommendation: don’t use :latest for Trivy in CI/CD if you care about reproducibility

latest (and other floating tags) can be retagged to different image contents over time, which breaks determinism. Docker explicitly recommends pinning by digest when you need fully repeatable builds/scans. [1]

Best practices for reproducible Trivy container-image scanning

  1. Pin the Trivy tool version (and ideally the image digest)

    • Prefer an explicit Trivy release version over latest.
    • If you use GitHub Actions, pin the action and the Trivy version it installs (the action supports version: latest or version: vX.Y.Z; don’t use latest for reproducibility). [2]
    • Avoid “canary” images/builds for production pipelines. [3]

  2. Control the vulnerability database lifecycle (this is a major source of “same image, different results”)

    • Trivy’s vulnerability DB is updated frequently (published every ~6 hours), so scan results can change even when the target image is unchanged. [4]
    • For reproducible runs, cache the DB between CI jobs and/or runs (Trivy explicitly calls out CI caching benefits). [4]
    • For “frozen” results (e.g., audits), run with --skip-db-update to prevent pulling a newer DB during the scan. [4]

  3. If you need strict repeatability across networks/regions, self-host or mirror the Trivy DB

    • Trivy distributes DBs as OCI artifacts (notably via GHCR), and its “air-gapped/offline” guidance is designed around controlling that dependency. [5]
    • In locked-down enterprise CI, mirror/self-host the DB and point Trivy at your controlled source (so the exact DB used is explicit and reviewable). [5]

  4. Separate “reproducible security gate” from “continuously updated security signal”

    • Common pattern:
      • PR gate: pinned Trivy version + cached/frozen DB (stable pass/fail).
      • Scheduled job (daily): same Trivy version (or intentionally updated), DB updates enabled to detect newly disclosed CVEs.

Sources: Docker pinning/digest guidance for determinism [1]; Trivy Action supports explicit Trivy version pinning [2]; Trivy warns about canary builds [3]; Trivy DB update frequency + skip-update + caching [4]; Trivy DB distribution via GHCR / offline control concepts [5].

🏁 Script executed:

# Examine the .just/security.just file to understand context
cat -n .just/security.just

Repository: evstack/ev-node

Length of output: 1566

🏁 Script executed:

# Check if there are other references to trivy_image or container images in the project
rg "trivy_image|docker|container|image" --type-list | head -20
fd -t f "*.just" -exec rg "trivy_image" {} +

Repository: evstack/ev-node

Length of output: 693

Pin Trivy version and consider database update strategy.

Line 3 uses :latest, which allows the Trivy vulnerability database to update automatically (~every 6 hours), causing identical image scans to produce different results. Pin an explicit Trivy version (and optionally the image digest) for reproducible security scans.

The project already caches the Trivy database via trivy_cache_volume, which helps. For further reproducibility, consider:

  • Pinning to a specific Trivy release (e.g., aquasec/trivy:0.48.0) or digest
  • Using --skip-db-update in PR/gate scans to freeze results; enable DB updates only in scheduled/continuous jobs
🔧 Suggested fix 
-trivy_image := "aquasec/trivy:latest"
+trivy_image := env("TRIVY_IMAGE", "aquasec/trivy:0.48.0")
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
trivy_image := "aquasec/trivy:latest"
trivy_image := env("TRIVY_IMAGE", "aquasec/trivy:0.48.0")
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In @.just/security.just at line 3, The Trivy image is pinned to "latest" via the
trivy_image variable which allows non-reproducible scans; change trivy_image to
a specific release tag or digest (e.g., aquasec/trivy:0.48.0 or the
image@sha256:...) to lock the tool version, and update any scan invocations
(where Trivy is run) to optionally include --skip-db-update for PR/gate scans
while reserving DB updates for scheduled jobs; update the trivy_image variable
and adjust the scan invocation logic that references trivy_image to implement
these changes.

trivy_severity := env("TRIVY_SEVERITY", "CRITICAL,HIGH")
trivy_cache_volume := "trivy-cache"
scan_images := env("SCAN_IMAGES", "evstack:local-dev")

trivy_run := "docker run --rm -v " + trivy_cache_volume + ":/root/.cache/ -e TRIVY_SEVERITY=" + trivy_severity

# Run all Trivy security scans (filesystem + Docker images)
trivy-scan: trivy-scan-fs trivy-scan-image

# Scan repo for dependency vulnerabilities, misconfigs, and secrets
trivy-scan-fs:
@echo "--> Scanning repository filesystem with Trivy"
@{{trivy_run}} \
-v {{justfile_directory()}}:/workspace \
{{trivy_image}} \
fs --scanners vuln,misconfig,secret \
--severity {{trivy_severity}} \
--exit-code 1 \
/workspace
@echo "--> Filesystem scan complete"

# Scan built Docker images for vulnerabilities
trivy-scan-image:
@echo "--> Scanning Docker images with Trivy"
@for img in {{scan_images}}; do \
echo "--> Scanning image: $img"; \
{{trivy_run}} \
-v /var/run/docker.sock:/var/run/docker.sock \
{{trivy_image}} \
image --severity {{trivy_severity}} \
--exit-code 1 \
$img; \
done
@echo "--> Image scan complete"
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Changes

- Store pending blocks separately from executed blocks key. [#3073](https://github.com/evstack/ev-node/pull/3073)
- **BREAKING:** Docker images for `evm`, `testapp`, and `local-da` now run as non-root user `ev-node` (uid 1000) instead of `root`. Existing volumes or bind mounts with root-owned files may require a `chown` to uid 1000. See the [migration guide](https://ev.xyz/guides/migrate-docker-nonroot). [`#3082`](https://github.com/evstack/ev-node/pull/3082)
- Fixes issues with force inclusion verification on sync nodes. [#3057](https://github.com/evstack/ev-node/pull/3057)
- Add flag to `local-da` to produce empty DA blocks (closer to the real
system). [#3057](https://github.com/evstack/ev-node/pull/3057)
Expand Down
30 changes: 18 additions & 12 deletions apps/evm/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,26 +1,32 @@
FROM golang:1.25-alpine AS build-env
FROM golang:1.26-bookworm AS build-env

WORKDIR /src

COPY core core

COPY go.mod go.sum ./
RUN go mod download
RUN --mount=type=cache,target=/go/pkg/mod \
go mod download

COPY . .

WORKDIR /src/apps/evm
RUN go mod tidy && CGO_ENABLED=0 GOOS=linux go build -o evm .
RUN --mount=type=cache,target=/go/pkg/mod \

Check failure on line 13 in apps/evm/Dockerfile

View workflow job for this annotation

GitHub Actions / lint / hadolint 

DL3003 warning: Use WORKDIR to switch to a directory
cd ./apps/evm && \
CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags "-s -w" -o /out/evm .

FROM alpine:3.22.2

#hadolint ignore=DL3018
RUN apk --no-cache add ca-certificates curl
FROM busybox:1.36.1-musl AS busybox

WORKDIR /root
FROM scratch

COPY --from=build-env /src/apps/evm/evm /usr/bin/evm
COPY apps/evm/entrypoint.sh /usr/bin/entrypoint.sh
RUN chmod +x /usr/bin/entrypoint.sh

ENTRYPOINT ["/usr/bin/entrypoint.sh"]
COPY --from=build-env /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=build-env /out/evm /usr/bin/evm
COPY --from=build-env /src/apps/evm/entrypoint.sh /usr/bin/entrypoint.sh
COPY --from=busybox /bin/busybox /bin/busybox

ENV HOME=/home/ev-node
WORKDIR /home/ev-node
USER 10001:10001

ENTRYPOINT ["/bin/busybox", "sh", "/usr/bin/entrypoint.sh"]
29 changes: 25 additions & 4 deletions apps/evm/docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ services:
volumes:
- ./chain:/root/chain:ro
- ./jwttoken:/root/jwt:ro
- reth:/home/reth/eth-home
- ev-reth-data:/home/reth/eth-home
entrypoint: /bin/sh -c
command:
- |
Expand All @@ -39,6 +39,13 @@ services:
--txpool.max-account-slots 2048 \
--txpool.max-new-txns 2048 \
--txpool.additional-validation-tasks 16 \
security_opt:
- no-new-privileges:true
cap_drop:
- ALL
read_only: true
tmpfs:
- /tmp
networks:
- evolve-network

Expand All @@ -47,6 +54,13 @@ services:
ports:
- "7980:7980"
command: ["-listen-all"]
security_opt:
- no-new-privileges:true
cap_drop:
- ALL
read_only: true
tmpfs:
- /tmp
networks:
- evolve-network

Expand All @@ -61,7 +75,7 @@ services:
- "7676:7676" # p2p
- "7331:7331" # rpc
volumes:
- evm-single-data:/root/.evm/
- ev-node-evm-data:/home/ev-node/.evm/
restart: always
entrypoint: /usr/bin/entrypoint.sh
command: start
Expand All @@ -75,12 +89,19 @@ services:
- DA_ADDRESS=http://local-da:7980 # http://localhost:26658 (Use if not using local-da)
# - DA_AUTH_TOKEN=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJBbGxvdyI6WyJwdWJsaWMiLCJyZWFkIiwid3JpdGUiXSwiTm9uY2UiOiJQcEswTmhyWi9IY05NWkVtUG9sSXNpRTRDcUpMdE9mbWtBMW0zMWFUaEswPSIsIkV4cGlyZXNBdCI6IjAwMDEtMDEtMDFUMDA6MDA6MDBaIn0.gaWh6tS6Rel1XFYclDkapNnZlaZVjrikCRNBxSDkCGk
# - DA_NAMESPACE=00000000000000000000000000000000000000000008e5f679bf7116c1
security_opt:
- no-new-privileges:true
cap_drop:
- ALL
read_only: true
tmpfs:
- /tmp
networks:
- evolve-network

volumes:
evm-single-data:
reth:
ev-node-evm-data:
ev-reth-data:

networks:
evolve-network:
Expand Down
53 changes: 13 additions & 40 deletions apps/grpc/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,61 +1,34 @@
# Build stage
FROM golang:1.25-alpine AS builder
FROM golang:1.26-bookworm AS builder

#hadolint ignore=DL3018
RUN apk add --no-cache git gcc musl-dev linux-headers

# Set working directory
WORKDIR /ev-node

# Copy go mod files
COPY go.mod go.sum ./
COPY apps/grpc/go.mod apps/grpc/go.sum ./apps/grpc/
COPY core/go.mod core/go.sum ./core/
COPY execution/grpc/go.mod execution/grpc/go.sum ./execution/grpc/

# Download dependencies
RUN go mod download
RUN --mount=type=cache,target=/go/pkg/mod \
go mod download

# Copy source code
COPY . .

# Build the application
WORKDIR /ev-node/apps/grpc
RUN go build -o evgrpc .

# Runtime stage
FROM alpine:3.22.2

#hadolint ignore=DL3018
RUN apk add --no-cache ca-certificates curl
RUN mkdir -p /home/ev-node
RUN --mount=type=cache,target=/go/pkg/mod \

Check failure on line 16 in apps/grpc/Dockerfile

View workflow job for this annotation

GitHub Actions / lint / hadolint 

DL3003 warning: Use WORKDIR to switch to a directory
cd ./apps/grpc && \
CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags "-s -w" -o /out/evgrpc .

# Create non-root user
RUN addgroup -g 1000 ev-node && \
adduser -u 1000 -G ev-node -s /bin/sh -D ev-node
FROM scratch

# Set working directory
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=builder --chown=10001:10001 /home/ev-node /home/ev-node
WORKDIR /home/ev-node

# Copy binary from builder
COPY --from=builder /ev-node/apps/grpc/evgrpc /usr/local/bin/
COPY --from=builder /out/evgrpc /usr/local/bin/evgrpc
USER 10001:10001

# Create necessary directories
RUN mkdir -p /home/ev-node/.evgrpc && \
chown -R ev-node:ev-node /home/ev-node

# Switch to non-root user
USER ev-node

# Expose ports
# P2P port
EXPOSE 26656
# RPC port
EXPOSE 26657
# Prometheus metrics
EXPOSE 26660

# Set entrypoint
ENTRYPOINT ["evgrpc"]

# Default command
ENTRYPOINT ["/usr/local/bin/evgrpc"]
CMD ["start"]
29 changes: 19 additions & 10 deletions apps/grpc/docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,17 +4,19 @@ services:
# Local DA service for development
local-da:
build:
context: ../../../
context: ../../
dockerfile: tools/local-da/Dockerfile
ports:
- "7980:7980"
environment:
- DA_NAMESPACE=00000000000000000000000000000000000000000000000000000000deadbeef
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:7980/health"]
interval: 5s
timeout: 3s
retries: 5
security_opt:
- no-new-privileges:true
cap_drop:
- ALL
read_only: true
tmpfs:
- /tmp

# Example gRPC execution service (replace with your implementation)
# execution-service:
Expand All @@ -32,11 +34,11 @@ services:
# Evolve node with gRPC execution client
evolve-grpc:
build:
context: ../../../
context: ../../
dockerfile: apps/grpc/Dockerfile
depends_on:
local-da:
condition: service_healthy
condition: service_started
Comment on lines 39 to +41
Copy link
Contributor

@coderabbitai coderabbitai bot Mar 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Removing healthcheck may cause startup race conditions.

Changing from service_healthy to service_started means evolve-grpc will start as soon as the local-da container starts, not when it's actually ready to accept connections on port 7980. This could cause transient connection failures on startup.

Consider adding a lightweight healthcheck back to local-da, or ensure evolve-grpc has retry logic for connecting to the DA service.

💡 Suggested healthcheck for local-da 
     tmpfs:
       - /tmp
+    healthcheck:
+      test: ["CMD-SHELL", "nc -z localhost 7980 || exit 1"]
+      interval: 5s
+      timeout: 3s
+      retries: 5

Then revert the depends_on condition:

     depends_on:
       local-da:
-        condition: service_started
+        condition: service_healthy
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@apps/grpc/docker-compose.yml` around lines 39 - 41, The change replaced the
depends_on condition for local-da from service_healthy to service_started, which
can cause evolve-grpc to start before local-da is accepting connections on port
7980; restore a proper readiness check by adding a lightweight Docker
healthcheck to the local-da service that probes the DA port (e.g., curl/tcp
check on 7980) and then revert the depends_on entry for evolve-grpc back to
condition: service_healthy so Docker waits for readiness, or alternatively
implement retry/backoff connection logic in evolve-grpc's startup (where it
connects to the DA service) to retry until the DA responds.

# execution-service:
# condition: service_healthy
ports:
Expand All @@ -48,17 +50,24 @@ services:
- DA_NAMESPACE=00000000000000000000000000000000000000000000000000deadbeef
- GRPC_EXECUTOR_URL=http://host.docker.internal:50051 # Change to your execution service
volumes:
- evolve-data:/home/evolve/.evgrpc
- evolve-data:/home/ev-node/.evgrpc
command:
- start
- --root-dir=/home/evolve/.evgrpc
- --root-dir=/home/ev-node/.evgrpc
- --da.address=${DA_ADDRESS:-http://local-da:7980}
- --da.namespace=${DA_NAMESPACE}
- --grpc-executor-url=${GRPC_EXECUTOR_URL:-http://host.docker.internal:50051}
- --p2p.listen-address=/ip4/0.0.0.0/tcp/26656
- --rpc.laddr=tcp://0.0.0.0:26657
- --instrumentation.prometheus=true
- --instrumentation.prometheus-listen-addr=0.0.0.0:26660
security_opt:
- no-new-privileges:true
cap_drop:
- ALL
read_only: true
tmpfs:
- /tmp

volumes:
evolve-data:
41 changes: 17 additions & 24 deletions apps/testapp/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,34 +1,27 @@
FROM golang:1.25 AS base

#hadolint ignore=DL3018
RUN apt-get update && \
apt-get install -y --no-install-recommends \
build-essential \
ca-certificates \
curl \
&& rm -rf /var/lib/apt/lists/*

# enable faster module downloading.
ENV GOPROXY=https://proxy.golang.org

## builder stage.
#
FROM base AS builder
FROM golang:1.26-bookworm AS builder

WORKDIR /ev-node

# Copy all source code first
COPY go.mod go.sum ./
COPY apps/testapp/go.mod apps/testapp/go.sum ./apps/testapp/
COPY core/go.mod core/go.sum ./core/
RUN --mount=type=cache,target=/go/pkg/mod \
go mod download

COPY . .
RUN mkdir -p /home/ev-node
RUN --mount=type=cache,target=/go/pkg/mod \

Check failure on line 13 in apps/testapp/Dockerfile

View workflow job for this annotation

GitHub Actions / lint / hadolint 

DL3003 warning: Use WORKDIR to switch to a directory
cd ./apps/testapp && \
CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags "-s -w" -o /out/testapp .

# Now download dependencies and build
RUN go mod download && cd apps/testapp && go install .
FROM scratch

## prep the final image.
#
FROM base
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=builder --chown=10001:10001 /home/ev-node /home/ev-node
COPY --from=builder /out/testapp /usr/bin/testapp

COPY --from=builder /go/bin/testapp /usr/bin
WORKDIR /home/ev-node

WORKDIR /apps
USER 10001:10001

ENTRYPOINT ["testapp"]
Binary file added apps/testapp/out/testapp
View file
Open in desktop
Binary file not shown.
4 changes: 4 additions & 0 deletions docs/.vitepress/config.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -309,6 +309,10 @@ function sidebarHome() {
text: "Migrating to ev-abci",
link: "/guides/migrating-to-ev-abci",
},
{
text: "Migrate Docker to non-root",
link: "/guides/migrate-docker-nonroot",
},
{
text: "Create genesis for your chain",
link: "/guides/create-genesis",
Expand Down
Loading
Loading