Merge branch 'main' into nix_junit

Author: dougch

Diff for: .github/PULL_REQUEST_TEMPLATE.md

@@ -1,17 +1,19 @@
 ### Resolved issues:
 
- resolves #ISSUE-NUMBER1, resolves #ISSUE-NUMBER2, etc.
+Resolves #ISSUE-NUMBER1, resolves #ISSUE-NUMBER2, etc.
 
 ### Description of changes: 
 
-Describe s2n’s current behavior and how your code changes that behavior. If there are no issues this pr is resolving, explain why this change is necessary.
+Describe s2n’s current behavior and how your code changes that behavior. If there are no issues this PR is resolving, explain why this change is necessary.
+
 ### Call-outs:
 
 Address any potentially confusing code. Is there code added that needs to be cleaned up later? Is there code that is missing because it’s still in development? 
+
 ### Testing:
 
- How is this change tested (unit tests, fuzz tests, etc.)? Are there any testing steps to be verified by the reviewer?
+How is this change tested (unit tests, fuzz tests, etc.)? Are there any testing steps to be verified by the reviewer?
 
- Is this a refactor change? If so, how have you proved that the intended behavior hasn't changed?
+Is this a refactor change? If so, how have you proved that the intended behavior hasn't changed?
 
 By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Diff for: .github/workflows/ci_compliance.yml

@@ -1,7 +1,8 @@
 ---
 name: Compliance
-
 on:
+  push:
+    branches: [main]
   pull_request:
     branches: [main]
   merge_group:
@@ -12,13 +13,18 @@ jobs:
   duvet:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v3
-      - uses: actions/checkout@v3
+      - name: Clone s2n-tls
+        uses: actions/checkout@v3
+
+      - name: Clone s2n-quic
+        uses: actions/checkout@v3
         with:
           repository: aws/s2n-quic
           path: ./s2n-quic
           submodules: true
-      - uses: ./s2n-quic/.github/actions/duvet
+
+      - name: Run duvet action
+        uses: ./s2n-quic/.github/actions/duvet
         with:
           s2n-quic-dir: ./s2n-quic
           report-script: compliance/generate_report.sh
@@ -27,3 +33,39 @@ jobs:
           aws-s3-bucket-name: s2n-tls-ci-artifacts
           aws-s3-region: us-west-2
           cdn: https://d3fqnyekunr9xg.cloudfront.net
+
+      # The `duvet report` command generates some artifacts (specs folder) that
+      # interfere with detecting uncommitted files. This step cleans up those
+      # artifacts. Since the cleanup runs prior to the “Extract RFC spec data”
+      # phase, this is a safe operation.
+      - name: Cleanup intermediate artifacts
+        run: rm -r specs
+        shell: bash
+
+      - name: Extract RFC spec data
+        working-directory: ./compliance
+        run: ./initialize_duvet.sh
+        shell: bash
+
+      - name: Check if there are uncommitted changes
+        run: |
+          # If this fails you need to run `cd compliance && ./compliance/initialize_duvet.sh`
+          #
+          # FIXME: https://github.com/aws/s2n-tls/issues/4219
+          # We generate and commit the spec files to avoid re-downloading them each time in
+          # the CI (avoid flaky network calls). However, this currently doesn't work in
+          # s2n-tls since duvet assumes that the specs folder live in the project's base
+          # folder.
+          #
+          # Use 'git status --porcelain' instead of 'git diff --exit-code' since git diff
+          # only detects diffs but fails to detect new files. Ignore the s2n-quic dir
+          # `(:!s2n-quic)` since we explicitly clone the repo as part of this job.
+          git_status=$(git status --porcelain -- ':!s2n-quic')
+          if [ -n "$git_status" ]; then
+            echo "Found uncommitted changes:"
+            echo "$git_status"
+            exit 1
+          else
+            echo "Workspace is clean"
+          fi
+        shell: bash

Diff for: .github/workflows/ci_rust.yml

@@ -37,7 +37,7 @@ jobs:
 
       - name: Tests
         working-directory: ${{env.ROOT_PATH}}
-        run: cargo test
+        run: cargo test --all-features
 
       - name: Test external build
         # if this test is failing, make sure that api headers are appropriately
@@ -59,6 +59,53 @@ jobs:
           ./generate.sh
           ldd target/debug/integration | grep libs2n.so
 
+  generate-openssl-102:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+
+      - uses: actions-rs/toolchain@v1
+        id: toolchain
+        with:
+          toolchain: stable
+          override: true
+
+      - uses: camshaft/rust-cache@v1
+
+      - name: Cache OpenSSL 1.0.2
+        id: cache-openssl
+        uses: actions/cache@v3
+        with:
+          path: ~/openssl-102/install
+          key: ${{ runner.os }}-openssl-102
+
+      - if: ${{ steps.cache-openssl.outputs.cache-hit != 'true' }}
+        name: Install OpenSSL 1.0.2
+        run: |
+          mkdir ~/openssl-102
+          pushd ~/openssl-102
+
+          mkdir install
+          install_dir="$(pwd)"/install
+
+          wget https://www.openssl.org/source/old/1.0.2/openssl-1.0.2u.tar.gz
+          tar -xzvf openssl-1.0.2u.tar.gz
+
+          pushd openssl-1.0.2u
+          ./config --prefix="${install_dir}" --openssldir="${install_dir}"/openssl
+          make
+          make install
+          popd
+
+          popd
+
+      - name: Generate
+        run: OPENSSL_DIR=~/openssl-102/install ${{env.ROOT_PATH}}/generate.sh
+
+      - name: Tests
+        working-directory: ${{env.ROOT_PATH}}
+        run: cargo test --all-features
+
   rustfmt:
     runs-on: ubuntu-latest
     steps:

Diff for: .github/workflows/proof_ci.yaml

@@ -62,7 +62,7 @@ jobs:
         run: |
           # Search within 5 most recent releases for latest available package
           CBMC_REL="https://api.github.com/repos/diffblue/cbmc/releases?page=1&per_page=5"
-          CBMC_DEB=$(curl -s $CBMC_REL | jq -r '.[].assets[].browser_download_url' | grep -e 'ubuntu-20.04' | head -n 1)
+          CBMC_DEB=$(curl -s $CBMC_REL --header 'authorization: Bearer ${{ secrets.GITHUB_TOKEN }}' | jq -r '.[].assets[].browser_download_url' | grep -e 'ubuntu-20.04' | head -n 1)
           CBMC_ARTIFACT_NAME=$(basename $CBMC_DEB)
           curl -o $CBMC_ARTIFACT_NAME -L $CBMC_DEB
           sudo dpkg -i $CBMC_ARTIFACT_NAME
@@ -80,7 +80,7 @@ jobs:
         shell: bash
         run: |
           CBMC_VIEWER_REL="https://api.github.com/repos/model-checking/cbmc-viewer/releases/latest"
-          CBMC_VIEWER_VERSION=$(curl -s $CBMC_VIEWER_REL | jq -r .name | sed  's/viewer-//')
+          CBMC_VIEWER_VERSION=$(curl -s $CBMC_VIEWER_REL --header 'authorization: Bearer ${{ secrets.GITHUB_TOKEN }}' | jq -r .name | sed  's/viewer-//')
           pip3 install cbmc-viewer==$CBMC_VIEWER_VERSION
       - name: Install CBMC viewer ${{ env.CBMC_VIEWER_VERSION }}
         if: ${{ env.CBMC_VIEWER_VERSION != 'latest' }}
@@ -96,7 +96,7 @@ jobs:
         run: |
           # Search within 5 most recent releases for latest available package
           LITANI_REL="https://api.github.com/repos/awslabs/aws-build-accumulator/releases?page=1&per_page=5"
-          LITANI_DEB=$(curl -s $LITANI_REL | jq -r '.[].assets[0].browser_download_url' | head -n 1)
+          LITANI_DEB=$(curl -s $LITANI_REL --header 'authorization: Bearer ${{ secrets.GITHUB_TOKEN }}' | jq -r '.[].assets[0].browser_download_url' | head -n 1)
           DBN_PKG_FILENAME=$(basename $LITANI_DEB)
           curl -L $LITANI_DEB -o $DBN_PKG_FILENAME
           sudo apt-get update
@@ -118,7 +118,7 @@ jobs:
           if ${{ env.KISSAT_TAG == 'latest' }}
           then
             KISSAT_REL="https://api.github.com/repos/arminbiere/kissat/releases/latest"
-            KISSAT_TAG_NAME=$(curl -s $KISSAT_REL | jq -r '.tag_name')
+            KISSAT_TAG_NAME=$(curl -s $KISSAT_REL --header 'authorization: Bearer ${{ secrets.GITHUB_TOKEN }}' | jq -r '.tag_name')
           else
             KISSAT_TAG_NAME=${{ env.KISSAT_TAG }}
           fi
@@ -137,7 +137,7 @@ jobs:
           if ${{ env.CADICAL_TAG == 'latest' }}
           then
             CADICAL_REL="https://api.github.com/repos/arminbiere/cadical/releases/latest"
-            CADICAL_TAG_NAME=$(curl -s $CADICAL_REL | jq -r '.tag_name')
+            CADICAL_TAG_NAME=$(curl -s $CADICAL_REL --header 'authorization: Bearer ${{ secrets.GITHUB_TOKEN }}' | jq -r '.tag_name')
           else
             CADICAL_TAG_NAME=${{ env.CADICAL_TAG }}
           fi

Diff for: CMakeLists.txt

@@ -38,11 +38,8 @@ option(COVERAGE "Enable profiling collection for code coverage calculation" OFF)
 option(S2N_INTEG_TESTS "Enable the integrationv2 tests" OFF)
 option(S2N_FAST_INTEG_TESTS "Enable the integrationv2 with more parallelism, only has effect if S2N_INTEG_TESTS=ON" OFF)
 option(S2N_INSTALL_S2NC_S2ND "Install the binaries s2nc and s2nd" OFF)
-option(EXPERIMENTAL_TREAT_WARNINGS_AS_ERRORS "Additional compiler warnings are treated as errors. Warnings may
-indicate danger points where you should verify with the S2N-TLS developers that the security of
-the library is not compromised. These warnings are currently failing for some builds; once the problems are fixed,
-they will be moved to UNSAFE_TREAT_WARNINGS_AS_ERRORS." OFF)
 option(TSAN "Enable ThreadSanitizer to test thread safety" OFF)
+option(ASAN "Enable AddressSanitizer to test memory safety" OFF)
 
 # Turn BUILD_TESTING=ON by default
 include(CTest)
@@ -164,13 +161,9 @@ set(CMAKE_C_FLAGS_DEBUGOPT "")
 
 target_compile_options(${PROJECT_NAME} PRIVATE -pedantic -std=gnu99 -Wall -Wimplicit -Wunused -Wcomment -Wchar-subscripts
         -Wuninitialized -Wshadow -Wcast-align -Wwrite-strings -Wno-deprecated-declarations -Wno-unknown-pragmas -Wformat-security
-        -Wno-missing-braces -Wno-strict-prototypes -Wa,--noexecstack
+        -Wno-missing-braces -Wsign-compare -Wno-strict-prototypes -Wa,--noexecstack
 )
 
-if (EXPERIMENTAL_TREAT_WARNINGS_AS_ERRORS)
-    target_compile_options(${PROJECT_NAME} PRIVATE -Wsign-compare )
-endif()
-
 if (UNSAFE_TREAT_WARNINGS_AS_ERRORS)
     target_compile_options(${PROJECT_NAME} PRIVATE -Werror )
 endif ()
@@ -222,10 +215,20 @@ if(S2N_UNSAFE_FUZZING_MODE)
 endif()
 
 if(TSAN)
-    target_compile_options(${PROJECT_NAME} PUBLIC -fsanitize=thread)
+    target_compile_options(${PROJECT_NAME} PUBLIC -fsanitize=thread -DS2N_THREAD_SANITIZER=1)
     target_link_options(${PROJECT_NAME} PUBLIC -fsanitize=thread)
 endif()
 
+if(ASAN)
+    target_compile_options(${PROJECT_NAME} PUBLIC -fsanitize=address -DS2N_ADDRESS_SANITIZER=1)
+    target_link_options(${PROJECT_NAME} PUBLIC -fsanitize=address)
+endif()
+
+if(TSAN OR ASAN)
+    # no-omit-frame-pointer and no-optimize-sibling-calls provide better stack traces
+    target_compile_options(${PROJECT_NAME} PUBLIC -fno-omit-frame-pointer -fno-optimize-sibling-calls)
+endif()
+
 list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/cmake/modules")
 
 if (NOT $ENV{S2N_LIBCRYPTO} MATCHES "awslc")
@@ -313,6 +316,10 @@ endfunction()
 
 # Tries to compile a feature probe and initializes the corresponding flags
 function(feature_probe PROBE_NAME)
+    # Load the global probe flags
+    file(READ "${CMAKE_CURRENT_LIST_DIR}/tests/features/GLOBAL.flags" GLOBAL_FILE)
+    string(REPLACE "\n" "" GLOBAL_FLAGS "${GLOBAL_FILE}")
+
     # Load the probe's flags
     file(READ "${CMAKE_CURRENT_LIST_DIR}/tests/features/${PROBE_NAME}.flags" PROBE_FILE)
     string(REPLACE "\n" "" PROBE_FLAGS "${PROBE_FILE}")
@@ -324,9 +331,12 @@ function(feature_probe PROBE_NAME)
         SOURCES "${CMAKE_CURRENT_LIST_DIR}/tests/features/${PROBE_NAME}.c"
         LINK_LIBRARIES ${LINK_LIB} ${OS_LIBS}
         CMAKE_FLAGS ${ADDITIONAL_FLAGS}
-        COMPILE_DEFINITIONS -c ${PROBE_FLAGS}
+        COMPILE_DEFINITIONS -c ${GLOBAL_FLAGS} ${PROBE_FLAGS}
         ${ARGN}
+        OUTPUT_VARIABLE TRY_COMPILE_OUTPUT
     )
+    # Uncomment the line below to get the output of the try_compile command
+    #message(STATUS "Output of try_compile: ${TRY_COMPILE_OUTPUT}")
 
     # Set the result of the probe
     feature_probe_result(${PROBE_NAME} ${IS_AVAILABLE})
@@ -504,13 +514,30 @@ if (BUILD_TESTING)
     add_library(allocator_overrides SHARED ${TEST_LD_PRELOAD})
 
     set(UNIT_TEST_ENVS S2N_DONT_MLOCK=1)
+    if (TSAN OR ASAN)
+        set(UNIT_TEST_ENVS ${UNIT_TEST_ENVS} S2N_ADDRESS_SANITIZER=1)
+    endif()
     if(TSAN)
         set(TSAN_SUPPRESSIONS_FILE ${CMAKE_SOURCE_DIR}/tests/.tsan_suppressions)
         if(NOT EXISTS ${TSAN_SUPPRESSIONS_FILE})
             message(FATAL_ERROR "TSAN suppression file ${TSAN_SUPPRESSIONS_FILE} missing")
         endif()
-        set(UNIT_TEST_ENVS ${UNIT_TEST_ENVS} S2N_ADDRESS_SANITIZER=1)
-        set(UNIT_TEST_ENVS ${UNIT_TEST_ENVS} TSAN_OPTIONS=suppressions=${TSAN_SUPPRESSIONS_FILE})
+        set(TSAN_OPTIONS suppressions=${TSAN_SUPPRESSIONS_FILE})
+        if(DEFINED ENV{TSAN_OPTIONS})
+            set(TSAN_OPTIONS "${TSAN_OPTIONS} $ENV{TSAN_OPTIONS}")
+        endif()
+        set(UNIT_TEST_ENVS ${UNIT_TEST_ENVS} TSAN_OPTIONS=${TSAN_OPTIONS})
+    endif()
+    if(ASAN)
+        # "detect_odr_violation" detects violations of the "one definition rule",
+        # ensuring that symbols are only defined once.
+        # But some of our unit tests intentionally include *.c files for testing,
+        # resulting in duplicate global values.
+        set(ASAN_OPTIONS detect_odr_violation=0)
+        if(DEFINED ENV{ASAN_OPTIONS})
+            set(ASAN_OPTIONS "${ASAN_OPTIONS} $ENV{ASAN_OPTIONS}")
+        endif()
+        set(UNIT_TEST_ENVS ${UNIT_TEST_ENVS} ASAN_OPTIONS=${ASAN_OPTIONS})
     endif()
     message(STATUS "Running tests with environment: ${UNIT_TEST_ENVS}")
 
@@ -589,6 +616,12 @@ if (BUILD_TESTING)
                 # For Nix and environments where LD_LIBRARY_PATH is already correct.
                 # We're also dropping tox and calling pytest directly, because
                 # Nix is already handling all of the python setup.
+                if (CMAKE_SYSTEM_PROCESSOR STREQUAL "aarch64" AND ${test_target} STREQUAL "integrationv2_sslyze" )
+                  # sslyze/nassl is not available on aarch64.
+                  message(WARNING "Skipping ${test_target} due to missing tools on ${CMAKE_SYSTEM_PROCESSOR}")
+                  continue()
+                endif()
+                message(STATUS "Adding integ test ${test_target}")
                 add_test(NAME ${test_target}
                         COMMAND
                         pytest

Diff for: README.md

@@ -14,55 +14,30 @@ s2n-tls is a C99 implementation of the TLS/SSL protocols that is designed to be
 [![Join the chat at https://gitter.im/awslabs/s2n](https://badges.gitter.im/awslabs/s2n.svg)](https://gitter.im/awslabs/s2n?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
 
 ## Quickstart for Ubuntu
-1. Fork s2n-tls on GitHub
-2. Run the following commands on Ubuntu.
-```
-git clone https://github.com/${YOUR_GITHUB_ACCOUNT_NAME}/s2n-tls.git
-cd s2n-tls
-
-# Pick an "env" line from the codebuild/codebuild.config file and run it, in this case choose the openssl-1.1.1 with GCC 9 build
-S2N_LIBCRYPTO=openssl-1.1.1 BUILD_S2N=true TESTS=integrationv2 GCC_VERSION=9
-
-sudo codebuild/bin/s2n_install_test_dependencies.sh
-codebuild/bin/s2n_codebuild.sh
-```
-
-## Quickstart for OSX (or other platforms)
-
-If you are building on OSX, or simply don't want to execute the entire build script above, you can use build tools like Ninja.
 
-### OSX
-
-An example of building on OSX:
-
-```sh
-# Install required dependencies using homebrew
-brew install ninja cmake coreutils [email protected]
-
-# Clone the s2n-tls source repository into the `s2n-tls` directory
-git clone https://github.com/${YOUR_GITHUB_ACCOUNT_NAME}/s2n-tls.git
+```bash
+# clone s2n-tls
+git clone https://github.com/aws/s2n-tls.git
 cd s2n-tls
 
-# Create a build directory, and build s2n-tls with debug symbols and a specific OpenSSL version.
-cmake . -Bbuild -GNinja \
-    -DCMAKE_BUILD_TYPE=Debug \
-    -DCMAKE_PREFIX_PATH=$(dirname $(dirname $(brew list [email protected]|grep libcrypto.dylib)))
-cmake --build ./build -j $(nproc)
-CTEST_PARALLEL_LEVEL=$(nproc) ninja -C build test
-```
+# install build dependencies
+sudo apt update
+sudo apt install cmake
 
-### Amazonlinux2
+# install a libcrypto
+sudo apt install libssl-dev
 
-Install dependencies with `./codebuild/bin/install_al2_dependencies.sh` after cloning.
-
-```sh
-git clone https://github.com/${YOUR_GITHUB_ACCOUNT_NAME}/s2n-tls.git
-cd s2n-tls
-cmake . -Bbuild -DCMAKE_EXE_LINKER_FLAGS="-lcrypto -lz" -DCMAKE_EXPORT_COMPILE_COMMANDS=ON
-cmake --build ./build -j $(nproc)
-CTEST_PARALLEL_LEVEL=$(nproc) make -C build test
+# build s2n-tls
+cmake . -Bbuild \
+    -DCMAKE_BUILD_TYPE=Release \
+    -DCMAKE_INSTALL_PREFIX=./s2n-tls-install
+cmake --build build -j $(nproc)
+CTEST_PARALLEL_LEVEL=$(nproc) ctest --test-dir build
+cmake --install build
 ```
 
+See the [s2n-tls build documentation](docs/BUILD.md) for further guidance on building s2n-tls for your platform.
+
 ## Have a Question?
 If you have any questions about Submitting PR's, Opening Issues, s2n-tls API usage, or something similar, we have a public chatroom available here to answer your questions: https://gitter.im/awslabs/s2n