Skip to content

Commit b7ec47b

Browse files
calaverasoftpropsmildanieljfussmndeveci
authored
feat: Rust cargo lambda workflow (#350)
* feat: support rust via cargo * make black reformat * use os.path.join in path assertion tests to address windows paths * address pylint ci errors * fix reformatted list comprehensions * try passing rust-lld linker on windows * update rust cargo design doc. windows support was added and tested * add test for cargo workspaces project * reformat test sources * build with musl on linux because glibc may differ on lambda * update linux copy and bin paths * update make test again, use the version appveyor is complaining about * update integration tests for rust cargo with latest aws rust runtime interfaces * align TestCustomMakeWorkflow integ test assumptions with appveyor reality * make x86_64-unknown-linux-musl a const for the rust cargo workflow * add integ test for failing cargo rust build * Update Rust workflow to use cargo-lambda Cargo-lambda takes care of cross compiling using Zig as linker. This works on Windows, Linux, and MacOS natively. Signed-off-by: David Calavera <[email protected]> * Fix deprecation warnings. Signed-off-by: David Calavera <[email protected]> * Install Zig on Windows manually Signed-off-by: David Calavera <[email protected]> * Print Zig version on Windows Signed-off-by: David Calavera <[email protected]> * Update version of cargo-lambda Signed-off-by: David Calavera <[email protected]> * Run windows tests in powershell Signed-off-by: David Calavera <[email protected]> * Fix powershell env notation Signed-off-by: David Calavera <[email protected]> * Print clang version on windows Signed-off-by: David Calavera <[email protected]> * Fix env variable name Signed-off-by: David Calavera <[email protected]> * Try new version of zigbuild that fixes some linker issues on Windows. Signed-off-by: David Calavera <[email protected]> * Update releases URL. Signed-off-by: David Calavera <[email protected]> * Upgrade LLVM and clang Signed-off-by: David Calavera <[email protected]> * Update visual studio image To check if that makes any difference. Signed-off-by: David Calavera <[email protected]> * Change the visual studio image everywhere. Signed-off-by: David Calavera <[email protected]> * Revert upgrade changes They didn't fix the problem Signed-off-by: David Calavera <[email protected]> * Print environment Signed-off-by: David Calavera <[email protected]> * Update to Visual Studio 2022 Signed-off-by: David Calavera <[email protected]> * Fix python variable Signed-off-by: David Calavera <[email protected]> * Fix package urls Signed-off-by: David Calavera <[email protected]> * Update missing vs 2019 reference. Signed-off-by: David Calavera <[email protected]> * Change windows package suffix Signed-off-by: David Calavera <[email protected]> * Update cargo-lambda to version 0.9.0 Signed-off-by: David Calavera <[email protected]> * Go back to the original VS version. Signed-off-by: David Calavera <[email protected]> * Cleanup options - Use architecture to setup the right build target. - Don't require `--bin` flag, the default behaviour should work for the majority of functions. - Add flags option to provide a list of additional flags for projects that need extra configuration, like projects within a workspace. Signed-off-by: David Calavera <[email protected]> * Add integration test with cargo_lambda_flags Signed-off-by: David Calavera <[email protected]> * Detect binaries when the project only includes one function. Signed-off-by: David Calavera <[email protected]> * Bring back handler as an optional argument. It saves some duplicated flags for working with workspaces. Signed-off-by: David Calavera <[email protected]> * Ignore errors if directory doesn't exist. Signed-off-by: David Calavera <[email protected]> * Add debug logs Set RUST_LOG=debug when debug is enabled. Signed-off-by: David Calavera <[email protected]> * Log the artifact destination path. Signed-off-by: David Calavera <[email protected]> * Add missing comma Signed-off-by: David Calavera <[email protected]> * Print out and err in the log when debug is enabled Signed-off-by: David Calavera <[email protected]> * Only set RUST_LOG when it's not already set Log it's value, so users know what's set at. Signed-off-by: David Calavera <[email protected]> * Add experimentalCargoLambda feature flag. Signed-off-by: David Calavera <[email protected]> * Add integration test for multi-function projects. Signed-off-by: David Calavera <[email protected]> * Remove type hints They're causing false positives in Python 3.9 with pylint. Signed-off-by: David Calavera <[email protected]> * Add new CI steps for GHA Signed-off-by: David Calavera <[email protected]> * Add build_in_source_support Signed-off-by: David Calavera <[email protected]> * Test rust logger Signed-off-by: David Calavera <[email protected]> * Fix assertion Signed-off-by: David Calavera <[email protected]> * Don't fail fast Signed-off-by: David Calavera <[email protected]> * fix: Fix failing esbuild integration tests (#423) * fix: Fix failing esbuild integration tests * Replace npm ci with npm install * Remove default shell Signed-off-by: David Calavera <[email protected]> * Revert "Remove default shell" This reverts commit 478c3b6. * Add check to ensure that Cargo Lambda is installed. Include a link to the gettings started guide that gives direct installation instructions based on the platform. Signed-off-by: David Calavera <[email protected]> * feat: Add support for mjs files with esbuild (#427) * feat: Add support for mjs files with esbuild * Black reformat * Test Cargo Lambda check Signed-off-by: David Calavera <[email protected]> * Don't redefine which Signed-off-by: David Calavera <[email protected]> * Organize code in more modules This structure follows other workflows, and provides better testeability. Signed-off-by: David Calavera <[email protected]> * Add more documentation Signed-off-by: David Calavera <[email protected]> * Format code Signed-off-by: David Calavera <[email protected]> * Capture exception Signed-off-by: David Calavera <[email protected]> * chore: Remove type/bug label for Bug Issue Template (#425) Co-authored-by: Jacob Fuss <[email protected]> Co-authored-by: Mehmet Nuri Deveci <[email protected]> * Clean design doc and tests Signed-off-by: David Calavera <[email protected]> * chore: Version bump to 1.25.0 (#429) * refactor: assign each workflow a default build directory (#428) * fix: remove unused symlinking (#432) * chore: Move to ruff from pylint (#435) When we started the project we defaulted to use pylint. Pylint has served it's purpose but it pretty slow. Ruff is a newer linter in the python ecosystem but is written in Rust. This makes Ruff was faster than pylint. On my machine (while testing in SAM CLI), pylint took about 70s to lint the repo but with ruff it took .04s. Co-authored-by: Jacob Fuss <[email protected]> * chore: Enable pylint within ruff (#436) Co-authored-by: Jacob Fuss <[email protected]> * refactor: esbuild refactor for readability (#433) * feat: use build_dir in esbuild workflow to support building in source (#437) * Fix formatting Signed-off-by: David Calavera <[email protected]> * Update build in source settings Signed-off-by: David Calavera <[email protected]> * fix: remove python3.6 support (#434) * chore: bump version to 1.26.0 (#441) * Remove default value for subprocess_cargo_lambda Signed-off-by: David Calavera <[email protected]> * feat: Pin ruff version, add dependabot config (#442) * feat: Pin ruff version, add dependabot config to keep our dependencies up to date * Exlude isort and flake8 from dependabot updates * feat: Add sources content flag to supported esbuild options (#439) * Better process management Signed-off-by: David Calavera <[email protected]> * Make release mode the default Signed-off-by: David Calavera <[email protected]> * Remove already default None Signed-off-by: David Calavera <[email protected]> * Turn debug message into warning Signed-off-by: David Calavera <[email protected]> * Update documentation format Signed-off-by: David Calavera <[email protected]> * Fix formatting Signed-off-by: David Calavera <[email protected]> * Preserve binary permissions Use copy2 instead of copyfile Signed-off-by: David Calavera <[email protected]> --------- Signed-off-by: David Calavera <[email protected]> Co-authored-by: softprops <[email protected]> Co-authored-by: Daniel Mil <[email protected]> Co-authored-by: Jacob Fuss <[email protected]> Co-authored-by: Jacob Fuss <[email protected]> Co-authored-by: Mehmet Nuri Deveci <[email protected]> Co-authored-by: Ruperto Torres <[email protected]>
1 parent 98ed70b commit b7ec47b

38 files changed

+4013
-2
lines changed

.github/workflows/build.yml

Lines changed: 57 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ jobs:
2121
- python-integration
2222
- ruby-integration
2323
- dotnet-integration
24+
- rust-cargo-lambda-integration
2425
steps:
2526
- name: report-failure
2627
if: |
@@ -33,7 +34,8 @@ jobs:
3334
needs.custom-make-integration.result != 'success' ||
3435
needs.python-integration.result != 'success' ||
3536
needs.ruby-integration.result != 'success' ||
36-
needs.dotnet-integration.result != 'success'
37+
needs.dotnet-integration.result != 'success' ||
38+
needs.rust-cargo-lambda-integration.result != 'success'
3739
run: exit 1
3840
- name: report-success
3941
run: exit 0
@@ -296,3 +298,57 @@ jobs:
296298
python-version: ${{ matrix.python }}
297299
- run: make init
298300
- run: pytest -vv tests/integration/workflows/dotnet_clipackage
301+
302+
rust-cargo-lambda-integration:
303+
name: ${{ matrix.os }} / ${{ matrix.python }} / rust-cargo-lambda
304+
if: github.repository_owner == 'aws'
305+
runs-on: ${{ matrix.os }}
306+
env:
307+
CARGO_LAMBDA_VERSION: 0.15.0
308+
defaults:
309+
run:
310+
shell: bash
311+
strategy:
312+
fail-fast: false
313+
matrix:
314+
os:
315+
- ubuntu-latest
316+
- windows-latest
317+
python:
318+
- "3.9"
319+
- "3.8"
320+
- "3.7"
321+
rust:
322+
- stable
323+
steps:
324+
- uses: actions/checkout@v3
325+
- uses: actions/setup-python@v4
326+
with:
327+
python-version: ${{ matrix.python }}
328+
329+
# Install and configure Rust
330+
- name: Install rustup
331+
run: |
332+
: install rustup if needed
333+
if ! command -v rustup &> /dev/null ; then
334+
curl --proto '=https' --tlsv1.2 --retry 10 --retry-connrefused -fsSL "https://sh.rustup.rs" | sh -s -- --default-toolchain none -y
335+
echo "${CARGO_HOME:-$HOME/.cargo}/bin" >> $GITHUB_PATH
336+
fi
337+
if: ${{ matrix.os }} == 'ubuntu-latest'
338+
- name: rustup toolchain install ${{ matrix.rust }}
339+
run: rustup toolchain install ${{ matrix.rust }} --profile minimal --no-self-update
340+
- run: rustup default ${{ matrix.rust }}
341+
- run: |
342+
: disable incremental compilation
343+
echo CARGO_INCREMENTAL=0 >> $GITHUB_ENV
344+
- run: |
345+
: enable colors in Cargo output
346+
echo CARGO_TERM_COLOR=always >> $GITHUB_ENV
347+
348+
# Install and configure Cargo Lambda
349+
- name: Install Cargo Lambda
350+
run: pip install cargo-lambda==$CARGO_LAMBDA_VERSION
351+
- run: echo "$HOME/.local/bin" >> $GITHUB_PATH
352+
353+
- run: make init
354+
- run: pytest -vv tests/integration/workflows/rust_cargo

Makefile

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,10 +2,13 @@ init:
22
LAMBDA_BUILDERS_DEV=1 pip install -e '.[dev]'
33

44
test:
5-
# Run unit tests
5+
# Run unit and functional tests
66
# Fail if coverage falls below 94%
77
LAMBDA_BUILDERS_DEV=1 pytest -vv --cov aws_lambda_builders --cov-report term-missing --cov-fail-under 94 tests/unit tests/functional
88

9+
unit-test:
10+
LAMBDA_BUILDERS_DEV=1 pytest tests/unit
11+
912
func-test:
1013
LAMBDA_BUILDERS_DEV=1 pytest tests/functional
1114

README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ Lambda Builders currently contains the following workflows
1818
* Typescript with esbuild
1919
* Ruby with Bundler
2020
* Go with Mod
21+
* Rust with Cargo
2122

2223
In Addition to above workflows, AWS Lambda Builders also supports *Custom Workflows* through a Makefile.
2324

aws_lambda_builders/workflows/__init__.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,3 +11,4 @@
1111
import aws_lambda_builders.workflows.dotnet_clipackage
1212
import aws_lambda_builders.workflows.custom_make
1313
import aws_lambda_builders.workflows.nodejs_npm_esbuild
14+
import aws_lambda_builders.workflows.rust_cargo
Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
# Rust Cargo Builder
2+
3+
## Scope
4+
5+
This package enables the creation of a Lambda deployment package for Rust projects managed using the [cargo](https://doc.rust-lang.org/cargo/) build tool targeting Lambda's "provided" runtime. Rust support for the provided runtime is bundled as a compilation dependency of these projects, provided by the [lambda](https://github.com/awslabs/aws-lambda-rust-runtime) crate.
6+
7+
## Implementation
8+
9+
This package uses [Cargo Lambda](https://www.cargo-lambda.info) to do all the heavy lifting for cross compilation, target validation, and other executable optimizations.
10+
11+
It supports X86-64 architectures with the target `x86_64-unknown-linux-gnu` by default. It also supports ARM architectures with the target option `aarch64-unknown-linux-gnu`. Those are the only two valid targets. The target is automatically configured based on the `architecture` option in the `RustCargoLambdaWorkflow`.
12+
13+
The general algorithm for preparing a rust executable for use on AWS Lambda is as follows.
14+
15+
### Build
16+
17+
It builds a binary in the standard cargo target directory. The binary's name is always `bootstrap`, and it's always located under `target/lambda/HANDLER_NAME/bootstrap`.
18+
19+
### Copy and Rename executable
20+
21+
It then copies the executable to the target directory honoring the provided runtime's [expectation on executable names](https://docs.aws.amazon.com/lambda/latest/dg/runtimes-custom.html).
22+
23+
## Notes
24+
25+
Like the go builders, the workflow argument `options.artifact_executable_name`
26+
interface can used to provide a handler name that resolves to an executable. This
27+
enables sam support for cargo workspaces allowing for one rust project to have multiple lambdas. Cargo workspaces have a notion of a `package` and `bin`. A `package` can have
28+
multiple bins but typically `packages` have a 1-to-1 relationship with a default `bin`: `main.rs`. The handler names must be uniques across a Rust project, regardless of how many packages and binaries that project includes.
Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
"""
2+
Builds Rust Lambda functions using Cargo Lambda
3+
"""
4+
5+
from .workflow import RustCargoLambdaWorkflow
Lines changed: 146 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,146 @@
1+
"""
2+
Rust Cargo build actions
3+
"""
4+
5+
import logging
6+
import os
7+
8+
from aws_lambda_builders.workflow import BuildMode
9+
from aws_lambda_builders.actions import ActionFailedError, BaseAction, Purpose
10+
from aws_lambda_builders.architecture import X86_64, ARM64
11+
from .exceptions import CargoLambdaExecutionException
12+
from .utils import OSUtils
13+
14+
15+
LOG = logging.getLogger(__name__)
16+
17+
18+
class RustCargoLambdaBuildAction(BaseAction):
19+
NAME = "CargoLambdaBuild"
20+
DESCRIPTION = "Building the project using Cargo Lambda"
21+
PURPOSE = Purpose.COMPILE_SOURCE
22+
23+
def __init__(
24+
self,
25+
source_dir,
26+
binaries,
27+
mode,
28+
subprocess_cargo_lambda,
29+
architecture=X86_64,
30+
handler=None,
31+
flags=None,
32+
):
33+
"""
34+
Build the a Rust executable
35+
36+
:type source_dir: str
37+
:param source_dir:
38+
Path to a folder containing the source code
39+
40+
:type binaries: dict
41+
:param binaries:
42+
Resolved path dependencies
43+
44+
:type mode: str
45+
:param mode:
46+
Mode the build should produce
47+
48+
:type architecture: str, optional
49+
:param architecture:
50+
Target architecture to build the binary, either arm64 or x86_64
51+
52+
:type handler: str, optional
53+
:param handler:
54+
Handler name in `bin_name` format
55+
56+
:type flags: list, optional
57+
:param flags:
58+
Extra list of flags to pass to `cargo lambda build`
59+
60+
:type subprocess_cargo_lambda: aws_lambda_builders.workflows.rust_cargo.cargo_lambda.SubprocessCargoLambda
61+
:param subprocess_cargo_lambda: An instance of the Cargo Lambda process wrapper
62+
"""
63+
64+
self._source_dir = source_dir
65+
self._mode = mode
66+
self._binaries = binaries
67+
self._handler = handler
68+
self._flags = flags
69+
self._architecture = architecture
70+
self._subprocess_cargo_lambda = subprocess_cargo_lambda
71+
72+
def build_command(self):
73+
cmd = [self._binaries["cargo"].binary_path, "lambda", "build"]
74+
if self._mode != BuildMode.DEBUG:
75+
cmd.append("--release")
76+
if self._architecture == ARM64:
77+
cmd.append("--arm64")
78+
if self._handler:
79+
cmd.extend(["--bin", self._handler])
80+
if self._flags:
81+
cmd.extend(self._flags)
82+
83+
return cmd
84+
85+
def execute(self):
86+
try:
87+
return self._subprocess_cargo_lambda.run(command=self.build_command(), cwd=self._source_dir)
88+
except CargoLambdaExecutionException as ex:
89+
raise ActionFailedError(str(ex))
90+
91+
92+
class RustCopyAndRenameAction(BaseAction):
93+
NAME = "RustCopyAndRename"
94+
DESCRIPTION = "Copy Rust executable, renaming if needed"
95+
PURPOSE = Purpose.COPY_SOURCE
96+
97+
def __init__(self, source_dir, artifacts_dir, handler=None, osutils=OSUtils()):
98+
"""
99+
Copy and rename Rust executable
100+
101+
Parameters
102+
----------
103+
source_dir : str
104+
Path to a folder containing the source code
105+
106+
artifacts_dir : str
107+
Path to a folder containing the deployable artifacts
108+
109+
handler : str, optional
110+
Handler name in `package.bin_name` or `bin_name` format
111+
112+
osutils : aws_lambda_builders.workflows.rust_cargo.utils.OSUtils, optional
113+
Optional, External IO utils
114+
"""
115+
self._source_dir = source_dir
116+
self._handler = handler
117+
self._artifacts_dir = artifacts_dir
118+
self._osutils = osutils
119+
120+
def base_path(self):
121+
return os.path.join(self._source_dir, "target", "lambda")
122+
123+
def binary_path(self):
124+
base = self.base_path()
125+
if self._handler:
126+
binary_path = os.path.join(base, self._handler, "bootstrap")
127+
LOG.debug("copying function binary from %s", binary_path)
128+
return binary_path
129+
130+
output = os.listdir(base)
131+
if len(output) == 1:
132+
binary_path = os.path.join(base, output[0], "bootstrap")
133+
LOG.debug("copying function binary from %s", binary_path)
134+
return binary_path
135+
136+
LOG.warning("unexpected list of binary directories: [%s]", ", ".join(output))
137+
raise CargoLambdaExecutionException(
138+
message="unable to find function binary, use the option `artifact_executable_name` to specify the name"
139+
)
140+
141+
def execute(self):
142+
self._osutils.makedirs(self._artifacts_dir)
143+
binary_path = self.binary_path()
144+
destination_path = os.path.join(self._artifacts_dir, "bootstrap")
145+
LOG.debug("copying function binary from %s to %s", binary_path, destination_path)
146+
self._osutils.copyfile(binary_path, destination_path)

0 commit comments

Comments
 (0)