Skip to content

Building on linux (Ubuntu)... getting permission problems (due to Docker running as root?) #449

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mflint opened this issue Jan 2, 2025 · 18 comments
Assignees
Labels
kind/bug Feature doesn't work as expected. platform/linux Linux platform specific issues. semver/none No version bump required. size/S Small task. (A couple of hours of work.) status/triage Collecting information required to triage the issue.

Comments

@mflint
Copy link

mflint commented Jan 2, 2025

Expected behavior

I'd like to be able to build the lambdas for Arm using a Linux host, because a buying a chonky Raspberry Pi is much cheaper than a new MacBook with Apple Silicon:

swift package archive --allow-network-connections docker

Actual behavior

The archiving stage fails with a permission error:

  {lots of build output deleted}

  [495/497] Compiling SuggestionGetFunction SuggestionGet.swift
  [495/497] Write Objects.LinkFileList
  /usr/lib/swift_static/linux/libFoundationEssentials.a(FoundationEssentials-1.o):FoundationEssentials-1.o:function $s20FoundationEssentials19createTemporaryFile33_FC9EC52B075D2ACCFF86F1C9F84293BELL2at6inPath6prefix7optionss5Int32V_SStSS_AA0Q5OrURLOSSAA4DataV14WritingOptionsVtKFTf4nnnd_n: warning: the use of `mktemp' is dangerous, better use `mkstemp' or `mkdtemp'
/usr/lib/swift_static/linux/libFoundationEssentials.a(FoundationEssentials-1.o):FoundationEssentials-1.o:function $s20FoundationEssentials19createTemporaryFile33_FC9EC52B075D2ACCFF86F1C9F84293BELL2at6inPath6prefix7optionss5Int32V_SStSS_AA0Q5OrURLOSSAA4DataV14WritingOptionsVtKFTf4nnnd_n: warning: the use of `mktemp' is dangerous, better use `mkstemp' or `mkdtemp'
/usr/lib/swift_static/linux/libFoundationEssentials.a(FoundationEssentials-1.o):FoundationEssentials-1.o:function $s20FoundationEssentials19createTemporaryFile33_FC9EC52B075D2ACCFF86F1C9F84293BELL2at6inPath6prefix7optionss5Int32V_SStSS_AA0Q5OrURLOSSAA4DataV14WritingOptionsVtKFTf4nnnd_n: warning: the use of `mktemp' is dangerous, better use `mkstemp' or `mkdtemp'
/usr/lib/swift_static/linux/libFoundationEssentials.a(FoundationEssentials-1.o):FoundationEssentials-1.o:function $s20FoundationEssentials19createTemporaryFile33_FC9EC52B075D2ACCFF86F1C9F84293BELL2at6inPath6prefix7optionss5Int32V_SStSS_AA0Q5OrURLOSSAA4DataV14WritingOptionsVtKFTf4nnnd_n: warning: the use of `mktemp' is dangerous, better use `mkstemp' or `mkdtemp'
/usr/lib/swift_static/linux/libFoundationEssentials.a(FoundationEssentials-1.o):FoundationEssentials-1.o:function $s20FoundationEssentials19createTemporaryFile33_FC9EC52B075D2ACCFF86F1C9F84293BELL2at6inPath6prefix7optionss5Int32V_SStSS_AA0Q5OrURLOSSAA4DataV14WritingOptionsVtKFTf4nnnd_n: warning: the use of `mktemp' is dangerous, better use `mkstemp' or `mkdtemp'
[496/497] Linking SuggestionGetFunction
  Build of product 'SuggestionGetFunction' complete! (482.85s)
-------------------------------------------------------------------------
archiving "SuggestionGetFunction"
-------------------------------------------------------------------------
error: Error Domain=NSCocoaErrorDomain Code=513 "You don’t have permission."
make: *** [Makefile:53: build-single-product] Error 1

(Wild guess: maybe this is due to the Docker container running as root, so the build artefacts are owned by root? ¯\_(ツ)_/¯ )

Steps to reproduce

  1. fresh installation of Ubuntu 24.10
  2. install make and zip
  3. install Swift 6.0.3 from tarball
  4. install the Docker engine using their installation docs
  5. clone my lambda repo
  6. swift package archive --allow-network-connections docker

If possible, minimal yet complete reproducer code (or URL to code)

No response

What version of this project (swift-aws-lambda-runtime) are you using?

38cc01a

Swift version

Swift version 6.0.3 (swift-6.0.3-RELEASE)
Target: aarch64-unknown-linux-gnu
Linux heslop 6.11.0-1005-raspi #5-Ubuntu SMP PREEMPT_DYNAMIC Fri Nov 22 13:24:30 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux

Amazon Linux 2 docker image version

SWIFT_PLATFORM=amazonlinux2, SWIFT_BRANCH=swift-6.0.3-release

@sebsto
Copy link
Contributor

sebsto commented Jan 4, 2025

Hello @mflint Matthew,

Sorry to read you're experiencing this issue.
In my experience, this often happens when building for multiple platforms (Linux and macOS) without a proper cleaning first.

Can you ensure to

  • rm -rf .build
  • rm Package.resolved

On linux, you might get into problems when docker runs as a different user than your current user (docker typically runs as root)
I'll try to reproduce this issue

@sebsto sebsto self-assigned this Jan 4, 2025
@sebsto sebsto added kind/bug Feature doesn't work as expected. semver/none No version bump required. size/S Small task. (A couple of hours of work.) status/triage Collecting information required to triage the issue. platform/linux Linux platform specific issues. labels Jan 4, 2025
@mflint
Copy link
Author

mflint commented Jan 4, 2025

I think I can close this - I’ve worked around it by making my own Dockerfile, and putting build steps into a makefile.

You were correct - it was a Docker permissions issue, running as root on Linux.

@mflint mflint closed this as completed Jan 4, 2025
@sebsto
Copy link
Contributor

sebsto commented Jan 4, 2025

Thank you for the feedback.
We have a test on the plugin that runs on Ubuntu + docker in the CI and this test passes. I don't think this is a systemic issue however, I'd like to understand the conditions that trigger this and document it clearly to avoid others to run into the same problem.

@mflint
Copy link
Author

mflint commented Jan 4, 2025

Interesting. I’ll look at your test suite to see what’s different with my setup.

@sebsto
Copy link
Contributor

sebsto commented Jan 5, 2025

One of the main difference is that the script runs inside a container on Amazon Linux 2. Builds on Amazon Linux do not trigger docker and start swift build directly.
See this line in the code of the plugin.

I started an Ubuntu VM to reproduce your issue.

@sebsto
Copy link
Contributor

sebsto commented Jan 5, 2025

I confirm that a docker build (swift package archive) on a fresh install does not work.

Error is

-------------------------------------------------------------------------
building "helloworld" in docker
-------------------------------------------------------------------------
updating "swift:amazonlinux2" docker image
  permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post "http://%2Fvar%2Frun%2Fdocker.sock/v1.47/images/create?fromImage=swift&tag=amazonlinux2": dial unix /var/run/docker.sock: connect: permission denied
error: /usr/bin/docker pull swift:amazonlinux2 failed with code 1

This is most probably due because the current user can not read/write the docker shared socket file.

@sebsto
Copy link
Contributor

sebsto commented Jan 5, 2025

The solution is to add your current user (ubuntu for me) to the docker group. Do not forget to logout and login again after that change.

sudo usermod -aG docker $USER

Here is the full script I used

#!/bin/bash

sudo apt update && sudo apt -y upgrade

# Install Swift 6.0.3
sudo apt-get -y install \
          binutils \
          git \
          gnupg2 \
          libc6-dev \
          libcurl4-openssl-dev \
          libedit2 \
          libgcc-13-dev \
          libncurses-dev \
          libpython3-dev \
          libsqlite3-0 \
          libstdc++-13-dev \
          libxml2-dev \
          libz3-dev \
          pkg-config \
          tzdata \
          unzip \
          zlib1g-dev

wget https://download.swift.org/swift-6.0.3-release/ubuntu2404-aarch64/swift-6.0.3-RELEASE/swift-6.0.3-RELEASE-ubuntu24.04-aarch64.tar.gz

tar xfvz swift-6.0.3-RELEASE-ubuntu24.04-aarch64.tar.gz

export PATH=/home/ubuntu/swift-6.0.3-RELEASE-ubuntu24.04-aarch64/usr/bin:"${PATH}"

swift --version

# Install Docker
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update

sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

# Add the current user to the docker group
sudo usermod -aG docker $USER

# LOGOUT and LOGIN to apply the changes
exit

# reconnect with ssh, then
export PATH=/home/ubuntu/swift-6.0.3-RELEASE-ubuntu24.04-aarch64/usr/bin:"${PATH}"

# clone a project 
git clone https://github.com/swift-server/swift-aws-lambda-runtime.git

# build the project
cd swift-aws-lambda-runtime/Examples/HelloWorld/
swift package archive --allow-network-connections docker

This produces

swift package archive --allow-network-connections docker

-------------------------------------------------------------------------
building "helloworld" in docker
-------------------------------------------------------------------------
updating "swift:amazonlinux2" docker image
  amazonlinux2: Pulling from library/swift
  Digest: sha256:df06a50f70e2e87f237bd904d2fc48195742ebda9f40b4a821c4d39766434009
Status: Image is up to date for swift:amazonlinux2
  docker.io/library/swift:amazonlinux2
building "MyLambda"
  [0/1] Planning build
  Building for production...
  [0/2] Write swift-version-24593BA9C3E375BF.txt
  Build of product 'MyLambda' complete! (4.78s)
-------------------------------------------------------------------------
archiving "MyLambda"
-------------------------------------------------------------------------
1 archive created
  * MyLambda at /home/ubuntu/swift-aws-lambda-runtime/Examples/HelloWorld/.build/plugins/AWSLambdaPackager/outputs/AWSLambdaPackager/MyLambda/MyLambda.zip

@sebsto
Copy link
Contributor

sebsto commented Jan 5, 2025

I have added a Note in the deployment guide.
#455

@sebsto sebsto reopened this Jan 5, 2025
@mflint
Copy link
Author

mflint commented Jan 5, 2025

Thank-you @sebsto, but that's different from the error that I reported. I was seeing problems with the archive stage reading files from the disk. (My user was already in the docker group, and I could successfully connect to the docker socket)

Don't worry about it - I'll do some investigations here :)

@sebsto
Copy link
Contributor

sebsto commented Jan 5, 2025

You’re correct. I saw the exact same error on macOS when multiple build attempts from different users or systems. (Docker and native for example)

Can you try again from scratch (after having deleted .build and Package.resolved ?

Is the error systematic ?

@mflint
Copy link
Author

mflint commented Jan 6, 2025

@sebsto When building Examples/HelloWorld, I see the same success as you... but I get a permissions error when trying to build Examples/S3_AWSSDK.

Can you try to build S3_AWSSDK please?

cd Examples/S3_AWSSDK
swift package archive --allow-network-connections docker

sebsto added a commit that referenced this issue Jan 6, 2025
Add a note in the deployment guide to inform Linux user they must have
correct permissions to use docker on their system.

### Motivation:

Build instructions fail on a fresh Ubuntu installation. See this error
report.
#449

### Modifications:

Add a note in the deployment guide that Linux user must add their user
in the `docker` group.

### Result:

Hopefully, Linux users will not experience error at first use of `swift
package archive`
@sebsto
Copy link
Contributor

sebsto commented Jan 8, 2025

@mflint ACK. Will do. Thanks !

@jsonfry
Copy link
Contributor

jsonfry commented Jan 16, 2025

I believe the failure in the archiving step for the S3 example is when copying resources from the build folder into the plugins/outputs folder just prior to zipping.

It's odd, it does seem to copy the .resources folder BUT then fails (I think with an underlying POSIX EPERM). It seems like the Swift/Foundation/FileManager/Something internals is trying to do something after copying that is not permitted.

The original resources files are owned by root (as they were made in docker) and the new copied files are owned by the user (in my case jason), see:

$ ls -lah .build/release/aws-crt-swift_AwsCommonRuntimeKit.resources
total 36K
drwxr-xr-x   2 root root 4.0K Jan 16 14:52 .
drwxr-xr-x 495 root root  24K Jan 16 14:53 ..
-r--rw-r--   1 root root  599 Jan 16 14:52 PrivacyInfo.xcprivacy

$ ls -lah .build/plugins/AWSLambdaPackager/outputs/AWSLambdaPackager/AWSSDKExample/aws-crt-swift_AwsCommonRuntimeKit.resources/
total 12K
drwxr-xr-x 2 jason jason 4.0K Jan 16 14:58 .
drwxrwxr-x 3 jason jason 4.0K Jan 16 14:58 ..
-r--rw-r-- 1 jason jason  599 Jan 16 14:58 PrivacyInfo.xcprivacy

I've also logged out (CocoaError) error.userInfo:

["NSUnderlyingError": AnyHashable(Error Domain=NSPOSIXErrorDomain Code=1 "Operation not permitted"), "NSFilePath": AnyHashable("/home/jason/S3_AWSSDK/.build/plugins/AWSLambdaPackager/outputs/AWSLambdaPackager/AWSSDKExample/aws-crt-swift_AwsCommonRuntimeKit.resources"), "NSURL": AnyHashable(file:///home/jason/S3_AWSSDK/.build/plugins/AWSLambdaPackager/outputs/AWSLambdaPackager/AWSSDKExample/aws-crt-swift_AwsCommonRuntimeKit.resources)]

The error is thrown on this line (try FileManager.default.copyItem)

try FileManager.default.copyItem(

Not sure how to go any further on this one.

(I should note that I'm also seeing this issue in my own project)

@sebsto
Copy link
Contributor

sebsto commented Jan 16, 2025

Thank you for having a look @jsonfry
@jrosen081 can you have a look if time permits ? I think it might be related to #386

jsonfry added a commit to jsonfry/swift-aws-lambda-runtime that referenced this issue Jan 16, 2025
sebsto added a commit to sebsto/swift-aws-lambda-runtime that referenced this issue Jan 18, 2025
@sebsto
Copy link
Contributor

sebsto commented Jan 18, 2025

I started a thread to try to figure out the root cause.

In the meantime, I built upon @jsonfry workaround's idea. I added a test to ensure we ignore the error only when it's a permission error and when the files are actually correctly copied, otherwise I let the error propagate.

I also added an example project and modified the CI integration test. It now runs the plugin test on the example project to ensure the case of a resource bundle is tested.

See #467 and tell me what you think.
@mflint @jsonfry

I also filed an issue on the Foundation project
swiftlang/swift-foundation#1125

@jsonfry
Copy link
Contributor

jsonfry commented Jan 20, 2025

That seems like a sensible move at this stage, much better than my dangerous quick fix to get things deploying again!

I'm still working with the v1 plugin (hence my own temporary fork with cherrypicked resources support) so I won't be deploying this until v2 is ready and I've had the time to migrate to it.

Glad you were able to dig into Foundation, I'd hoped to get to have a look at that later this week so I'm pleased that's off my list already, thank you :D

sebsto added a commit that referenced this issue Jan 21, 2025
When including resources in the package and packaging on Ubuntu,
`FileManager` throws a FilePermission error. The docker daemon runs as
root and files to be copied are owned by `root` while the archiver runs
as the current user (`ubuntu` on EC2 Ubuntu). The `FileManager` manages
to copy the files but throws an error after the copy. We suspect the
`FileManager` to perform some kind of operation after the copy and it
fails because of the `root` permission of the files.

See
#449 (comment)
for a description of the problem.

This PR contains code to reproduce the problem, a very simple
workaround, and an integration test.
The workaround consists of 
- trapping all errors
- verify if the error is the permission error (Code = 513)
- verify if the files have been copied or not 
- if the two above conditions are met, ignore the error, otherwise
re-throw it

I would rather prefer a solution that solves the root cause rather than
just ignoring the error.
We're still investigating the root cause (see [this
thread](https://forums.swift.org/t/filemanager-copyitem-on-linux-fails-after-copying-the-files/77282)
on the Swift Forum and this issue on Swift Foundation
swiftlang/swift-foundation#1125
@sebsto
Copy link
Contributor

sebsto commented Jan 21, 2025

#467 is now merged, I'm closing this. Thank you @mflint for having reported it and @jsonfry for the investigation.

@jsonfry I will not piggy back this change to 1.0.0, we're close to tag a v2.0.0-alpha and we prefer to focus on this objective

@jsonfry
Copy link
Contributor

jsonfry commented Jan 21, 2025

Thanks

@jsonfry I will not piggy back this change to 1.0.0, we're close to tag a v2.0.0-alpha and we prefer to focus on this objective

Absolutely, makes sense. Looking forward to v2!

@sebsto sebsto closed this as completed Feb 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Feature doesn't work as expected. platform/linux Linux platform specific issues. semver/none No version bump required. size/S Small task. (A couple of hours of work.) status/triage Collecting information required to triage the issue.
Projects
None yet
Development

No branches or pull requests

3 participants