Support running in background #72

monthonk · 2023-02-07T16:03:19Z

This change allows the file connector to run in background and it will be running there by default. Users can change this behavior by passing in mount option --foreground or -f to run it in foreground instead.

I'm still figuring out how we can create some tests for this change. Any ideas are welcome. Another thing is that the logs are currently writing to the stdout directly even if the process is running in background. It's a bit annoying but I think that would be a part of PR that addresses #39.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

monthonk · 2023-02-09T14:23:11Z

From this PR, I think AutoUnmount should be set as a default config, otherwise the mount state would be left hanging when we kill the background process.

monthonk · 2023-02-09T16:57:36Z

ASan is not happy with the tests. Any thoughts?

s3-file-connector/Cargo.toml

s3-file-connector/src/main.rs

s3-file-connector/tests/common/mod.rs

s3-file-connector/tests/cli.rs

s3-file-connector/src/main.rs

s3-file-connector/tests/cli.rs

s3-file-connector/src/main.rs

s3-file-connector/tests/cli.rs

s3-file-connector/src/main.rs

Makefile

s3-file-connector/src/main.rs

dannycjones · 2023-02-13T15:04:12Z

s3-file-connector/src/main.rs

+                init_tracing_subscriber();
+
+                let (interrupt_tx, interrupt_rx) = crossbeam::channel::unbounded();
+                std::thread::spawn(move || await_interrupt(interrupt_tx));


Is there a risk this happens too late? What if the mount completes in the child thread?

That's not likely to happen for success case due to network latency. Could be possible in some error cases that send the signal very soon. I'm not sure we should move this to before forking because I think the child process would copy the channel too. @jamesbornholt do you have any suggestions here?

Hmm, yeah, this is unlikely but we should do it properly. That means creating a pipe (pipe2) before the fork, writing a character into the write side of the pipe when the child wants to send its status (probably File::from_raw_fd), and reading a character off the read side of the pipe in the parent to check the status (same). Doing timeouts will be more annoying this way, since you can't configure a timeout for file/pipe reads. Might need to spawn a thread to do it.

s3-file-connector/src/main.rs

Makefile

s3-file-connector/src/main.rs

jamesbornholt · 2023-02-13T16:24:50Z

s3-file-connector/src/main.rs

+                match session {
+                    Ok(_) => {
+                        let _ = nix::sys::signal::kill(parent_pid, Signal::SIGCONT);
+                        thread::park();
+
+                        return Ok(());


This is kind of subtle and we should be more explicit -- the session stays running because its lifetime is bound to the match statement, so doesn't drop until after the park. Probably we should write Ok(_session) and put a comment above the park explaining this.

s3-file-connector/src/main.rs

s3-file-connector/tests/common/mod.rs

s3-file-connector/tests/fuse_tests/fork_test.rs

s3-file-connector/src/main.rs

jamesbornholt · 2023-02-13T18:11:32Z

s3-file-connector/src/main.rs

+                init_tracing_subscriber();
+
+                let (interrupt_tx, interrupt_rx) = crossbeam::channel::unbounded();
+                std::thread::spawn(move || await_interrupt(interrupt_tx));


Hmm, yeah, this is unlikely but we should do it properly. That means creating a pipe (pipe2) before the fork, writing a character into the write side of the pipe when the child wants to send its status (probably File::from_raw_fd), and reading a character off the read side of the pipe in the parent to check the status (same). Doing timeouts will be more annoying this way, since you can't configure a timeout for file/pipe reads. Might need to spawn a thread to do it.

dannycjones

I'm a little uneasy on some of the multi-process logic, not that I understand the area well at this point.

Also, I expect the logic to change anyway with the pipe change.

s3-file-connector/src/main.rs

s3-file-connector/tests/fuse_tests/fork_test.rs

s3-file-connector/src/main.rs

dannycjones · 2023-02-15T17:09:33Z

s3-file-connector/src/main.rs

+                        // also `park()` does not guarantee to remain parked forever. so, we put it inside a loop.
+                        loop {
+                            thread::park();
+                        }


I think we need some kind of stopping condition.

When we receive SIGINT, we should exit. Maybe another thread waits for SIGINT, and sets an atomic bool?

Does this sound right, @jamesbornholt?

I think the standard library already handle that. I have tested this by running kill -2 {pid} which send SIGINT to the running child process and it works just fine.

We need to figure something different out here, otherwise the process runs forever even if the user unmounts it. But we can do that as a followup, I think, because we'll probably have to make fuser changes.

Yeah, that is existing problem, it keep running forever even for foreground process. It's just less visible when run in the background. Let's do it as a follow up.

Let's open an issue and then we can resolve this.

Opened #93.

s3-file-connector/src/main.rs

jamesbornholt · 2023-02-15T18:53:34Z

s3-file-connector/src/main.rs

+                        // also `park()` does not guarantee to remain parked forever. so, we put it inside a loop.
+                        loop {
+                            thread::park();
+                        }


We need to figure something different out here, otherwise the process runs forever even if the user unmounts it. But we can do that as a followup, I think, because we'll probably have to make fuser changes.

s3-file-connector/src/main.rs

jamesbornholt · 2023-02-15T18:57:33Z

s3-file-connector/src/main.rs

+                    // SAFETY: `read_fd` is a valid file descriptor.
+                    let mut f = unsafe { File::from_raw_fd(read_fd) };


Maybe do this conversion outside the thread and right below closing the write_fd, to make clearer the fate of the pipe (and give f a better name)

This change allows the file connector to run in background and it will be running there by default. Users can change this behavior by passing in mount option `--foreground` or `-f` to run it in foreground instead. Signed-off-by: Monthon Klongklaew <[email protected]>

dannycjones

LGTM, pending new issue to follow up on failing to exit if unmounted.

After #72, our file connector will be running in background by default. This breaks the benchmark clean up process because we're now using the wrong pid to kill the process after unmount. Technically, running in background should not be a problem but because of #93 we will have to use `--foreground` flag until we have it fixed. Signed-off-by: Monthon Klongklaew <[email protected]>

* Fix benchmark script After #72, our file connector will be running in background by default. This breaks the benchmark clean up process because we're now using the wrong pid to kill the process after unmount. Technically, running in background should not be a problem but because of #93 we will have to use `--foreground` flag until we have it fixed. Signed-off-by: Monthon Klongklaew <[email protected]> * Update file system name in bench script Signed-off-by: Monthon Klongklaew <[email protected]> --------- Signed-off-by: Monthon Klongklaew <[email protected]>

Submodule mountpoint-s3-crt-sys/crt/aws-c-auth 5bc67797..b513db4b: > A bunch of CMake fixes (awslabs#258) > Add Account Id to Credentials (awslabs#260) > Skip Transfer-Encoding from signing (awslabs#261) Submodule mountpoint-s3-crt-sys/crt/aws-c-cal fbbe2612..7299c6ab: > Fix Findcrypto.cmake (awslabs#205) > A bunch of CMake fixes (awslabs#203) > Switch CI to use roles (awslabs#202) Submodule mountpoint-s3-crt-sys/crt/aws-c-common 7a6f5df2..0e7637fa: > A bunch of CMake fixes (awslabs#1178) > Fix heap overflow on uri parsing (awslabs#1185) > (take 2) Detect when AVX is disabled via OSXSAVE (awslabs#1184) > Fixup IPv6 validation logic (awslabs#1180) > Detect when AVX is disabled via OSXSAVE (awslabs#1182) > proof_ci.yaml must use latest upload-artifact (awslabs#1183) > change PR template to ask for clearer wording (awslabs#1177) Submodule mountpoint-s3-crt-sys/crt/aws-c-compression c6c1191e..f951ab2b: > A bunch of CMake fixes (awslabs#72) > Switch CI to use roles (awslabs#71) > chore: Modified bug issue template to add checkbox to report potential regression. (awslabs#69) Submodule mountpoint-s3-crt-sys/crt/aws-c-http fc3eded2..590c7b59: > A bunch of CMake fixes (awslabs#497) > Fix CI for GCC-13 on Ubuntu-18 (awslabs#496) > Switch CI to use roles (awslabs#494) Submodule mountpoint-s3-crt-sys/crt/aws-c-io fcb38c80..3041dabf: > A bunch of CMake fixes (awslabs#701) > Event Loop & Socket Type Multi-Support (awslabs#692) > fix typo in log message (awslabs#702) > Fix CI for GCC-13 on Ubuntu-18 (awslabs#700) > Switch CI to use roles (awslabs#698) Submodule mountpoint-s3-crt-sys/crt/aws-c-s3 a3b401bf..6eb8be53: > A bunch of CMake fixes (awslabs#480) > S3Express CreateSession Allowlist Headers (awslabs#492) > Auto - Update S3 Ruleset & Partition (awslabs#491) Submodule mountpoint-s3-crt-sys/crt/aws-c-sdkutils 1ae8664f..ba6a28fa: > A bunch of CMake fixes (awslabs#50) Submodule mountpoint-s3-crt-sys/crt/aws-checksums 3e4101b9..fb8bd0b8: > A bunch of CMake fixes (awslabs#101) > Switch CI to use roles (awslabs#100) Submodule mountpoint-s3-crt-sys/crt/aws-lc ffd6fb71..138a6ad3: > Prepare AWS-LC v1.44.0 (#2153) > Fix issue with ML-DSA key parsing (#2152) > Add support for PKCS7_set/get_detached (#2134) > Prepare Docker image for CI integration jobs (#2126) > Delete OpenVPN mainline patch from our integration build (#2149) > SHA3/SHAKE Init Updates via FIPS202 API layer (#2101) > Support keypair calculation for PQDSA PKEY (#2145) > Optimize x86/aarch64 MD5 implementation (#2137) > Check for MIPSEB in target.h (#2143) > Ed25519ph and Ed25519ctx Support (#2120) > Support for ML-DSA public key generation from private key (#2142) > Avoid mixing SSE and AVX in XTS-mode AVX512 implementation (#2140) > Remove remaining support for Trusty and Fuchsia operating systems (#2136) > ACVP test harness for ML-DSA (#2127) > Minor symbols to work with Ruby's mainline (#2132) Signed-off-by: Alessandro Passaro <[email protected]>

Update the CRT libraries to the latest releases. In particular, include: * S3Express CreateSession Allowlist Headers ([awslabs/aws-c-s3#492](awslabs/aws-c-s3#492)) <details> <summary>Full CRT changelog:</summary> ``` Submodule mountpoint-s3-crt-sys/crt/aws-c-auth 5bc67797..b513db4b: > A bunch of CMake fixes (#258) > Add Account Id to Credentials (#260) > Skip Transfer-Encoding from signing (#261) Submodule mountpoint-s3-crt-sys/crt/aws-c-cal fbbe2612..7299c6ab: > Fix Findcrypto.cmake (#205) > A bunch of CMake fixes (#203) > Switch CI to use roles (#202) Submodule mountpoint-s3-crt-sys/crt/aws-c-common 7a6f5df2..0e7637fa: > A bunch of CMake fixes (#1178) > Fix heap overflow on uri parsing (#1185) > (take 2) Detect when AVX is disabled via OSXSAVE (#1184) > Fixup IPv6 validation logic (#1180) > Detect when AVX is disabled via OSXSAVE (#1182) > proof_ci.yaml must use latest upload-artifact (#1183) > change PR template to ask for clearer wording (#1177) Submodule mountpoint-s3-crt-sys/crt/aws-c-compression c6c1191e..f951ab2b: > A bunch of CMake fixes (#72) > Switch CI to use roles (#71) > chore: Modified bug issue template to add checkbox to report potential regression. (#69) Submodule mountpoint-s3-crt-sys/crt/aws-c-http fc3eded2..590c7b59: > A bunch of CMake fixes (#497) > Fix CI for GCC-13 on Ubuntu-18 (#496) > Switch CI to use roles (#494) Submodule mountpoint-s3-crt-sys/crt/aws-c-io fcb38c80..3041dabf: > A bunch of CMake fixes (#701) > Event Loop & Socket Type Multi-Support (#692) > fix typo in log message (#702) > Fix CI for GCC-13 on Ubuntu-18 (#700) > Switch CI to use roles (#698) Submodule mountpoint-s3-crt-sys/crt/aws-c-s3 a3b401bf..6eb8be53: > A bunch of CMake fixes (#480) > S3Express CreateSession Allowlist Headers (#492) > Auto - Update S3 Ruleset & Partition (#491) Submodule mountpoint-s3-crt-sys/crt/aws-c-sdkutils 1ae8664f..ba6a28fa: > A bunch of CMake fixes (#50) Submodule mountpoint-s3-crt-sys/crt/aws-checksums 3e4101b9..fb8bd0b8: > A bunch of CMake fixes (#101) > Switch CI to use roles (#100) Submodule mountpoint-s3-crt-sys/crt/aws-lc ffd6fb71..138a6ad3: > Prepare AWS-LC v1.44.0 (#2153) > Fix issue with ML-DSA key parsing (#2152) > Add support for PKCS7_set/get_detached (#2134) > Prepare Docker image for CI integration jobs (#2126) > Delete OpenVPN mainline patch from our integration build (#2149) > SHA3/SHAKE Init Updates via FIPS202 API layer (#2101) > Support keypair calculation for PQDSA PKEY (#2145) > Optimize x86/aarch64 MD5 implementation (#2137) > Check for MIPSEB in target.h (#2143) > Ed25519ph and Ed25519ctx Support (#2120) > Support for ML-DSA public key generation from private key (#2142) > Avoid mixing SSE and AVX in XTS-mode AVX512 implementation (#2140) > Remove remaining support for Trusty and Fuchsia operating systems (#2136) > ACVP test harness for ML-DSA (#2127) > Minor symbols to work with Ruby's mainline (#2132) ``` </details> ### Does this change impact existing behavior? No. ### Does this change need a changelog entry? Does it require a version change? No. --- By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and I agree to the terms of the [Developer Certificate of Origin (DCO)](https://developercertificate.org/). Signed-off-by: Alessandro Passaro <[email protected]>

Update the CRT libraries to the latest releases. In particular, include: * S3Express CreateSession Allowlist Headers ([awslabs/aws-c-s3#492](awslabs/aws-c-s3#492)) <details> <summary>Full CRT changelog:</summary> ``` Submodule mountpoint-s3-crt-sys/crt/aws-c-auth 5bc67797..b513db4b: > A bunch of CMake fixes (awslabs#258) > Add Account Id to Credentials (awslabs#260) > Skip Transfer-Encoding from signing (awslabs#261) Submodule mountpoint-s3-crt-sys/crt/aws-c-cal fbbe2612..7299c6ab: > Fix Findcrypto.cmake (awslabs#205) > A bunch of CMake fixes (awslabs#203) > Switch CI to use roles (awslabs#202) Submodule mountpoint-s3-crt-sys/crt/aws-c-common 7a6f5df2..0e7637fa: > A bunch of CMake fixes (awslabs#1178) > Fix heap overflow on uri parsing (awslabs#1185) > (take 2) Detect when AVX is disabled via OSXSAVE (awslabs#1184) > Fixup IPv6 validation logic (awslabs#1180) > Detect when AVX is disabled via OSXSAVE (awslabs#1182) > proof_ci.yaml must use latest upload-artifact (awslabs#1183) > change PR template to ask for clearer wording (awslabs#1177) Submodule mountpoint-s3-crt-sys/crt/aws-c-compression c6c1191e..f951ab2b: > A bunch of CMake fixes (awslabs#72) > Switch CI to use roles (awslabs#71) > chore: Modified bug issue template to add checkbox to report potential regression. (awslabs#69) Submodule mountpoint-s3-crt-sys/crt/aws-c-http fc3eded2..590c7b59: > A bunch of CMake fixes (awslabs#497) > Fix CI for GCC-13 on Ubuntu-18 (awslabs#496) > Switch CI to use roles (awslabs#494) Submodule mountpoint-s3-crt-sys/crt/aws-c-io fcb38c80..3041dabf: > A bunch of CMake fixes (awslabs#701) > Event Loop & Socket Type Multi-Support (awslabs#692) > fix typo in log message (awslabs#702) > Fix CI for GCC-13 on Ubuntu-18 (awslabs#700) > Switch CI to use roles (awslabs#698) Submodule mountpoint-s3-crt-sys/crt/aws-c-s3 a3b401bf..6eb8be53: > A bunch of CMake fixes (awslabs#480) > S3Express CreateSession Allowlist Headers (awslabs#492) > Auto - Update S3 Ruleset & Partition (awslabs#491) Submodule mountpoint-s3-crt-sys/crt/aws-c-sdkutils 1ae8664f..ba6a28fa: > A bunch of CMake fixes (awslabs#50) Submodule mountpoint-s3-crt-sys/crt/aws-checksums 3e4101b9..fb8bd0b8: > A bunch of CMake fixes (awslabs#101) > Switch CI to use roles (awslabs#100) Submodule mountpoint-s3-crt-sys/crt/aws-lc ffd6fb71..138a6ad3: > Prepare AWS-LC v1.44.0 (#2153) > Fix issue with ML-DSA key parsing (#2152) > Add support for PKCS7_set/get_detached (#2134) > Prepare Docker image for CI integration jobs (#2126) > Delete OpenVPN mainline patch from our integration build (#2149) > SHA3/SHAKE Init Updates via FIPS202 API layer (#2101) > Support keypair calculation for PQDSA PKEY (#2145) > Optimize x86/aarch64 MD5 implementation (#2137) > Check for MIPSEB in target.h (#2143) > Ed25519ph and Ed25519ctx Support (#2120) > Support for ML-DSA public key generation from private key (#2142) > Avoid mixing SSE and AVX in XTS-mode AVX512 implementation (#2140) > Remove remaining support for Trusty and Fuchsia operating systems (#2136) > ACVP test harness for ML-DSA (#2127) > Minor symbols to work with Ruby's mainline (#2132) ``` </details> ### Does this change impact existing behavior? No. ### Does this change need a changelog entry? Does it require a version change? No. --- By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and I agree to the terms of the [Developer Certificate of Origin (DCO)](https://developercertificate.org/). Signed-off-by: Alessandro Passaro <[email protected]>

monthonk force-pushed the running_in_background branch from 3939a18 to 147d0bd Compare February 7, 2023 16:23

dannycjones self-requested a review February 8, 2023 18:04

monthonk force-pushed the running_in_background branch 3 times, most recently from e308c2f to 9b7b556 Compare February 9, 2023 14:14

monthonk force-pushed the running_in_background branch 4 times, most recently from 707c137 to 56f2f9e Compare February 9, 2023 16:39

monthonk requested a review from jamesbornholt February 9, 2023 16:58

dannycjones reviewed Feb 9, 2023

View reviewed changes

jamesbornholt reviewed Feb 9, 2023

View reviewed changes

monthonk force-pushed the running_in_background branch 11 times, most recently from 131cb22 to 9817ad8 Compare February 13, 2023 10:08

monthonk commented Feb 13, 2023

View reviewed changes

s3-file-connector/src/main.rs Outdated Show resolved Hide resolved

monthonk commented Feb 13, 2023

View reviewed changes

Makefile Outdated Show resolved Hide resolved

monthonk requested review from jamesbornholt and dannycjones February 13, 2023 10:23

monthonk force-pushed the running_in_background branch from 9817ad8 to 43703f5 Compare February 13, 2023 11:12

dannycjones reviewed Feb 13, 2023

View reviewed changes

jamesbornholt reviewed Feb 13, 2023

View reviewed changes

monthonk force-pushed the running_in_background branch from fdc740d to 88d87ab Compare February 13, 2023 17:59

jamesbornholt reviewed Feb 13, 2023

View reviewed changes

monthonk force-pushed the running_in_background branch from 88d87ab to d6dc79c Compare February 14, 2023 10:04

dannycjones requested changes Feb 14, 2023

View reviewed changes

s3-file-connector/src/main.rs Show resolved Hide resolved

s3-file-connector/src/main.rs Outdated Show resolved Hide resolved

s3-file-connector/tests/fuse_tests/fork_test.rs Show resolved Hide resolved

monthonk force-pushed the running_in_background branch 5 times, most recently from 00ac0ea to 67cedc7 Compare February 15, 2023 14:48

monthonk requested review from dannycjones and jamesbornholt February 15, 2023 15:28

monthonk force-pushed the running_in_background branch from 67cedc7 to 6e52bb5 Compare February 15, 2023 16:22

dannycjones reviewed Feb 15, 2023

View reviewed changes

monthonk force-pushed the running_in_background branch from 6e52bb5 to d465085 Compare February 15, 2023 18:19

jamesbornholt reviewed Feb 15, 2023

View reviewed changes

monthonk force-pushed the running_in_background branch from d465085 to 17b1af2 Compare February 16, 2023 10:47

dannycjones approved these changes Feb 16, 2023

View reviewed changes

jamesbornholt approved these changes Feb 16, 2023

View reviewed changes

jamesbornholt merged commit f6e936f into main Feb 16, 2023

jamesbornholt deleted the running_in_background branch February 16, 2023 15:42

monthonk mentioned this pull request Feb 16, 2023

Fix benchmark script #94

Merged

		// SAFETY: `read_fd` is a valid file descriptor.
		let mut f = unsafe { File::from_raw_fd(read_fd) };

Support running in background #72

Support running in background #72

Uh oh!

Conversation

monthonk commented Feb 7, 2023

Uh oh!

monthonk commented Feb 9, 2023

Uh oh!

monthonk commented Feb 9, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dannycjones left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dannycjones left a comment

Choose a reason for hiding this comment

Uh oh!