Implement robustness improvements in the compiler downloader #6056

galargh · 2024-12-16T14:24:40Z

Because this PR includes a bug fix, relevant tests have been included.
Because this PR includes a new feature, the change was previously discussed on an Issue or with someone from the team.
I didn't do anything of this.

Resolves #5990

This PR implements changes outlined in #5990. Please note, that trimming down the solc dependency has been extracted to a separate issue #6071 as it will be easier to review that way.

changeset-bot · 2024-12-16T14:24:44Z

⚠️ No Changeset found

Latest commit: 6745a2a

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

vercel · 2024-12-16T14:24:44Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
hardhat	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jan 3, 2025 10:48am

galargh · 2024-12-22T11:19:56Z

v-next/hardhat-utils/src/synchronization.ts

I have copied the multi-process mutex from the v2.

galargh · 2024-12-22T11:21:45Z

v-next/hardhat/test/internal/builtin-plugins/solidity/build-system/compiler/downloader.ts

I have copied the compiler downloader tests from v2 and updated them to work with the v3 implementation.

galargh · 2024-12-22T11:23:46Z

v-next/hardhat/test/internal/builtin-plugins/solidity/build-system/compiler/index.ts

I have copied the compiler tests from v2 and updated them to work with the v3 implementation. One significant update is the installation of tsx for the solcjs compiler tests.

galargh · 2024-12-22T11:28:56Z

v-next/hardhat/test/internal/builtin-plugins/solidity/build-system/compiler/downloader.ts

+        assertRejectsWithHardhatError(
+          () => mockDownloader.downloadCompiler("0.4.12"),
+          HardhatError.ERRORS.SOLIDITY.INVALID_DOWNLOAD,
+          {
+            remoteVersion: "0.4.12+commit.194ff033",
+          },
+        );


I've seen this check being flaky already as we sometimes get HardhatError.ERRORS.SOLIDITY.DOWNLOAD_FAILED instead. What do you think about adding a retry mechanism to the downloader?

I like the idea, but let's discuss it with @alcuadrado. Either way, this should be addressed in a separate PR.

I had another look at that and I don't quite understand why this is being flaky. We're using a mock downloader in this test case, which just writes to the disk. I assumed it was flaky because of the network use, but we're not even touching it here.

I'm trying to capture the full error we get when this flakes on another branch. I'll report back here when I find something.

I still think a retry mechanism might be an idea worth exploring, but I would definitely like to understand the source of flakiness in this test case better.

Found it! It was throwing the dir doesn't exist error because we weren't awaiting the assert 🤦 We should be able to catch it with a linting rule.

Suggested change

assertRejectsWithHardhatError(

() => mockDownloader.downloadCompiler("0.4.12"),

HardhatError.ERRORS.SOLIDITY.INVALID_DOWNLOAD,

{

remoteVersion: "0.4.12+commit.194ff033",

},

);

await assertRejectsWithHardhatError(

() => mockDownloader.downloadCompiler("0.4.12"),

HardhatError.ERRORS.SOLIDITY.INVALID_DOWNLOAD,

{

remoteVersion: "0.4.12+commit.194ff033",

},

);

schaable

Hey Piotr! Sorry for the delay in reviewing this. It looks good. I just left a few comments.

v-next/hardhat-utils/src/synchronization.ts

v-next/hardhat/test/internal/builtin-plugins/solidity/build-system/compiler/downloader.ts

schaable · 2024-12-30T14:58:07Z

v-next/hardhat/test/internal/builtin-plugins/solidity/build-system/compiler/downloader.ts

+        assertRejectsWithHardhatError(
+          () => mockDownloader.downloadCompiler("0.4.12"),
+          HardhatError.ERRORS.SOLIDITY.INVALID_DOWNLOAD,
+          {
+            remoteVersion: "0.4.12+commit.194ff033",
+          },
+        );


I like the idea, but let's discuss it with @alcuadrado. Either way, this should be addressed in a separate PR.

v-next/hardhat/test/internal/builtin-plugins/solidity/build-system/compiler/index.ts

github-actions · 2024-12-30T17:04:02Z

hardhat

Total size of the bundle: 219M
Total number of dependencies (including transitive): 53

List of dependencies (sorted by size)

214M	total
30M	@ignored/edr-optimism-linux-x64-musl
30M	@ignored/edr-optimism-linux-x64-gnu
27M	@ignored/edr-optimism-linux-arm64-musl
27M	@ignored/edr-optimism-linux-arm64-gnu
22M	@ignored/edr-optimism-win32-x64-msvc
21M	@ignored/edr-optimism-darwin-x64
20M	esbuild
20M	@ignored/edr-optimism-darwin-arm64
2.8M	@sentry/tracing
2.5M	micro-eth-signer
1.9M	@noble/curves
1.7M	undici
1.2M	@sentry/types
1.2M	@noble/hashes
932K	@sentry/node
920K	@sentry/utils
856K	zod
844K	@ignored/hardhat-vnext-utils
616K	micro-packed
576K	tsx
548K	@sentry/core
504K	fast-equals
492K	@scure/bip39
460K	@ignored/edr
384K	@ignored/edr-optimism
368K	ethereum-cryptography
344K	@sentry/hub
320K	enquirer
320K	@ignored/hardhat-vnext-errors
284K	semver
192K	ws
168K	@scure/base
136K	adm-zip
128K	get-tsconfig
96K	@scure/bip32
92K	chalk
88K	tslib
88K	@sentry/minimal
76K	agent-base
72K	@nomicfoundation/solidity-analyzer
68K	debug
64K	lru_map
64K	https-proxy-agent
60K	@ignored/hardhat-vnext-zod-utils
56K	rfdc
48K	ansi-colors
40K	resolve-pkg-maps
36K	p-map
32K	cookie
24K	strip-ansi
24K	env-paths
24K	ansi-regex
20K	ms

galargh added 3 commits December 16, 2024 12:58

feat: add process mutex to the compiler downloader

72d8175

test: port the multi-process-mutex tests

5082048

test: port the compiler downloader tests

b85b358

github-actions bot added the status:ready This issue is ready to be worked on label Dec 16, 2024

vercel bot deployed to Preview December 18, 2024 12:32 View deployment

galargh force-pushed the robust-compiler-downloader branch from a2c8b5c to 70d8632 Compare December 18, 2024 12:32

vercel bot deployed to Preview December 18, 2024 12:33 View deployment

galargh force-pushed the robust-compiler-downloader branch from 70d8632 to c7b302e Compare December 18, 2024 12:35

vercel bot deployed to Preview December 18, 2024 12:36 View deployment

galargh force-pushed the robust-compiler-downloader branch from c7b302e to d7ad1d3 Compare December 18, 2024 13:10

vercel bot deployed to Preview December 18, 2024 13:11 View deployment

fix: compilation with solcjs on windows

50e983f

galargh force-pushed the robust-compiler-downloader branch from d7ad1d3 to 50e983f Compare December 18, 2024 13:18

vercel bot deployed to Preview December 18, 2024 13:19 View deployment

feat: always download native and wasm compilers

133ab74

vercel bot deployed to Preview December 18, 2024 13:46 View deployment

Merge branch 'v-next' into robust-compiler-downloader

ed94118

vercel bot deployed to Preview December 22, 2024 11:19 View deployment

galargh commented Dec 22, 2024

View reviewed changes

v-next/hardhat-utils/src/synchronization.ts

Copy link

Member Author

galargh Dec 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have copied the multi-process mutex from the v2.

galargh commented Dec 22, 2024

View reviewed changes

galargh changed the title ~~[WIP - not ready for review] Implement robustness improvements in the compiler downloader~~ Implement robustness improvements in the compiler downloader Dec 22, 2024

galargh requested review from alcuadrado and schaable December 22, 2024 11:29

galargh marked this pull request as ready for review December 22, 2024 11:29

chore: replace solc dependency with a thin solidity_compile wrapper

9904353

galargh mentioned this pull request Dec 23, 2024

Replace solc dependency with a thin solidity_compile wrapper #6073

Merged

3 tasks

schaable added the v-next A Hardhat v3 development task label Dec 30, 2024

schaable reviewed Dec 30, 2024

View reviewed changes

galargh added 5 commits December 30, 2024 17:40

chore: throw hardhat errors from synchronization util

97ab1f7

chore: do not reimplement the sleep function

064bdce

docs: explain difference between tsx and node loader better

be49e78

chore: plaftorm -> platform

4e9c17e

test: apply review comments

b27f710

vercel bot deployed to Preview December 30, 2024 17:03 View deployment

galargh requested a review from schaable December 30, 2024 19:53

Merge branch 'v-next' into robust-compiler-downloader

c48a049

vercel bot deployed to Preview December 30, 2024 19:54 View deployment

schaable approved these changes Dec 31, 2024

View reviewed changes

Merge branch 'v-next' into robust-compiler-downloader

a6fab0e

vercel bot deployed to Preview January 3, 2025 10:01 View deployment

fix: await assert rejects with hardhat error call

6745a2a

vercel bot deployed to Preview January 3, 2025 10:48 View deployment

galargh added this pull request to the merge queue Jan 3, 2025

Merged via the queue into v-next with commit a8c02e1 Jan 3, 2025
142 checks passed

galargh deleted the robust-compiler-downloader branch January 3, 2025 10:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement robustness improvements in the compiler downloader #6056

Implement robustness improvements in the compiler downloader #6056

galargh commented Dec 16, 2024 •

edited

Loading

changeset-bot bot commented Dec 16, 2024 •

edited

Loading

vercel bot commented Dec 16, 2024 •

edited

Loading

galargh Dec 22, 2024

galargh Dec 22, 2024

galargh Dec 22, 2024

galargh Dec 22, 2024

schaable Dec 30, 2024

galargh Jan 3, 2025

galargh Jan 3, 2025

schaable left a comment

schaable Dec 30, 2024

github-actions bot commented Dec 30, 2024

Implement robustness improvements in the compiler downloader #6056

Implement robustness improvements in the compiler downloader #6056

Conversation

galargh commented Dec 16, 2024 • edited Loading

changeset-bot bot commented Dec 16, 2024 • edited Loading

⚠️ No Changeset found

vercel bot commented Dec 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

schaable left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Dec 30, 2024

hardhat

galargh commented Dec 16, 2024 •

edited

Loading

changeset-bot bot commented Dec 16, 2024 •

edited

Loading

vercel bot commented Dec 16, 2024 •

edited

Loading