Use SierraGenerator for testing (part 1) #1125

FrancoGiachetta · 2025-02-28T12:49:06Z

This PR changes some tests to use Sierra code, rather than Cairo code.

This has the following advantages:

Reduces test execution time by omitting compilation.
We ensure that the test is not modified by changes in the compiler

Related to #1152

Checklist

Linked to Github Issue

github-actions · 2025-02-28T13:14:07Z

Benchmark results Main vs HEAD.

Base

Command	Mean [s]	Min [s]	Max [s]	Relative
`base dict_insert.cairo (JIT)`	4.965 ± 0.075	4.783	5.061	1.06 ± 0.02
`base dict_insert.cairo (AOT)`	4.681 ± 0.042	4.637	4.760	1.00

Head

Command	Mean [s]	Min [s]	Max [s]	Relative
`head dict_insert.cairo (JIT)`	4.816 ± 0.014	4.794	4.840	1.02 ± 0.01
`head dict_insert.cairo (AOT)`	4.734 ± 0.021	4.714	4.778	1.00

Base

Command	Mean [s]	Min [s]	Max [s]	Relative
`base dict_snapshot.cairo (JIT)`	4.627 ± 0.018	4.606	4.662	1.02 ± 0.01
`base dict_snapshot.cairo (AOT)`	4.519 ± 0.017	4.494	4.543	1.00

Head

Command	Mean [s]	Min [s]	Max [s]	Relative
`head dict_snapshot.cairo (JIT)`	4.717 ± 0.038	4.654	4.781	1.03 ± 0.01
`head dict_snapshot.cairo (AOT)`	4.595 ± 0.028	4.553	4.654	1.00

Base

Command	Mean [s]	Min [s]	Max [s]	Relative
`base factorial_2M.cairo (JIT)`	5.002 ± 0.044	4.916	5.051	1.01 ± 0.01
`base factorial_2M.cairo (AOT)`	4.943 ± 0.030	4.887	4.991	1.00

Head

Command	Mean [s]	Min [s]	Max [s]	Relative
`head factorial_2M.cairo (JIT)`	5.046 ± 0.030	4.990	5.086	1.01 ± 0.01
`head factorial_2M.cairo (AOT)`	4.998 ± 0.017	4.959	5.016	1.00

Base

Command	Mean [s]	Min [s]	Max [s]	Relative
`base fib_2M.cairo (JIT)`	4.504 ± 0.025	4.482	4.566	1.01 ± 0.01
`base fib_2M.cairo (AOT)`	4.477 ± 0.012	4.459	4.496	1.00

Head

Command	Mean [s]	Min [s]	Max [s]	Relative
`head fib_2M.cairo (JIT)`	4.608 ± 0.031	4.575	4.665	1.02 ± 0.01
`head fib_2M.cairo (AOT)`	4.538 ± 0.027	4.493	4.581	1.00

Base

Command	Mean [s]	Min [s]	Max [s]	Relative
`base linear_search.cairo (JIT)`	4.722 ± 0.045	4.680	4.799	1.03 ± 0.01
`base linear_search.cairo (AOT)`	4.578 ± 0.026	4.529	4.615	1.00

Head

Command	Mean [s]	Min [s]	Max [s]	Relative
`head linear_search.cairo (JIT)`	4.767 ± 0.035	4.723	4.815	1.03 ± 0.01
`head linear_search.cairo (AOT)`	4.634 ± 0.041	4.563	4.683	1.00

Base

Command	Mean [s]	Min [s]	Max [s]	Relative
`base logistic_map.cairo (JIT)`	4.779 ± 0.016	4.751	4.800	1.03 ± 0.01
`base logistic_map.cairo (AOT)`	4.628 ± 0.022	4.592	4.649	1.00

Head

Command	Mean [s]	Min [s]	Max [s]	Relative
`head logistic_map.cairo (JIT)`	4.867 ± 0.018	4.842	4.904	1.03 ± 0.01
`head logistic_map.cairo (AOT)`	4.729 ± 0.027	4.710	4.803	1.00

codecov-commenter · 2025-02-28T13:15:54Z

Codecov Report

Attention: Patch coverage is 98.16934% with 8 lines in your changes missing coverage. Please review.

Project coverage is 80.46%. Comparing base (fe539d2) to head (c57e219).

Files with missing lines	Patch %	Lines
src/libfuncs/bounded_int.rs	92.98%	4 Missing ⚠️
src/values.rs	70.00%	3 Missing ⚠️
src/libfuncs/bytes31.rs	88.88%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1125      +/-   ##
==========================================
+ Coverage   80.34%   80.46%   +0.11%     
==========================================
  Files         110      110              
  Lines       29498    29506       +8     
==========================================
+ Hits        23701    23742      +41     
+ Misses       5797     5764      -33

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

github-actions · 2025-02-28T13:25:07Z

Benchmarking results

Benchmark for program `dict_insert`

Open benchmarks

Command	Mean [s]	Min [s]	Max [s]	Relative
`Cairo-vm (Rust, Cairo 1)`	15.281 ± 0.082	15.211	15.469	3.16 ± 0.03
`cairo-native (embedded AOT)`	4.840 ± 0.043	4.761	4.905	1.00
`cairo-native (embedded JIT using LLVM's ORC Engine)`	4.928 ± 0.038	4.890	5.017	1.02 ± 0.01

Benchmark for program `dict_snapshot`

Open benchmarks

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`Cairo-vm (Rust, Cairo 1)`	673.2 ± 7.0	663.6	686.8	1.00
`cairo-native (embedded AOT)`	4646.8 ± 37.6	4593.4	4723.2	6.90 ± 0.09
`cairo-native (embedded JIT using LLVM's ORC Engine)`	4776.5 ± 43.2	4706.2	4827.0	7.10 ± 0.10

Benchmark for program `factorial_2M`

Open benchmarks

Command	Mean [s]	Min [s]	Max [s]	Relative
`Cairo-vm (Rust, Cairo 1)`	9.295 ± 0.038	9.211	9.352	1.83 ± 0.02
`cairo-native (embedded AOT)`	5.104 ± 0.024	5.073	5.137	1.01 ± 0.01
`cairo-native (embedded JIT using LLVM's ORC Engine)`	5.076 ± 0.046	4.983	5.151	1.00

Benchmark for program `fib_2M`

Open benchmarks

Command	Mean [s]	Min [s]	Max [s]	Relative
`Cairo-vm (Rust, Cairo 1)`	8.785 ± 0.032	8.740	8.838	1.95 ± 0.01
`cairo-native (embedded AOT)`	4.506 ± 0.022	4.469	4.543	1.00
`cairo-native (embedded JIT using LLVM's ORC Engine)`	4.580 ± 0.041	4.503	4.633	1.02 ± 0.01

Benchmark for program `linear_search`

Open benchmarks

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`Cairo-vm (Rust, Cairo 1)`	692.7 ± 16.7	671.1	723.5	1.00
`cairo-native (embedded AOT)`	4620.0 ± 26.5	4577.6	4679.5	6.67 ± 0.17
`cairo-native (embedded JIT using LLVM's ORC Engine)`	4763.8 ± 30.3	4739.5	4830.3	6.88 ± 0.17

Benchmark for program `logistic_map`

Open benchmarks

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`Cairo-vm (Rust, Cairo 1)`	421.3 ± 2.7	418.4	427.4	1.00
`cairo-native (embedded AOT)`	4719.3 ± 31.2	4662.8	4762.0	11.20 ± 0.10
`cairo-native (embedded JIT using LLVM's ORC Engine)`	4889.0 ± 47.0	4843.6	4973.7	11.60 ± 0.13

JulianGCalderon · 2025-03-10T18:08:03Z

src/values.rs

+        match ty {
+            CoreTypeConcrete::Box(info)
+            | CoreTypeConcrete::NonZero(info)
+            | CoreTypeConcrete::Nullable(info)
+            | CoreTypeConcrete::Snapshot(info) => {
+                return self.to_ptr(arena, registry, &info.ty, find_dict_drop_override);
+            }
+            _ => {}
+        }
+


I don't fully understand this: if we try to call to_ptr with a Box<uint32>, then this function would return a pointer to an uint32, and not a double pointer to the uint32 (which is the correct type), right?

At that point, the value being transformed to a pointer is a Value from the jit, it has no knowledge it is a pointer. I think @azteca1998 or @edg-l can give a better answer.

At a glance, the match should work for NonZero and Snapshot, but should fail for Box and Nullable.

Both NonZero and Snapshot have the same internal representation as the original type, but Box and Nullable are heap pointers, so we need to allocate and initialize them.
Additionally, Nullable should handle the Value::Null variant.

Your approach is not wrong, but it's missing those parts.

mmm, both box and nullable tests are passing without any failures. Perhaps this is because the values are not really pointers since they are passed as an arguments through the test. Does this have an impact outside of tests? If not maybe I should leave it only for tests.

Right now It will only be used for tests, but its better to implement it properly in case we need it later, right? If not we could be leaving bugs that show up sooner or later.

src/libfuncs/bytes31.rs

src/libfuncs/bounded_int.rs

JulianGCalderon · 2025-03-10T18:30:55Z

src/values.rs

-                CoreTypeConcrete::Uint128MulGuarantee(_) => {
-                    native_panic!("todo: implement uint128mulguarantee from_ptr")
-                }
+                CoreTypeConcrete::Uint128MulGuarantee(_) => Self::Null,


As Null is only for nullables, i'd add a new variant Unit that could be used in these situations.

src/libfuncs/cast.rs

…ve into tests-to-sierra-1

FrancoGiachetta added 2 commits February 28, 2025 09:37

some tests to sierra

05b25d2

fix test

f066443

FrancoGiachetta marked this pull request as ready for review February 28, 2025 13:49

FrancoGiachetta requested review from edg-l, igaray, azteca1998, jrchatruc, entropidelic, fmoletta, Oppen, pefontana and gabrielbosio as code owners February 28, 2025 13:49

clean cast.rs

a3502c9

edg-l previously approved these changes Feb 28, 2025

View reviewed changes

Merge branch 'main' into tests-to-sierra-1

c5eb0c5

azteca1998 previously approved these changes Mar 4, 2025

View reviewed changes

edg-l added this pull request to the merge queue Mar 5, 2025

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Mar 5, 2025

edg-l added this pull request to the merge queue Mar 5, 2025

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Mar 5, 2025

FrancoGiachetta added 2 commits March 6, 2025 16:42

Merge branch 'main' into tests-to-sierra-1

10cfd4a

fix clippy

bc11b41

FrancoGiachetta dismissed stale reviews from edg-l and azteca1998 via bc11b41 March 7, 2025 14:11

Merge branch 'main' into tests-to-sierra-1

96a0a53

JulianGCalderon reviewed Mar 10, 2025

View reviewed changes

src/libfuncs/bytes31.rs Outdated Show resolved Hide resolved

JulianGCalderon reviewed Mar 10, 2025

View reviewed changes

src/libfuncs/bounded_int.rs Outdated Show resolved Hide resolved

JulianGCalderon reviewed Mar 10, 2025

View reviewed changes

src/libfuncs/cast.rs Outdated Show resolved Hide resolved

FrancoGiachetta added 4 commits March 10, 2025 16:07

reviews

1243c32

remove cairo program

b2abe2c

fix casts tests and clippy

377cf01

Merge branch 'main' into tests-to-sierra-1

c0362e3

gabrielbosio linked an issue Mar 28, 2025 that may be closed by this pull request

Use SierraGenerator for testing #1152

Open

JulianGCalderon assigned FrancoGiachetta Mar 31, 2025

FrancoGiachetta added 2 commits April 3, 2025 13:00

merge main

6f984dd

Merge branch 'tests-to-sierra-1' of github.com:lambdaclass/cairo_nati…

c57e219

…ve into tests-to-sierra-1

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Use SierraGenerator for testing (part 1) #1125

Use SierraGenerator for testing (part 1) #1125

Uh oh!

FrancoGiachetta commented Feb 28, 2025 •

edited by JulianGCalderon

Loading

Uh oh!

github-actions bot commented Feb 28, 2025 •

edited

Loading

Uh oh!

codecov-commenter commented Feb 28, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Feb 28, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

JulianGCalderon Mar 10, 2025 •

edited

Loading

Uh oh!

FrancoGiachetta Mar 10, 2025

Uh oh!

azteca1998 Mar 11, 2025 •

edited

Loading

Uh oh!

FrancoGiachetta Mar 11, 2025

Uh oh!

JulianGCalderon Apr 7, 2025

Uh oh!

Uh oh!

Uh oh!

JulianGCalderon Mar 10, 2025

Uh oh!

Uh oh!

Uh oh!

Use SierraGenerator for testing (part 1) #1125

Are you sure you want to change the base?

Use SierraGenerator for testing (part 1) #1125

Uh oh!

Conversation

FrancoGiachetta commented Feb 28, 2025 • edited by JulianGCalderon Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

github-actions bot commented Feb 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmark results Main vs HEAD.

Base

Head

Base

Head

Base

Head

Base

Head

Base

Head

Base

Head

Uh oh!

codecov-commenter commented Feb 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions bot commented Feb 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarking results

Benchmark for program dict_insert

Benchmark for program dict_snapshot

Benchmark for program factorial_2M

Benchmark for program fib_2M

Benchmark for program linear_search

Benchmark for program logistic_map

Uh oh!

Uh oh!

Uh oh!

JulianGCalderon Mar 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FrancoGiachetta Mar 10, 2025

Choose a reason for hiding this comment

Uh oh!

azteca1998 Mar 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FrancoGiachetta Mar 11, 2025

Choose a reason for hiding this comment

Uh oh!

JulianGCalderon Apr 7, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

JulianGCalderon Mar 10, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

FrancoGiachetta commented Feb 28, 2025 •

edited by JulianGCalderon

Loading

github-actions bot commented Feb 28, 2025 •

edited

Loading

codecov-commenter commented Feb 28, 2025 •

edited

Loading

github-actions bot commented Feb 28, 2025 •

edited

Loading

Benchmark for program `dict_insert`

Benchmark for program `dict_snapshot`

Benchmark for program `factorial_2M`

Benchmark for program `fib_2M`

Benchmark for program `linear_search`

Benchmark for program `logistic_map`

JulianGCalderon Mar 10, 2025 •

edited

Loading

azteca1998 Mar 11, 2025 •

edited

Loading