feat(evals): add bare React Native quickstart eval - #166
Draft
subhankarmaiti wants to merge 3 commits into
Draft
Conversation
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
The Expo and bare RN scaffolds now ship an AGENTS.md stating that no native toolchain is available and that npm run typecheck is the only verification needed. The framework renames it to whichever context file the active runner reads, so all four runners get it. Environment facts only — SDK API guidance stays in the skills so the no-skills configs keep measuring what the model knows on its own.
Two bugs made judge graders fail for reasons unrelated to agent output. The workspace walker reads every file as UTF-8, so the bare React Native scaffold fed the judge gradle-wrapper.jar and the launcher PNGs as mojibake — 115KB of a 172KB corpus, 5x over maxCodeChars. Every judge grader in that eval errored without the model being asked. Exclude binaries and generated native project files; .plist, .xml and .gradle stay in, since judges assert on Info.plist, AndroidManifest.xml and app/build.gradle. RN is now ~24K chars. Separately, an oversized corpus threw. That turns an infrastructure limit into a grader failure, so truncate with a visible notice instead. Also stop excluding .txt. Baseline mode writes extracted code to llm_response.txt, so the exclusion left every baseline judge grading an empty corpus and reporting "the code block is empty" for correct solutions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a
react_native_quickstarteval covering Auth0 integration in a bare (non-Expo) React Native app, complementing the existing Expo eval. The scaffold is a minimal RN 0.86.2 project — Barkbook login/logout stub, fullandroid/andios/projects, no test or lint tooling — verified to typecheck and bundle.What it exposes: 20 graders across L1–L5, including L4 checks that the agent wired the native callback config for both platforms —
auth0Domain/auth0SchememanifestPlaceholders inandroid/app/build.gradleandCFBundleURLTypesinInfo.plist— plus L2 checks that it didn't reach for the Expo config plugin orreact-native-app-auth.The Android graders are file-anchored rather than plain substring matches.
matchesruns against every workspace file concatenated into// FILE: <path>blocks, so an unanchoredmanifestPlaceholders[\s\S]*?auth0Domaincan straddle two files and pass on code that never touchedbuild.gradle: