Separate parser token inspection from consumption

CHANGELOG.md

@@ -38,6 +38,8 @@
 
 - Fix `reanalyze` reporting labels and variant cases of a re-exported type (`type y = x = {...}`) as dead in the editor. The re-export linking existed only in the batch pipeline, while the editor runs the reactive one; both are now the same pipeline. The `reanalyze -reactive` flag is gone with it, since analysis is always reactive. https://github.com/rescript-lang/rescript/issues/8647
 - Fix `reanalyze` reporting record labels reached through a record coercion as dead. The typed tree now keeps the source type of a coercion, so reading a label on the target counts as reading the source label of the same name. https://github.com/rescript-lang/rescript/issues/8643
+- Fix speculative parser lookahead suppressing syntax errors for malformed attributes and inline records in non-arrow external declarations. https://github.com/rescript-lang/rescript/pull/8633
+- Preserve list elements when recovering from unexpected delimiters, and report invalid type-argument parentheses at their opening. https://github.com/rescript-lang/rescript/pull/8633
 - Fix constant folding of pattern matches on unboxed variants whose payload overlaps a literal constructor, so inlined calls agree with runtime matching. Reject multi-argument unboxed constructors instead of crashing. https://github.com/rescript-lang/rescript/pull/8631
 - Fix escaped backticks and interpolation openers in backquoted `%raw`, `%ffi`, and `%re` payloads leaking into emitted JavaScript. https://github.com/rescript-lang/rescript/pull/8630
 - Fix the side-effect analysis treating bigint exponentiation and bounds-checked array and string reads as pure, which let dead-code elimination drop an unused one that throws: `let _ = 2n ** -1n` no longer raised. https://github.com/rescript-lang/rescript/pull/8617
@@ -81,6 +83,7 @@
 
 #### :house: Internal
 
+- Refactor parser token handling to separate inspection (`peek`/`peek2`) from consumption, removing Diamond mode and `prev_end_pos` bookkeeping and moving missing JSX prop recovery from an editor-analysis regex heuristic into the parser. https://github.com/rescript-lang/rescript/pull/8633
 - Developer playground: Make panes resizable with wrapping text. https://github.com/rescript-lang/rescript/pull/8628
 - Normalize Lambda terms where they are built: a match guard stays structured data until its fallthrough is known, and `apply` and `mk_builtin` go through the folding constructors. https://github.com/rescript-lang/rescript/pull/8615
 - Replace non-escaping local mutable blocks with scalar bindings when all uses are direct field accesses, generalizing reference unboxing to multi-field records and references captured by JavaScript closures. https://github.com/rescript-lang/rescript/pull/8617

analysis/src/completion_jsx.ml

@@ -284,32 +284,6 @@ type jsx_props = {
   children_start: (int * int) option;
 }
 
-(**
-<div muted= />
-
-This is a special case for JSX props, where the above code is parsed
-as <div muted=//, a regexp literal. We leverage that fact to trigger completion
-for the JSX prop value.
-
-This code is safe because we also check that the location of the expression is broken,
-which only happens when the expression is a parse error/not complete.
-*)
-let is_regexp_jsx_heuristic_expr expr =
-  match expr.Parsetree.pexp_desc with
-  | Pexp_extension
-      ( {txt = "re"},
-        PStr
-          [
-            {
-              pstr_desc =
-                Pstr_eval
-                  ({pexp_desc = Pexp_constant (Pconst_raw_source "//")}, _);
-            };
-          ] )
-    when expr.pexp_loc |> Loc.end_ = (Location.none |> Loc.end_) ->
-    true
-  | _ -> false
-
 let find_jsx_props_completable ~jsx_props ~end_pos ~pos_before_cursor
     ~first_char_before_cursor_no_white ~char_at_cursor ~pos_after_comp_name =
   let all_labels =
@@ -392,14 +366,9 @@ let find_jsx_props_completable ~jsx_props ~end_pos ~pos_before_cursor
       else if prop.exp.pexp_loc |> Loc.end_ = (Location.none |> Loc.end_) then (
         if Debug.verbose () then
           print_endline "[jsx_props_completable]--> Loc is broken";
-        if
-          Completion_expressions.is_expr_hole prop.exp
-          || is_regexp_jsx_heuristic_expr prop.exp
-        then (
+        if Completion_expressions.is_expr_hole prop.exp then (
           if Debug.verbose () then
-            print_endline
-              "[jsx_props_completable]--> Expr was expr hole or regexp literal \
-               heuristic";
+            print_endline "[jsx_props_completable]--> Expr was expr hole";
           Some
             (Cexpression
                {

compiler/jsoo/jsoo_playground_main.ml

@@ -280,8 +280,8 @@ module Res_driver = struct
         | _ as diagnostics -> (true, diagnostics)
       in
       {
-        filename = engine.scanner.filename;
-        source = engine.scanner.src;
+        filename = engine.filename;
+        source = engine.source;
         parsetree = structure;
         diagnostics;
         invalid;

compiler/syntax/ParserCursor.md

@@ -0,0 +1,110 @@
+# Parser token cursor
+
+The parser now separates token inspection from consumption inside `Res_parser`.
+`next` consumes the current token without scanning its successor; `make` does no
+scanning. This keeps the recursive-descent grammar and driver results while
+removing scanner modes and manual scanner restoration from grammar code.
+
+The implementation stays in the existing parser module. Most changes in
+`res_core.ml` replace field reads with `peek`, `start_pos`, and `end_pos`; the
+existing `next`, `expect`, and `optional` call sites retain their names.
+
+## Inspection and consumption
+
+```ocaml
+let p = Parser.make "let /* comment */ x = 1" "Example.res" in
+Parser.peek p;   (* Let; committed byte offset is still 0 *)
+Parser.peek2 p;  (* Lident "x"; comment is still pending *)
+Parser.next p;   (* consumes only Let; committed byte offset is 3 *)
+Parser.peek p;   (* reuses the cached Lident *)
+Parser.next p   (* consumes x and commits its leading comment once *)
+```
+
+A logical cursor records the consumed source position. Separate caches hold the
+current token and, when requested, one successor.
+The two slots have separate scanners so lookahead cannot advance the current
+token's physical boundary. Both slots are reused for the entire file; inspection
+does not allocate a new cursor or scanner. Each slot retains the scanner's
+immutable result tuple directly, avoiding separate writes to three mutable fields
+for each token. Checkpoints retain that same tuple. Simple arrow checks use
+`peek2` or reject impossible first tokens before taking a checkpoint.
+
+Comments and lexical diagnostics are pending until consumption. Reporting a
+grammar error first publishes pending lexical diagnostics, preserving their
+order. Diagnostic queries also include pending diagnostics to avoid duplicate
+string errors.
+
+`position` reads the logical cursor without scanning. AST boundaries, missing-token
+diagnostics, and recovery progress checks all use this position. Consumption reuses
+the token's immutable end position, so there is no separate previous-token position
+or byte offset to update and restore. Before the first consumption, the cursor is
+`Lexing.dummy_pos`, preserving existing location behavior. `finish` retains EOF
+trivia and publishes pending warnings without moving the cursor.
+
+## Contextual readers, without Diamond
+
+The scanner returns individual `>` tokens and individual `<` tokens unless the
+latter starts `<=`. In expression context, `peek_binary_operator` extends
+an adjacent prefix into `>=`, `>>`, `>>>`, or `<<`. It never joins tokens across
+whitespace or comments. Balanced lookahead uses the same query, keeping `>`
+separate when it is the expected generic closer. Type arguments need no mode
+stack:
+
+```rescript
+let value: array<option<int>>= [Some(1)]
+let shifted = value >>> count
+```
+
+Regex reading restarts at the opening slash of `/` or `/.`, so the grammar no
+longer reconstructs a missing dot. Template reading consumes the current opening
+backtick or interpolation delimiter before reading raw text. Both readers discard
+ordinary lookahead; raw text may already have been provisionally read as code.
+
+## Speculation
+
+`lookahead` always rolls back. `try_parse` commits `Some result` and rolls back
+`None`; both restore on exceptions. One checkpoint implementation owns scanner
+positions, cached token data, comments, diagnostics, breadcrumbs, committed
+position, recovery regions, and pending warnings.
+
+```ocaml
+Parser.try_parse p (fun p ->
+  let attrs = parse_attributes p in
+  if Parser.peek p = And then Some attrs else None)
+```
+
+Recovery regions use persistent values rather than shared mutable refs. A failed
+probe cannot suppress a later real error. In particular, an unquoted record field
+in a non-arrow external type now reports the existing forbidden-inline-record
+error; the valid object-type fixtures use quoted fields explicitly.
+
+## List recovery
+
+Comma-separated lists share one parser, with a reversal at the boundary for
+callers that need source order. Recovery of unexpected `()`, `[]`, `{}`, or `<>`
+groups uses that same list grammar recursively, retaining recoverable elements
+and consuming the group's own closer. Closers belonging to an enclosing recovery
+group remain available to it; stray closers do not end the list. The enclosing
+closers are tracked only during error recovery, with an empty list on the normal
+path. This replaces the type-argument-specific rule that discarded extra `<`
+tokens without accounting for their closing `>`.
+
+Type arguments report an invalid opening `(` before parsing its contents and
+recover through the matching `)`. The existing diagnostic region suppresses
+secondary errors within that declaration. When displayed, the diagnostic uses
+the existing type printer to show the constructor with its recovered arguments,
+such as `Nullable.t<'a>`. Formatting is deferred until the message is requested.
+No scanner mode or additional persistent recovery state is needed.
+
+## Validation
+
+Cursor unit tests cover lazy consumption, cached lookahead, trivia, diagnostics,
+EOF, nested rollback, exceptions, raw readers, and UTF-16/CRLF positions. Syntax
+and runtime fixtures exercise nested generic closers, shifts, regex prefixes, and
+template interpolation. Runtime compilation also runs the Lambda invariant check.
+
+Compare parser ASTs and locations, syntax snapshots, round trips, the full test
+suite, and the playground build against the same upstream revision. Benchmark
+saved release binaries on identical real and synthetic inputs, alternating their
+order; measure both elapsed parsing time and allocations. Earlier Diamond-only
+PoC measurements do not establish the performance of this cursor implementation.