Rust edition upgrade, DX fix for generated token macros (#2)

# Breaking changes

## Generated token macros are no longer `#[macro_export]`

The `Token` derive macro generates a `macro_rules!` macro for each of the enum
variants. E.g., for Lox's `Token` enum in `examples/lox`:

```rs
// examples/lox/src/tokens.rs

#[derive(DebugLispToken, PartialEq, Token, Lexer)]
pub enum Token {
    #[subset_of(Ident)]
    #[pattern = "and|class|else|false|for|fun|if|nil|or|print|return|super|this|true|var|while"]
    Keyword(Substr, Span),

    #[pattern = "[a-zA-Z_][a-zA-Z0-9_]*"]
    Ident(Substr, Span),

    #[pattern = r"[(){}]"]
    Brace(Substr, Span),

    #[pattern = "[,.;]"]
    Punct(Substr, Span),

    #[pattern = "[=!<>]=?"]
    #[pattern = "[-+*/]"]
    Operator(Substr, Span),

    #[pattern = "[0-9]+"]
    NumLit(Substr, Span),

    #[pattern = r#""[^"]*""#]
    StrLit(Substr, Span),
}
```

We get `macro_rules!` macros named `keyword`, `ident`, `brace`, `punct`,
`operator`, `num_lit` and `str_lit`. These are mostly useful for the `Parse`
implementations, e.g.:

```rs
// examples/lox/src/decl.rs

impl Parse for ClassDecl {
    type Stream = ParseStream;

    fn parse(input: &mut Self::Stream) -> Result<Self> {
        // ...
        let superclass = if input.check(operator![<]) {
            input.consume(operator![<])?;
            Some(input.consume_kind(TokenKind::Ident)?)
        } else {
            None
        };
        // ...
    }
}
```

Previously, those generated macros were declared like this:

```rs
#(
    #[macro_export]
    macro_rules! #snake_case_variant_ident {
        // ...
    }
)*
```

That has always [caused some headaches](rust-lang/rust#52234 (comment))
which I was never really sure how to deal with. In the examples here, and in my
consuming projects, I had resorted to using `*` imports everywhere to work
around the problem (in hindsight though, I think that likely just hides the lint
by obscuring what we're doing instead of actually addressing the issue).

This PR attempts an alternative solution. `#[macro_export]` is intended for
libraries to expose macros to external consumers, but I don't foresee the macros
generated by the `Token` derive being actually useful in that context. So
instead, the generated macros are now declared like this:

```rs
#(
    macro_rules! #snake_case_variant_ident {
        // ...
    }
    pub(crate) use #snake_case_variant_ident;
)*
```

This is a breaking change for two reasons:

1. If you were depending on those `#[macro_export]` attributes to make the
   generated macros available to external consumers of your library, that is no
   longer going to work. Again, I don't imagine this was actually a real-world
   use case for anyone, but I've been wrong before! Let me know if this is a
   problem for you and I'll see what we can do about it.

2. If you had been importing those macros from the root of your crate, but your
   `Token` enum is _not_ declared in the crate root, you'll need to update your
   import paths to instead import them from the module where the enum is
   declared. E.g.:
  
   ```rs
   mod token {
       use gramatika::{DebugLisp, Span, Substr, Token as _};
   
       #[derive(DebugLispToken, PartialEq, Token, Lexer)]
       pub enum Token {
           #[pattern = ".+"]
           Word(Substr, Span),
       }
   }
   
   mod foo {
       // use crate::word;     // 👎
       use crate::token::word; // 👍
   }
   ```
  
   On the bright side, tools like `rust-analyzer` should now find and
   automatically suggest the correct import paths for those macros, so the
   fastest way to migrate will probably be to just delete your existing `use`
   statement and invoke your editor's suggestion feature to re-import any
   unresolved symbols from their correct paths.

# Other changes

* Updated the Rust edition to 2021 and fixed any resulting errors
* Fixed any new `clippy` lints as a result of upgrading my environment to
  v1.77.2
* Performed some low-hanging-fruit dependency upgrades.
  
  `regex-automata` and `syn` are still out of date for now &mdash; I attempted
  to update the former, but couldn't figure out how to migrate after ~10 minutes
  of poking around, and unfortunately I have other priorities that need to take
  precedence. Didn't even attempt `syn` because it's a major version bump, and
  that crate is basically the backbone of this whole project, so it'll have to
  wait for now.

Author: dannymcgee

crates/gramatika-macro/Cargo.toml

@@ -1,7 +1,7 @@
 [package]
 name = "gramatika-macro"
-version = "0.4.3"
-edition = "2018"
+version = "0.5.0"
+edition = "2021"
 authors = ["Danny McGee <[email protected]>"]
 license = "MIT OR Apache-2.0"
 readme = "../../README.md"
@@ -18,8 +18,8 @@ path = "tests/tests.rs"
 [dependencies]
 anyhow = "1"
 arcstr = "1.1"
-convert_case = "0.4"
-itertools = "0.10"
+convert_case = "0.6"
+itertools = "0.12"
 once_cell = "1"
 proc-macro2 = "1.0"
 quote = "1.0"

crates/gramatika-macro/src/common.rs

@@ -29,12 +29,12 @@ impl VariantIdents {
 					| "for" | "if" | "impl"
 					| "in" | "let" | "loop"
 					| "macro" | "match" | "mod"
-					| "move" | "mut" | "pub"
-					| "ref" | "return" | "static"
-					| "struct" | "super" | "trait"
-					| "type" | "union" | "unsafe"
-					| "use" | "where" | "while"
-					| "yield"
+					| "move" | "mut" | "path"
+					| "pub" | "ref" | "return"
+					| "static" | "struct"
+					| "super" | "trait" | "type"
+					| "union" | "unsafe" | "use"
+					| "where" | "while" | "yield"
 			) {
 				format_ident!("{}_", snake)
 			} else {

crates/gramatika-macro/src/debug_lisp.rs

@@ -43,15 +43,15 @@ fn derive_debug_struct(
 	};
 
 	let stream = quote! {
-		impl#generics ::gramatika::DebugLisp for #ident#generics {
+		impl #generics ::gramatika::DebugLisp for #ident #generics {
 			fn fmt(&self, f: &mut ::core::fmt::Formatter<'_>, indent: usize) -> ::core::fmt::Result {
 				::gramatika::DebugLispStruct::new(f, indent, stringify!(#ident))
 					#(#field_method_call)*
 					.finish()
 			}
 		}
 
-		impl#generics ::core::fmt::Debug for #ident#generics {
+		impl #generics ::core::fmt::Debug for #ident #generics {
 			fn fmt(&self, f: &mut ::core::fmt::Formatter<'_>) -> ::core::fmt::Result {
 				::gramatika::DebugLisp::fmt(self, f, 0)
 			}
@@ -63,22 +63,25 @@ fn derive_debug_struct(
 
 fn derive_debug_enum(ident: &Ident, generics: &Generics, data: &DataEnum) -> TokenStream {
 	let variant_name = data.variants.iter().map(|variant| &variant.ident);
-	let variant_inner_type = data
-		.variants
-		.iter()
-		.map(|variant| match &variant.fields {
-			Fields::Unnamed(fields) => fields.unnamed.iter().map(|field| &field.ty),
-			Fields::Named(_) => {
-				panic!("`#[derive(DebugLisp)]` is not supported for enum variants with named fields")
-			}
-			Fields::Unit => {
-				panic!("`#[derive(DebugLisp)]` is not supported for unit enum variants")
-			}
-		})
-		.flatten();
+	let variant_inner_type =
+		data.variants
+			.iter()
+			.flat_map(|variant| match &variant.fields {
+				Fields::Unnamed(fields) => fields.unnamed.iter().map(|field| &field.ty),
+				Fields::Named(_) => {
+					panic!(
+						"`#[derive(DebugLisp)]` is not supported for enum variants with named fields"
+					)
+				}
+				Fields::Unit => {
+					panic!(
+						"`#[derive(DebugLisp)]` is not supported for unit enum variants"
+					)
+				}
+			});
 
 	let stream = quote! {
-		impl#generics ::gramatika::DebugLisp for #ident#generics {
+		impl #generics ::gramatika::DebugLisp for #ident #generics {
 			fn fmt(&self, f: &mut ::std::fmt::Formatter<'_>, indent: usize) -> ::std::fmt::Result {
 				write!(f, "({}::", stringify!(#ident))?;
 
@@ -93,7 +96,7 @@ fn derive_debug_enum(ident: &Ident, generics: &Generics, data: &DataEnum) -> Tok
 			}
 		}
 
-		impl#generics ::core::fmt::Debug for #ident#generics {
+		impl #generics ::core::fmt::Debug for #ident #generics {
 			fn fmt(&self, f: &mut ::core::fmt::Formatter<'_>) -> ::core::fmt::Result {
 				::gramatika::DebugLisp::fmt(self, f, 0)
 			}
@@ -109,7 +112,7 @@ pub fn derive_token(input: TokenStream) -> TokenStream {
 	let generics = &ast.generics;
 
 	let stream = quote! {
-		impl#generics ::gramatika::DebugLisp for #ident#generics {
+		impl #generics ::gramatika::DebugLisp for #ident #generics {
 			fn fmt(&self, f: &mut ::std::fmt::Formatter<'_>, _: usize) -> ::std::fmt::Result {
 				write!(
 					f,

crates/gramatika-macro/src/lexer.rs

@@ -31,6 +31,7 @@ pub fn derive(input: TokenStream) -> TokenStream {
 			>>
 		}
 
+		#[allow(non_camel_case_types)]
 		type __TOKEN_CTOR = fn(::gramatika::Substr, ::gramatika::Span) -> #enum_ident;
 
 		impl ::gramatika::Lexer for #lexer_ident {

crates/gramatika-macro/src/token.rs

@@ -100,7 +100,7 @@ pub fn derive(input: pm::TokenStream) -> pm::TokenStream {
 			}
 		}
 
-		impl#generics #ident#generics {
+		impl #generics #ident #generics {
 			pub fn as_inner(&self) -> (::gramatika::Substr, ::gramatika::Span) {
 				match self {#(
 					Self::#variant_ident(lexeme, span) => (lexeme.clone(), *span)
@@ -115,7 +115,7 @@ pub fn derive(input: pm::TokenStream) -> pm::TokenStream {
 		}
 
 		#(
-			#[macro_export]
+			#[allow(unused_macros)]
 			macro_rules! #ctor_ident {
 				($lexeme:literal) => {
 					#ident::#ctor_ident(
@@ -130,9 +130,12 @@ pub fn derive(input: pm::TokenStream) -> pm::TokenStream {
 					)
 				};
 			}
+
+			#[allow(unused)]
+			pub(crate) use #ctor_ident;
 		)*
 
-		impl#generics ::gramatika::Token for #ident#generics {
+		impl #generics ::gramatika::Token for #ident #generics {
 			type Kind = #kind_ident;
 
 			fn lexeme(&self) -> ::gramatika::Substr {
@@ -156,13 +159,13 @@ pub fn derive(input: pm::TokenStream) -> pm::TokenStream {
 			}
 		}
 
-		impl#generics ::gramatika::Spanned for #ident#generics {
+		impl #generics ::gramatika::Spanned for #ident #generics {
 			fn span(&self) -> ::gramatika::Span {
 				self.as_inner().1
 			}
 		}
 
-		impl#generics Clone for #ident#generics {
+		impl #generics Clone for #ident #generics {
 			fn clone(&self) -> Self {
 				match self {#(
 					#ident::#variant_ident(lexeme, span) => #ident::#variant_ident(lexeme.clone(), *span)

crates/gramatika-macro/src/traversal/codegen.rs

@@ -21,7 +21,7 @@ impl ToTokens for VisitorDef {
 		});
 
 		if let Some(Walk) = &self.walker_ident {
-			if let Some(signature) = self.signatures.get(0) {
+			if let Some(signature) = self.signatures.first() {
 				let swn = signature.param_type.ownership;
 				let pwn = self.receiver.ownership;
 				let walk = format_ident!("{}", Walk.to_string().to_case(Case::Snake));

crates/gramatika/Cargo.toml

@@ -1,7 +1,7 @@
 [package]
 name = "gramatika"
-version = "0.4.3"
-edition = "2018"
+version = "0.5.0"
+edition = "2021"
 authors = ["Danny McGee <[email protected]>"]
 license = "MIT OR Apache-2.0"
 readme = "../../README.md"
@@ -17,7 +17,7 @@ once_cell = { version = "1.8", optional = true }
 regex-automata = { version = "0.1", optional = true }
 
 [dependencies.gramatika-macro]
-version = "0.4.3"
+version = "0.5.0"
 path = "../gramatika-macro"
 optional = true
 

crates/gramatika/src/span.rs

@@ -24,19 +24,19 @@ pub struct Position {
 	pub character: usize,
 }
 
-impl PartialOrd for Position {
-	fn partial_cmp(&self, other: &Self) -> Option<Ordering> {
+impl Ord for Position {
+	fn cmp(&self, other: &Self) -> Ordering {
 		if self.line == other.line {
-			Some(self.character.cmp(&other.character))
+			self.character.cmp(&other.character)
 		} else {
-			Some(self.line.cmp(&other.line))
+			self.line.cmp(&other.line)
 		}
 	}
 }
 
-impl Ord for Position {
-	fn cmp(&self, other: &Self) -> Ordering {
-		self.partial_cmp(other).unwrap()
+impl PartialOrd for Position {
+	fn partial_cmp(&self, other: &Self) -> Option<Ordering> {
+		Some(self.cmp(other))
 	}
 }
 

examples/expand/src/main.rs

@@ -168,14 +168,9 @@ pub struct ExprStmt;
 
 pub enum Expr {}
 
-#[derive(Clone, Copy, PartialEq, Eq)]
+#[derive(Clone, Copy, Default, PartialEq, Eq)]
 pub enum FlowControl {
+	#[default]
 	Continue,
 	Break,
 }
-
-impl Default for FlowControl {
-	fn default() -> Self {
-		FlowControl::Continue
-	}
-}

examples/lox/src/decl.rs

@@ -1,6 +1,10 @@
 use gramatika::{Parse, ParseStreamer, Result, Spanned, SpannedError, Token as _};
 
-use crate::*;
+use crate::{
+	expr::{Expr, FunExpr},
+	tokens::{brace, operator, punct, Token, TokenKind},
+	ParseStream,
+};
 
 #[derive(DebugLisp)]
 pub enum Decl {