Skip to content

Fallback Ident is not normalized #500

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
kennytm opened this issue Apr 17, 2025 · 0 comments
Open

Fallback Ident is not normalized #500

kennytm opened this issue Apr 17, 2025 · 0 comments

Comments

@kennytm
Copy link

kennytm commented Apr 17, 2025

Starting from Rust 1.42.0 (rust-lang/rust#66670) Unicode identifiers are normalized in NFC form, that means the following proc-macro will produce true:

extern crate proc_macro;
use proc_macro::{Ident, Span, TokenStream};

#[proc_macro]
pub fn check(_: TokenStream) -> TokenStream {
    let decomposed = Ident::new("e\u{301}", Span::call_site());
    let composed = Ident::new("\u{e9}", Span::call_site());
    (composed.to_string() == decomposed.to_string()).to_string().parse().unwrap()
}

However, the proc_macro2 fallback does not perform NFC normalization, meaning the equivalent code in proc_macro2 will produce false:

use proc_macro2::{Ident, Span, TokenStream};

pub fn check(_: TokenStream) -> TokenStream {
    let decomposed = Ident::new("e\u{301}", Span::call_site());
    let composed = Ident::new("\u{e9}", Span::call_site());
    (composed.to_string() == decomposed.to_string()).to_string().parse().unwrap()
}

fn main() {
    dbg!(check(TokenStream::default()));
    // Ident { sym: false, span: bytes(1..6) }
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant