Fix hygiene issues in declare_program! and declare_loader! (bp #10905) (#11073)

* Fix hygiene issues in `declare_program!` and `declare_loader!`

The `declare_program!` and `declare_loader!` macros both expand to
new macro definitions (based on the `$name` argument). These 'inner'
macros make use of the special `$crate` metavariable to access items in
the crate where the 'inner' macros is defined.

However, this only works due to a bug in rustc. When a macro is
expanded, all `$crate` tokens in its output are 'marked' as being
resolved in the defining crate of that macro. An inner macro (including
the body of its arms) is 'just' another set of tokens that appears in
the body of the outer macro, so any `$crate` identifiers used there are
resolved relative to the 'outer' macro.

For example, consider the following code:

```rust
macro_rules! outer {
    () => {
        macro_rules! inner {
            () => {
                $crate::Foo
            }
        }
    }
}
```

The path `$crate::Foo` will be resolved relative to the crate that defines `outer`,
**not** the crate which defines `inner`.

However, rustc currently loses this extra resolution information
(referred to as 'hygiene' information) when a crate is serialized.
In the above example, this means that the macro `inner` (which gets
defined in whatever crate invokes `outer!`) will behave differently
depending on which crate it is invoked from:

When `inner` is invoked from the same crate in which it is defined,
the hygiene information will still be available,
which will cause `$crate::Foo` to be resolved in the crate which defines 'outer'.

When `inner` is invoked from a different crate, it will be loaded from
the metadata of the crate which defines 'inner'. Since the hygiene
information is currently lost, rust will 'forget' that `$crate::Foo` is
supposed to be resolved in the context of 'outer'. Instead, it will be
resolved relative to the crate which defines 'inner', which can cause
incorrect code to compile.

This bug will soon be fixed in rust (see rust-lang/rust#72121),
which will break `declare_program!` and `declare_loader!`. Fortunately,
it's possible to obtain the desired behavior (`$crate` resolving in the
context of the 'inner' macro) by use of a procedural macro.

This commit adds a `respan!` proc-macro to the `sdk/macro` crate.
Using the newly-stabilized (on Nightly) `Span::resolved_at` method,
the `$crate` identifier can be made to be resolved in the context of the
proper crate.

Since `Span::resolved_at` is only stable on the latest nightly,
referencing it on an earlier version of Rust will cause a compilation error.
This requires the `rustversion` crate to be used, which allows conditionally
compiling code epending on the Rust compiler version in use. Since this method is already
stabilized in the latest nightly, there will never be a situation where
the hygiene bug is fixed (e.g. rust-lang/rust#72121)
is merged but we are unable to call `Span::resolved_at`.

(cherry picked from commit 05445c7)

# Conflicts:
#	Cargo.lock
#	sdk/Cargo.toml

* Replace FIXME with an issue link

(cherry picked from commit b0cb2b0)

* Update lock files

(cherry picked from commit 42f8848)

# Conflicts:
#	programs/bpf/Cargo.lock
#	programs/librapay/Cargo.lock
#	programs/move_loader/Cargo.lock

* Split comment over multiple lines

Due to rust-lang/rustfmt#4325, leaving this as
one line causes rustfmt to add extra indentation to the surrounding
code.

(cherry picked from commit fed69e9)

* Fix clippy lints

(cherry picked from commit e7387f6)

* Apply #![feature(proc_macro_hygiene)] when needed

This allows the rust-bpf-builder toolchain to build the sdk

(cherry picked from commit 95490ff)

# Conflicts:
#	sdk/build.rs
#	sdk/src/lib.rs

* Update Cargo.toml

* Update lib.rs

* Add rustc_version

* lock file updates

Co-authored-by: Aaron Hill <[email protected]>
Co-authored-by: Jack May <[email protected]>
Co-authored-by: Michael Vines <[email protected]>

Author: 4 people

Cargo.lock

@@ -23,6 +23,7 @@ thiserror = "1.0"
 
 [dev-dependencies]
 rand = "0.7.3"
+rustversion = "1.0.3"
 
 [lib]
 crate-type = ["lib", "cdylib"]

programs/bpf/Cargo.lock

@@ -297,6 +297,14 @@ mod tests {
         }
     }
 
+    #[rustversion::since(1.46.0)]
+    #[test]
+    fn test_bpf_loader_same_crate() {
+        // Ensure that we can invoke this macro from the same crate
+        // where it is defined.
+        solana_bpf_loader_program!();
+    }
+
     #[test]
     #[should_panic(expected = "ExceededMaxInstructions(10)")]
     fn test_bpf_loader_non_terminating_program() {

programs/bpf_loader/Cargo.toml

@@ -54,9 +54,13 @@ ed25519-dalek = { version = "=1.0.0-pre.3", optional = true }
 solana-crate-features = { path = "../crate-features", version = "1.2.11", optional = true }
 solana-logger = { path = "../logger", version = "1.2.11", optional = true }
 solana-sdk-macro = { path = "macro", version = "1.2.11" }
+rustversion = "1.0.3"
 
 [dev-dependencies]
 tiny-bip39 = "0.7.0"
 
 [package.metadata.docs.rs]
 targets = ["x86_64-unknown-linux-gnu"]
+
+[build-dependencies]
+rustc_version = "0.2"

programs/bpf_loader/src/lib.rs

@@ -0,0 +1,27 @@
+extern crate rustc_version;
+use rustc_version::{version_meta, Channel};
+
+fn main() {
+    // Copied and adapted from
+    // https://github.com/Kimundi/rustc-version-rs/blob/1d692a965f4e48a8cb72e82cda953107c0d22f47/README.md#example
+    // Licensed under Apache-2.0 + MIT
+    match version_meta().unwrap().channel {
+        Channel::Stable => {
+            println!("cargo:rustc-cfg=RUSTC_WITHOUT_SPECIALIZATION");
+        }
+        Channel::Beta => {
+            println!("cargo:rustc-cfg=RUSTC_WITHOUT_SPECIALIZATION");
+        }
+        Channel::Nightly => {
+            println!("cargo:rustc-cfg=RUSTC_WITH_SPECIALIZATION");
+        }
+        Channel::Dev => {
+            println!("cargo:rustc-cfg=RUSTC_WITH_SPECIALIZATION");
+            // See https://github.com/solana-labs/solana/issues/11055
+            // We may be running the custom `rust-bpf-builder` toolchain,
+            // which currently needs `#![feature(proc_macro_hygiene)]` to
+            // be applied.
+            println!("cargo:rustc-cfg=RUSTC_NEEDS_PROC_MACRO_HYGIENE");
+        }
+    }
+}

programs/librapay/Cargo.lock

@@ -16,6 +16,7 @@ bs58 = "0.3.0"
 proc-macro2 = "1.0"
 quote = "1.0"
 syn = { version = "1.0", features = ["full", "extra-traits"] }
+rustversion = "1.0.3"
 
 [package.metadata.docs.rs]
 targets = ["x86_64-unknown-linux-gnu"]

programs/move_loader/Cargo.lock

@@ -5,7 +5,7 @@
 extern crate proc_macro;
 
 use proc_macro::TokenStream;
-use proc_macro2::Span;
+use proc_macro2::{Delimiter, Span, TokenTree};
 use quote::{quote, ToTokens};
 use std::convert::TryFrom;
 use syn::{
@@ -14,7 +14,7 @@ use syn::{
     parse_macro_input,
     punctuated::Punctuated,
     token::Bracket,
-    Expr, Ident, LitByte, LitStr, Token,
+    Expr, Ident, LitByte, LitStr, Path, Token,
 };
 
 struct Id(proc_macro2::TokenStream);
@@ -63,6 +63,75 @@ impl ToTokens for Id {
     }
 }
 
+#[allow(dead_code)] // `respan` may be compiled out
+struct RespanInput {
+    to_respan: Path,
+    respan_using: Span,
+}
+
+impl Parse for RespanInput {
+    fn parse(input: ParseStream) -> Result<Self> {
+        let to_respan: Path = input.parse()?;
+        let _comma: Token![,] = input.parse()?;
+        let respan_tree: TokenTree = input.parse()?;
+        match respan_tree {
+            TokenTree::Group(g) if g.delimiter() == Delimiter::None => {
+                let ident: Ident = syn::parse2(g.stream())?;
+                Ok(RespanInput {
+                    to_respan,
+                    respan_using: ident.span(),
+                })
+            }
+            val => Err(syn::Error::new_spanned(
+                val,
+                "expected None-delimited group",
+            )),
+        }
+    }
+}
+
+/// A proc-macro which respans the tokens in its first argument (a `Path`)
+/// to be resolved at the tokens of its second argument.
+/// For internal use only.
+///
+/// There must be exactly one comma in the input,
+/// which is used to separate the two arguments.
+/// The second argument should be exactly one token.
+///
+/// For example, `respan!($crate::foo, with_span)`
+/// produces the tokens `$crate::foo`, but resolved
+/// at the span of `with_span`.
+///
+/// The input to this function should be very short -
+/// its only purpose is to override the span of a token
+/// sequence containing `$crate`. For all other purposes,
+/// a more general proc-macro should be used.
+#[rustversion::since(1.46.0)] // `Span::resolved_at` is stable in 1.46.0 and above
+#[proc_macro]
+pub fn respan(input: TokenStream) -> TokenStream {
+    // Obtain the `Path` we are going to respan, and the ident
+    // whose span we will be using.
+    let RespanInput {
+        to_respan,
+        respan_using,
+    } = parse_macro_input!(input as RespanInput);
+    // Respan all of the tokens in the `Path`
+    let to_respan: proc_macro2::TokenStream = to_respan
+        .into_token_stream()
+        .into_iter()
+        .map(|mut t| {
+            // Combine the location of the token with the resolution behavior of `respan_using`
+            // Note: `proc_macro2::Span::resolved_at` is currently gated with cfg(procmacro2_semver_exempt)
+            // Once this gate is removed, we will no longer need to use 'unwrap()' to call
+            // the underling `proc_macro::Span::resolved_at` method.
+            let new_span: Span = t.span().unwrap().resolved_at(respan_using.unwrap()).into();
+            t.set_span(new_span);
+            t
+        })
+        .collect();
+    TokenStream::from(to_respan)
+}
+
 #[proc_macro]
 pub fn declare_id(input: TokenStream) -> TokenStream {
     let id = parse_macro_input!(input as Id);