Error originates in the attribute macro `subxt::subxt` for asset-hub-westend

[paulormart]

Hi @jsdw, I'm currently migrating some staking tooling for AH, but subxt is currently not compiling asset-hub-westend, please see below:

#![allow(missing_docs)]
use subxt::{OnlineClient, SubstrateConfig};

// subxt metadata --url wss://westend-asset-hub-rpc.polkadot.io -f bytes > artifacts/asset_hub_westend_metadata.scale
#[subxt::subxt(runtime_metadata_path = "../artifacts/asset_hub_westend_metadata.scale")]
pub mod polkadot {}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Create a client to use:
    let api = OnlineClient::<SubstrateConfig>::from_url("wss://westend-asset-hub-rpc.polkadot.io")
        .await?;

    // A query to obtain some contant:
    let constant_query = polkadot::constants().system().block_length();

    // Obtain the value:
    let value = api.constants().at(&constant_query)?;

    println!("Block length: {value:?}");
    Ok(())
}

error[E0277]: the trait bound `PagedExposureMetadata<_1>: subxt::ext::parity_scale_codec::Decode` is not satisfied
 --> subxt/examples/constants_static.rs:9:1
  |
9 | #[subxt::subxt(runtime_metadata_path = "../artifacts/asset_hub_westend_metadata.scale")]
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `subxt::ext::parity_scale_codec::Decode` is not implemented for `PagedExposureMetadata<_1>`
  |
  = note: this error originates in the attribute macro `subxt::subxt` (in Nightly builds, run with -Z macro-backtrace for more info)
help: consider extending the `where` clause, but there might be an alternative better way to express this requirement
  |
9 | #[subxt::subxt(runtime_metadata_path = "../artifacts/asset_hub_westend_metadata.scale")], PagedExposureMetadata<_1>: subxt::ext::parity_scale_codec::Decode
  |                                                                                         +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

error[E0277]: the trait bound `ExposurePage<_0, _1>: subxt::ext::parity_scale_codec::Decode` is not satisfied
 --> subxt/examples/constants_static.rs:9:1
  |
9 | #[subxt::subxt(runtime_metadata_path = "../artifacts/asset_hub_westend_metadata.scale")]
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `subxt::ext::parity_scale_codec::Decode` is not implemented for `ExposurePage<_0, _1>`
  |
  = note: this error originates in the attribute macro `subxt::subxt` (in Nightly builds, run with -Z macro-backtrace for more info)
help: consider extending the `where` clause, but there might be an alternative better way to express this requirement
  |
9 | #[subxt::subxt(runtime_metadata_path = "../artifacts/asset_hub_westend_metadata.scale")], ExposurePage<_0, _1>: subxt::ext::parity_scale_codec::Decode
  |                                                                                         ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

error[E0277]: the trait bound `PagedExposureMetadata<_1>: subxt::ext::parity_scale_codec::Encode` is not satisfied
 --> subxt/examples/constants_static.rs:9:1
  |
9 | #[subxt::subxt(runtime_metadata_path = "../artifacts/asset_hub_westend_metadata.scale")]
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `subxt::ext::parity_scale_codec::Encode` is not implemented for `PagedExposureMetadata<_1>`
  |
  = note: this error originates in the attribute macro `subxt::subxt` (in Nightly builds, run with -Z macro-backtrace for more info)
help: consider extending the `where` clause, but there might be an alternative better way to express this requirement
  |
9 | #[subxt::subxt(runtime_metadata_path = "../artifacts/asset_hub_westend_metadata.scale")], PagedExposureMetadata<_1>: subxt::ext::parity_scale_codec::Encode
  |                                                                                         +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

error[E0277]: the trait bound `ExposurePage<_0, _1>: subxt::ext::parity_scale_codec::Encode` is not satisfied
 --> subxt/examples/constants_static.rs:9:1
  |
9 | #[subxt::subxt(runtime_metadata_path = "../artifacts/asset_hub_westend_metadata.scale")]
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `subxt::ext::parity_scale_codec::Encode` is not implemented for `ExposurePage<_0, _1>`
  |
  = note: this error originates in the attribute macro `subxt::subxt` (in Nightly builds, run with -Z macro-backtrace for more info)
help: consider extending the `where` clause, but there might be an alternative better way to express this requirement
  |
9 | #[subxt::subxt(runtime_metadata_path = "../artifacts/asset_hub_westend_metadata.scale")], ExposurePage<_0, _1>: subxt::ext::parity_scale_codec::Encode
  |                                                                                         ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

For more information about this error, try `rustc --explain E0277`.
error: could not compile `subxt` (example "constants_static") due to 6 previous errors

[jsdw]

@paulormart I looked into this and found a partiy-scale-codec issue, which I'll dig more into and see how easily a fix can be produced!

[jsdw]

Ok, so I dug into the issue; see the comment I left over there.

TL;DR I don't think there's a reasonable way to fix this from the parity-scale-codec PoV. From the subxt PoV, the options are:

1. Use custom types to override the ones that cause a problem. This is bad UX, but in most cases is the workaround that works here. So in this case we can just duplicate the offending types, remove the #[codec(compact)] annotations that are present, acknowledge that Encode/Decode will no longer line up (which Subxt doesn't care much about anyway (it uses DecodeAsType and EncodeAsType, which understand how to handle Compact values anyway). Code below.
2. We could add something to Subxt to allow custom attrs and not just custom derives on types. You'd then have to add an attr like #[codec(encode_bound(_1: ::codec::HasCompact + ::codec::Encode))] or similar to types where the _1 generic ends up needing to be a compact compatible type.
3. We could figure out an approach in Subxt to recursively spot generics that need to ultimately be compact encoded, and apply the relevant trait bounds automatically to all generated types which use them. I expect this will be tricky to add to the existing type generation code but should be possible.
4. We could remove Decode/Encode bound generation entirely from Subxt. Subxt doesn't really need these bounds anyway since it relies on DecodeAsType and EncodeAsType for the most part, which can handle things like compact types automatically. This would lead to cleaner APIs and less fuss getting types compiling (but we'd need to see if/where Subxt still benefitted from Decode/Encode before making such a step).
5. I wonder whether we could tweak the staking interface to avoid these generic compact bounds in the first place.

For 1, the code can look something like this:

#![allow(missing_docs)]

#[subxt::subxt(
    runtime_metadata_insecure_url = "wss://westend-asset-hub-rpc.polkadot.io",
    substitute_type(path = "sp_staking::ExposurePage<A,B>", with = "crate::custom_types::ExposurePage<A,B>"),
    substitute_type(path = "sp_staking::IndividualExposure<A>", with = "crate::custom_types::IndividualExposure<A>"),
    substitute_type(path = "sp_staking::PagedExposureMetadata<A>", with = "crate::custom_types::PagedExposureMetadata<A>"),
)]  
pub mod polkadot {}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    Ok(())
}

mod custom_types {
    use codec::{Decode, Encode};
    use scale_encode::EncodeAsType;
    use scale_decode::DecodeAsType;

    #[derive(Decode, Encode, DecodeAsType, EncodeAsType, Debug)]
    pub struct ExposurePage<_0, _1> {
        pub page_total: _1,
        pub others: Vec<IndividualExposure<_0, _1>>,
    }

    #[derive(Decode, Encode, DecodeAsType, EncodeAsType, Debug)]
    pub struct IndividualExposure<_0, _1> {
        pub who: _0,
        pub value: _1,
    }

    #[derive(Decode, Encode, DecodeAsType, EncodeAsType, Debug)]
    pub struct PagedExposureMetadata<_0> {
        pub total: _0,
        pub own: _0,
        pub nominator_count: ::core::primitive::u32,
        pub page_count: ::core::primitive::u32,
    }
}

@paulormart I hope that workaround will suffice for now, and in the meantime I'll need to think on a more permanent solution. I would prefer 4 personally as that's the way I always intended Subxt to head, but if that's possible then I'd need to see how feasible 3 is!

[paulormart]

Many thanks for the feedback @jsdw, adding those custom types works well for me. Regarding point 5. I've also raised the question with the staking team.

[kianenigma]

I wonder whether we could tweak the staking interface to avoid these generic compact bounds in the first place.

I don't think there is a way around having HasCompact in eg.

pub struct Exposure<AccountId, Balance: HasCompact> {
	/// The total balance backing this validator.
	#[codec(compact)]
	pub total: Balance,
	/// The validator's own stash that is exposed.
	#[codec(compact)]
	pub own: Balance,
	/// The portions of nominators stashes that are exposed.
	pub others: Vec<IndividualExposure<AccountId, Balance>>,
}

If we want to use the #[codec(compact)] macro. How is this not an issue in other places in FRAME storage types that we use #[codec(compact)]? 🤔

[jsdw]

If we want to use the #[codec(compact)] macro. How is this not an issue in other places in FRAME storage types that we use #[codec(compact)]?

It's quite a specific/niche issue; for the issue to occur:

• The type with #[codec(compact)] on it must be a generic param (this means that derive will require the generic to have HasCompact).
• Another type must make use of that type but also pass a generic param to it (the derive here will not know that this generic type needs to impl HasCompact and will only expect it to have Encode, which is what leads to the error)
• The #[codec(dumb_trait_bounds)] attr is used (we added it because of: [0.34.0] the trait bound Vec<XYZ> : subxt::ext::parity_scale_codec::Decode is not satisfied #1603). This makes the derive work much like how most other derive macros do, and add bounds only to the generics of a type, allowing for the above. Otherwise the bounds work ok in this case.

In other words, this is an issue:

#[derive(Encode, Decode)]
#[codec(dumb_trait_bound)]
pub struct Inner<_0> {
    #[codec(compact)]
    pub total: _0,
}

#[derive(Encode, Decode)]
#[codec(dumb_trait_bound)]
pub struct Outer<_1> {
    // Generic type passed from outer to inner. Inner needs it to impl Encode + HasCompact,
    // but Outer doesn't know this and just expects it to impl Encode.
    exposure: Inner<_1>, 
}

But this works ok:

#[derive(Encode, Decode)]
#[codec(dumb_trait_bound)]
pub struct Inner<_0> {
    #[codec(compact)]
    pub total: _0,
}

#[derive(Encode, Decode)]
#[codec(dumb_trait_bound)]
pub struct Outer<_1> {
    // u64 impls HasCompact + Encode and _1 here will be expected to impl Encode. All is well
    exposure: Inner<u64>, 
    // ...
}

And ofc not having the generic go to #[codec(compact)] would work ok. Given that Compact<u16>, Compact<u32> etc all encode to the same representation, I imagine it's not a super common desire to pass a generic to it. Even if that was done before, the type may not have then been used from some other type which also passed through the generic. So I can see that this is something we may just have never hit before.

[ggwpez]

If we want to use the #[codec(compact)] macro. How is this not an issue in other places in FRAME storage types that we use #[codec(compact)]? 🤔

Alternatively we should be able to use the wrapper type instead of the macro like: pub total: Compact<Balance>.
Maybe that is more macro friendly...

[jsdw]

Alternatively we should be able to use the wrapper type instead of the macro like: pub total: Compact.
Maybe that is more macro friendly...

#[derive(Encode, Decode)]
#[codec(dumb_trait_bound)]
pub struct Inner<_0> {
    pub total: Compact<_0>,
}

Do you mean like this?

Whether or not the macro is used, _0 must still be a type that can be compact encoded in order for Compact<_0> to impl Encode, and so we'd hit the same issue that struct Outer<_1> { exposure: Inner<_1> } would not know to add the relevant extra bounds to _1, no?

[jsdw]

@paulormart with #2008 now merged, would you be able to try the latest master of Subxt and see if this resolves your issue? In my testing, the codegen now works OK and you don't need any weird workarounds :)

[paulormart]

@jsdw looking into this. Yes, from branch master it works well and solves the initial issue raised here.

However it comes with some side effects when decoding extrinsics -- at least on the one i'm currently tracking. I'm keen to open a different issue, but wanted to leave it here in case you have some nice solution/workaround. An example below to be easily reproducible:

#![allow(missing_docs)]
use subxt::{OnlineClient, PolkadotConfig};

#[subxt::subxt(runtime_metadata_path = "../artifacts/westend_metadata.scale")]
pub mod node_runtime {}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Create a client that subscribes to blocks of the Polkadot network.
    let api = OnlineClient::<PolkadotConfig>::from_url("wss://westend-rpc.polkadot.io").await?;

    let mut blocks_sub = api.blocks().subscribe_finalized().await?;

    while let Some(block) = blocks_sub.next().await {
        let block = block?;
        let block_number = block.header().number;
        let block_hash = block.hash();
        println!("Block #{block_number}  ({block_hash}):");

        let extrinsics = block.extrinsics().await?;
        for extrinsic in extrinsics.find::<node_runtime::para_inherent::calls::types::Enter>() {
            let extrinsic = extrinsic?;

            for availability_bitfield in extrinsic.value.data.bitfields.iter() {
                println!(
                    "    validator_index: {:?}",
                    availability_bitfield.validator_index
                );
            }
        }
    }

    Ok(())
}

Response:

Block #26413894  (0x3443…9e92):
Error: Decode(Error { context: Context { path: [Location { inner: Index(0) }, Location { inner: Field("bitfields") }, Location { inner: Field("data") }] }, kind: CannotFindField { name: "__ignore" } })

[jsdw]

Thankyou! I opened a new issue for that; sounds sortof unrelated to me offhand; what may have happened is that the above change made visible a new issue but need to look into what it is!