Refactor `substrate-test-runtime`

[bkchr]

The substrate-test-runtime is some special kind of runtime that incorporates some FRAME stuff and some custom logic.
As the name suggests, it is used for testing. The idea here being to rewrite the test runtime into some "pure" FRAME runtime and to remove other cruft. The runtime contains a lot of stuff for testing sp-api behavior in wasm vs native. Most of this stuff can probably be removed as we will gonna deprecate the native runtime in the future. This means that we can remove this huge cfg-if block. Then the runtime contains the custom extrinsic. These are all the calls this test runtime supports. They are being used all over the code base to test different things. These calls need to rewritten as some FRAME pallet. This means we there should be written one pallet as part of this runtime that exposes the same kind of calls, but through the "FRAME abstraction". All the code that is calling these calls need to be changed to use the FRAME Call enum to achieve the same outcome.

The runtime currently also only supports BABE. There should be two features with-babe and with-aura. Depending on which feature is activated, the respective pallets are activated. As we already enable the disable-logging feature while building the wasm binary, it should be changed in the same way to build with once with BABE and once with AURA.

The general idea of this whole issue is that we can continue using the runtime for our unit tests, but we also can write "proper" unit tests for AURA. In a later future we would also like to have zombienet tests based with AURA so that we can test things like warp syncing etc. Currently we use the kitchensink-runtime for the zombienet tests. Maybe it would be a good idea to move the "special testing" pallet just to the kithchensink-runtime and to make this runtime build able with AURA/BABE. Then remove the testing runtime and only use the kitchenshink-runtime for all tests. However, this may adds more complexity and things that can fail in the unit tests. A simple runtime with just the consensus pallets, the testing pallets and some minimal set of required pallets is probably easier to maintain and to use.

[gilescope]

Sorry I should ask questions sooner rather than waste time trying to figure things out by myself. The substrate-test-runtime is used by the cumulus test environment so I am assuming that we are looking to make the substrate-test-runtime more general and split it into a babe and a aura (and maybe a manual seal one?).

I have been copying across the cumulus as an example, stripping out the relay chain and seeing if I can make that the basis to start from and port the tests to use that rather than using the substrate-test-runtime directly. (and then subsume the bits of substrate-test-runtime that are still needed).

So there will be a substrate/tests dir like cumulus and polkadot and the substrate/test-utils dir will hopefully disappear.

I am not sure if there was a question in there - more a statement of what I am trying to do...

[bkchr]

The substrate-test-runtime is used by the cumulus test environment

No it is not used in Cumulus.

the substrate-test-runtime more general and split it into a babe and a aura (and maybe a manual seal one?)

Yes that should be the way forward. A manual seal one should probably used in most of the tests that just want to interact with "a runtime" to test stuff. And a Babe + Aura version should be used by the respective crates to test their code, these runtimes could be stripped down to the minimum.

So there will be a substrate/tests dir like cumulus and polkadot and the substrate/test-utils dir will hopefully disappear.

This is only about Substrate, nothing that touches Polkadot. test-utils also contains the test-client which would stay, but could also be moved tests.

[gilescope]

Ah yes I was thinking of the references to sustrate-test-client - there are no direct references to the substrate-test-runtime.

[bkchr]

As I just the days thought about all this code here:

substrate/test-utils/runtime/src/lib.rs

Lines 696 to 1183 in a76ed8c

	cfg_if! {
	if #[cfg(feature = "std")] {
	impl_runtime_apis! {
	impl sp_api::Core<Block> for Runtime {
	fn version() -> RuntimeVersion {
	version()
	}
	fn execute_block(block: Block) {
	system::execute_block(block);
	}
	fn initialize_block(header: &<Block as BlockT>::Header) {
	system::initialize_block(header)
	}
	}
	impl sp_api::Metadata<Block> for Runtime {
	fn metadata() -> OpaqueMetadata {
	unimplemented!()
	}
	}
	impl sp_transaction_pool::runtime_api::TaggedTransactionQueue<Block> for Runtime {
	fn validate_transaction(
	_source: TransactionSource,
	utx: <Block as BlockT>::Extrinsic,
	_: <Block as BlockT>::Hash,
	) -> TransactionValidity {
	if let Extrinsic::IncludeData(data) = utx {
	return Ok(ValidTransaction {
	priority: data.len() as u64,
	requires: vec![],
	provides: vec![data],
	longevity: 1,
	propagate: false,
	});
	}
	system::validate_transaction(utx)
	}
	}
	impl sp_block_builder::BlockBuilder<Block> for Runtime {
	fn apply_extrinsic(extrinsic: <Block as BlockT>::Extrinsic) -> ApplyExtrinsicResult {
	system::execute_transaction(extrinsic)
	}
	fn finalize_block() -> <Block as BlockT>::Header {
	system::finalize_block()
	}
	fn inherent_extrinsics(_data: InherentData) -> Vec<<Block as BlockT>::Extrinsic> {
	vec![]
	}
	fn check_inherents(_block: Block, _data: InherentData) -> CheckInherentsResult {
	CheckInherentsResult::new()
	}
	}
	impl self::TestAPI<Block> for Runtime {
	fn balance_of(id: AccountId) -> u64 {
	system::balance_of(id)
	}
	fn benchmark_add_one(val: &u64) -> u64 {
	val + 1
	}
	fn benchmark_vector_add_one(vec: &Vec<u64>) -> Vec<u64> {
	let mut vec = vec.clone();
	vec.iter_mut().for_each(|v| *v += 1);
	vec
	}
	fn fail_convert_parameter(_: DecodeFails<Block>) {}
	fn fail_convert_return_value() -> DecodeFails<Block> {
	DecodeFails::new()
	}
	fn function_signature_changed() -> u64 {
	1
	}
	fn fail_on_native() -> u64 {
	panic!("Failing because we are on native")
	}
	fn fail_on_wasm() -> u64 {
	1
	}
	fn use_trie() -> u64 {
	code_using_trie()
	}
	fn benchmark_indirect_call() -> u64 {
	let function = benchmark_add_one;
	(0..1000).fold(0, |p, i| p + function(i))
	}
	fn benchmark_direct_call() -> u64 {
	(0..1000).fold(0, |p, i| p + benchmark_add_one(i))
	}
	fn vec_with_capacity(_size: u32) -> Vec<u8> {
	unimplemented!("is not expected to be invoked from non-wasm builds");
	}
	fn get_block_number() -> u64 {
	system::get_block_number().expect("Block number is initialized")
	}
	fn take_block_number() -> Option<u64> {
	system::take_block_number()
	}
	fn test_ed25519_crypto() -> (ed25519::AppSignature, ed25519::AppPublic) {
	test_ed25519_crypto()
	}
	fn test_sr25519_crypto() -> (sr25519::AppSignature, sr25519::AppPublic) {
	test_sr25519_crypto()
	}
	fn test_ecdsa_crypto() -> (ecdsa::AppSignature, ecdsa::AppPublic) {
	test_ecdsa_crypto()
	}
	fn test_storage() {
	test_read_storage();
	test_read_child_storage();
	}
	fn test_witness(proof: StorageProof, root: crate::Hash) {
	test_witness(proof, root);
	}
	fn test_multiple_arguments(data: Vec<u8>, other: Vec<u8>, num: u32) {
	assert_eq!(&data[..], &other[..]);
	assert_eq!(data.len(), num as usize);
	}
	fn do_trace_log() {
	log::trace!("Hey I'm runtime");
	}
	}
	impl sp_consensus_aura::AuraApi<Block, AuraId> for Runtime {
	fn slot_duration() -> sp_consensus_aura::SlotDuration {
	sp_consensus_aura::SlotDuration::from_millis(1000)
	}
	fn authorities() -> Vec<AuraId> {
	system::authorities().into_iter().map(|a| {
	let authority: sr25519::Public = a.into();
	AuraId::from(authority)
	}).collect()
	}
	}
	impl sp_consensus_babe::BabeApi<Block> for Runtime {
	fn configuration() -> sp_consensus_babe::BabeGenesisConfiguration {
	sp_consensus_babe::BabeGenesisConfiguration {
	slot_duration: 1000,
	epoch_length: EpochDuration::get(),
	c: (3, 10),
	genesis_authorities: system::authorities()
	.into_iter().map(|x|(x, 1)).collect(),
	randomness: <pallet_babe::Pallet<Runtime>>::randomness(),
	allowed_slots: AllowedSlots::PrimaryAndSecondaryPlainSlots,
	}
	}
	fn current_epoch_start() -> Slot {
	<pallet_babe::Pallet<Runtime>>::current_epoch_start()
	}
	fn current_epoch() -> sp_consensus_babe::Epoch {
	<pallet_babe::Pallet<Runtime>>::current_epoch()
	}
	fn next_epoch() -> sp_consensus_babe::Epoch {
	<pallet_babe::Pallet<Runtime>>::next_epoch()
	}
	fn submit_report_equivocation_unsigned_extrinsic(
	_equivocation_proof: sp_consensus_babe::EquivocationProof<
	<Block as BlockT>::Header,
	>,
	_key_owner_proof: sp_consensus_babe::OpaqueKeyOwnershipProof,
	) -> Option<()> {
	None
	}
	fn generate_key_ownership_proof(
	_slot: sp_consensus_babe::Slot,
	_authority_id: sp_consensus_babe::AuthorityId,
	) -> Option<sp_consensus_babe::OpaqueKeyOwnershipProof> {
	None
	}
	}
	impl sp_offchain::OffchainWorkerApi<Block> for Runtime {
	fn offchain_worker(header: &<Block as BlockT>::Header) {
	let ex = Extrinsic::IncludeData(header.number.encode());
	sp_io::offchain::submit_transaction(ex.encode()).unwrap();
	}
	}
	impl sp_session::SessionKeys<Block> for Runtime {
	fn generate_session_keys(_: Option<Vec<u8>>) -> Vec<u8> {
	SessionKeys::generate(None)
	}
	fn decode_session_keys(
	encoded: Vec<u8>,
	) -> Option<Vec<(Vec<u8>, sp_core::crypto::KeyTypeId)>> {
	SessionKeys::decode_into_raw_public_keys(&encoded)
	}
	}
	impl sp_finality_grandpa::GrandpaApi<Block> for Runtime {
	fn grandpa_authorities() -> sp_finality_grandpa::AuthorityList {
	Vec::new()
	}
	fn current_set_id() -> sp_finality_grandpa::SetId {
	0
	}
	fn submit_report_equivocation_unsigned_extrinsic(
	_equivocation_proof: sp_finality_grandpa::EquivocationProof<
	<Block as BlockT>::Hash,
	NumberFor<Block>,
	>,
	_key_owner_proof: sp_finality_grandpa::OpaqueKeyOwnershipProof,
	) -> Option<()> {
	None
	}
	fn generate_key_ownership_proof(
	_set_id: sp_finality_grandpa::SetId,
	_authority_id: sp_finality_grandpa::AuthorityId,
	) -> Option<sp_finality_grandpa::OpaqueKeyOwnershipProof> {
	None
	}
	}
	impl frame_system_rpc_runtime_api::AccountNonceApi<Block, AccountId, Index> for Runtime {
	fn account_nonce(_account: AccountId) -> Index {
	0
	}
	}
	}
	} else {
	impl_runtime_apis! {
	impl sp_api::Core<Block> for Runtime {
	fn version() -> RuntimeVersion {
	version()
	}
	fn execute_block(block: Block) {
	system::execute_block(block);
	}
	fn initialize_block(header: &<Block as BlockT>::Header) {
	system::initialize_block(header)
	}
	}
	impl sp_api::Metadata<Block> for Runtime {
	fn metadata() -> OpaqueMetadata {
	unimplemented!()
	}
	}
	impl sp_transaction_pool::runtime_api::TaggedTransactionQueue<Block> for Runtime {
	fn validate_transaction(
	_source: TransactionSource,
	utx: <Block as BlockT>::Extrinsic,
	_: <Block as BlockT>::Hash,
	) -> TransactionValidity {
	if let Extrinsic::IncludeData(data) = utx {
	return Ok(ValidTransaction{
	priority: data.len() as u64,
	requires: vec![],
	provides: vec![data],
	longevity: 1,
	propagate: false,
	});
	}
	system::validate_transaction(utx)
	}
	}
	impl sp_block_builder::BlockBuilder<Block> for Runtime {
	fn apply_extrinsic(extrinsic: <Block as BlockT>::Extrinsic) -> ApplyExtrinsicResult {
	system::execute_transaction(extrinsic)
	}
	fn finalize_block() -> <Block as BlockT>::Header {
	system::finalize_block()
	}
	fn inherent_extrinsics(_data: InherentData) -> Vec<<Block as BlockT>::Extrinsic> {
	vec![]
	}
	fn check_inherents(_block: Block, _data: InherentData) -> CheckInherentsResult {
	CheckInherentsResult::new()
	}
	}
	impl self::TestAPI<Block> for Runtime {
	fn balance_of(id: AccountId) -> u64 {
	system::balance_of(id)
	}
	fn benchmark_add_one(val: &u64) -> u64 {
	val + 1
	}
	fn benchmark_vector_add_one(vec: &Vec<u64>) -> Vec<u64> {
	let mut vec = vec.clone();
	vec.iter_mut().for_each(|v| *v += 1);
	vec
	}
	fn fail_convert_parameter(_: DecodeFails<Block>) {}
	fn fail_convert_return_value() -> DecodeFails<Block> {
	DecodeFails::new()
	}
	fn function_signature_changed() -> Vec<u64> {
	let mut vec = Vec::new();
	vec.push(1);
	vec.push(2);
	vec
	}
	fn fail_on_native() -> u64 {
	1
	}
	fn fail_on_wasm() -> u64 {
	panic!("Failing because we are on wasm")
	}
	fn use_trie() -> u64 {
	code_using_trie()
	}
	fn benchmark_indirect_call() -> u64 {
	(0..10000).fold(0, |p, i| p + BENCHMARK_ADD_ONE.get()(i))
	}
	fn benchmark_direct_call() -> u64 {
	(0..10000).fold(0, |p, i| p + benchmark_add_one(i))
	}
	fn vec_with_capacity(size: u32) -> Vec<u8> {
	Vec::with_capacity(size as usize)
	}
	fn get_block_number() -> u64 {
	system::get_block_number().expect("Block number is initialized")
	}
	fn take_block_number() -> Option<u64> {
	system::take_block_number()
	}
	fn test_ed25519_crypto() -> (ed25519::AppSignature, ed25519::AppPublic) {
	test_ed25519_crypto()
	}
	fn test_sr25519_crypto() -> (sr25519::AppSignature, sr25519::AppPublic) {
	test_sr25519_crypto()
	}
	fn test_ecdsa_crypto() -> (ecdsa::AppSignature, ecdsa::AppPublic) {
	test_ecdsa_crypto()
	}
	fn test_storage() {
	test_read_storage();
	test_read_child_storage();
	}
	fn test_witness(proof: StorageProof, root: crate::Hash) {
	test_witness(proof, root);
	}
	fn test_multiple_arguments(data: Vec<u8>, other: Vec<u8>, num: u32) {
	assert_eq!(&data[..], &other[..]);
	assert_eq!(data.len(), num as usize);
	}
	fn do_trace_log() {
	log::trace!("Hey I'm runtime: {}", log::STATIC_MAX_LEVEL);
	}
	}
	impl sp_consensus_aura::AuraApi<Block, AuraId> for Runtime {
	fn slot_duration() -> sp_consensus_aura::SlotDuration {
	sp_consensus_aura::SlotDuration::from_millis(1000)
	}
	fn authorities() -> Vec<AuraId> {
	system::authorities().into_iter().map(|a| {
	let authority: sr25519::Public = a.into();
	AuraId::from(authority)
	}).collect()
	}
	}
	impl sp_consensus_babe::BabeApi<Block> for Runtime {
	fn configuration() -> sp_consensus_babe::BabeGenesisConfiguration {
	sp_consensus_babe::BabeGenesisConfiguration {
	slot_duration: 1000,
	epoch_length: EpochDuration::get(),
	c: (3, 10),
	genesis_authorities: system::authorities()
	.into_iter().map(|x|(x, 1)).collect(),
	randomness: <pallet_babe::Pallet<Runtime>>::randomness(),
	allowed_slots: AllowedSlots::PrimaryAndSecondaryPlainSlots,
	}
	}
	fn current_epoch_start() -> Slot {
	<pallet_babe::Pallet<Runtime>>::current_epoch_start()
	}
	fn current_epoch() -> sp_consensus_babe::Epoch {
	<pallet_babe::Pallet<Runtime>>::current_epoch()
	}
	fn next_epoch() -> sp_consensus_babe::Epoch {
	<pallet_babe::Pallet<Runtime>>::next_epoch()
	}
	fn submit_report_equivocation_unsigned_extrinsic(
	_equivocation_proof: sp_consensus_babe::EquivocationProof<
	<Block as BlockT>::Header,
	>,
	_key_owner_proof: sp_consensus_babe::OpaqueKeyOwnershipProof,
	) -> Option<()> {
	None
	}
	fn generate_key_ownership_proof(
	_slot: sp_consensus_babe::Slot,
	_authority_id: sp_consensus_babe::AuthorityId,
	) -> Option<sp_consensus_babe::OpaqueKeyOwnershipProof> {
	None
	}
	}
	impl sp_offchain::OffchainWorkerApi<Block> for Runtime {
	fn offchain_worker(header: &<Block as BlockT>::Header) {
	let ex = Extrinsic::IncludeData(header.number.encode());
	sp_io::offchain::submit_transaction(ex.encode()).unwrap()
	}
	}
	impl sp_session::SessionKeys<Block> for Runtime {
	fn generate_session_keys(_: Option<Vec<u8>>) -> Vec<u8> {
	SessionKeys::generate(None)
	}
	fn decode_session_keys(
	encoded: Vec<u8>,
	) -> Option<Vec<(Vec<u8>, sp_core::crypto::KeyTypeId)>> {
	SessionKeys::decode_into_raw_public_keys(&encoded)
	}
	}
	impl frame_system_rpc_runtime_api::AccountNonceApi<Block, AccountId, Index> for Runtime {
	fn account_nonce(_account: AccountId) -> Index {
	0
	}
	}
	}
	}
	}

In this new "model" we should put this code in its own crate that is compiled to WASM etc and is only used by sp-api-test.

[gilescope]

Agreed. impl_runtime_apis! feels like a hack to do hand rolled stuff.

[bkchr]

Agreed. impl_runtime_apis! feels like a hack to do hand rolled stuff.

Not sure what you mean by hand rolled stuff here. You can just use the macro, also for sp-api-test