From 650979ae405afc8b87935172189774cb1f24a8a3 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Mon, 8 Aug 2022 11:01:57 -0500 Subject: [PATCH] Implement strings in adapter modules (#4623) * Implement strings in adapter modules This commit is a hefty addition to Wasmtime's support for the component model. This implements the final remaining type (in the current type hierarchy) unimplemented in adapter module trampolines: strings. Strings are the most complicated type to implement in adapter trampolines because they are highly structured chunks of data in memory (according to specific encodings). Additionally each lift/lower operation can choose its own encoding for strings meaning that Wasmtime, the host, may have to convert between any pairwise ordering of string encodings. The `CanonicalABI.md` in the component-model repo in general specifies all the fiddly bits of string encoding so there's not a ton of wiggle room for Wasmtime to get creative. This PR largely "just" implements that. The high-level architecture of this implementation is: * Fused adapters are first identified to determine src/dst string encodings. This statically fixes what transcoding operation is being performed. * The generated adapter will be responsible for managing calls to `realloc` and performing bounds checks. The adapter itself does not perform memory copies or validation of string contents, however. Instead each transcoding operation is modeled as an imported function into the adapter module. This means that the adapter module dynamically, during compile time, determines what string transcoders are needed. Note that an imported transcoder is not only parameterized over the transcoding operation but additionally which memory is the source and which is the destination. * The imported core wasm functions are modeled as a new `CoreDef::Transcoder` structure. These transcoders end up being small Cranelift-compiled trampolines. The Cranelift-compiled trampoline will load the actual base pointer of memory and add it to the relative pointers passed as function arguments. This trampoline then calls a transcoder "libcall" which enters Rust-defined functions for actual transcoding operations. * Each possible transcoding operation is implemented in Rust with a unique name and a unique signature depending on the needs of the transcoder. I've tried to document inline what each transcoder does. This means that the `Module::translate_string` in adapter modules is by far the largest translation method. The main reason for this is due to the management around calling the imported transcoder functions in the face of validating string pointer/lengths and performing the dance of `realloc`-vs-transcode at the right time. I've tried to ensure that each individual case in transcoding is documented well enough to understand what's going on as well. Additionally in this PR is a full implementation in the host for the `latin1+utf16` encoding which means that both lifting and lowering host strings now works with this encoding. Currently the implementation of each transcoder function is likely far from optimal. Where possible I've leaned on the standard library itself and for latin1-related things I'm leaning on the `encoding_rs` crate. I initially tried to implement everything with `encoding_rs` but was unable to uniformly do so easily. For now I settled on trying to get a known-correct (even in the face of endianness) implementation for all of these transcoders. If an when performance becomes an issue it should be possible to implement more optimized versions of each of these transcoding operations. Testing this commit has been somewhat difficult and my general plan, like with the `(list T)` type, is to rely heavily on fuzzing to cover the various cases here. In this PR though I've added a simple test that pushes some statically known strings through all the pairs of encodings between source and destination. I've attempted to pick "interesting" strings that one way or another stress the various paths in each transcoding operation to ideally get full branch coverage there. Additionally a suite of "negative" tests have also been added to ensure that validity of encoding is actually checked. * Fix a temporarily commented out case * Fix wasmtime-runtime tests * Update deny.toml configuration * Add `BSD-3-Clause` for the `encoding_rs` crate * Remove some unused licenses * Add an exemption for `encoding_rs` for now * Split up the `translate_string` method Move out all the closures and package up captured state into smaller lists of arguments. * Test out-of-bounds for zero-length strings --- Cargo.lock | 11 + crates/component-util/src/lib.rs | 17 +- crates/cranelift/src/compiler/component.rs | 399 +++++++- crates/cranelift/src/func_environ.rs | 14 +- .../fuzz/fuzz_targets/fact-valid-module.rs | 64 +- crates/environ/src/component.rs | 22 + crates/environ/src/component/compiler.rs | 21 + crates/environ/src/component/dfg.rs | 50 + crates/environ/src/component/info.rs | 51 + .../environ/src/component/translate/adapt.rs | 57 +- crates/environ/src/component/types.rs | 7 + .../src/component/vmcomponent_offsets.rs | 33 +- crates/environ/src/fact.rs | 71 +- crates/environ/src/fact/trampoline.rs | 875 +++++++++++++++++- crates/environ/src/fact/transcode.rs | 178 ++++ crates/environ/src/fact/traps.rs | 4 + crates/environ/src/module.rs | 2 +- crates/environ/src/vmoffsets.rs | 73 +- crates/misc/component-test-util/src/lib.rs | 8 +- crates/runtime/Cargo.toml | 6 +- crates/runtime/src/component.rs | 45 +- crates/runtime/src/component/transcode.rs | 446 +++++++++ crates/runtime/src/vmcontext.rs | 8 +- crates/wasmtime/Cargo.toml | 2 + crates/wasmtime/src/component/component.rs | 64 +- crates/wasmtime/src/component/func/typed.rs | 127 ++- crates/wasmtime/src/component/instance.rs | 71 +- crates/wasmtime/src/config.rs | 9 + deny.toml | 3 +- supply-chain/config.toml | 4 + tests/all/component_model.rs | 1 + tests/all/component_model/strings.rs | 578 ++++++++++++ .../component-model/strings.wast | 108 +++ 33 files changed, 3239 insertions(+), 190 deletions(-) create mode 100644 crates/environ/src/fact/transcode.rs create mode 100644 crates/runtime/src/component/transcode.rs create mode 100644 tests/all/component_model/strings.rs create mode 100644 tests/misc_testsuite/component-model/strings.wast diff --git a/Cargo.lock b/Cargo.lock index 518b7d8b0c29..b952743ceae2 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1082,6 +1082,15 @@ version = "0.3.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a357d28ed41a50f9c765dbfe56cbc04a64e53e5fc58ba79fbc34c10ef3df831f" +[[package]] +name = "encoding_rs" +version = "0.8.31" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9852635589dc9f9ea1b6fe9f05b50ef208c85c834a562f0c6abb1c475736ec2b" +dependencies = [ + "cfg-if", +] + [[package]] name = "env_logger" version = "0.7.1" @@ -3333,6 +3342,7 @@ dependencies = [ "async-trait", "bincode", "cfg-if", + "encoding_rs", "indexmap", "libc", "log", @@ -3653,6 +3663,7 @@ dependencies = [ "anyhow", "cc", "cfg-if", + "encoding_rs", "indexmap", "libc", "log", diff --git a/crates/component-util/src/lib.rs b/crates/component-util/src/lib.rs index 409ba551c8f4..2c4a2a461dc9 100644 --- a/crates/component-util/src/lib.rs +++ b/crates/component-util/src/lib.rs @@ -88,6 +88,8 @@ pub const REALLOC_AND_FREE: &str = r#" (param $new_size i32) (result i32) + (local $ret i32) + ;; Test if the old pointer is non-null local.get $old_ptr if @@ -101,8 +103,8 @@ pub const REALLOC_AND_FREE: &str = r#" return end - ;; ... otherwise this is unimplemented - unreachable + ;; otherwise fall through to allocate a new chunk which will later + ;; copy data over end ;; align up `$last` @@ -121,6 +123,7 @@ pub const REALLOC_AND_FREE: &str = r#" ;; save the current value of `$last` as the return value global.get $last + local.tee $ret ;; ensure anything necessary is set to valid data by spraying a bit ;; pattern that is invalid @@ -129,6 +132,16 @@ pub const REALLOC_AND_FREE: &str = r#" local.get $new_size memory.fill + ;; If the old pointer is present then that means this was a reallocation + ;; of an existing chunk which means the existing data must be copied. + local.get $old_ptr + if + local.get $ret ;; destination + local.get $old_ptr ;; source + local.get $old_size ;; size + memory.copy + end + ;; bump our pointer (global.set $last (i32.add diff --git a/crates/cranelift/src/compiler/component.rs b/crates/cranelift/src/compiler/component.rs index 318764d11e56..33bd9d74cdbc 100644 --- a/crates/cranelift/src/compiler/component.rs +++ b/crates/cranelift/src/compiler/component.rs @@ -10,8 +10,9 @@ use object::write::Object; use std::any::Any; use std::ops::Range; use wasmtime_environ::component::{ - AlwaysTrapInfo, CanonicalOptions, Component, ComponentCompiler, ComponentTypes, FunctionInfo, - LowerImport, LoweredIndex, RuntimeAlwaysTrapIndex, VMComponentOffsets, + AlwaysTrapInfo, CanonicalOptions, Component, ComponentCompiler, ComponentTypes, FixedEncoding, + FunctionInfo, LowerImport, LoweredIndex, RuntimeAlwaysTrapIndex, RuntimeMemoryIndex, + RuntimeTranscoderIndex, Transcode, Transcoder, VMComponentOffsets, }; use wasmtime_environ::{PrimaryMap, PtrSize, SignatureIndex, Trampoline, TrapCode, WasmFuncType}; @@ -47,42 +48,7 @@ impl ComponentCompiler for Compiler { let vmctx = builder.func.dfg.block_params(block0)[0]; // Save the exit FP and return address for stack walking purposes. - // - // First we need to get the `VMRuntimeLimits`. - let limits = builder.ins().load( - pointer_type, - MemFlags::trusted(), - vmctx, - i32::try_from(offsets.limits()).unwrap(), - ); - // Then save the exit Wasm FP to the limits. We dereference the current - // FP to get the previous FP because the current FP is the trampoline's - // FP, and we want the Wasm function's FP, which is the caller of this - // trampoline. - let trampoline_fp = builder.ins().get_frame_pointer(pointer_type); - let wasm_fp = builder.ins().load( - pointer_type, - MemFlags::trusted(), - trampoline_fp, - // The FP always points to the next older FP for all supported - // targets. See assertion in - // `crates/runtime/src/traphandlers/backtrace.rs`. - 0, - ); - builder.ins().store( - MemFlags::trusted(), - wasm_fp, - limits, - offsets.ptr.vmruntime_limits_last_wasm_exit_fp(), - ); - // Finally save the Wasm return address to the limits. - let wasm_pc = builder.ins().get_return_address(pointer_type); - builder.ins().store( - MemFlags::trusted(), - wasm_pc, - limits, - offsets.ptr.vmruntime_limits_last_wasm_exit_pc(), - ); + self.save_last_wasm_fp_and_pc(&mut builder, &offsets, vmctx); // Below this will incrementally build both the signature of the host // function we're calling as well as the list of arguments since the @@ -219,15 +185,53 @@ impl ComponentCompiler for Compiler { Ok(Box::new(func)) } + fn compile_transcoder( + &self, + component: &Component, + transcoder: &Transcoder, + types: &ComponentTypes, + ) -> Result> { + let ty = &types[transcoder.signature]; + let isa = &*self.isa; + let offsets = VMComponentOffsets::new(isa.pointer_bytes(), component); + + let CompilerContext { + mut func_translator, + codegen_context: mut context, + } = self.take_context(); + + context.func = ir::Function::with_name_signature( + ir::ExternalName::user(0, 0), + crate::indirect_signature(isa, ty), + ); + + let mut builder = FunctionBuilder::new(&mut context.func, func_translator.context()); + let block0 = builder.create_block(); + builder.append_block_params_for_function_params(block0); + builder.switch_to_block(block0); + builder.seal_block(block0); + + self.translate_transcode(&mut builder, &offsets, transcoder, block0); + + let func: CompiledFunction = self.finish_trampoline(&mut context, isa)?; + self.save_context(CompilerContext { + func_translator, + codegen_context: context, + }); + Ok(Box::new(func)) + } + fn emit_obj( &self, lowerings: PrimaryMap>, always_trap: PrimaryMap>, + transcoders: PrimaryMap>, trampolines: Vec<(SignatureIndex, Box)>, obj: &mut Object<'static>, ) -> Result<( PrimaryMap, PrimaryMap, + PrimaryMap, Vec, )> { let module = Default::default(); @@ -264,6 +268,16 @@ impl ComponentCompiler for Compiler { }) .collect(); + let ret_transcoders = transcoders + .iter() + .map(|(i, func)| { + let func = func.downcast_ref::().unwrap(); + let name = format!("_wasmtime_transcoder{}", i.as_u32()); + let range = text.named_func(&name, func); + range2info(range) + }) + .collect(); + let ret_trampolines = trampolines .iter() .map(|(i, func)| { @@ -275,6 +289,313 @@ impl ComponentCompiler for Compiler { text.finish()?; - Ok((ret_lowerings, ret_always_trap, ret_trampolines)) + Ok(( + ret_lowerings, + ret_always_trap, + ret_transcoders, + ret_trampolines, + )) + } +} + +impl Compiler { + fn save_last_wasm_fp_and_pc( + &self, + builder: &mut FunctionBuilder<'_>, + offsets: &VMComponentOffsets, + vmctx: ir::Value, + ) { + let pointer_type = self.isa.pointer_type(); + // First we need to get the `VMRuntimeLimits`. + let limits = builder.ins().load( + pointer_type, + MemFlags::trusted(), + vmctx, + i32::try_from(offsets.limits()).unwrap(), + ); + // Then save the exit Wasm FP to the limits. We dereference the current + // FP to get the previous FP because the current FP is the trampoline's + // FP, and we want the Wasm function's FP, which is the caller of this + // trampoline. + let trampoline_fp = builder.ins().get_frame_pointer(pointer_type); + let wasm_fp = builder.ins().load( + pointer_type, + MemFlags::trusted(), + trampoline_fp, + // The FP always points to the next older FP for all supported + // targets. See assertion in + // `crates/runtime/src/traphandlers/backtrace.rs`. + 0, + ); + builder.ins().store( + MemFlags::trusted(), + wasm_fp, + limits, + offsets.ptr.vmruntime_limits_last_wasm_exit_fp(), + ); + // Finally save the Wasm return address to the limits. + let wasm_pc = builder.ins().get_return_address(pointer_type); + builder.ins().store( + MemFlags::trusted(), + wasm_pc, + limits, + offsets.ptr.vmruntime_limits_last_wasm_exit_pc(), + ); + } + + fn translate_transcode( + &self, + builder: &mut FunctionBuilder<'_>, + offsets: &VMComponentOffsets, + transcoder: &Transcoder, + block: ir::Block, + ) { + let pointer_type = self.isa.pointer_type(); + let vmctx = builder.func.dfg.block_params(block)[0]; + + // Save the exit FP and return address for stack walking purposes. This + // is used when an invalid encoding is encountered and a trap is raised. + self.save_last_wasm_fp_and_pc(builder, &offsets, vmctx); + + // Determine the static signature of the host libcall for this transcode + // operation and additionally calculate the static offset within the + // transode libcalls array. + let func = &mut builder.func; + let (sig, offset) = match transcoder.op { + Transcode::Copy(FixedEncoding::Utf8) => host::utf8_to_utf8(self, func), + Transcode::Copy(FixedEncoding::Utf16) => host::utf16_to_utf16(self, func), + Transcode::Copy(FixedEncoding::Latin1) => host::latin1_to_latin1(self, func), + Transcode::Latin1ToUtf16 => host::latin1_to_utf16(self, func), + Transcode::Latin1ToUtf8 => host::latin1_to_utf8(self, func), + Transcode::Utf16ToCompactProbablyUtf16 => { + host::utf16_to_compact_probably_utf16(self, func) + } + Transcode::Utf16ToCompactUtf16 => host::utf16_to_compact_utf16(self, func), + Transcode::Utf16ToLatin1 => host::utf16_to_latin1(self, func), + Transcode::Utf16ToUtf8 => host::utf16_to_utf8(self, func), + Transcode::Utf8ToCompactUtf16 => host::utf8_to_compact_utf16(self, func), + Transcode::Utf8ToLatin1 => host::utf8_to_latin1(self, func), + Transcode::Utf8ToUtf16 => host::utf8_to_utf16(self, func), + }; + + // Load the host function pointer for this transcode which comes from a + // function pointer within the VMComponentContext's libcall array. + let transcode_libcalls_array = builder.ins().load( + pointer_type, + MemFlags::trusted(), + vmctx, + i32::try_from(offsets.transcode_libcalls()).unwrap(), + ); + let transcode_libcall = builder.ins().load( + pointer_type, + MemFlags::trusted(), + transcode_libcalls_array, + i32::try_from(offset * u32::from(offsets.ptr.size())).unwrap(), + ); + + // Load the base pointers for the from/to linear memories. + let from_base = self.load_runtime_memory_base(builder, vmctx, offsets, transcoder.from); + let to_base = self.load_runtime_memory_base(builder, vmctx, offsets, transcoder.to); + + // Helper function to cast a core wasm input to a host pointer type + // which will go into the host libcall. + let cast_to_pointer = |builder: &mut FunctionBuilder<'_>, val: ir::Value, is64: bool| { + let host64 = pointer_type == ir::types::I64; + if is64 == host64 { + val + } else if !is64 { + assert!(host64); + builder.ins().uextend(pointer_type, val) + } else { + assert!(!host64); + builder.ins().ireduce(pointer_type, val) + } + }; + + // Helper function to cast an input parameter to the host pointer type. + let len_param = |builder: &mut FunctionBuilder<'_>, param: usize, is64: bool| { + let val = builder.func.dfg.block_params(block)[2 + param]; + cast_to_pointer(builder, val, is64) + }; + + // Helper function to interpret an input parameter as a pointer into + // linear memory. This will cast the input parameter to the host integer + // type and then add that value to the base. + // + // Note that bounds-checking happens in adapter modules, and this + // trampoline is simply calling the host libcall. + let ptr_param = + |builder: &mut FunctionBuilder<'_>, param: usize, is64: bool, base: ir::Value| { + let val = len_param(builder, param, is64); + builder.ins().iadd(base, val) + }; + + let Transcoder { to64, from64, .. } = *transcoder; + let mut args = Vec::new(); + + // Most transcoders share roughly the same signature despite doing very + // different things internally, so most libcalls are lumped together + // here. + match transcoder.op { + Transcode::Copy(_) + | Transcode::Latin1ToUtf16 + | Transcode::Utf16ToCompactProbablyUtf16 + | Transcode::Utf8ToLatin1 + | Transcode::Utf16ToLatin1 + | Transcode::Utf8ToUtf16 => { + args.push(ptr_param(builder, 0, from64, from_base)); + args.push(len_param(builder, 1, from64)); + args.push(ptr_param(builder, 2, to64, to_base)); + } + + Transcode::Utf16ToUtf8 | Transcode::Latin1ToUtf8 => { + args.push(ptr_param(builder, 0, from64, from_base)); + args.push(len_param(builder, 1, from64)); + args.push(ptr_param(builder, 2, to64, to_base)); + args.push(len_param(builder, 3, to64)); + } + + Transcode::Utf8ToCompactUtf16 | Transcode::Utf16ToCompactUtf16 => { + args.push(ptr_param(builder, 0, from64, from_base)); + args.push(len_param(builder, 1, from64)); + args.push(ptr_param(builder, 2, to64, to_base)); + args.push(len_param(builder, 3, to64)); + args.push(len_param(builder, 4, to64)); + } + }; + let call = builder.ins().call_indirect(sig, transcode_libcall, &args); + let results = builder.func.dfg.inst_results(call).to_vec(); + let mut raw_results = Vec::new(); + + // Helper to cast a host pointer integer type to the destination type. + let cast_from_pointer = |builder: &mut FunctionBuilder<'_>, val: ir::Value, is64: bool| { + let host64 = pointer_type == ir::types::I64; + if is64 == host64 { + val + } else if !is64 { + assert!(host64); + builder.ins().ireduce(ir::types::I32, val) + } else { + assert!(!host64); + builder.ins().uextend(ir::types::I64, val) + } + }; + + // Like the arguments the results are fairly similar across libcalls, so + // they're lumped into various buckets here. + match transcoder.op { + Transcode::Copy(_) | Transcode::Latin1ToUtf16 => {} + + Transcode::Utf8ToUtf16 + | Transcode::Utf16ToCompactProbablyUtf16 + | Transcode::Utf8ToCompactUtf16 + | Transcode::Utf16ToCompactUtf16 => { + raw_results.push(cast_from_pointer(builder, results[0], to64)); + } + + Transcode::Latin1ToUtf8 + | Transcode::Utf16ToUtf8 + | Transcode::Utf8ToLatin1 + | Transcode::Utf16ToLatin1 => { + raw_results.push(cast_from_pointer(builder, results[0], from64)); + raw_results.push(cast_from_pointer(builder, results[1], to64)); + } + }; + + builder.ins().return_(&raw_results); + builder.finalize(); + } + + fn load_runtime_memory_base( + &self, + builder: &mut FunctionBuilder<'_>, + vmctx: ir::Value, + offsets: &VMComponentOffsets, + mem: RuntimeMemoryIndex, + ) -> ir::Value { + let pointer_type = self.isa.pointer_type(); + let from_vmmemory_definition = builder.ins().load( + pointer_type, + MemFlags::trusted(), + vmctx, + i32::try_from(offsets.runtime_memory(mem)).unwrap(), + ); + builder.ins().load( + pointer_type, + MemFlags::trusted(), + from_vmmemory_definition, + i32::from(offsets.ptr.vmmemory_definition_base()), + ) + } +} + +/// Module with macro-generated contents that will return the signature and +/// offset for each of the host transcoder functions. +/// +/// Note that a macro is used here to keep this in sync with the actual +/// transcoder functions themselves which are also defined via a macro. +#[allow(unused_mut)] +mod host { + use crate::compiler::Compiler; + use cranelift_codegen::ir::{self, AbiParam}; + + macro_rules! host_transcode { + ( + $( + $( #[$attr:meta] )* + $name:ident( $( $pname:ident: $param:ident ),* ) $( -> $result:ident )?; + )* + ) => { + $( + pub(super) fn $name(compiler: &Compiler, func: &mut ir::Function) -> (ir::SigRef, u32) { + let pointer_type = compiler.isa.pointer_type(); + let params = vec![ + $( AbiParam::new(host_transcode!(@ty pointer_type $param)) ),* + ]; + let mut returns = Vec::new(); + $(host_transcode!(@push_return pointer_type params returns $result);)? + let sig = func.import_signature(ir::Signature { + params, + returns, + call_conv: crate::wasmtime_call_conv(&*compiler.isa), + }); + + (sig, offsets::$name) + } + )* + }; + + (@ty $ptr:ident size) => ($ptr); + (@ty $ptr:ident ptr_u8) => ($ptr); + (@ty $ptr:ident ptr_u16) => ($ptr); + + (@push_return $ptr:ident $params:ident $returns:ident size) => ($returns.push(AbiParam::new($ptr));); + (@push_return $ptr:ident $params:ident $returns:ident size_pair) => ({ + $returns.push(AbiParam::new($ptr)); + $returns.push(AbiParam::new($ptr)); + }); + } + + wasmtime_environ::foreach_transcoder!(host_transcode); + + mod offsets { + macro_rules! offsets { + ( + $( + $( #[$attr:meta] )* + $name:ident($($t:tt)*) $( -> $result:ident )?; + )* + ) => { + offsets!(@declare (0) $($name)*); + }; + + (@declare ($n:expr)) => (); + (@declare ($n:expr) $name:ident $($rest:tt)*) => ( + pub static $name: u32 = $n; + offsets!(@declare ($n + 1) $($rest)*); + ); + } + + wasmtime_environ::foreach_transcoder!(offsets); } } diff --git a/crates/cranelift/src/func_environ.rs b/crates/cranelift/src/func_environ.rs index 7d5e053b02eb..7576f423c081 100644 --- a/crates/cranelift/src/func_environ.rs +++ b/crates/cranelift/src/func_environ.rs @@ -1387,9 +1387,9 @@ impl<'module_environment> cranelift_wasm::FuncEnvironment for FuncEnvironment<'m global_type: pointer_type, readonly: true, }); - let base_offset = i32::from(self.offsets.vmmemory_definition_base()); + let base_offset = i32::from(self.offsets.ptr.vmmemory_definition_base()); let current_length_offset = - i32::from(self.offsets.vmmemory_definition_current_length()); + i32::from(self.offsets.ptr.vmmemory_definition_current_length()); (memory, base_offset, current_length_offset) } else { let owned_index = self.module.owned_memory_index(def_index); @@ -1410,9 +1410,9 @@ impl<'module_environment> cranelift_wasm::FuncEnvironment for FuncEnvironment<'m global_type: pointer_type, readonly: true, }); - let base_offset = i32::from(self.offsets.vmmemory_definition_base()); + let base_offset = i32::from(self.offsets.ptr.vmmemory_definition_base()); let current_length_offset = - i32::from(self.offsets.vmmemory_definition_current_length()); + i32::from(self.offsets.ptr.vmmemory_definition_current_length()); (memory, base_offset, current_length_offset) } }; @@ -1726,7 +1726,7 @@ impl<'module_environment> cranelift_wasm::FuncEnvironment for FuncEnvironment<'m pos.ins() .load(pointer_type, ir::MemFlags::trusted(), base, offset); let vmmemory_definition_offset = - i64::from(self.offsets.vmmemory_definition_current_length()); + i64::from(self.offsets.ptr.vmmemory_definition_current_length()); let vmmemory_definition_ptr = pos.ins().iadd_imm(vmmemory_ptr, vmmemory_definition_offset); // This atomic access of the @@ -1758,7 +1758,7 @@ impl<'module_environment> cranelift_wasm::FuncEnvironment for FuncEnvironment<'m .load(pointer_type, ir::MemFlags::trusted(), base, offset); if is_shared { let vmmemory_definition_offset = - i64::from(self.offsets.vmmemory_definition_current_length()); + i64::from(self.offsets.ptr.vmmemory_definition_current_length()); let vmmemory_definition_ptr = pos.ins().iadd_imm(vmmemory_ptr, vmmemory_definition_offset); pos.ins().atomic_load( @@ -1771,7 +1771,7 @@ impl<'module_environment> cranelift_wasm::FuncEnvironment for FuncEnvironment<'m pointer_type, ir::MemFlags::trusted(), vmmemory_ptr, - i32::from(self.offsets.vmmemory_definition_current_length()), + i32::from(self.offsets.ptr.vmmemory_definition_current_length()), ) } } diff --git a/crates/environ/fuzz/fuzz_targets/fact-valid-module.rs b/crates/environ/fuzz/fuzz_targets/fact-valid-module.rs index 58f2488cf174..5c7114a4a348 100644 --- a/crates/environ/fuzz/fuzz_targets/fact-valid-module.rs +++ b/crates/environ/fuzz/fuzz_targets/fact-valid-module.rs @@ -45,9 +45,11 @@ enum GenStringEncoding { CompactUtf16, } -fuzz_target!(|module: GenAdapterModule| { drop(target(module)) }); +fuzz_target!(|module: GenAdapterModule| { + target(module); +}); -fn target(module: GenAdapterModule) -> Result<(), ()> { +fn target(module: GenAdapterModule) { drop(env_logger::try_init()); let mut types = ComponentTypesBuilder::default(); @@ -57,7 +59,7 @@ fn target(module: GenAdapterModule) -> Result<(), ()> { let mut next_def = 0; let mut dummy_def = || { next_def += 1; - CoreDef::Adapter(AdapterIndex::from_u32(next_def)) + dfg::CoreDef::Adapter(dfg::AdapterId::from_u32(next_def)) }; // Manufactures a `CoreExport` for a memory with the shape specified. Note @@ -80,8 +82,8 @@ fn target(module: GenAdapterModule) -> Result<(), ()> { } else { dst[0] }; - CoreExport { - instance: RuntimeInstanceIndex::from_u32(idx), + dfg::CoreExport { + instance: dfg::InstanceId::from_u32(idx), item: ExportItem::Name(String::new()), } }; @@ -90,9 +92,9 @@ fn target(module: GenAdapterModule) -> Result<(), ()> { for adapter in module.adapters.iter() { let mut params = Vec::new(); for param in adapter.ty.params.iter() { - params.push((None, intern(&mut types, param)?)); + params.push((None, intern(&mut types, param))); } - let result = intern(&mut types, &adapter.ty.result)?; + let result = intern(&mut types, &adapter.ty.result); let signature = types.add_func_type(TypeFunc { params: params.into(), result, @@ -143,7 +145,7 @@ fn target(module: GenAdapterModule) -> Result<(), ()> { .validate_all(&wasm); let err = match result { - Ok(_) => return Ok(()), + Ok(_) => return, Err(e) => e, }; eprintln!("invalid wasm module: {err:?}"); @@ -159,8 +161,8 @@ fn target(module: GenAdapterModule) -> Result<(), ()> { panic!() } -fn intern(types: &mut ComponentTypesBuilder, ty: &ValType) -> Result { - Ok(match ty { +fn intern(types: &mut ComponentTypesBuilder, ty: &ValType) -> InterfaceType { + match ty { ValType::Unit => InterfaceType::Unit, ValType::Bool => InterfaceType::Bool, ValType::U8 => InterfaceType::U8, @@ -174,8 +176,9 @@ fn intern(types: &mut ComponentTypesBuilder, ty: &ValType) -> Result InterfaceType::Float32, ValType::Float64 => InterfaceType::Float64, ValType::Char => InterfaceType::Char, + ValType::String => InterfaceType::String, ValType::List(ty) => { - let ty = intern(types, ty)?; + let ty = intern(types, ty); InterfaceType::List(types.add_interface_type(ty)) } ValType::Record(tys) => { @@ -183,13 +186,11 @@ fn intern(types: &mut ComponentTypesBuilder, ty: &ValType) -> Result>()?, + .collect(), }; InterfaceType::Record(types.add_record_type(ty)) } @@ -201,10 +202,7 @@ fn intern(types: &mut ComponentTypesBuilder, ty: &ValType) -> Result { let ty = TypeTuple { - types: tys - .iter() - .map(|ty| intern(types, ty)) - .collect::>()?, + types: tys.iter().map(|ty| intern(types, ty)).collect(), }; InterfaceType::Tuple(types.add_tuple_type(ty)) } @@ -213,22 +211,17 @@ fn intern(types: &mut ComponentTypesBuilder, ty: &ValType) -> Result>()?, + .collect(), }; InterfaceType::Variant(types.add_variant_type(ty)) } ValType::Union(tys) => { let ty = TypeUnion { - types: tys - .iter() - .map(|ty| intern(types, ty)) - .collect::>()?, + types: tys.iter().map(|ty| intern(types, ty)).collect(), }; InterfaceType::Union(types.add_union_type(ty)) } @@ -239,16 +232,15 @@ fn intern(types: &mut ComponentTypesBuilder, ty: &ValType) -> Result { - let ty = intern(types, ty)?; + let ty = intern(types, ty); InterfaceType::Option(types.add_interface_type(ty)) } ValType::Expected { ok, err } => { - let ok = intern(types, ok)?; - let err = intern(types, err)?; + let ok = intern(types, ok); + let err = intern(types, err); InterfaceType::Expected(types.add_expected_type(TypeExpected { ok, err })) } - ValType::String => return Err(()), - }) + } } impl From for StringEncoding { diff --git a/crates/environ/src/component.rs b/crates/environ/src/component.rs index d6c1c2838492..0961b356e9a1 100644 --- a/crates/environ/src/component.rs +++ b/crates/environ/src/component.rs @@ -48,3 +48,25 @@ pub use self::info::*; pub use self::translate::*; pub use self::types::*; pub use self::vmcomponent_offsets::*; + +/// Helper macro to iterate over the transcoders that the host will provide +/// adapter modules through libcalls. +#[macro_export] +macro_rules! foreach_transcoder { + ($mac:ident) => { + $mac! { + utf8_to_utf8(src: ptr_u8, len: size, dst: ptr_u8); + utf16_to_utf16(src: ptr_u16, len: size, dst: ptr_u16); + latin1_to_latin1(src: ptr_u8, len: size, dst: ptr_u8); + latin1_to_utf16(src: ptr_u8, len: size, dst: ptr_u16); + utf8_to_utf16(src: ptr_u8, len: size, dst: ptr_u16) -> size; + utf16_to_utf8(src: ptr_u16, src_len: size, dst: ptr_u8, dst_len: size) -> size_pair; + latin1_to_utf8(src: ptr_u8, src_len: size, dst: ptr_u8, dst_len: size) -> size_pair; + utf16_to_compact_probably_utf16(src: ptr_u16, len: size, dst: ptr_u16) -> size; + utf8_to_latin1(src: ptr_u8, len: size, dst: ptr_u8) -> size_pair; + utf16_to_latin1(src: ptr_u16, len: size, dst: ptr_u8) -> size_pair; + utf8_to_compact_utf16(src: ptr_u8, src_len: size, dst: ptr_u16, dst_len: size, bytes_so_far: size) -> size; + utf16_to_compact_utf16(src: ptr_u16, src_len: size, dst: ptr_u16, dst_len: size, bytes_so_far: size) -> size; + } + }; +} diff --git a/crates/environ/src/component/compiler.rs b/crates/environ/src/component/compiler.rs index ac07a3c1d540..212b6ca3c265 100644 --- a/crates/environ/src/component/compiler.rs +++ b/crates/environ/src/component/compiler.rs @@ -1,5 +1,6 @@ use crate::component::{ Component, ComponentTypes, LowerImport, LoweredIndex, RuntimeAlwaysTrapIndex, + RuntimeTranscoderIndex, Transcoder, }; use crate::{PrimaryMap, SignatureIndex, Trampoline, WasmFuncType}; use anyhow::Result; @@ -61,6 +62,24 @@ pub trait ComponentCompiler: Send + Sync { /// `canon lift`'d function immediately being `canon lower`'d. fn compile_always_trap(&self, ty: &WasmFuncType) -> Result>; + /// Compiles a trampoline to implement string transcoding from adapter + /// modules. + /// + /// The generated trampoline will invoke the `transcoder.op` libcall with + /// the various memory configuration provided in `transcoder`. This is used + /// to pass raw pointers to host functions to avoid the host having to deal + /// with base pointers, offsets, memory32-vs-64, etc. + /// + /// Note that all bounds checks for memories are present in adapters + /// themselves, and the host libcalls simply assume that the pointers are + /// valid. + fn compile_transcoder( + &self, + component: &Component, + transcoder: &Transcoder, + types: &ComponentTypes, + ) -> Result>; + /// Emits the `lowerings` and `trampolines` specified into the in-progress /// ELF object specified by `obj`. /// @@ -73,11 +92,13 @@ pub trait ComponentCompiler: Send + Sync { &self, lowerings: PrimaryMap>, always_trap: PrimaryMap>, + transcoders: PrimaryMap>, tramplines: Vec<(SignatureIndex, Box)>, obj: &mut Object<'static>, ) -> Result<( PrimaryMap, PrimaryMap, + PrimaryMap, Vec, )>; } diff --git a/crates/environ/src/component/dfg.rs b/crates/environ/src/component/dfg.rs index 7aa35d603d13..ae986b55a2cd 100644 --- a/crates/environ/src/component/dfg.rs +++ b/crates/environ/src/component/dfg.rs @@ -71,6 +71,9 @@ pub struct ComponentDfg { /// out of the inlining pass of translation. pub adapters: Intern, + /// Metadata about string transcoders needed by adapter modules. + pub transcoders: Intern, + /// Metadata about all known core wasm instances created. /// /// This is mostly an ordered list and is not deduplicated based on contents @@ -125,6 +128,7 @@ id! { pub struct PostReturnId(u32); pub struct AlwaysTrapId(u32); pub struct AdapterModuleId(u32); + pub struct TranscoderId(u32); } /// Same as `info::InstantiateModule` @@ -158,6 +162,7 @@ pub enum CoreDef { Lowered(LowerImportId), AlwaysTrap(AlwaysTrapId), InstanceFlags(RuntimeComponentInstanceIndex), + Transcoder(TranscoderId), /// This is a special variant not present in `info::CoreDef` which /// represents that this definition refers to a fused adapter function. This @@ -220,6 +225,18 @@ pub struct CanonicalOptions { pub post_return: Option, } +/// Same as `info::Transcoder` +#[derive(Clone, Hash, Eq, PartialEq)] +#[allow(missing_docs)] +pub struct Transcoder { + pub op: Transcode, + pub from: MemoryId, + pub from64: bool, + pub to: MemoryId, + pub to64: bool, + pub signature: SignatureIndex, +} + /// A helper structure to "intern" and deduplicate values of type `V` with an /// identifying key `K`. /// @@ -292,6 +309,7 @@ impl ComponentDfg { runtime_instances: Default::default(), runtime_always_trap: Default::default(), runtime_lowerings: Default::default(), + runtime_transcoders: Default::default(), }; // First the instances are all processed for instantiation. This will, @@ -324,6 +342,7 @@ impl ComponentDfg { num_runtime_instances: linearize.runtime_instances.len() as u32, num_always_trap: linearize.runtime_always_trap.len() as u32, num_lowerings: linearize.runtime_lowerings.len() as u32, + num_transcoders: linearize.runtime_transcoders.len() as u32, imports: self.imports, import_types: self.import_types, @@ -342,6 +361,7 @@ struct LinearizeDfg<'a> { runtime_instances: HashMap, runtime_always_trap: HashMap, runtime_lowerings: HashMap, + runtime_transcoders: HashMap, } #[derive(Copy, Clone, Hash, Eq, PartialEq)] @@ -460,6 +480,7 @@ impl LinearizeDfg<'_> { CoreDef::Lowered(id) => info::CoreDef::Lowered(self.runtime_lowering(*id)), CoreDef::InstanceFlags(i) => info::CoreDef::InstanceFlags(*i), CoreDef::Adapter(id) => info::CoreDef::Export(self.adapter(*id)), + CoreDef::Transcoder(id) => info::CoreDef::Transcoder(self.runtime_transcoder(*id)), } } @@ -497,6 +518,35 @@ impl LinearizeDfg<'_> { ) } + fn runtime_transcoder(&mut self, id: TranscoderId) -> RuntimeTranscoderIndex { + self.intern( + id, + |me| &mut me.runtime_transcoders, + |me, id| { + let info = &me.dfg.transcoders[id]; + ( + info.op, + me.runtime_memory(info.from), + info.from64, + me.runtime_memory(info.to), + info.to64, + info.signature, + ) + }, + |index, (op, from, from64, to, to64, signature)| { + GlobalInitializer::Transcoder(info::Transcoder { + index, + op, + from, + from64, + to, + to64, + signature, + }) + }, + ) + } + fn core_export(&mut self, export: &CoreExport) -> info::CoreExport where T: Clone, diff --git a/crates/environ/src/component/info.rs b/crates/environ/src/component/info.rs index 63f8545b825a..d0c32bd0f1a8 100644 --- a/crates/environ/src/component/info.rs +++ b/crates/environ/src/component/info.rs @@ -147,6 +147,10 @@ pub struct Component { /// The number of functions which "always trap" used to implement /// `canon.lower` of `canon.lift`'d functions within the same component. pub num_always_trap: u32, + + /// The number of host transcoder functions needed for strings in adapter + /// modules. + pub num_transcoders: u32, } /// GlobalInitializer instructions to get processed when instantiating a component @@ -207,6 +211,11 @@ pub enum GlobalInitializer { /// Same as `SaveModuleUpvar`, but for imports. SaveModuleImport(RuntimeImportIndex), + + /// Similar to `ExtractMemory` and friends and indicates that a + /// `VMCallerCheckedAnyfunc` needs to be initialized for a transcoder + /// function and this will later be used to instantiate an adapter module. + Transcoder(Transcoder), } /// Metadata for extraction of a memory of what's being extracted and where it's @@ -316,6 +325,9 @@ pub enum CoreDef { /// This is a reference to a wasm global which represents the /// runtime-managed flags for a wasm instance. InstanceFlags(RuntimeComponentInstanceIndex), + /// This refers to a cranelift-generated trampoline which calls to a + /// host-defined transcoding function. + Transcoder(RuntimeTranscoderIndex), } impl From> for CoreDef @@ -433,3 +445,42 @@ pub enum StringEncoding { Utf16, CompactUtf16, } + +/// Information about a string transcoding function required by an adapter +/// module. +/// +/// A transcoder is used when strings are passed between adapter modules, +/// optionally changing string encodings at the same time. The transcoder is +/// implemented in a few different layers: +/// +/// * Each generated adapter module has some glue around invoking the transcoder +/// represented by this item. This involves bounds-checks and handling +/// `realloc` for example. +/// * Each transcoder gets a cranelift-generated trampoline which has the +/// appropriate signature for the adapter module in question. Existence of +/// this initializer indicates that this should be compiled by Cranelift. +/// * The cranelift-generated trampoline will invoke a "transcoder libcall" +/// which is implemented natively in Rust that has a signature independent of +/// memory64 configuration options for example. +#[derive(Debug, Clone, Serialize, Deserialize, Hash, Eq, PartialEq)] +pub struct Transcoder { + /// The index of the transcoder being defined and initialized. + /// + /// This indicates which `VMCallerCheckedAnyfunc` slot is written to in a + /// `VMComponentContext`. + pub index: RuntimeTranscoderIndex, + /// The transcoding operation being performed. + pub op: Transcode, + /// The linear memory that the string is being read from. + pub from: RuntimeMemoryIndex, + /// Whether or not the source linear memory is 64-bit or not. + pub from64: bool, + /// The linear memory that the string is being written to. + pub to: RuntimeMemoryIndex, + /// Whether or not the destination linear memory is 64-bit or not. + pub to64: bool, + /// The wasm signature of the cranelift-generated trampoline. + pub signature: SignatureIndex, +} + +pub use crate::fact::{FixedEncoding, Transcode}; diff --git a/crates/environ/src/component/translate/adapt.rs b/crates/environ/src/component/translate/adapt.rs index 7289bb555881..1b8c6c6830be 100644 --- a/crates/environ/src/component/translate/adapt.rs +++ b/crates/environ/src/component/translate/adapt.rs @@ -116,7 +116,8 @@ //! created. use crate::component::translate::*; -use crate::fact::Module; +use crate::fact; +use crate::EntityType; use std::collections::HashSet; use wasmparser::WasmFeatures; @@ -183,7 +184,7 @@ impl<'data> Translator<'_, 'data> { // the module using standard core wasm translation, and then fills out // the dfg metadata for each adapter. for (module_id, adapter_module) in state.adapter_modules.iter() { - let mut module = Module::new( + let mut module = fact::Module::new( self.types.component_types(), self.tunables.debug_adapter_modules, ); @@ -194,7 +195,7 @@ impl<'data> Translator<'_, 'data> { names.push(name); } let wasm = module.encode(); - let args = module.imports().to_vec(); + let imports = module.imports().to_vec(); // Extend the lifetime of the owned `wasm: Vec` on the stack to // a higher scope defined by our original caller. That allows to @@ -240,6 +241,12 @@ impl<'data> Translator<'_, 'data> { // module is also recorded in the dfg. This metadata will be used // to generate `GlobalInitializer` entries during the linearization // final phase. + assert_eq!(imports.len(), translation.module.imports().len()); + let args = imports + .iter() + .zip(translation.module.imports()) + .map(|(arg, (_, _, ty))| fact_import_to_core_def(component, arg, ty)) + .collect::>(); let static_index = self.static_modules.push(translation); let id = component.adapter_modules.push((static_index, args.into())); assert_eq!(id, module_id); @@ -247,6 +254,47 @@ impl<'data> Translator<'_, 'data> { } } +fn fact_import_to_core_def( + dfg: &mut dfg::ComponentDfg, + import: &fact::Import, + ty: EntityType, +) -> dfg::CoreDef { + match import { + fact::Import::CoreDef(def) => def.clone(), + fact::Import::Transcode { + op, + from, + from64, + to, + to64, + } => { + fn unwrap_memory(def: &dfg::CoreDef) -> dfg::CoreExport { + match def { + dfg::CoreDef::Export(e) => e.clone().map_index(|i| match i { + EntityIndex::Memory(i) => i, + _ => unreachable!(), + }), + _ => unreachable!(), + } + } + + let from = dfg.memories.push_uniq(unwrap_memory(from)); + let to = dfg.memories.push_uniq(unwrap_memory(to)); + dfg::CoreDef::Transcoder(dfg.transcoders.push_uniq(dfg::Transcoder { + op: *op, + from, + from64: *from64, + to, + to64: *to64, + signature: match ty { + EntityType::Function(signature) => signature, + _ => unreachable!(), + }, + })) + } + } +} + #[derive(Default)] struct PartitionAdapterModules { /// The next adapter module that's being created. This may be empty. @@ -336,6 +384,9 @@ impl PartitionAdapterModules { dfg::CoreDef::Lowered(_) | dfg::CoreDef::AlwaysTrap(_) | dfg::CoreDef::InstanceFlags(_) => {} + + // should not be in the dfg yet + dfg::CoreDef::Transcoder(_) => unreachable!(), } } diff --git a/crates/environ/src/component/types.rs b/crates/environ/src/component/types.rs index 3f1ade9d7748..da13b7de5c19 100644 --- a/crates/environ/src/component/types.rs +++ b/crates/environ/src/component/types.rs @@ -166,6 +166,13 @@ indices! { /// Index that represents an exported module from a component since that's /// currently the only use for saving the entire module state at runtime. pub struct RuntimeModuleIndex(u32); + + /// Index into the list of transcoders identified during compilation. + /// + /// This is used to index the `VMCallerCheckedAnyfunc` slots reserved for + /// string encoders which reference linear memories defined within a + /// component. + pub struct RuntimeTranscoderIndex(u32); } // Reexport for convenience some core-wasm indices which are also used in the diff --git a/crates/environ/src/component/vmcomponent_offsets.rs b/crates/environ/src/component/vmcomponent_offsets.rs index 649aae50f9a4..8d5a1388673d 100644 --- a/crates/environ/src/component/vmcomponent_offsets.rs +++ b/crates/environ/src/component/vmcomponent_offsets.rs @@ -2,11 +2,13 @@ // // struct VMComponentContext { // magic: u32, +// transcode_libcalls: &'static VMBuiltinTranscodeArray, // store: *mut dyn Store, // limits: *const VMRuntimeLimits, // flags: [VMGlobalDefinition; component.num_runtime_component_instances], // lowering_anyfuncs: [VMCallerCheckedAnyfunc; component.num_lowerings], // always_trap_anyfuncs: [VMCallerCheckedAnyfunc; component.num_always_trap], +// transcoder_anyfuncs: [VMCallerCheckedAnyfunc; component.num_transcoders], // lowerings: [VMLowering; component.num_lowerings], // memories: [*mut VMMemoryDefinition; component.num_memories], // reallocs: [*mut VMCallerCheckedAnyfunc; component.num_reallocs], @@ -15,7 +17,7 @@ use crate::component::{ Component, LoweredIndex, RuntimeAlwaysTrapIndex, RuntimeComponentInstanceIndex, - RuntimeMemoryIndex, RuntimePostReturnIndex, RuntimeReallocIndex, + RuntimeMemoryIndex, RuntimePostReturnIndex, RuntimeReallocIndex, RuntimeTranscoderIndex, }; use crate::PtrSize; @@ -57,14 +59,18 @@ pub struct VMComponentOffsets

{ /// Number of "always trap" functions which have their /// `VMCallerCheckedAnyfunc` stored inline in the `VMComponentContext`. pub num_always_trap: u32, + /// Number of transcoders needed for string conversion. + pub num_transcoders: u32, // precalculated offsets of various member fields magic: u32, + transcode_libcalls: u32, store: u32, limits: u32, flags: u32, lowering_anyfuncs: u32, always_trap_anyfuncs: u32, + transcoder_anyfuncs: u32, lowerings: u32, memories: u32, reallocs: u32, @@ -93,12 +99,15 @@ impl VMComponentOffsets

{ .try_into() .unwrap(), num_always_trap: component.num_always_trap, + num_transcoders: component.num_transcoders, magic: 0, + transcode_libcalls: 0, store: 0, limits: 0, flags: 0, lowering_anyfuncs: 0, always_trap_anyfuncs: 0, + transcoder_anyfuncs: 0, lowerings: 0, memories: 0, reallocs: 0, @@ -133,6 +142,7 @@ impl VMComponentOffsets

{ fields! { size(magic) = 4u32, align(u32::from(ret.ptr.size())), + size(transcode_libcalls) = ret.ptr.size(), size(store) = cmul(2, ret.ptr.size()), size(limits) = ret.ptr.size(), align(16), @@ -140,6 +150,7 @@ impl VMComponentOffsets

{ align(u32::from(ret.ptr.size())), size(lowering_anyfuncs) = cmul(ret.num_lowerings, ret.ptr.size_of_vmcaller_checked_anyfunc()), size(always_trap_anyfuncs) = cmul(ret.num_always_trap, ret.ptr.size_of_vmcaller_checked_anyfunc()), + size(transcoder_anyfuncs) = cmul(ret.num_transcoders, ret.ptr.size_of_vmcaller_checked_anyfunc()), size(lowerings) = cmul(ret.num_lowerings, ret.ptr.size() * 2), size(memories) = cmul(ret.num_runtime_memories, ret.ptr.size()), size(reallocs) = cmul(ret.num_runtime_reallocs, ret.ptr.size()), @@ -168,6 +179,12 @@ impl VMComponentOffsets

{ self.magic } + /// The offset of the `transcode_libcalls` field. + #[inline] + pub fn transcode_libcalls(&self) -> u32 { + self.transcode_libcalls + } + /// The offset of the `flags` field. #[inline] pub fn instance_flags(&self, index: RuntimeComponentInstanceIndex) -> u32 { @@ -215,6 +232,20 @@ impl VMComponentOffsets

{ + index.as_u32() * u32::from(self.ptr.size_of_vmcaller_checked_anyfunc()) } + /// The offset of the `transcoder_anyfuncs` field. + #[inline] + pub fn transcoder_anyfuncs(&self) -> u32 { + self.transcoder_anyfuncs + } + + /// The offset of `VMCallerCheckedAnyfunc` for the `index` specified. + #[inline] + pub fn transcoder_anyfunc(&self, index: RuntimeTranscoderIndex) -> u32 { + assert!(index.as_u32() < self.num_transcoders); + self.transcoder_anyfuncs() + + index.as_u32() * u32::from(self.ptr.size_of_vmcaller_checked_anyfunc()) + } + /// The offset of the `lowerings` field. #[inline] pub fn lowerings(&self) -> u32 { diff --git a/crates/environ/src/fact.rs b/crates/environ/src/fact.rs index 2c9f7b73dd80..3cb16a68c800 100644 --- a/crates/environ/src/fact.rs +++ b/crates/environ/src/fact.rs @@ -20,7 +20,7 @@ use crate::component::dfg::CoreDef; use crate::component::{Adapter, AdapterOptions, ComponentTypes, StringEncoding, TypeFuncIndex}; -use crate::{FuncIndex, GlobalIndex, MemoryIndex}; +use crate::{FuncIndex, GlobalIndex, MemoryIndex, PrimaryMap}; use std::collections::HashMap; use std::mem; use wasm_encoder::*; @@ -28,8 +28,11 @@ use wasm_encoder::*; mod core_types; mod signature; mod trampoline; +mod transcode; mod traps; +pub use self::transcode::{FixedEncoding, Transcode}; + /// Representation of an adapter module. pub struct Module<'a> { /// Whether or not debug code is inserted into the adapters themselves. @@ -46,9 +49,10 @@ pub struct Module<'a> { core_imports: ImportSection, /// Final list of imports that this module ended up using, in the same order /// as the imports in the import section. - imports: Vec, + imports: Vec, /// Intern'd imports and what index they were assigned. imported: HashMap, + imported_memories: PrimaryMap, // Current status of index spaces from the imports generated so far. core_funcs: u32, @@ -100,6 +104,7 @@ impl<'a> Module<'a> { imported: Default::default(), adapters: Default::default(), imports: Default::default(), + imported_memories: PrimaryMap::new(), core_funcs: 0, core_memories: 0, core_globals: 0, @@ -246,19 +251,50 @@ impl<'a> Module<'a> { let ret = *cnt - 1; self.core_imports.import(module, name, ty); self.imported.insert(def.clone(), ret); - self.imports.push(def); + if let EntityType::Memory(_) = ty { + self.imported_memories.push(def.clone()); + } + self.imports.push(Import::CoreDef(def)); ret } /// Encodes this module into a WebAssembly binary. pub fn encode(&mut self) -> Vec { + let mut types = mem::take(&mut self.core_types); + let mut transcoders = transcode::Transcoders::new(self.core_funcs); + let mut adapter_funcs = Vec::new(); + for adapter in self.adapters.iter() { + adapter_funcs.push(trampoline::compile( + self, + &mut types, + &mut transcoders, + adapter, + )); + } + + // If any string transcoding imports were needed add imported items + // associated with them. + for (module, name, ty, transcoder) in transcoders.imports() { + self.core_imports.import(module, name, ty); + let from = self.imported_memories[transcoder.from_memory].clone(); + let to = self.imported_memories[transcoder.to_memory].clone(); + self.imports.push(Import::Transcode { + op: transcoder.op, + from, + from64: transcoder.from_memory64, + to, + to64: transcoder.to_memory64, + }); + self.core_funcs += 1; + } + + // Now that all functions are known as well as all imports the actual + // bodies of all adapters are assembled into a final module. let mut funcs = FunctionSection::new(); let mut code = CodeSection::new(); let mut exports = ExportSection::new(); let mut traps = traps::TrapSection::default(); - - let mut types = mem::take(&mut self.core_types); - for adapter in self.adapters.iter() { + for (adapter, (function, func_traps)) in self.adapters.iter().zip(adapter_funcs) { let idx = self.core_funcs + funcs.len(); exports.export(&adapter.name, ExportKind::Func, idx); @@ -266,7 +302,6 @@ impl<'a> Module<'a> { let ty = types.function(&signature.params, &signature.results); funcs.function(ty); - let (function, func_traps) = trampoline::compile(self, &mut types, adapter); code.raw(&function); traps.append(idx, func_traps); } @@ -290,11 +325,31 @@ impl<'a> Module<'a> { /// Returns the imports that were used, in order, to create this adapter /// module. - pub fn imports(&self) -> &[CoreDef] { + pub fn imports(&self) -> &[Import] { &self.imports } } +/// Possible imports into an adapter module. +#[derive(Clone)] +pub enum Import { + /// A definition required in the configuration of an `Adapter`. + CoreDef(CoreDef), + /// A transcoding function from the host to convert between string encodings. + Transcode { + /// The transcoding operation this performs. + op: Transcode, + /// The memory being read + from: CoreDef, + /// Whether or not `from` is a 64-bit memory + from64: bool, + /// The memory being written + to: CoreDef, + /// Whether or not `to` is a 64-bit memory + to64: bool, + }, +} + impl Options { fn ptr(&self) -> ValType { if self.memory64 { diff --git a/crates/environ/src/fact/trampoline.rs b/crates/environ/src/fact/trampoline.rs index 16e528dd1b4f..b5337229a2f8 100644 --- a/crates/environ/src/fact/trampoline.rs +++ b/crates/environ/src/fact/trampoline.rs @@ -16,12 +16,13 @@ //! can be somewhat arbitrary, an intentional decision. use crate::component::{ - InterfaceType, TypeEnumIndex, TypeExpectedIndex, TypeFlagsIndex, TypeInterfaceIndex, - TypeRecordIndex, TypeTupleIndex, TypeUnionIndex, TypeVariantIndex, FLAG_MAY_ENTER, - FLAG_MAY_LEAVE, MAX_FLAT_PARAMS, MAX_FLAT_RESULTS, + InterfaceType, StringEncoding, TypeEnumIndex, TypeExpectedIndex, TypeFlagsIndex, + TypeInterfaceIndex, TypeRecordIndex, TypeTupleIndex, TypeUnionIndex, TypeVariantIndex, + FLAG_MAY_ENTER, FLAG_MAY_LEAVE, MAX_FLAT_PARAMS, MAX_FLAT_RESULTS, }; use crate::fact::core_types::CoreTypes; use crate::fact::signature::{align_to, Signature}; +use crate::fact::transcode::{FixedEncoding as FE, Transcode, Transcoder, Transcoders}; use crate::fact::traps::Trap; use crate::fact::{AdapterData, Context, Module, Options}; use crate::GlobalIndex; @@ -31,6 +32,9 @@ use std::ops::Range; use wasm_encoder::{BlockType, Encode, Instruction, Instruction::*, MemArg, ValType}; use wasmtime_component_util::{DiscriminantSize, FlagsSize}; +const MAX_STRING_BYTE_LENGTH: u32 = 1 << 31; +const UTF16_TAG: u32 = 1 << 31; + struct Compiler<'a, 'b> { /// The module that the adapter will eventually be inserted into. module: &'a Module<'a>, @@ -38,6 +42,9 @@ struct Compiler<'a, 'b> { /// The type section of `module` types: &'b mut CoreTypes, + /// Imported functions to transcode between various string encodings. + transcoders: &'b mut Transcoders, + /// Metadata about the adapter that is being compiled. adapter: &'a AdapterData, @@ -71,6 +78,7 @@ struct Compiler<'a, 'b> { pub(super) fn compile( module: &Module<'_>, types: &mut CoreTypes, + transcoders: &mut Transcoders, adapter: &AdapterData, ) -> (Vec, Vec<(usize, Trap)>) { let lower_sig = &module.signature(&adapter.lower, Context::Lower); @@ -79,6 +87,7 @@ pub(super) fn compile( module, types, adapter, + transcoders, code: Vec::new(), locals: Vec::new(), nlocals: lower_sig.params.len() as u32, @@ -356,6 +365,7 @@ impl Compiler<'_, '_> { InterfaceType::Float32 => self.translate_f32(src, dst_ty, dst), InterfaceType::Float64 => self.translate_f64(src, dst_ty, dst), InterfaceType::Char => self.translate_char(src, dst_ty, dst), + InterfaceType::String => self.translate_string(src, dst_ty, dst), InterfaceType::List(t) => self.translate_list(*t, src, dst_ty, dst), InterfaceType::Record(t) => self.translate_record(*t, src, dst_ty, dst), InterfaceType::Flags(f) => self.translate_flags(*f, src, dst_ty, dst), @@ -365,13 +375,6 @@ impl Compiler<'_, '_> { InterfaceType::Enum(t) => self.translate_enum(*t, src, dst_ty, dst), InterfaceType::Option(t) => self.translate_option(*t, src, dst_ty, dst), InterfaceType::Expected(t) => self.translate_expected(*t, src, dst_ty, dst), - - InterfaceType::String => { - // consider this field used for now until this is fully - // implemented. - drop(&self.adapter.lift.string_encoding); - unimplemented!("don't know how to translate strings") - } } } @@ -636,6 +639,768 @@ impl Compiler<'_, '_> { } } + fn translate_string(&mut self, src: &Source<'_>, dst_ty: &InterfaceType, dst: &Destination) { + assert!(matches!(dst_ty, InterfaceType::String)); + let src_opts = src.opts(); + let dst_opts = dst.opts(); + + // Load the pointer/length of this string into temporary locals. These + // will be referenced a good deal so this just makes it easier to deal + // with them consistently below rather than trying to reload from memory + // for example. + let src_ptr = self.gen_local(src_opts.ptr()); + let src_len = self.gen_local(src_opts.ptr()); + match src { + Source::Stack(s) => { + assert_eq!(s.locals.len(), 2); + self.stack_get(&s.slice(0..1), src_opts.ptr()); + self.instruction(LocalSet(src_ptr)); + self.stack_get(&s.slice(1..2), src_opts.ptr()); + self.instruction(LocalSet(src_len)); + } + Source::Memory(mem) => { + self.ptr_load(mem); + self.instruction(LocalSet(src_ptr)); + self.ptr_load(&mem.bump(src_opts.ptr_size().into())); + self.instruction(LocalSet(src_len)); + } + } + let src_str = &WasmString { + ptr: src_ptr, + len: src_len, + opts: src_opts, + }; + + let dst_str = match src_opts.string_encoding { + StringEncoding::Utf8 => match dst_opts.string_encoding { + StringEncoding::Utf8 => self.string_copy(src_str, FE::Utf8, dst_opts, FE::Utf8), + StringEncoding::Utf16 => self.string_utf8_to_utf16(src_str, dst_opts), + StringEncoding::CompactUtf16 => self.string_to_compact(src_str, FE::Utf8, dst_opts), + }, + + StringEncoding::Utf16 => { + self.verify_aligned(src_opts, src_ptr, 2); + match dst_opts.string_encoding { + StringEncoding::Utf8 => { + self.string_deflate_to_utf8(src_str, FE::Utf16, dst_opts) + } + StringEncoding::Utf16 => { + self.string_copy(src_str, FE::Utf16, dst_opts, FE::Utf16) + } + StringEncoding::CompactUtf16 => { + self.string_to_compact(src_str, FE::Utf16, dst_opts) + } + } + } + + StringEncoding::CompactUtf16 => { + self.verify_aligned(src_opts, src_ptr, 2); + + // Test the tag big to see if this is a utf16 or a latin1 string + // at runtime... + self.instruction(LocalGet(src_len)); + self.ptr_uconst(src_opts, UTF16_TAG); + self.ptr_and(src_opts); + self.ptr_if(src_opts, BlockType::Empty); + + // In the utf16 block unset the upper bit from the length local + // so further calculations have the right value. Afterwards the + // string transcode proceeds assuming utf16. + self.instruction(LocalGet(src_len)); + self.ptr_uconst(src_opts, UTF16_TAG); + self.ptr_xor(src_opts); + self.instruction(LocalSet(src_len)); + let s1 = match dst_opts.string_encoding { + StringEncoding::Utf8 => { + self.string_deflate_to_utf8(src_str, FE::Utf16, dst_opts) + } + StringEncoding::Utf16 => { + self.string_copy(src_str, FE::Utf16, dst_opts, FE::Utf16) + } + StringEncoding::CompactUtf16 => { + self.string_compact_utf16_to_compact(src_str, dst_opts) + } + }; + + self.instruction(Else); + + // In the latin1 block the `src_len` local is already the number + // of code units, so the string transcoding is all that needs to + // happen. + let s2 = match dst_opts.string_encoding { + StringEncoding::Utf16 => { + self.string_copy(src_str, FE::Latin1, dst_opts, FE::Utf16) + } + StringEncoding::Utf8 => { + self.string_deflate_to_utf8(src_str, FE::Latin1, dst_opts) + } + StringEncoding::CompactUtf16 => { + self.string_copy(src_str, FE::Latin1, dst_opts, FE::Latin1) + } + }; + // Set our `s2` generated locals to the `s2` generated locals + // as the resulting pointer of this transcode. + self.instruction(LocalGet(s2.ptr)); + self.instruction(LocalSet(s1.ptr)); + self.instruction(LocalGet(s2.len)); + self.instruction(LocalSet(s1.len)); + self.instruction(End); + s1 + } + }; + + // Store the ptr/length in the desired destination + match dst { + Destination::Stack(s, _) => { + self.instruction(LocalGet(dst_str.ptr)); + self.stack_set(&s[..1], dst_opts.ptr()); + self.instruction(LocalGet(dst_str.len)); + self.stack_set(&s[1..], dst_opts.ptr()); + } + Destination::Memory(mem) => { + self.instruction(LocalGet(mem.addr_local)); + self.instruction(LocalGet(dst_str.ptr)); + self.ptr_store(mem); + self.instruction(LocalGet(mem.addr_local)); + self.instruction(LocalGet(dst_str.len)); + self.ptr_store(&mem.bump(dst_opts.ptr_size().into())); + } + } + } + + // Corresponding function for `store_string_copy` in the spec. + // + // This performs a transcoding of the string with a one-pass copy from + // the `src` encoding to the `dst` encoding. This is only possible for + // fixed encodings where the first allocation is guaranteed to be an + // appropriate fit so it's not suitable for all encodings. + // + // Imported host transcoding functions here take the src/dst pointers as + // well as the number of code units in the source (which always matches + // the number of code units in the destination). There is no return + // value from the transcode function since the encoding should always + // work on the first pass. + fn string_copy<'a>( + &mut self, + src: &WasmString<'_>, + src_enc: FE, + dst_opts: &'a Options, + dst_enc: FE, + ) -> WasmString<'a> { + assert!(dst_enc.width() >= src_enc.width()); + self.validate_string_length(src, dst_enc); + + // Calculate the source byte length given the size of each code + // unit. Note that this shouldn't overflow given + // `validate_string_length` above. + let src_byte_len = if src_enc.width() == 1 { + src.len + } else { + assert_eq!(src_enc.width(), 2); + let tmp = self.gen_local(src.opts.ptr()); + self.instruction(LocalGet(src.len)); + self.ptr_uconst(src.opts, 1); + self.ptr_shl(src.opts); + self.instruction(LocalSet(tmp)); + tmp + }; + + // Convert the source code units length to the destination byte + // length type. + self.convert_src_len_to_dst(src.len, src.opts.ptr(), dst_opts.ptr()); + let dst_len = self.gen_local(dst_opts.ptr()); + self.instruction(LocalTee(dst_len)); + if dst_enc.width() > 1 { + assert_eq!(dst_enc.width(), 2); + self.ptr_uconst(dst_opts, 1); + self.ptr_shl(dst_opts); + } + let dst_byte_len = self.gen_local(dst_opts.ptr()); + self.instruction(LocalSet(dst_byte_len)); + + // Allocate space in the destination using the calculated byte + // length. + let dst = { + let dst_mem = self.malloc( + dst_opts, + MallocSize::Local(dst_byte_len), + dst_enc.width().into(), + ); + WasmString { + ptr: dst_mem.addr_local, + len: dst_len, + opts: dst_opts, + } + }; + + // Validate that `src_len + src_ptr` and + // `dst_mem.addr_local + dst_byte_len` are both in-bounds. This + // is done by loading the last byte of the string and if that + // doesn't trap then it's known valid. + self.validate_string_inbounds(src, src_byte_len); + self.validate_string_inbounds(&dst, dst_byte_len); + + // If the validations pass then the host `transcode` intrinsic + // is invoked. This will either raise a trap or otherwise succeed + // in which case we're done. + let op = if src_enc == dst_enc { + Transcode::Copy(src_enc) + } else { + assert_eq!(src_enc, FE::Latin1); + assert_eq!(dst_enc, FE::Utf16); + Transcode::Latin1ToUtf16 + }; + let transcode = self.transcoder(src, &dst, op); + self.instruction(LocalGet(src.ptr)); + self.instruction(LocalGet(src.len)); + self.instruction(LocalGet(dst.ptr)); + self.instruction(Call(transcode)); + + dst + } + // Corresponding function for `store_string_to_utf8` in the spec. + // + // This translation works by possibly performing a number of + // reallocations. First a buffer of size input-code-units is used to try + // to get the transcoding correct on the first try. If that fails the + // maximum worst-case size is used and then that is resized down if it's + // too large. + // + // The host transcoding function imported here will receive src ptr/len + // and dst ptr/len and return how many code units were consumed on both + // sides. The amount of code units consumed in the source dictates which + // branches are taken in this conversion. + fn string_deflate_to_utf8<'a>( + &mut self, + src: &WasmString<'_>, + src_enc: FE, + dst_opts: &'a Options, + ) -> WasmString<'a> { + self.validate_string_length(src, src_enc); + + // Optimistically assume that the code unit length of the source is + // all that's needed in the destination. Perform that allocaiton + // here and proceed to transcoding below. + self.convert_src_len_to_dst(src.len, src.opts.ptr(), dst_opts.ptr()); + let dst_len = self.gen_local(dst_opts.ptr()); + self.instruction(LocalTee(dst_len)); + let dst_byte_len = self.gen_local(dst_opts.ptr()); + self.instruction(LocalSet(dst_byte_len)); + + let dst = { + let dst_mem = self.malloc(dst_opts, MallocSize::Local(dst_byte_len), 1); + WasmString { + ptr: dst_mem.addr_local, + len: dst_len, + opts: dst_opts, + } + }; + + // Ensure buffers are all in-bounds + let src_byte_len = match src_enc { + FE::Latin1 => src.len, + FE::Utf16 => { + let tmp = self.gen_local(src.opts.ptr()); + self.instruction(LocalGet(src.len)); + self.ptr_uconst(src.opts, 1); + self.ptr_shl(src.opts); + self.instruction(LocalSet(tmp)); + tmp + } + FE::Utf8 => unreachable!(), + }; + self.validate_string_inbounds(src, src_byte_len); + self.validate_string_inbounds(&dst, dst_byte_len); + + // Perform the initial transcode + let op = match src_enc { + FE::Latin1 => Transcode::Latin1ToUtf8, + FE::Utf16 => Transcode::Utf16ToUtf8, + FE::Utf8 => unreachable!(), + }; + let transcode = self.transcoder(src, &dst, op); + self.instruction(LocalGet(src.ptr)); + self.instruction(LocalGet(src.len)); + self.instruction(LocalGet(dst.ptr)); + self.instruction(LocalGet(dst_byte_len)); + self.instruction(Call(transcode)); + self.instruction(LocalSet(dst.len)); + let src_len_tmp = self.gen_local(src.opts.ptr()); + self.instruction(LocalSet(src_len_tmp)); + + // Test if the source was entirely transcoded by comparing + // `src_len_tmp`, the number of code units transcoded from the + // source, with `src_len`, the original number of code units. + self.instruction(LocalGet(src_len_tmp)); + self.instruction(LocalGet(src.len)); + self.ptr_ne(src.opts); + self.instruction(If(BlockType::Empty)); + + // Here a worst-case reallocation is performed to grow `dst_mem`. + // In-line a check is also performed that the worst-case byte size + // fits within the maximum size of strings. + self.instruction(LocalGet(dst.ptr)); // old_ptr + self.instruction(LocalGet(dst_byte_len)); // old_size + self.ptr_uconst(dst.opts, 1); // align + let factor = match src_enc { + FE::Latin1 => 2, + FE::Utf16 => 3, + _ => unreachable!(), + }; + self.validate_string_length_u8(src, factor); + self.convert_src_len_to_dst(src.len, src.opts.ptr(), dst_opts.ptr()); + self.ptr_uconst(dst_opts, factor.into()); + self.ptr_mul(dst_opts); + self.instruction(LocalTee(dst_byte_len)); + self.instruction(Call(dst_opts.realloc.unwrap().as_u32())); + self.instruction(LocalSet(dst.ptr)); + + // Verify that the destination is still in-bounds + self.validate_string_inbounds(&dst, dst_byte_len); + + // Perform another round of transcoding that should be guaranteed + // to succeed. Note that all the parameters here are offset by the + // results of the first transcoding to only perform the remaining + // transcode on the final units. + self.instruction(LocalGet(src.ptr)); + self.instruction(LocalGet(src_len_tmp)); + if let FE::Utf16 = src_enc { + self.ptr_uconst(src.opts, 1); + self.ptr_shl(src.opts); + } + self.ptr_add(src.opts); + self.instruction(LocalGet(src.len)); + self.instruction(LocalGet(src_len_tmp)); + self.ptr_sub(src.opts); + self.instruction(LocalGet(dst.ptr)); + self.instruction(LocalGet(dst.len)); + self.ptr_add(dst.opts); + self.instruction(LocalGet(dst_byte_len)); + self.instruction(LocalGet(dst.len)); + self.ptr_sub(dst.opts); + self.instruction(Call(transcode)); + + // Add the second result, the amount of destination units encoded, + // to `dst_len` so it's an accurate reflection of the final size of + // the destination buffer. + self.instruction(LocalGet(dst.len)); + self.ptr_add(dst.opts); + self.instruction(LocalSet(dst.len)); + + // In debug mode verify the first result consumed the entire string, + // otherwise simply discard it. + if self.module.debug { + self.instruction(LocalGet(src.len)); + self.instruction(LocalGet(src_len_tmp)); + self.ptr_sub(src.opts); + self.ptr_ne(src.opts); + self.instruction(If(BlockType::Empty)); + self.trap(Trap::AssertFailed("should have finished encoding")); + self.instruction(End); + } else { + self.instruction(Drop); + } + + // Perform a downsizing if the worst-case size was too large + self.instruction(LocalGet(dst.len)); + self.instruction(LocalGet(dst_byte_len)); + self.ptr_ne(dst.opts); + self.instruction(If(BlockType::Empty)); + self.instruction(LocalGet(dst.ptr)); // old_ptr + self.instruction(LocalGet(dst_byte_len)); // old_size + self.ptr_uconst(dst.opts, 1); // align + self.instruction(LocalGet(dst.len)); // new_size + self.instruction(Call(dst.opts.realloc.unwrap().as_u32())); + self.instruction(LocalSet(dst.ptr)); + self.instruction(End); + + // If the first transcode was enough then assert that the returned + // amount of destination items written equals the byte size. + if self.module.debug { + self.instruction(Else); + + self.instruction(LocalGet(dst.len)); + self.instruction(LocalGet(dst_byte_len)); + self.ptr_ne(dst_opts); + self.instruction(If(BlockType::Empty)); + self.trap(Trap::AssertFailed("should have finished encoding")); + self.instruction(End); + } + + self.instruction(End); // end of "first transcode not enough" + + dst + } + + // Corresponds to the `store_utf8_to_utf16` function in the spec. + // + // When converting utf-8 to utf-16 a pessimistic allocation is + // done which is twice the byte length of the utf-8 string. + // The host then transcodes and returns how many code units were + // actually used during the transcoding and if it's beneath the + // pessimistic maximum then the buffer is reallocated down to + // a smaller amount. + // + // The host-imported transcoding function takes the src/dst pointer as + // well as the code unit size of both the source and destination. The + // destination should always be big enough to hold the result of the + // transcode and so the result of the host function is how many code + // units were written to the destination. + fn string_utf8_to_utf16<'a>( + &mut self, + src: &WasmString<'_>, + dst_opts: &'a Options, + ) -> WasmString<'a> { + self.validate_string_length(src, FE::Utf16); + self.convert_src_len_to_dst(src.len, src.opts.ptr(), dst_opts.ptr()); + let dst_len = self.gen_local(dst_opts.ptr()); + self.instruction(LocalTee(dst_len)); + self.ptr_uconst(dst_opts, 1); + self.ptr_shl(dst_opts); + let dst_byte_len = self.gen_local(dst_opts.ptr()); + self.instruction(LocalSet(dst_byte_len)); + let dst = { + let dst_mem = self.malloc(dst_opts, MallocSize::Local(dst_byte_len), 2); + WasmString { + ptr: dst_mem.addr_local, + len: dst_len, + opts: dst_opts, + } + }; + + self.validate_string_inbounds(src, src.len); + self.validate_string_inbounds(&dst, dst_byte_len); + + let transcode = self.transcoder(src, &dst, Transcode::Utf8ToUtf16); + self.instruction(LocalGet(src.ptr)); + self.instruction(LocalGet(src.len)); + self.instruction(LocalGet(dst.ptr)); + self.instruction(Call(transcode)); + self.instruction(LocalSet(dst.len)); + + // If the number of code units returned by transcode is not + // equal to the original number of code units then + // the buffer must be shrunk. + // + // Note that the byte length of the final allocation we + // want is twice the code unit length returned by the + // transcoding function. + self.convert_src_len_to_dst(src.len, src.opts.ptr(), dst.opts.ptr()); + self.instruction(LocalGet(dst.len)); + self.ptr_ne(dst_opts); + self.instruction(If(BlockType::Empty)); + self.instruction(LocalGet(dst.ptr)); + self.instruction(LocalGet(dst_byte_len)); + self.ptr_uconst(dst.opts, 2); + self.instruction(LocalGet(dst.len)); + self.ptr_uconst(dst.opts, 1); + self.ptr_shl(dst.opts); + self.instruction(Call(dst.opts.realloc.unwrap().as_u32())); + self.instruction(LocalSet(dst.ptr)); + self.instruction(End); // end of shrink-to-fit + + dst + } + + // Corresponds to `store_probably_utf16_to_latin1_or_utf16` in the spec. + // + // This will try to transcode the input utf16 string to utf16 in the + // destination. If utf16 isn't needed though and latin1 could be used + // then that's used instead and a reallocation to downsize occurs + // afterwards. + // + // The host transcode function here will take the src/dst pointers as + // well as src length. The destination byte length is twice the src code + // unit length. The return value is the tagged length of the returned + // string. If the upper bit is set then utf16 was used and the + // conversion is done. If the upper bit is not set then latin1 was used + // and a downsizing needs to happen. + fn string_compact_utf16_to_compact<'a>( + &mut self, + src: &WasmString<'_>, + dst_opts: &'a Options, + ) -> WasmString<'a> { + self.validate_string_length(src, FE::Utf16); + self.convert_src_len_to_dst(src.len, src.opts.ptr(), dst_opts.ptr()); + let dst_len = self.gen_local(dst_opts.ptr()); + self.instruction(LocalTee(dst_len)); + self.ptr_uconst(dst_opts, 1); + self.ptr_shl(dst_opts); + let dst_byte_len = self.gen_local(dst_opts.ptr()); + self.instruction(LocalSet(dst_byte_len)); + let dst = { + let dst_mem = self.malloc(dst_opts, MallocSize::Local(dst_byte_len), 2); + WasmString { + ptr: dst_mem.addr_local, + len: dst_len, + opts: dst_opts, + } + }; + + self.validate_string_inbounds(src, dst_byte_len); + self.validate_string_inbounds(&dst, dst_byte_len); + + let transcode = self.transcoder(src, &dst, Transcode::Utf16ToCompactProbablyUtf16); + self.instruction(LocalGet(src.ptr)); + self.instruction(LocalGet(src.len)); + self.instruction(LocalGet(dst.ptr)); + self.instruction(Call(transcode)); + self.instruction(LocalSet(dst.len)); + + // Assert that the untagged code unit length is the same as the + // source code unit length. + if self.module.debug { + self.instruction(LocalGet(dst.len)); + self.ptr_uconst(dst.opts, !UTF16_TAG); + self.ptr_and(dst.opts); + self.convert_src_len_to_dst(src.len, src.opts.ptr(), dst.opts.ptr()); + self.ptr_ne(dst.opts); + self.instruction(If(BlockType::Empty)); + self.trap(Trap::AssertFailed("expected equal code units")); + self.instruction(End); + } + + // If the UTF16_TAG is set then utf16 was used and the destination + // should be appropriately sized. Bail out of the "is this string + // empty" block and fall through otherwise to resizing. + self.instruction(LocalGet(dst.len)); + self.ptr_uconst(dst.opts, UTF16_TAG); + self.ptr_and(dst.opts); + self.ptr_br_if(dst.opts, 0); + + // Here `realloc` is used to downsize the string + self.instruction(LocalGet(dst.ptr)); // old_ptr + self.instruction(LocalGet(dst_byte_len)); // old_size + self.ptr_uconst(dst.opts, 2); // align + self.instruction(LocalGet(dst.len)); // new_size + self.instruction(Call(dst.opts.realloc.unwrap().as_u32())); + self.instruction(LocalSet(dst.ptr)); + + dst + } + + // Corresponds to `store_string_to_latin1_or_utf16` in the spec. + // + // This will attempt a first pass of transcoding to latin1 and on + // failure a larger buffer is allocated for utf16 and then utf16 is + // encoded in-place into the buffer. After either latin1 or utf16 the + // buffer is then resized to fit the final string allocation. + fn string_to_compact<'a>( + &mut self, + src: &WasmString<'_>, + src_enc: FE, + dst_opts: &'a Options, + ) -> WasmString<'a> { + self.validate_string_length(src, src_enc); + self.convert_src_len_to_dst(src.len, src.opts.ptr(), dst_opts.ptr()); + let dst_len = self.gen_local(dst_opts.ptr()); + self.instruction(LocalTee(dst_len)); + let dst_byte_len = self.gen_local(dst_opts.ptr()); + self.instruction(LocalSet(dst_byte_len)); + let dst = { + let dst_mem = self.malloc(dst_opts, MallocSize::Local(dst_byte_len), 2); + WasmString { + ptr: dst_mem.addr_local, + len: dst_len, + opts: dst_opts, + } + }; + + self.validate_string_inbounds(src, src.len); + self.validate_string_inbounds(&dst, dst_byte_len); + + // Perform the initial latin1 transcode. This returns the number of + // source code units consumed and the number of destination code + // units (bytes) written. + let (latin1, utf16) = match src_enc { + FE::Utf8 => (Transcode::Utf8ToLatin1, Transcode::Utf8ToCompactUtf16), + FE::Utf16 => (Transcode::Utf16ToLatin1, Transcode::Utf16ToCompactUtf16), + FE::Latin1 => unreachable!(), + }; + let transcode_latin1 = self.transcoder(src, &dst, latin1); + let transcode_utf16 = self.transcoder(src, &dst, utf16); + self.instruction(LocalGet(src.ptr)); + self.instruction(LocalGet(src.len)); + self.instruction(LocalGet(dst.ptr)); + self.instruction(Call(transcode_latin1)); + self.instruction(LocalSet(dst.len)); + let src_len_tmp = self.gen_local(src.opts.ptr()); + self.instruction(LocalSet(src_len_tmp)); + + // If the source was entirely consumed then the transcode completed + // and all that's necessary is to optionally shrink the buffer. + self.instruction(LocalGet(src_len_tmp)); + self.instruction(LocalGet(src.len)); + self.ptr_eq(src.opts); + self.instruction(If(BlockType::Empty)); // if latin1-or-utf16 block + + // Test if the original byte length of the allocation is the same as + // the number of written bytes, and if not then shrink the buffer + // with a call to `realloc`. + self.instruction(LocalGet(dst_byte_len)); + self.instruction(LocalGet(dst.len)); + self.ptr_ne(dst.opts); + self.instruction(If(BlockType::Empty)); + self.instruction(LocalGet(dst.ptr)); // old_ptr + self.instruction(LocalGet(dst_byte_len)); // old_size + self.ptr_uconst(dst.opts, 2); // align + self.instruction(LocalGet(dst.len)); // new_size + self.instruction(Call(dst.opts.realloc.unwrap().as_u32())); + self.instruction(LocalSet(dst.ptr)); + self.instruction(End); + + // In this block the latin1 encoding failed. The host transcode + // returned how many units were consumed from the source and how + // many bytes were written to the destination. Here the buffer is + // inflated and sized and the second utf16 intrinsic is invoked to + // perform the final inflation. + self.instruction(Else); // else latin1-or-utf16 block + + // For utf8 validate that the inflated size is still within bounds. + if src_enc.width() == 1 { + self.validate_string_length_u8(src, 2); + } + + // Reallocate the buffer with twice the source code units in byte + // size. + self.instruction(LocalGet(dst.ptr)); // old_ptr + self.instruction(LocalGet(dst_byte_len)); // old_size + self.ptr_uconst(dst.opts, 2); // align + self.convert_src_len_to_dst(src.len, src.opts.ptr(), dst.opts.ptr()); + self.ptr_uconst(dst.opts, 1); + self.ptr_shl(dst.opts); + self.instruction(LocalTee(dst_byte_len)); + self.instruction(Call(dst.opts.realloc.unwrap().as_u32())); + self.instruction(LocalSet(dst.ptr)); + + // Call the host utf16 transcoding function. This will inflate the + // prior latin1 bytes and then encode the rest of the source string + // as utf16 into the remaining space in the destination buffer. + self.instruction(LocalGet(src.ptr)); + self.instruction(LocalGet(src_len_tmp)); + if let FE::Utf16 = src_enc { + self.ptr_uconst(src.opts, 1); + self.ptr_shl(src.opts); + } + self.ptr_add(src.opts); + self.instruction(LocalGet(src.len)); + self.instruction(LocalGet(src_len_tmp)); + self.ptr_sub(src.opts); + self.instruction(LocalGet(dst.ptr)); + self.convert_src_len_to_dst(src.len, src.opts.ptr(), dst.opts.ptr()); + self.instruction(LocalGet(dst.len)); + self.instruction(Call(transcode_utf16)); + self.instruction(LocalSet(dst.len)); + + // If the returned number of code units written to the destination + // is not equal to the size of the allocation then the allocation is + // resized down to the appropriate size. + // + // Note that the byte size desired is `2*dst_len` and the current + // byte buffer size is `2*src_len` so the `2` factor isn't checked + // here, just the lengths. + self.instruction(LocalGet(dst.len)); + self.convert_src_len_to_dst(src.len, src.opts.ptr(), dst.opts.ptr()); + self.ptr_ne(dst.opts); + self.instruction(If(BlockType::Empty)); + self.instruction(LocalGet(dst.ptr)); // old_ptr + self.instruction(LocalGet(dst_byte_len)); // old_size + self.ptr_uconst(dst.opts, 2); // align + self.instruction(LocalGet(dst.len)); + self.ptr_uconst(dst.opts, 1); + self.ptr_shl(dst.opts); + self.instruction(Call(dst.opts.realloc.unwrap().as_u32())); + self.instruction(LocalSet(dst.ptr)); + self.instruction(End); + + // Tag the returned pointer as utf16 + self.instruction(LocalGet(dst.len)); + self.ptr_uconst(dst.opts, UTF16_TAG); + self.ptr_or(dst.opts); + self.instruction(LocalSet(dst.len)); + + self.instruction(End); // end latin1-or-utf16 block + + dst + } + + fn validate_string_length(&mut self, src: &WasmString<'_>, dst: FE) { + self.validate_string_length_u8(src, dst.width()) + } + + fn validate_string_length_u8(&mut self, s: &WasmString<'_>, dst: u8) { + // Check to see if the source byte length is out of bounds in + // which case a trap is generated. + self.instruction(LocalGet(s.len)); + let max = MAX_STRING_BYTE_LENGTH / u32::from(dst); + self.ptr_uconst(s.opts, max); + self.ptr_ge_u(s.opts); + self.instruction(If(BlockType::Empty)); + self.trap(Trap::StringLengthTooBig); + self.instruction(End); + } + + fn transcoder(&mut self, src: &WasmString<'_>, dst: &WasmString<'_>, op: Transcode) -> u32 { + self.transcoders.import( + self.types, + Transcoder { + from_memory: src.opts.memory.unwrap(), + from_memory64: src.opts.memory64, + to_memory: dst.opts.memory.unwrap(), + to_memory64: dst.opts.memory64, + op, + }, + ) + } + + fn validate_string_inbounds(&mut self, s: &WasmString<'_>, byte_len: u32) { + let extend_to_64 = |me: &mut Self| { + if !s.opts.memory64 { + me.instruction(I64ExtendI32U); + } + }; + + self.instruction(Block(BlockType::Empty)); + self.instruction(Block(BlockType::Empty)); + + // Calculate the full byte size of memory with `memory.size`. Note that + // arithmetic here is done always in 64-bits to accomodate 4G memories. + // Additionally it's assumed that 64-bit memories never fill up + // entirely. + self.instruction(MemorySize(s.opts.memory.unwrap().as_u32())); + extend_to_64(self); + self.instruction(I64Const(16)); + self.instruction(I64Shl); + + // Calculate the end address of the string. This is done by adding the + // base pointer to the byte length. For 32-bit memories there's no need + // to check for overflow since everything is extended to 64-bit, but for + // 64-bit memories overflow is checked. + self.instruction(LocalGet(s.ptr)); + extend_to_64(self); + self.instruction(LocalGet(byte_len)); + extend_to_64(self); + self.instruction(I64Add); + if s.opts.memory64 { + let tmp = self.gen_local(ValType::I64); + self.instruction(LocalTee(tmp)); + self.instruction(LocalGet(s.ptr)); + self.ptr_lt_u(s.opts); + self.ptr_br_if(s.opts, 0); + self.instruction(LocalGet(tmp)); + } + + // If the byte size of memory is greater than the final address of the + // string then the string is invalid. Note that if it's precisely equal + // then that's ok. + self.instruction(I64GtU); + self.instruction(BrIf(1)); + + self.instruction(End); + self.trap(Trap::StringLengthOverflow); + self.instruction(End); + } + fn translate_list( &mut self, src_ty: TypeInterfaceIndex, @@ -1467,17 +2232,17 @@ impl Compiler<'_, '_> { self.instruction(GlobalSet(flags_global.as_u32())); } - fn verify_aligned(&mut self, memory: &Memory, align: usize) { + fn verify_aligned(&mut self, opts: &Options, addr_local: u32, align: usize) { // If the alignment is 1 then everything is trivially aligned and the // check can be omitted. if align == 1 { return; } - self.instruction(LocalGet(memory.addr_local)); + self.instruction(LocalGet(addr_local)); assert!(align.is_power_of_two()); - self.ptr_uconst(memory.opts, u32::try_from(align - 1).unwrap()); - self.ptr_and(memory.opts); - self.ptr_if(memory.opts, BlockType::Empty); + self.ptr_uconst(opts, u32::try_from(align - 1).unwrap()); + self.ptr_and(opts); + self.ptr_if(opts, BlockType::Empty); self.trap(Trap::UnalignedPointer); self.instruction(End); } @@ -1527,7 +2292,7 @@ impl Compiler<'_, '_> { offset: 0, opts, }; - self.verify_aligned(&ret, align); + self.verify_aligned(opts, ret.addr_local, align); ret } @@ -1711,6 +2476,46 @@ impl Compiler<'_, '_> { } } + fn ptr_sub(&mut self, opts: &Options) { + if opts.memory64 { + self.instruction(I64Sub); + } else { + self.instruction(I32Sub); + } + } + + fn ptr_mul(&mut self, opts: &Options) { + if opts.memory64 { + self.instruction(I64Mul); + } else { + self.instruction(I32Mul); + } + } + + fn ptr_ge_u(&mut self, opts: &Options) { + if opts.memory64 { + self.instruction(I64GeU); + } else { + self.instruction(I32GeU); + } + } + + fn ptr_lt_u(&mut self, opts: &Options) { + if opts.memory64 { + self.instruction(I64LtU); + } else { + self.instruction(I32LtU); + } + } + + fn ptr_shl(&mut self, opts: &Options) { + if opts.memory64 { + self.instruction(I64Shl); + } else { + self.instruction(I32Shl); + } + } + fn ptr_eqz(&mut self, opts: &Options) { if opts.memory64 { self.instruction(I64Eqz); @@ -1735,6 +2540,22 @@ impl Compiler<'_, '_> { } } + fn ptr_eq(&mut self, opts: &Options) { + if opts.memory64 { + self.instruction(I64Eq); + } else { + self.instruction(I32Eq); + } + } + + fn ptr_ne(&mut self, opts: &Options) { + if opts.memory64 { + self.instruction(I64Ne); + } else { + self.instruction(I32Ne); + } + } + fn ptr_and(&mut self, opts: &Options) { if opts.memory64 { self.instruction(I64And); @@ -1743,6 +2564,22 @@ impl Compiler<'_, '_> { } } + fn ptr_or(&mut self, opts: &Options) { + if opts.memory64 { + self.instruction(I64Or); + } else { + self.instruction(I32Or); + } + } + + fn ptr_xor(&mut self, opts: &Options) { + if opts.memory64 { + self.instruction(I64Xor); + } else { + self.instruction(I32Xor); + } + } + fn ptr_if(&mut self, opts: &Options, ty: BlockType) { if opts.memory64 { self.instruction(I64Const(0)); @@ -1974,3 +2811,9 @@ enum MallocSize { Const(usize), Local(u32), } + +struct WasmString<'a> { + ptr: u32, + len: u32, + opts: &'a Options, +} diff --git a/crates/environ/src/fact/transcode.rs b/crates/environ/src/fact/transcode.rs new file mode 100644 index 000000000000..865fef316ed6 --- /dev/null +++ b/crates/environ/src/fact/transcode.rs @@ -0,0 +1,178 @@ +use crate::fact::core_types::CoreTypes; +use crate::MemoryIndex; +use serde::{Deserialize, Serialize}; +use std::collections::HashMap; +use wasm_encoder::{EntityType, ValType}; + +pub struct Transcoders { + imported: HashMap, + prev_func_imports: u32, + imports: Vec<(String, EntityType, Transcoder)>, +} + +#[derive(Copy, Clone, Hash, Eq, PartialEq)] +pub struct Transcoder { + pub from_memory: MemoryIndex, + pub from_memory64: bool, + pub to_memory: MemoryIndex, + pub to_memory64: bool, + pub op: Transcode, +} + +/// Possible transcoding operations that must be provided by the host. +/// +/// Note that each transcoding operation may have a unique signature depending +/// on the precise operation. +#[allow(missing_docs)] +#[derive(Debug, Copy, Clone, Hash, Eq, PartialEq, Serialize, Deserialize)] +pub enum Transcode { + Copy(FixedEncoding), + Latin1ToUtf16, + Latin1ToUtf8, + Utf16ToCompactProbablyUtf16, + Utf16ToCompactUtf16, + Utf16ToLatin1, + Utf16ToUtf8, + Utf8ToCompactUtf16, + Utf8ToLatin1, + Utf8ToUtf16, +} + +#[derive(Debug, Copy, Clone, Hash, Eq, PartialEq, Serialize, Deserialize)] +#[allow(missing_docs)] +pub enum FixedEncoding { + Utf8, + Utf16, + Latin1, +} + +impl Transcoders { + pub fn new(prev_func_imports: u32) -> Transcoders { + Transcoders { + imported: HashMap::new(), + prev_func_imports, + imports: Vec::new(), + } + } + + pub fn import(&mut self, types: &mut CoreTypes, transcoder: Transcoder) -> u32 { + *self.imported.entry(transcoder).or_insert_with(|| { + let idx = self.prev_func_imports + (self.imports.len() as u32); + self.imports + .push((transcoder.name(), transcoder.ty(types), transcoder)); + idx + }) + } + + pub fn imports(&self) -> impl Iterator { + self.imports + .iter() + .map(|(name, ty, transcoder)| ("transcode", &name[..], *ty, transcoder)) + } +} + +impl Transcoder { + fn name(&self) -> String { + format!( + "{} (mem{} => mem{})", + self.op.desc(), + self.from_memory.as_u32(), + self.to_memory.as_u32(), + ) + } + + fn ty(&self, types: &mut CoreTypes) -> EntityType { + let from_ptr = if self.from_memory64 { + ValType::I64 + } else { + ValType::I32 + }; + let to_ptr = if self.to_memory64 { + ValType::I64 + } else { + ValType::I32 + }; + + let ty = match self.op { + // These direct transcodings take the source pointer, the source + // code units, and the destination pointer. + // + // The memories being copied between are part of each intrinsic and + // the destination code units are the same as the source. + // Note that the pointers are dynamically guaranteed to be aligned + // and in-bounds for the code units length as defined by the string + // encoding. + Transcode::Copy(_) | Transcode::Latin1ToUtf16 => { + types.function(&[from_ptr, from_ptr, to_ptr], &[]) + } + + // Transcoding from utf8 to utf16 takes the from ptr/len as well as + // a destination. The destination is valid for len*2 bytes. The + // return value is how many code units were written to the + // destination. + Transcode::Utf8ToUtf16 => types.function(&[from_ptr, from_ptr, to_ptr], &[to_ptr]), + + // Transcoding to utf8 as a smaller format takes all the parameters + // and returns the amount of space consumed in the src/destination + Transcode::Utf16ToUtf8 | Transcode::Latin1ToUtf8 => { + types.function(&[from_ptr, from_ptr, to_ptr, to_ptr], &[from_ptr, to_ptr]) + } + + // The return type is a tagged length which indicates which was + // used + Transcode::Utf16ToCompactProbablyUtf16 => { + types.function(&[from_ptr, from_ptr, to_ptr], &[to_ptr]) + } + + // The initial step of transcoding from a fixed format to a compact + // format. Takes the ptr/len of the source the the destination + // pointer. The destination length is implicitly the same. Returns + // how many code units were consumed in the source, which is also + // how many bytes were written to the destination. + Transcode::Utf8ToLatin1 | Transcode::Utf16ToLatin1 => { + types.function(&[from_ptr, from_ptr, to_ptr], &[from_ptr, to_ptr]) + } + + // The final step of transcoding to a compact format when the fixed + // transcode has failed. This takes the ptr/len of the source that's + // remaining to transcode. Then this takes the destination ptr/len + // as well as the destination bytes written so far with latin1. + // Finally this returns the number of code units written to the + // destination. + Transcode::Utf8ToCompactUtf16 | Transcode::Utf16ToCompactUtf16 => { + types.function(&[from_ptr, from_ptr, to_ptr, to_ptr, to_ptr], &[to_ptr]) + } + }; + EntityType::Function(ty) + } +} + +impl Transcode { + /// Returns a human-readable description for this transcoding operation. + pub fn desc(&self) -> &'static str { + match self { + Transcode::Copy(FixedEncoding::Utf8) => "utf8-to-utf8", + Transcode::Copy(FixedEncoding::Utf16) => "utf16-to-utf16", + Transcode::Copy(FixedEncoding::Latin1) => "latin1-to-latin1", + Transcode::Latin1ToUtf16 => "latin1-to-utf16", + Transcode::Latin1ToUtf8 => "latin1-to-utf8", + Transcode::Utf16ToCompactProbablyUtf16 => "utf16-to-compact-probably-utf16", + Transcode::Utf16ToCompactUtf16 => "utf16-to-compact-utf16", + Transcode::Utf16ToLatin1 => "utf16-to-latin1", + Transcode::Utf16ToUtf8 => "utf16-to-utf8", + Transcode::Utf8ToCompactUtf16 => "utf8-to-compact-utf16", + Transcode::Utf8ToLatin1 => "utf8-to-latin1", + Transcode::Utf8ToUtf16 => "utf8-to-utf16", + } + } +} + +impl FixedEncoding { + pub(crate) fn width(&self) -> u8 { + match self { + FixedEncoding::Utf8 => 1, + FixedEncoding::Utf16 => 2, + FixedEncoding::Latin1 => 1, + } + } +} diff --git a/crates/environ/src/fact/traps.rs b/crates/environ/src/fact/traps.rs index 393194f1012a..e68bccfc13d5 100644 --- a/crates/environ/src/fact/traps.rs +++ b/crates/environ/src/fact/traps.rs @@ -30,6 +30,8 @@ pub enum Trap { InvalidDiscriminant, InvalidChar, ListByteLengthOverflow, + StringLengthTooBig, + StringLengthOverflow, AssertFailed(&'static str), } @@ -105,6 +107,8 @@ impl fmt::Display for Trap { Trap::InvalidDiscriminant => "invalid variant discriminant".fmt(f), Trap::InvalidChar => "invalid char value specified".fmt(f), Trap::ListByteLengthOverflow => "byte size of list too large for i32".fmt(f), + Trap::StringLengthTooBig => "string byte size exceeds maximum".fmt(f), + Trap::StringLengthOverflow => "string byte size overflows i32".fmt(f), Trap::AssertFailed(s) => write!(f, "assertion failure: {}", s), } } diff --git a/crates/environ/src/module.rs b/crates/environ/src/module.rs index c937af024bad..23d352f5b052 100644 --- a/crates/environ/src/module.rs +++ b/crates/environ/src/module.rs @@ -974,7 +974,7 @@ impl Module { /// Returns an iterator of all the imports in this module, along with their /// module name, field name, and type that's being imported. - pub fn imports(&self) -> impl Iterator { + pub fn imports(&self) -> impl ExactSizeIterator { self.initializers.iter().map(move |i| match i { Initializer::Import { name, field, index } => { (name.as_str(), field.as_str(), self.type_of(*index)) diff --git a/crates/environ/src/vmoffsets.rs b/crates/environ/src/vmoffsets.rs index d301765b40bd..de131eab4ad4 100644 --- a/crates/environ/src/vmoffsets.rs +++ b/crates/environ/src/vmoffsets.rs @@ -136,6 +136,8 @@ pub trait PtrSize { 16 } + // Offsets within `VMRuntimeLimits` + /// Return the offset of the `stack_limit` field of `VMRuntimeLimits` #[inline] fn vmruntime_limits_stack_limit(&self) -> u8 { @@ -168,6 +170,34 @@ pub trait PtrSize { fn vmruntime_limits_last_wasm_entry_sp(&self) -> u8 { self.vmruntime_limits_last_wasm_exit_pc() + self.size() } + + // Offsets within `VMMemoryDefinition` + + /// The offset of the `base` field. + #[allow(clippy::erasing_op)] + #[inline] + fn vmmemory_definition_base(&self) -> u8 { + 0 * self.size() + } + + /// The offset of the `current_length` field. + #[allow(clippy::identity_op)] + #[inline] + fn vmmemory_definition_current_length(&self) -> u8 { + 1 * self.size() + } + + /// Return the size of `VMMemoryDefinition`. + #[inline] + fn size_of_vmmemory_definition(&self) -> u8 { + 2 * self.size() + } + + /// Return the size of `*mut VMMemoryDefinition`. + #[inline] + fn size_of_vmmemory_pointer(&self) -> u8 { + self.size() + } } /// Type representing the size of a pointer for the current compilation host @@ -395,9 +425,9 @@ impl From> for VMOffsets

{ size(defined_tables) = cmul(ret.num_defined_tables, ret.size_of_vmtable_definition()), size(defined_memories) - = cmul(ret.num_defined_memories, ret.size_of_vmmemory_pointer()), + = cmul(ret.num_defined_memories, ret.ptr.size_of_vmmemory_pointer()), size(owned_memories) - = cmul(ret.num_owned_memories, ret.size_of_vmmemory_definition()), + = cmul(ret.num_owned_memories, ret.ptr.size_of_vmmemory_definition()), align(16), size(defined_globals) = cmul(ret.num_defined_globals, ret.ptr.size_of_vmglobal_definition()), @@ -523,35 +553,6 @@ impl VMOffsets

{ } } -/// Offsets for `VMMemoryDefinition`. -impl VMOffsets

{ - /// The offset of the `base` field. - #[allow(clippy::erasing_op)] - #[inline] - pub fn vmmemory_definition_base(&self) -> u8 { - 0 * self.pointer_size() - } - - /// The offset of the `current_length` field. - #[allow(clippy::identity_op)] - #[inline] - pub fn vmmemory_definition_current_length(&self) -> u8 { - 1 * self.pointer_size() - } - - /// Return the size of `VMMemoryDefinition`. - #[inline] - pub fn size_of_vmmemory_definition(&self) -> u8 { - 2 * self.pointer_size() - } - - /// Return the size of `*mut VMMemoryDefinition`. - #[inline] - pub fn size_of_vmmemory_pointer(&self) -> u8 { - self.pointer_size() - } -} - /// Offsets for `VMGlobalImport`. impl VMOffsets

{ /// The offset of the `from` field. @@ -733,7 +734,8 @@ impl VMOffsets

{ #[inline] pub fn vmctx_vmmemory_pointer(&self, index: DefinedMemoryIndex) -> u32 { assert!(index.as_u32() < self.num_defined_memories); - self.vmctx_memories_begin() + index.as_u32() * u32::from(self.size_of_vmmemory_pointer()) + self.vmctx_memories_begin() + + index.as_u32() * u32::from(self.ptr.size_of_vmmemory_pointer()) } /// Return the offset to the owned `VMMemoryDefinition` at index `index`. @@ -741,7 +743,7 @@ impl VMOffsets

{ pub fn vmctx_vmmemory_definition(&self, index: OwnedMemoryIndex) -> u32 { assert!(index.as_u32() < self.num_owned_memories); self.vmctx_owned_memories_begin() - + index.as_u32() * u32::from(self.size_of_vmmemory_definition()) + + index.as_u32() * u32::from(self.ptr.size_of_vmmemory_definition()) } /// Return the offset to the `VMGlobalDefinition` index `index`. @@ -807,13 +809,14 @@ impl VMOffsets

{ /// Return the offset to the `base` field in `VMMemoryDefinition` index `index`. #[inline] pub fn vmctx_vmmemory_definition_base(&self, index: OwnedMemoryIndex) -> u32 { - self.vmctx_vmmemory_definition(index) + u32::from(self.vmmemory_definition_base()) + self.vmctx_vmmemory_definition(index) + u32::from(self.ptr.vmmemory_definition_base()) } /// Return the offset to the `current_length` field in `VMMemoryDefinition` index `index`. #[inline] pub fn vmctx_vmmemory_definition_current_length(&self, index: OwnedMemoryIndex) -> u32 { - self.vmctx_vmmemory_definition(index) + u32::from(self.vmmemory_definition_current_length()) + self.vmctx_vmmemory_definition(index) + + u32::from(self.ptr.vmmemory_definition_current_length()) } /// Return the offset to the `from` field in `VMGlobalImport` index `index`. diff --git a/crates/misc/component-test-util/src/lib.rs b/crates/misc/component-test-util/src/lib.rs index 364708250941..eb79e0f96989 100644 --- a/crates/misc/component-test-util/src/lib.rs +++ b/crates/misc/component-test-util/src/lib.rs @@ -35,7 +35,7 @@ impl FuncExt for Func { } } -pub fn engine() -> Engine { +pub fn config() -> Config { drop(env_logger::try_init()); let mut config = Config::new(); @@ -48,7 +48,11 @@ pub fn engine() -> Engine { config.static_memory_maximum_size(0); config.dynamic_memory_guard_size(0); } - Engine::new(&config).unwrap() + config +} + +pub fn engine() -> Engine { + Engine::new(&config()).unwrap() } /// Newtype wrapper for `f32` whose `PartialEq` impl considers NaNs equal to each other. diff --git a/crates/runtime/Cargo.toml b/crates/runtime/Cargo.toml index f209a2459453..3121c35de351 100644 --- a/crates/runtime/Cargo.toml +++ b/crates/runtime/Cargo.toml @@ -25,6 +25,7 @@ rand = "0.8.3" anyhow = "1.0.38" memfd = { version = "0.6.1", optional = true } paste = "1.0.3" +encoding_rs = { version = "0.8.31", optional = true } [target.'cfg(target_os = "macos")'.dependencies] mach = "0.3.2" @@ -62,4 +63,7 @@ pooling-allocator = [] # need portable signal handling. posix-signals-on-macos = [] -component-model = ["wasmtime-environ/component-model"] +component-model = [ + "wasmtime-environ/component-model", + "dep:encoding_rs", +] diff --git a/crates/runtime/src/component.rs b/crates/runtime/src/component.rs index 96db80b1979a..f450caaf0a7b 100644 --- a/crates/runtime/src/component.rs +++ b/crates/runtime/src/component.rs @@ -18,13 +18,16 @@ use std::ops::Deref; use std::ptr::{self, NonNull}; use wasmtime_environ::component::{ Component, LoweredIndex, RuntimeAlwaysTrapIndex, RuntimeComponentInstanceIndex, - RuntimeMemoryIndex, RuntimePostReturnIndex, RuntimeReallocIndex, StringEncoding, - VMComponentOffsets, FLAG_MAY_ENTER, FLAG_MAY_LEAVE, FLAG_NEEDS_POST_RETURN, VMCOMPONENT_MAGIC, + RuntimeMemoryIndex, RuntimePostReturnIndex, RuntimeReallocIndex, RuntimeTranscoderIndex, + StringEncoding, VMComponentOffsets, FLAG_MAY_ENTER, FLAG_MAY_LEAVE, FLAG_NEEDS_POST_RETURN, + VMCOMPONENT_MAGIC, }; use wasmtime_environ::HostPtr; const INVALID_PTR: usize = 0xdead_dead_beef_beef_u64 as usize; +mod transcode; + /// Runtime representation of a component instance and all state necessary for /// the instance itself. /// @@ -255,6 +258,14 @@ impl ComponentInstance { unsafe { self.anyfunc(self.offsets.always_trap_anyfunc(idx)) } } + /// Same as `lowering_anyfunc` except for the transcoding functions. + pub fn transcoder_anyfunc( + &self, + idx: RuntimeTranscoderIndex, + ) -> NonNull { + unsafe { self.anyfunc(self.offsets.transcoder_anyfunc(idx)) } + } + unsafe fn anyfunc(&self, offset: u32) -> NonNull { let ret = self.vmctx_plus_offset::(offset); debug_assert!((*ret).func_ptr.as_ptr() as usize != INVALID_PTR); @@ -349,6 +360,16 @@ impl ComponentInstance { unsafe { self.set_anyfunc(self.offsets.always_trap_anyfunc(idx), func_ptr, type_index) } } + /// Same as `set_lowering` but for the transcoder functions. + pub fn set_transcoder( + &mut self, + idx: RuntimeTranscoderIndex, + func_ptr: NonNull, + type_index: VMSharedSignatureIndex, + ) { + unsafe { self.set_anyfunc(self.offsets.transcoder_anyfunc(idx), func_ptr, type_index) } + } + unsafe fn set_anyfunc( &mut self, offset: u32, @@ -366,6 +387,8 @@ impl ComponentInstance { unsafe fn initialize_vmctx(&mut self, store: *mut dyn Store) { *self.vmctx_plus_offset(self.offsets.magic()) = VMCOMPONENT_MAGIC; + *self.vmctx_plus_offset(self.offsets.transcode_libcalls()) = + &transcode::VMBuiltinTranscodeArray::INIT; *self.vmctx_plus_offset(self.offsets.store()) = store; *self.vmctx_plus_offset(self.offsets.limits()) = (*store).vmruntime_limits(); @@ -395,6 +418,11 @@ impl ComponentInstance { let offset = self.offsets.always_trap_anyfunc(i); *self.vmctx_plus_offset(offset) = INVALID_PTR; } + for i in 0..self.offsets.num_transcoders { + let i = RuntimeTranscoderIndex::from_u32(i); + let offset = self.offsets.transcoder_anyfunc(i); + *self.vmctx_plus_offset(offset) = INVALID_PTR; + } for i in 0..self.offsets.num_runtime_memories { let i = RuntimeMemoryIndex::from_u32(i); let offset = self.offsets.runtime_memory(i); @@ -522,6 +550,19 @@ impl OwnedComponentInstance { .set_always_trap(idx, func_ptr, type_index) } } + + /// See `ComponentInstance::set_transcoder` + pub fn set_transcoder( + &mut self, + idx: RuntimeTranscoderIndex, + func_ptr: NonNull, + type_index: VMSharedSignatureIndex, + ) { + unsafe { + self.instance_mut() + .set_transcoder(idx, func_ptr, type_index) + } + } } impl Deref for OwnedComponentInstance { diff --git a/crates/runtime/src/component/transcode.rs b/crates/runtime/src/component/transcode.rs new file mode 100644 index 000000000000..e5f77a7e245c --- /dev/null +++ b/crates/runtime/src/component/transcode.rs @@ -0,0 +1,446 @@ +//! Implementation of string transcoding required by the component model. + +use anyhow::{anyhow, Result}; +use std::cell::Cell; +use std::slice; + +const UTF16_TAG: usize = 1 << 31; + +/// Macro to define the `VMBuiltinTranscodeArray` type which contains all of the +/// function pointers to the actual transcoder functions. This structure is read +/// by Cranelift-generated code, hence the `repr(C)`. +/// +/// Note that this references the `trampolines` module rather than the functions +/// below as the `trampolines` module has the raw ABI. +/// +/// This is modeled after the similar macros and usages in `libcalls.rs` and +/// `vmcontext.rs` +macro_rules! define_transcoders { + ( + $( + $( #[$attr:meta] )* + $name:ident( $( $pname:ident: $param:ident ),* ) $( -> $result:ident )?; + )* + ) => { + /// An array that stores addresses of builtin functions. We translate code + /// to use indirect calls. This way, we don't have to patch the code. + #[repr(C)] + pub struct VMBuiltinTranscodeArray { + $( + $name: unsafe extern "C" fn( + $(define_transcoders!(@ty $param),)* + $(define_transcoders!(@retptr $result),)? + ) $( -> define_transcoders!(@ty $result))?, + )* + } + + impl VMBuiltinTranscodeArray { + pub const INIT: VMBuiltinTranscodeArray = VMBuiltinTranscodeArray { + $($name: trampolines::$name,)* + }; + } + }; + + (@ty size) => (usize); + (@ty size_pair) => (usize); + (@ty ptr_u8) => (*mut u8); + (@ty ptr_u16) => (*mut u16); + + (@retptr size_pair) => (*mut usize); + (@retptr size) => (()); +} + +wasmtime_environ::foreach_transcoder!(define_transcoders); + +/// Submodule with macro-generated constants which are the actual libcall +/// transcoders that are invoked by Cranelift. These functions have a specific +/// ABI defined by the macro itself and will defer to the actual bodies of each +/// implementation following this submodule. +#[allow(improper_ctypes_definitions)] +mod trampolines { + macro_rules! transcoders { + ( + $( + $( #[$attr:meta] )* + $name:ident( $( $pname:ident: $param:ident ),* ) $( -> $result:ident )?; + )* + ) => ( + $( + pub unsafe extern "C" fn $name( + $($pname : define_transcoders!(@ty $param),)* + // If a result is given then a `size_pair` results gets its + // second result value passed via a return pointer here, so + // optionally indicate a return pointer. + $(_retptr: define_transcoders!(@retptr $result))? + ) $( -> define_transcoders!(@ty $result))? { + $(transcoders!(@validate_param $pname $param);)* + + // Always catch panics to avoid trying to unwind from Rust + // into Cranelift-generated code which would lead to a Bad + // Time. + // + // Additionally assume that every function below returns a + // `Result` where errors turn into traps. + let result = std::panic::catch_unwind(|| { + super::$name($($pname),*) + }); + match result { + Ok(Ok(ret)) => transcoders!(@convert_ret ret _retptr $($result)?), + Ok(Err(err)) => crate::traphandlers::raise_trap(err.into()), + Err(panic) => crate::traphandlers::resume_panic(panic), + } + } + )* + ); + + (@convert_ret $ret:ident $retptr:ident) => ($ret); + (@convert_ret $ret:ident $retptr:ident size) => ($ret); + (@convert_ret $ret:ident $retptr:ident size_pair) => ({ + let (a, b) = $ret; + *$retptr = b; + a + }); + + (@validate_param $arg:ident ptr_u16) => ({ + // This should already be guaranteed by the canonical ABI and our + // adapter modules, but double-check here to be extra-sure. If this + // is a perf concern it can become a `debug_assert!`. + assert!(($arg as usize) % 2 == 0, "unaligned 16-bit pointer"); + }); + (@validate_param $arg:ident $ty:ident) => (); + } + + wasmtime_environ::foreach_transcoder!(transcoders); +} + +/// This property should already be guaranteed by construction in the component +/// model but assert it here to be extra sure. Nothing below is sound if regions +/// can overlap. +fn assert_no_overlap(a: &[T], b: &[U]) { + let a_start = a.as_ptr() as usize; + let a_end = a_start + (a.len() * std::mem::size_of::()); + let b_start = b.as_ptr() as usize; + let b_end = b_start + (b.len() * std::mem::size_of::()); + + if a_start < b_start { + assert!(a_end < b_start); + } else { + assert!(b_end < a_start); + } +} + +/// Converts a utf8 string to a utf8 string. +/// +/// The length provided is length of both the source and the destination +/// buffers. No value is returned other than whether an invalid string was +/// found. +unsafe fn utf8_to_utf8(src: *mut u8, len: usize, dst: *mut u8) -> Result<()> { + let src = slice::from_raw_parts(src, len); + let dst = slice::from_raw_parts_mut(dst, len); + assert_no_overlap(src, dst); + log::trace!("utf8-to-utf8 {len}"); + let src = std::str::from_utf8(src).map_err(|_| anyhow!("invalid utf8 encoding"))?; + dst.copy_from_slice(src.as_bytes()); + Ok(()) +} + +/// Converts a utf16 string to a utf16 string. +/// +/// The length provided is length of both the source and the destination +/// buffers. No value is returned other than whether an invalid string was +/// found. +unsafe fn utf16_to_utf16(src: *mut u16, len: usize, dst: *mut u16) -> Result<()> { + let src = slice::from_raw_parts(src, len); + let dst = slice::from_raw_parts_mut(dst, len); + assert_no_overlap(src, dst); + log::trace!("utf16-to-utf16 {len}"); + run_utf16_to_utf16(src, dst)?; + Ok(()) +} + +/// Transcodes utf16 to itself, returning whether all code points were inside of +/// the latin1 space. +fn run_utf16_to_utf16(src: &[u16], mut dst: &mut [u16]) -> Result { + let mut all_latin1 = true; + for ch in std::char::decode_utf16(src.iter().map(|i| u16::from_le(*i))) { + let ch = ch.map_err(|_| anyhow!("invalid utf16 encoding"))?; + all_latin1 = all_latin1 && u8::try_from(u32::from(ch)).is_ok(); + let result = ch.encode_utf16(dst); + let size = result.len(); + for item in result { + *item = item.to_le(); + } + dst = &mut dst[size..]; + } + Ok(all_latin1) +} + +/// Converts a latin1 string to a latin1 string. +/// +/// Given that all byte sequences are valid latin1 strings this is simply a +/// memory copy. +unsafe fn latin1_to_latin1(src: *mut u8, len: usize, dst: *mut u8) -> Result<()> { + let src = slice::from_raw_parts(src, len); + let dst = slice::from_raw_parts_mut(dst, len); + assert_no_overlap(src, dst); + log::trace!("latin1-to-latin1 {len}"); + dst.copy_from_slice(src); + Ok(()) +} + +/// Converts a latin1 string to a utf16 string. +/// +/// This simply inflates the latin1 characters to the u16 code points. The +/// length provided is the same length of the source and destination buffers. +unsafe fn latin1_to_utf16(src: *mut u8, len: usize, dst: *mut u16) -> Result<()> { + let src = slice::from_raw_parts(src, len); + let dst = slice::from_raw_parts_mut(dst, len); + assert_no_overlap(src, dst); + for (src, dst) in src.iter().zip(dst) { + *dst = u16::from(*src).to_le(); + } + log::trace!("latin1-to-utf16 {len}"); + Ok(()) +} + +/// Converts utf8 to utf16. +/// +/// The length provided is the same unit length of both buffers, and the +/// returned value from this function is how many u16 units were written. +unsafe fn utf8_to_utf16(src: *mut u8, len: usize, dst: *mut u16) -> Result { + let src = slice::from_raw_parts(src, len); + let dst = slice::from_raw_parts_mut(dst, len); + assert_no_overlap(src, dst); + + let result = run_utf8_to_utf16(src, dst)?; + log::trace!("utf8-to-utf16 {len} => {result}"); + Ok(result) +} + +fn run_utf8_to_utf16(src: &[u8], dst: &mut [u16]) -> Result { + let src = std::str::from_utf8(src).map_err(|_| anyhow!("invalid utf8 encoding"))?; + let mut amt = 0; + for (i, dst) in src.encode_utf16().zip(dst) { + *dst = i.to_le(); + amt += 1; + } + Ok(amt) +} + +/// Converts utf16 to utf8. +/// +/// Each buffer is specified independently here and the returned value is a pair +/// of the number of code units read and code units written. This might perform +/// a partial transcode if the destination buffer is not large enough to hold +/// the entire contents. +unsafe fn utf16_to_utf8( + src: *mut u16, + src_len: usize, + dst: *mut u8, + dst_len: usize, +) -> Result<(usize, usize)> { + let src = slice::from_raw_parts(src, src_len); + let mut dst = slice::from_raw_parts_mut(dst, dst_len); + assert_no_overlap(src, dst); + + // This iterator will convert to native endianness and additionally count + // how many items have been read from the iterator so far. This + // count is used to return how many of the source code units were read. + let src_iter_read = Cell::new(0); + let src_iter = src.iter().map(|i| { + src_iter_read.set(src_iter_read.get() + 1); + u16::from_le(*i) + }); + + let mut src_read = 0; + let mut dst_written = 0; + + for ch in std::char::decode_utf16(src_iter) { + let ch = ch.map_err(|_| anyhow!("invalid utf16 encoding"))?; + + // If the destination doesn't have enough space for this character + // then the loop is ended and this function will be called later with a + // larger destination buffer. + if dst.len() < 4 && dst.len() < ch.len_utf8() { + break; + } + + // Record that characters were read and then convert the `char` to + // utf-8, advancing the destination buffer. + src_read = src_iter_read.get(); + let len = ch.encode_utf8(dst).len(); + dst_written += len; + dst = &mut dst[len..]; + } + + log::trace!("utf16-to-utf8 {src_len}/{dst_len} => {src_read}/{dst_written}"); + Ok((src_read, dst_written)) +} + +/// Converts latin1 to utf8. +/// +/// Receives the independent size of both buffers and returns the number of code +/// units read and code units written (both bytes in this case). +/// +/// This may perform a partial encoding if the destination is not large enough. +unsafe fn latin1_to_utf8( + src: *mut u8, + src_len: usize, + dst: *mut u8, + dst_len: usize, +) -> Result<(usize, usize)> { + let src = slice::from_raw_parts(src, src_len); + let dst = slice::from_raw_parts_mut(dst, dst_len); + assert_no_overlap(src, dst); + let (read, written) = encoding_rs::mem::convert_latin1_to_utf8_partial(src, dst); + log::trace!("latin1-to-utf8 {src_len}/{dst_len} => ({read}, {written})"); + Ok((read, written)) +} + +/// Converts utf16 to "latin1+utf16", probably using a utf16 encoding. +/// +/// The length specified is the length of both the source and destination +/// buffers. If the source string has any characters that don't fit in the +/// latin1 code space (0xff and below) then a utf16-tagged length will be +/// returned. Otherwise the string is "deflated" from a utf16 string to a latin1 +/// string and the latin1 length is returned. +unsafe fn utf16_to_compact_probably_utf16( + src: *mut u16, + len: usize, + dst: *mut u16, +) -> Result { + let src = slice::from_raw_parts(src, len); + let dst = slice::from_raw_parts_mut(dst, len); + assert_no_overlap(src, dst); + let all_latin1 = run_utf16_to_utf16(src, dst)?; + if all_latin1 { + let (left, dst, right) = dst.align_to_mut::(); + assert!(left.is_empty()); + assert!(right.is_empty()); + for i in 0..len { + dst[i] = dst[2 * i]; + } + log::trace!("utf16-to-compact-probably-utf16 {len} => latin1 {len}"); + Ok(len) + } else { + log::trace!("utf16-to-compact-probably-utf16 {len} => utf16 {len}"); + Ok(len | UTF16_TAG) + } +} + +/// Converts a utf8 string to latin1. +/// +/// The length specified is the same length of both the input and the output +/// buffers. +/// +/// Returns the number of code units read from the source and the number of code +/// units written to the destination. +/// +/// Note that this may not convert the entire source into the destination if the +/// original utf8 string has usvs not representable in latin1. +unsafe fn utf8_to_latin1(src: *mut u8, len: usize, dst: *mut u8) -> Result<(usize, usize)> { + let src = slice::from_raw_parts(src, len); + let dst = slice::from_raw_parts_mut(dst, len); + assert_no_overlap(src, dst); + let read = encoding_rs::mem::utf8_latin1_up_to(src); + let written = encoding_rs::mem::convert_utf8_to_latin1_lossy(&src[..read], dst); + log::trace!("utf8-to-latin1 {len} => ({read}, {written})"); + Ok((read, written)) +} + +/// Converts a utf16 string to latin1 +/// +/// This is the same as `utf8_to_latin1` in terms of parameters/results. +unsafe fn utf16_to_latin1(src: *mut u16, len: usize, dst: *mut u8) -> Result<(usize, usize)> { + let src = slice::from_raw_parts(src, len); + let dst = slice::from_raw_parts_mut(dst, len); + assert_no_overlap(src, dst); + + let mut size = 0; + for (src, dst) in src.iter().zip(dst) { + let src = u16::from_le(*src); + match u8::try_from(src) { + Ok(src) => *dst = src, + Err(_) => break, + } + size += 1; + } + log::trace!("utf16-to-latin1 {len} => {size}"); + Ok((size, size)) +} + +/// Converts a utf8 string to a utf16 string which has been partially converted +/// as latin1 prior. +/// +/// The original string has already been partially transcoded with +/// `utf8_to_latin1` and that was determined to not be able to transcode the +/// entire string. The substring of the source that couldn't be encoded into +/// latin1 is passed here via `src` and `src_len`. +/// +/// The destination buffer is specified by `dst` and `dst_len`. The first +/// `latin1_bytes_so_far` bytes (not code units) of the `dst` buffer have +/// already been filled in with latin1 characters and need to be inflated +/// in-place to their utf16 equivalents. +/// +/// After the initial latin1 code units have been inflated the entirety of `src` +/// is then transcoded into the remaining space within `dst`. +unsafe fn utf8_to_compact_utf16( + src: *mut u8, + src_len: usize, + dst: *mut u16, + dst_len: usize, + latin1_bytes_so_far: usize, +) -> Result { + let src = slice::from_raw_parts(src, src_len); + let dst = slice::from_raw_parts_mut(dst, dst_len); + assert_no_overlap(src, dst); + + let dst = inflate_latin1_bytes(dst, latin1_bytes_so_far); + let result = run_utf8_to_utf16(src, dst)?; + log::trace!("utf8-to-compact-utf16 {src_len}/{dst_len}/{latin1_bytes_so_far} => {result}"); + Ok(result + latin1_bytes_so_far) +} + +/// Same as `utf8_to_compact_utf16` but for utf16 source strings. +unsafe fn utf16_to_compact_utf16( + src: *mut u16, + src_len: usize, + dst: *mut u16, + dst_len: usize, + latin1_bytes_so_far: usize, +) -> Result { + let src = slice::from_raw_parts(src, src_len); + let dst = slice::from_raw_parts_mut(dst, dst_len); + assert_no_overlap(src, dst); + + let dst = inflate_latin1_bytes(dst, latin1_bytes_so_far); + run_utf16_to_utf16(src, dst)?; + let result = src.len(); + log::trace!("utf16-to-compact-utf16 {src_len}/{dst_len}/{latin1_bytes_so_far} => {result}"); + Ok(result + latin1_bytes_so_far) +} + +/// Inflates the `latin1_bytes_so_far` number of bytes written to the beginning +/// of `dst` into u16 codepoints. +/// +/// Returns the remaining space in the destination that can be transcoded into, +/// slicing off the prefix of the string that was inflated from the latin1 +/// bytes. +fn inflate_latin1_bytes(dst: &mut [u16], latin1_bytes_so_far: usize) -> &mut [u16] { + // Note that `latin1_bytes_so_far` is a byte measure while `dst` is a region + // of u16 units. This `split_at_mut` uses the byte index as an index into + // the u16 unit because each of the latin1 bytes will become a whole code + // unit in the destination which is 2 bytes large. + let (to_inflate, rest) = dst.split_at_mut(latin1_bytes_so_far); + + // Use a byte-oriented view to inflate the original latin1 bytes. + let (left, mid, right) = unsafe { to_inflate.align_to_mut::() }; + assert!(left.is_empty()); + assert!(right.is_empty()); + for i in (0..latin1_bytes_so_far).rev() { + mid[2 * i] = mid[i]; + mid[2 * i + 1] = 0; + } + + return rest; +} diff --git a/crates/runtime/src/vmcontext.rs b/crates/runtime/src/vmcontext.rs index 73eb7bf8563a..c2ee49a9b775 100644 --- a/crates/runtime/src/vmcontext.rs +++ b/crates/runtime/src/vmcontext.rs @@ -255,7 +255,7 @@ mod test_vmmemory_definition { use super::VMMemoryDefinition; use memoffset::offset_of; use std::mem::size_of; - use wasmtime_environ::{Module, VMOffsets}; + use wasmtime_environ::{Module, PtrSize, VMOffsets}; #[test] fn check_vmmemory_definition_offsets() { @@ -263,15 +263,15 @@ mod test_vmmemory_definition { let offsets = VMOffsets::new(size_of::<*mut u8>() as u8, &module); assert_eq!( size_of::(), - usize::from(offsets.size_of_vmmemory_definition()) + usize::from(offsets.ptr.size_of_vmmemory_definition()) ); assert_eq!( offset_of!(VMMemoryDefinition, base), - usize::from(offsets.vmmemory_definition_base()) + usize::from(offsets.ptr.vmmemory_definition_base()) ); assert_eq!( offset_of!(VMMemoryDefinition, current_length), - usize::from(offsets.vmmemory_definition_current_length()) + usize::from(offsets.ptr.vmmemory_definition_current_length()) ); /* TODO: Assert that the size of `current_length` matches. assert_eq!( diff --git a/crates/wasmtime/Cargo.toml b/crates/wasmtime/Cargo.toml index 65223caab723..c0643f039a82 100644 --- a/crates/wasmtime/Cargo.toml +++ b/crates/wasmtime/Cargo.toml @@ -37,6 +37,7 @@ once_cell = "1.12.0" rayon = { version = "1.0", optional = true } object = { version = "0.29", default-features = false, features = ['read_core', 'elf'] } async-trait = { version = "0.1.51", optional = true } +encoding_rs = { version = "0.8.31", optional = true } [target.'cfg(target_os = "windows")'.dependencies.windows-sys] version = "0.36.0" @@ -116,4 +117,5 @@ component-model = [ "wasmtime-runtime/component-model", "dep:wasmtime-component-macro", "dep:wasmtime-component-util", + "dep:encoding_rs", ] diff --git a/crates/wasmtime/src/component/component.rs b/crates/wasmtime/src/component/component.rs index f60aa99fb29a..f4afc9789f73 100644 --- a/crates/wasmtime/src/component/component.rs +++ b/crates/wasmtime/src/component/component.rs @@ -11,7 +11,7 @@ use std::ptr::NonNull; use std::sync::Arc; use wasmtime_environ::component::{ AlwaysTrapInfo, ComponentTypes, FunctionInfo, GlobalInitializer, LoweredIndex, - RuntimeAlwaysTrapIndex, StaticModuleIndex, Translator, + RuntimeAlwaysTrapIndex, RuntimeTranscoderIndex, StaticModuleIndex, Translator, }; use wasmtime_environ::{PrimaryMap, ScopeVec, SignatureIndex, Trampoline, TrapCode}; use wasmtime_jit::CodeMemory; @@ -64,6 +64,10 @@ struct ComponentInner { /// These functions are "degenerate functions" here solely to implement /// functions that are `canon lift`'d then immediately `canon lower`'d. always_trap: PrimaryMap, + + /// Where all the cranelift-generated transcode functions are located in the + /// compiled image of this component. + transcoders: PrimaryMap, } impl Component { @@ -158,7 +162,7 @@ impl Component { || Component::compile_component(engine, &component, &types, &provided_trampolines), ); let static_modules = static_modules?; - let (lowerings, always_trap, trampolines, trampoline_obj) = trampolines?; + let (lowerings, always_trap, transcoders, trampolines, trampoline_obj) = trampolines?; let mut trampoline_obj = CodeMemory::new(trampoline_obj); let code = trampoline_obj.publish()?; let text = wasmtime_jit::subslice_range(code.text, code.mmap); @@ -218,6 +222,7 @@ impl Component { text, lowerings, always_trap, + transcoders, }), }) } @@ -231,6 +236,7 @@ impl Component { ) -> Result<( PrimaryMap, PrimaryMap, + PrimaryMap, Vec, wasmtime_runtime::MmapVec, )> { @@ -239,22 +245,31 @@ impl Component { || -> Result<_> { Ok(engine.join_maybe_parallel( || compile_always_trap(engine, component, types), - || compile_trampolines(engine, component, types, provided_trampolines), + || -> Result<_> { + Ok(engine.join_maybe_parallel( + || compile_transcoders(engine, component, types), + || compile_trampolines(engine, component, types, provided_trampolines), + )) + }, )) }, ); let (lowerings, other) = results; - let (always_trap, trampolines) = other?; + let (always_trap, other) = other?; + let (transcoders, trampolines) = other?; let mut obj = engine.compiler().object()?; - let (lower, traps, trampolines) = engine.compiler().component_compiler().emit_obj( - lowerings?, - always_trap?, - trampolines?, - &mut obj, - )?; + let (lower, traps, transcoders, trampolines) = + engine.compiler().component_compiler().emit_obj( + lowerings?, + always_trap?, + transcoders?, + trampolines?, + &mut obj, + )?; return Ok(( lower, traps, + transcoders, trampolines, wasmtime_jit::mmap_vec_from_obj(obj)?, )); @@ -307,6 +322,30 @@ impl Component { .collect()) } + fn compile_transcoders( + engine: &Engine, + component: &wasmtime_environ::component::Component, + types: &ComponentTypes, + ) -> Result>> { + let always_trap = component + .initializers + .iter() + .filter_map(|init| match init { + GlobalInitializer::Transcoder(i) => Some(i), + _ => None, + }) + .collect::>(); + Ok(engine + .run_maybe_parallel(always_trap, |info| { + engine + .compiler() + .component_compiler() + .compile_transcoder(component, info, types) + })? + .into_iter() + .collect()) + } + fn compile_trampolines( engine: &Engine, component: &wasmtime_environ::component::Component, @@ -376,6 +415,11 @@ impl Component { self.func(&info.info) } + pub(crate) fn transcoder_ptr(&self, index: RuntimeTranscoderIndex) -> NonNull { + let info = &self.inner.transcoders[index]; + self.func(info) + } + fn func(&self, info: &FunctionInfo) -> NonNull { let text = self.text(); let trampoline = &text[info.start as usize..][..info.length as usize]; diff --git a/crates/wasmtime/src/component/func/typed.rs b/crates/wasmtime/src/component/func/typed.rs index 0300ac4e72e6..ec65231f5b5d 100644 --- a/crates/wasmtime/src/component/func/typed.rs +++ b/crates/wasmtime/src/component/func/typed.rs @@ -1,7 +1,7 @@ use crate::component::func::{Func, Memory, MemoryMut, Options}; use crate::store::StoreOpaque; use crate::{AsContext, AsContextMut, StoreContext, StoreContextMut, ValRaw}; -use anyhow::{bail, Context, Result}; +use anyhow::{anyhow, bail, Context, Result}; use std::borrow::Cow; use std::fmt; use std::marker; @@ -801,6 +801,10 @@ unsafe impl Lift for char { } } +// TODO: these probably need different constants for memory64 +const UTF16_TAG: usize = 1 << 31; +const MAX_STRING_BYTE_LENGTH: usize = (1 << 31) - 1; + // Note that this is similar to `ComponentType for WasmStr` except it can only // be used for lowering, not lifting. unsafe impl ComponentType for str { @@ -843,16 +847,51 @@ unsafe impl Lower for str { } fn lower_string(mem: &mut MemoryMut<'_, T>, string: &str) -> Result<(usize, usize)> { + // Note that in general the wasm module can't assume anything about what the + // host strings are encoded as. Additionally hosts are allowed to have + // differently-encoded strings at runtime. Finally when copying a string + // into wasm it's somewhat strict in the sense that the various patterns of + // allocation and such are already dictated for us. + // + // In general what this means is that when copying a string from the host + // into the destination we need to follow one of the cases of copying into + // WebAssembly. It doesn't particularly matter which case as long as it ends + // up in the right encoding. For example a destination encoding of + // latin1+utf16 has a number of ways to get copied into and we do something + // here that isn't the default "utf8 to latin1+utf16" since we have access + // to simd-accelerated helpers in the `encoding_rs` crate. This is ok though + // because we can fake that the host string was already stored in latin1 + // format and follow that copy pattern instead. match mem.string_encoding() { + // This corresponds to `store_string_copy` in the canonical ABI where + // the host's representation is utf-8 and the wasm module wants utf-8 so + // a copy is all that's needed (and the `realloc` can be precise for the + // initial memory allocation). StringEncoding::Utf8 => { + if string.len() > MAX_STRING_BYTE_LENGTH { + bail!( + "string length of {} too large to copy into wasm", + string.len() + ); + } let ptr = mem.realloc(0, 0, 1, string.len())?; if string.len() > 0 { mem.as_slice_mut()[ptr..][..string.len()].copy_from_slice(string.as_bytes()); } Ok((ptr, string.len())) } + + // This corresponds to `store_utf8_to_utf16` in the canonical ABI. Here + // an over-large allocation is performed and then shrunk afterwards if + // necessary. StringEncoding::Utf16 => { let size = string.len() * 2; + if size > MAX_STRING_BYTE_LENGTH { + bail!( + "string length of {} too large to copy into wasm", + string.len() + ); + } let mut ptr = mem.realloc(0, 0, 2, size)?; let mut copied = 0; if size > 0 { @@ -869,8 +908,60 @@ fn lower_string(mem: &mut MemoryMut<'_, T>, string: &str) -> Result<(usize, u } Ok((ptr, copied)) } + StringEncoding::CompactUtf16 => { - unimplemented!("compact-utf-16"); + // This corresponds to `store_string_to_latin1_or_utf16` + let bytes = string.as_bytes(); + let mut iter = string.char_indices(); + let mut ptr = mem.realloc(0, 0, 2, bytes.len())?; + let mut dst = &mut mem.as_slice_mut()[ptr..][..bytes.len()]; + let mut result = 0; + while let Some((i, ch)) = iter.next() { + // Test if this `char` fits into the latin1 encoding. + if let Ok(byte) = u8::try_from(u32::from(ch)) { + dst[result] = byte; + result += 1; + continue; + } + + // .. if utf16 is forced to be used then the allocation is + // bumped up to the maximum size. + let worst_case = bytes + .len() + .checked_mul(2) + .ok_or_else(|| anyhow!("byte length overflow"))?; + if worst_case > MAX_STRING_BYTE_LENGTH { + bail!("byte length too large"); + } + ptr = mem.realloc(ptr, bytes.len(), 2, worst_case)?; + dst = &mut mem.as_slice_mut()[ptr..][..worst_case]; + + // Previously encoded latin1 bytes are inflated to their 16-bit + // size for utf16 + for i in (0..result).rev() { + dst[2 * i] = dst[i]; + dst[2 * i + 1] = 0; + } + + // and then the remainder of the string is encoded. + for (u, bytes) in string[i..] + .encode_utf16() + .zip(dst[2 * result..].chunks_mut(2)) + { + let u_bytes = u.to_le_bytes(); + bytes[0] = u_bytes[0]; + bytes[1] = u_bytes[1]; + result += 1; + } + if worst_case > 2 * result { + ptr = mem.realloc(ptr, worst_case, 2, 2 * result)?; + } + return Ok((ptr, result | UTF16_TAG)); + } + if result < bytes.len() { + ptr = mem.realloc(ptr, bytes.len(), 2, result)?; + } + Ok((ptr, result)) } } } @@ -898,7 +989,13 @@ impl WasmStr { let byte_len = match memory.string_encoding() { StringEncoding::Utf8 => Some(len), StringEncoding::Utf16 => len.checked_mul(2), - StringEncoding::CompactUtf16 => unimplemented!(), + StringEncoding::CompactUtf16 => { + if len & UTF16_TAG == 0 { + Some(len) + } else { + (len ^ UTF16_TAG).checked_mul(2) + } + } }; match byte_len.and_then(|len| ptr.checked_add(len)) { Some(n) if n <= memory.as_slice().len() => {} @@ -939,8 +1036,14 @@ impl WasmStr { fn to_str_from_store<'a>(&self, store: &'a StoreOpaque) -> Result> { match self.options.string_encoding() { StringEncoding::Utf8 => self.decode_utf8(store), - StringEncoding::Utf16 => self.decode_utf16(store), - StringEncoding::CompactUtf16 => unimplemented!(), + StringEncoding::Utf16 => self.decode_utf16(store, self.len), + StringEncoding::CompactUtf16 => { + if self.len & UTF16_TAG == 0 { + self.decode_latin1(store) + } else { + self.decode_utf16(store, self.len ^ UTF16_TAG) + } + } } } @@ -952,10 +1055,10 @@ impl WasmStr { Ok(str::from_utf8(&memory[self.ptr..][..self.len])?.into()) } - fn decode_utf16<'a>(&self, store: &'a StoreOpaque) -> Result> { + fn decode_utf16<'a>(&self, store: &'a StoreOpaque, len: usize) -> Result> { let memory = self.options.memory(store); // See notes in `decode_utf8` for why this is panicking indexing. - let memory = &memory[self.ptr..][..self.len * 2]; + let memory = &memory[self.ptr..][..len * 2]; Ok(std::char::decode_utf16( memory .chunks(2) @@ -964,6 +1067,14 @@ impl WasmStr { .collect::>()? .into()) } + + fn decode_latin1<'a>(&self, store: &'a StoreOpaque) -> Result> { + // See notes in `decode_utf8` for why this is panicking indexing. + let memory = self.options.memory(store); + Ok(encoding_rs::mem::decode_latin1( + &memory[self.ptr..][..self.len], + )) + } } // Note that this is similar to `ComponentType for str` except it can only be @@ -1068,7 +1179,7 @@ where let size = list .len() .checked_mul(elem_size) - .ok_or_else(|| anyhow::anyhow!("size overflow copying a list"))?; + .ok_or_else(|| anyhow!("size overflow copying a list"))?; let ptr = mem.realloc(0, 0, T::ALIGN32, size)?; let mut cur = ptr; for item in list { diff --git a/crates/wasmtime/src/component/instance.rs b/crates/wasmtime/src/component/instance.rs index 1f836051d33c..7abc81bfeafa 100644 --- a/crates/wasmtime/src/component/instance.rs +++ b/crates/wasmtime/src/component/instance.rs @@ -10,9 +10,9 @@ use std::sync::Arc; use wasmtime_environ::component::{ AlwaysTrap, ComponentTypes, CoreDef, CoreExport, Export, ExportItem, ExtractMemory, ExtractPostReturn, ExtractRealloc, GlobalInitializer, InstantiateModule, LowerImport, - RuntimeImportIndex, RuntimeInstanceIndex, RuntimeModuleIndex, + RuntimeImportIndex, RuntimeInstanceIndex, RuntimeModuleIndex, Transcoder, }; -use wasmtime_environ::{EntityIndex, Global, GlobalInit, PrimaryMap, WasmType}; +use wasmtime_environ::{EntityIndex, EntityType, Global, GlobalInit, PrimaryMap, WasmType}; use wasmtime_runtime::component::{ComponentInstance, OwnedComponentInstance}; /// An instantiated component. @@ -142,6 +142,11 @@ impl InstanceData { }, }) } + CoreDef::Transcoder(idx) => { + wasmtime_runtime::Export::Function(wasmtime_runtime::ExportFunction { + anyfunc: self.state.transcoder_anyfunc(*idx), + }) + } } } @@ -287,6 +292,8 @@ impl<'a> Instantiator<'a> { _ => unreachable!(), }); } + + GlobalInitializer::Transcoder(e) => self.transcoder(e), } } Ok(()) @@ -328,6 +335,17 @@ impl<'a> Instantiator<'a> { ); } + fn transcoder(&mut self, transcoder: &Transcoder) { + self.data.state.set_transcoder( + transcoder.index, + self.component.transcoder_ptr(transcoder.index), + self.component + .signatures() + .shared_signature(transcoder.signature) + .expect("found unregistered signature"), + ); + } + fn extract_memory(&mut self, store: &mut StoreOpaque, memory: &ExtractMemory) { let mem = match self.data.lookup_export(store, &memory.export) { wasmtime_runtime::Export::Memory(m) => m, @@ -371,24 +389,16 @@ impl<'a> Instantiator<'a> { // core wasm instantiations internally within a component are // unnecessary and superfluous. Naturally though mistakes may be // made, so double-check this property of wasmtime in debug mode. + if cfg!(debug_assertions) { - let export = self.data.lookup_def(store, arg); let (_, _, expected) = imports.next().unwrap(); - let val = unsafe { crate::Extern::from_wasmtime_export(export, store) }; - crate::types::matching::MatchCx { - store, - engine: store.engine(), - signatures: module.signatures(), - types: module.types(), - } - .extern_(&expected, &val) - .expect("unexpected typecheck failure"); + self.assert_type_matches(store, module, arg, expected); } - let export = self.data.lookup_def(store, arg); // The unsafety here should be ok since the `export` is loaded // directly from an instance which should only give us valid export // items. + let export = self.data.lookup_def(store, arg); unsafe { self.core_imports.push_export(&export); } @@ -397,6 +407,41 @@ impl<'a> Instantiator<'a> { &self.core_imports } + + fn assert_type_matches( + &mut self, + store: &mut StoreOpaque, + module: &Module, + arg: &CoreDef, + expected: EntityType, + ) { + let export = self.data.lookup_def(store, arg); + + // If this value is a core wasm function then the type check is inlined + // here. This can otherwise fail `Extern::from_wasmtime_export` because + // there's no guarantee that there exists a trampoline for `f` so this + // can't fall through to the case below + if let wasmtime_runtime::Export::Function(f) = &export { + match expected { + EntityType::Function(expected) => { + let actual = unsafe { f.anyfunc.as_ref().type_index }; + assert_eq!(module.signatures().shared_signature(expected), Some(actual)); + return; + } + _ => panic!("function not expected"), + } + } + + let val = unsafe { crate::Extern::from_wasmtime_export(export, store) }; + crate::types::matching::MatchCx { + store, + engine: store.engine(), + signatures: module.signatures(), + types: module.types(), + } + .extern_(&expected, &val) + .expect("unexpected typecheck failure"); + } } /// A "pre-instantiated" [`Instance`] which has all of its arguments already diff --git a/crates/wasmtime/src/config.rs b/crates/wasmtime/src/config.rs index 2b3cf1202c07..6bb890be3ff5 100644 --- a/crates/wasmtime/src/config.rs +++ b/crates/wasmtime/src/config.rs @@ -1462,6 +1462,15 @@ impl Config { compiler.build() } + + /// Internal setting for whether adapter modules for components will have + /// extra WebAssembly instructions inserted performing more debug checks + /// then are necessary. + #[cfg(feature = "component-model")] + pub fn debug_adapter_modules(&mut self, debug: bool) -> &mut Self { + self.tunables.debug_adapter_modules = debug; + self + } } fn round_up_to_pages(val: u64) -> u64 { diff --git a/deny.toml b/deny.toml index 036c996d0ce8..ee47bb543e10 100644 --- a/deny.toml +++ b/deny.toml @@ -14,10 +14,9 @@ allow = [ "Apache-2.0 WITH LLVM-exception", "Apache-2.0", "BSD-2-Clause", - "CC0-1.0", + "BSD-3-Clause", "ISC", "MIT", - "MPL-2.0", "Zlib", ] diff --git a/supply-chain/config.toml b/supply-chain/config.toml index f46040a05d00..e8e3985ae852 100644 --- a/supply-chain/config.toml +++ b/supply-chain/config.toml @@ -311,6 +311,10 @@ criteria = "safe-to-deploy" version = "0.3.6" criteria = "safe-to-deploy" +[[exemptions.encoding_rs]] +version = "0.8.31" +criteria = "safe-to-deploy" + [[exemptions.env_logger]] version = "0.7.1" criteria = "safe-to-deploy" diff --git a/tests/all/component_model.rs b/tests/all/component_model.rs index a9a2117ad151..325ad1f965e3 100644 --- a/tests/all/component_model.rs +++ b/tests/all/component_model.rs @@ -12,6 +12,7 @@ mod instance; mod macros; mod nested; mod post_return; +mod strings; #[test] fn components_importing_modules() -> Result<()> { diff --git a/tests/all/component_model/strings.rs b/tests/all/component_model/strings.rs new file mode 100644 index 000000000000..59997090deba --- /dev/null +++ b/tests/all/component_model/strings.rs @@ -0,0 +1,578 @@ +use super::REALLOC_AND_FREE; +use anyhow::Result; +use wasmtime::component::{Component, Linker}; +use wasmtime::{Engine, Store, StoreContextMut, Trap, TrapCode}; + +const UTF16_TAG: u32 = 1 << 31; + +// Special cases that this tries to test: +// +// * utf8 -> utf8 +// * various code point sizes +// +// * utf8 -> utf16 - the adapter here will make a pessimistic allocation that's +// twice the size of the utf8 encoding for the utf16 destination +// * utf16 byte size is twice the utf8 size +// * utf16 byte size is less than twice the utf8 size +// +// * utf8 -> latin1+utf16 - attempts to convert to latin1 then falls back to a +// pessimistic utf16 allocation that's downsized if necessary +// * utf8 fits exactly in latin1 +// * utf8 fits latin1 but is bigger byte-wise +// * utf8 is not latin1 and fits utf16 allocation precisely (NOT POSSIBLE) +// * utf8 is not latin1 and utf16 is smaller than allocation +// +// * utf16 -> utf8 - this starts with an optimistic size and then reallocates to +// a pessimistic size, interesting cases are: +// * utf8 size is 0.5x the utf16 byte size (perfect fit in initial alloc) +// * utf8 size is 1.5x the utf16 byte size (perfect fit in larger alloc) +// * utf8 size is 0.5x-1.5x the utf16 size (larger alloc is downsized) +// +// * utf16 -> utf16 +// * various code point sizes +// +// * utf16 -> latin1+utf16 - attempts to convert to latin1 then falls back to a +// pessimistic utf16 allocation that's downsized if necessary +// * utf16 fits exactly in latin1 +// * utf16 fits latin1 but is bigger byte-wise (NOT POSSIBLE) +// * utf16 is not latin1 and fits utf16 allocation precisely +// * utf16 is not latin1 and utf16 is smaller than allocation (NOT POSSIBLE) +// +// * compact-utf16 -> utf8 dynamically determines between one of +// * latin1 -> utf8 +// * latin1 size matches utf8 size +// * latin1 is smaller than utf8 size +// * utf16 -> utf8 +// * covered above +// +// * compact-utf16 -> utf16 dynamically determines between one of +// * latin1 -> utf16 - latin1 size always matches utf16 +// * test various code points +// * utf16 -> utf16 +// * covered above +// +// * compact-utf16 -> compact-utf16 dynamically determines between one of +// * latin1 -> latin1 +// * not much interesting here +// * utf16 -> compact-utf16-to-compact-probably-utf16 +// * utf16 actually fits within latin1 +// * otherwise not more interesting than utf16 -> utf16 +// +const STRINGS: &[&str] = &[ + "", + // 1 byte in utf8, 2 bytes in utf16 + "x", + "hello this is a particularly long string yes it is it keeps going", + // 35 bytes in utf8, 23 units in utf16, 23 bytes in latin1 + "à á â ã ä å æ ç è é ê ë", + // 47 bytes in utf8, 31 units in utf16 + "Ξ Ο Π Ρ Σ Τ Υ Φ Χ Ψ Ω Ϊ Ϋ ά έ ή", + // 24 bytes in utf8, 8 units in utf16 + "ＳＴＵＶＷＸＹＺ", + // 16 bytes in utf8, 8 units in utf16 + "ËÌÍÎÏÐÑÒ", + // 4 bytes in utf8, 1 unit in utf16 + "\u{10000}", + // latin1-compatible prefix followed by utf8/16-requiring suffix + // + // 24 bytes in utf8, 13 units in utf16, first 8 usvs are latin1-compatible + "à ascii ＶＷＸＹＺ", +]; + +static ENCODINGS: [&str; 3] = ["utf8", "utf16", "latin1+utf16"]; + +#[test] +fn roundtrip() -> Result<()> { + for debug in [true, false] { + let mut config = component_test_util::config(); + config.debug_adapter_modules(debug); + let engine = Engine::new(&config)?; + for src in ENCODINGS { + for dst in ENCODINGS { + test_roundtrip(&engine, src, dst)?; + } + } + } + Ok(()) +} + +fn test_roundtrip(engine: &Engine, src: &str, dst: &str) -> Result<()> { + println!("src={src} dst={dst}"); + + let mk_echo = |name: &str, encoding: &str| { + format!( + r#" +(component {name} + (import "echo" (func $echo (param string) (result string))) + (core instance $libc (instantiate $libc)) + (core func $echo (canon lower (func $echo) + (memory $libc "memory") + (realloc (func $libc "realloc")) + string-encoding={encoding} + )) + (core instance $echo (instantiate $echo + (with "libc" (instance $libc)) + (with "" (instance (export "echo" (func $echo)))) + )) + (func (export "echo") (param string) (result string) + (canon lift + (core func $echo "echo") + (memory $libc "memory") + (realloc (func $libc "realloc")) + string-encoding={encoding} + ) + ) +) + "# + ) + }; + + let src = mk_echo("$src", src); + let dst = mk_echo("$dst", dst); + let component = format!( + r#" +(component + (import "host" (func $host (param string) (result string))) + + (core module $libc + (memory (export "memory") 1) + {REALLOC_AND_FREE} + ) + (core module $echo + (import "" "echo" (func $echo (param i32 i32 i32))) + (import "libc" "memory" (memory 0)) + (import "libc" "realloc" (func $realloc (param i32 i32 i32 i32) (result i32))) + + (func (export "echo") (param i32 i32) (result i32) + (local $retptr i32) + (local.set $retptr + (call $realloc + (i32.const 0) + (i32.const 0) + (i32.const 4) + (i32.const 8))) + (call $echo + (local.get 0) + (local.get 1) + (local.get $retptr)) + local.get $retptr + ) + ) + + {src} + {dst} + + (instance $dst (instantiate $dst (with "echo" (func $host)))) + (instance $src (instantiate $src (with "echo" (func $dst "echo")))) + (export "echo" (func $src "echo")) +) +"# + ); + let component = Component::new(engine, &component)?; + let mut store = Store::new(engine, String::new()); + let mut linker = Linker::new(engine); + linker + .root() + .func_wrap("host", |store: StoreContextMut, arg: String| { + assert_eq!(*store.data(), arg); + Ok(arg) + })?; + let instance = linker.instantiate(&mut store, &component)?; + let func = instance.get_typed_func::<(String,), String, _>(&mut store, "echo")?; + + for string in STRINGS { + println!("testing string {string:?}"); + *store.data_mut() = string.to_string(); + let ret = func.call(&mut store, (string.to_string(),))?; + assert_eq!(ret, *string); + func.post_return(&mut store)?; + } + Ok(()) +} + +#[test] +fn ptr_out_of_bounds() -> Result<()> { + let engine = component_test_util::engine(); + for src in ENCODINGS { + for dst in ENCODINGS { + test_ptr_out_of_bounds(&engine, src, dst)?; + } + } + Ok(()) +} + +fn test_ptr_out_of_bounds(engine: &Engine, src: &str, dst: &str) -> Result<()> { + let test = |len: u32| -> Result<()> { + let component = format!( + r#" +(component + (component $c + (core module $m + (func (export "") (param i32 i32)) + (func (export "realloc") (param i32 i32 i32 i32) (result i32) i32.const 0) + (memory (export "memory") 1) + ) + (core instance $m (instantiate $m)) + (func (export "") (param string) + (canon lift (core func $m "") (realloc (func $m "realloc")) (memory $m "memory") + string-encoding={dst}) + ) + ) + + (component $c2 + (import "" (func $f (param string))) + (core module $libc + (memory (export "memory") 1) + ) + (core instance $libc (instantiate $libc)) + (core func $f (canon lower (func $f) string-encoding={src} (memory $libc "memory"))) + (core module $m + (import "" "" (func $f (param i32 i32))) + + (func $start (call $f (i32.const 0x8000_0000) (i32.const {len}))) + (start $start) + ) + (core instance (instantiate $m (with "" (instance (export "" (func $f)))))) + ) + + (instance $c (instantiate $c)) + (instance $c2 (instantiate $c2 (with "" (func $c "")))) +) +"# + ); + let component = Component::new(engine, &component)?; + let mut store = Store::new(engine, ()); + let trap = Linker::new(engine) + .instantiate(&mut store, &component) + .err() + .unwrap() + .downcast::()?; + assert_eq!(trap.trap_code(), Some(TrapCode::UnreachableCodeReached)); + Ok(()) + }; + + test(0)?; + test(1)?; + + Ok(()) +} + +// Test that even if the ptr+len calculation overflows then a trap still +// happens. +#[test] +fn ptr_overflow() -> Result<()> { + let engine = component_test_util::engine(); + for src in ENCODINGS { + for dst in ENCODINGS { + test_ptr_overflow(&engine, src, dst)?; + } + } + Ok(()) +} + +fn test_ptr_overflow(engine: &Engine, src: &str, dst: &str) -> Result<()> { + let component = format!( + r#" +(component + (component $c + (core module $m + (func (export "") (param i32 i32)) + (func (export "realloc") (param i32 i32 i32 i32) (result i32) i32.const 0) + (memory (export "memory") 1) + ) + (core instance $m (instantiate $m)) + (func (export "") (param string) + (canon lift (core func $m "") (realloc (func $m "realloc")) (memory $m "memory") + string-encoding={dst}) + ) + ) + + (component $c2 + (import "" (func $f (param string))) + (core module $libc + (memory (export "memory") 1) + ) + (core instance $libc (instantiate $libc)) + (core func $f (canon lower (func $f) string-encoding={src} (memory $libc "memory"))) + (core module $m + (import "" "" (func $f (param i32 i32))) + + (func (export "f") (param i32) (call $f (i32.const 1000) (local.get 0))) + ) + (core instance $m (instantiate $m (with "" (instance (export "" (func $f)))))) + (func (export "f") (param u32) (canon lift (core func $m "f"))) + ) + + (instance $c (instantiate $c)) + (instance $c2 (instantiate $c2 (with "" (func $c "")))) + (export "f" (func $c2 "f")) +) +"# + ); + + let component = Component::new(engine, &component)?; + let mut store = Store::new(engine, ()); + + let mut test_overflow = |size: u32| -> Result<()> { + println!("src={src} dst={dst} size={size:#x}"); + let instance = Linker::new(engine).instantiate(&mut store, &component)?; + let func = instance.get_typed_func::<(u32,), (), _>(&mut store, "f")?; + let trap = func + .call(&mut store, (size,)) + .unwrap_err() + .downcast::()?; + assert_eq!(trap.trap_code(), Some(TrapCode::UnreachableCodeReached)); + Ok(()) + }; + + let max = 1 << 31; + + match src { + "utf8" => { + // This exceeds MAX_STRING_BYTE_LENGTH + test_overflow(max)?; + + if dst == "utf16" { + // exceeds MAX_STRING_BYTE_LENGTH when multiplied + test_overflow(max / 2)?; + + // Technically this fails on the first string, not the second. + // Ideally this would test the overflow check on the second + // string though. + test_overflow(max / 2 - 100)?; + } else { + // This will point into unmapped memory + test_overflow(max - 100)?; + } + } + + "utf16" => { + test_overflow(max / 2)?; + test_overflow(max / 2 - 100)?; + } + + "latin1+utf16" => { + test_overflow((max / 2) | UTF16_TAG)?; + // tag a utf16 string with the max length and it should overflow. + test_overflow((max / 2 - 100) | UTF16_TAG)?; + } + + _ => unreachable!(), + } + + Ok(()) +} + +// Test that that the pointer returned from `realloc` is bounds-checked. +#[test] +fn realloc_oob() -> Result<()> { + let engine = component_test_util::engine(); + for src in ENCODINGS { + for dst in ENCODINGS { + test_realloc_oob(&engine, src, dst)?; + } + } + Ok(()) +} + +fn test_realloc_oob(engine: &Engine, src: &str, dst: &str) -> Result<()> { + let component = format!( + r#" +(component + (component $c + (core module $m + (func (export "") (param i32 i32)) + (func (export "realloc") (param i32 i32 i32 i32) (result i32) i32.const 100_000) + (memory (export "memory") 1) + ) + (core instance $m (instantiate $m)) + (func (export "") (param string) + (canon lift (core func $m "") (realloc (func $m "realloc")) (memory $m "memory") + string-encoding={dst}) + ) + ) + + (component $c2 + (import "" (func $f (param string))) + (core module $libc + (memory (export "memory") 1) + ) + (core instance $libc (instantiate $libc)) + (core func $f (canon lower (func $f) string-encoding={src} (memory $libc "memory"))) + (core module $m + (import "" "" (func $f (param i32 i32))) + + (func (export "f") (call $f (i32.const 1000) (i32.const 10))) + ) + (core instance $m (instantiate $m (with "" (instance (export "" (func $f)))))) + (func (export "f") (canon lift (core func $m "f"))) + ) + + (instance $c (instantiate $c)) + (instance $c2 (instantiate $c2 (with "" (func $c "")))) + (export "f" (func $c2 "f")) +) +"# + ); + + let component = Component::new(engine, &component)?; + let mut store = Store::new(engine, ()); + + let instance = Linker::new(engine).instantiate(&mut store, &component)?; + let func = instance.get_typed_func::<(), (), _>(&mut store, "f")?; + let trap = func.call(&mut store, ()).unwrap_err().downcast::()?; + assert_eq!(trap.trap_code(), Some(TrapCode::UnreachableCodeReached)); + Ok(()) +} + +// Test that that the pointer returned from `realloc` is bounds-checked. +#[test] +fn raw_string_encodings() -> Result<()> { + let engine = component_test_util::engine(); + test_invalid_string_encoding(&engine, "utf8", "utf8", &[0xff], 1)?; + let array = b"valid string until \xffthen valid again"; + test_invalid_string_encoding(&engine, "utf8", "utf8", array, array.len() as u32)?; + test_invalid_string_encoding(&engine, "utf8", "utf16", array, array.len() as u32)?; + let array = b"symbol \xce\xa3 until \xffthen valid"; + test_invalid_string_encoding(&engine, "utf8", "utf8", array, array.len() as u32)?; + test_invalid_string_encoding(&engine, "utf8", "utf16", array, array.len() as u32)?; + test_invalid_string_encoding(&engine, "utf8", "latin1+utf16", array, array.len() as u32)?; + test_invalid_string_encoding(&engine, "utf16", "utf8", &[0x01, 0xd8], 1)?; + test_invalid_string_encoding(&engine, "utf16", "utf16", &[0x01, 0xd8], 1)?; + test_invalid_string_encoding( + &engine, + "utf16", + "latin1+utf16", + &[0xff, 0xff, 0x01, 0xd8], + 2, + )?; + test_invalid_string_encoding( + &engine, + "latin1+utf16", + "utf8", + &[0x01, 0xd8], + 1 | UTF16_TAG, + )?; + test_invalid_string_encoding( + &engine, + "latin1+utf16", + "utf16", + &[0x01, 0xd8], + 1 | UTF16_TAG, + )?; + test_invalid_string_encoding( + &engine, + "latin1+utf16", + "utf16", + &[0xff, 0xff, 0x01, 0xd8], + 2 | UTF16_TAG, + )?; + test_invalid_string_encoding( + &engine, + "latin1+utf16", + "latin1+utf16", + &[0xab, 0x00, 0xff, 0xff, 0x01, 0xd8], + 3 | UTF16_TAG, + )?; + + // This latin1+utf16 string should get compressed to latin1 across the + // boundary. + test_valid_string_encoding( + &engine, + "latin1+utf16", + "latin1+utf16", + &[0xab, 0x00, 0xff, 0x00], + 2 | UTF16_TAG, + )?; + Ok(()) +} + +fn test_invalid_string_encoding( + engine: &Engine, + src: &str, + dst: &str, + bytes: &[u8], + len: u32, +) -> Result<()> { + let trap = test_raw_when_encoded(engine, src, dst, bytes, len)?.unwrap(); + let src = src.replace("latin1+", ""); + assert!( + trap.to_string() + .contains(&format!("invalid {src} encoding")), + "bad error: {}", + trap, + ); + Ok(()) +} + +fn test_valid_string_encoding( + engine: &Engine, + src: &str, + dst: &str, + bytes: &[u8], + len: u32, +) -> Result<()> { + let err = test_raw_when_encoded(engine, src, dst, bytes, len)?; + assert!(err.is_none()); + Ok(()) +} + +fn test_raw_when_encoded( + engine: &Engine, + src: &str, + dst: &str, + bytes: &[u8], + len: u32, +) -> Result> { + let component = format!( + r#" +(component + (component $c + (core module $m + (func (export "") (param i32 i32)) + (func (export "realloc") (param i32 i32 i32 i32) (result i32) i32.const 0) + (memory (export "memory") 1) + ) + (core instance $m (instantiate $m)) + (func (export "") (param string) + (canon lift (core func $m "") (realloc (func $m "realloc")) (memory $m "memory") + string-encoding={dst}) + ) + ) + + (component $c2 + (import "" (func $f (param string))) + (core module $libc + (memory (export "memory") 1) + (func (export "realloc") (param i32 i32 i32 i32) (result i32) i32.const 0) + ) + (core instance $libc (instantiate $libc)) + (core func $f (canon lower (func $f) string-encoding={src} (memory $libc "memory"))) + (core module $m + (import "" "" (func $f (param i32 i32))) + + (func (export "f") (param i32 i32 i32) (call $f (local.get 0) (local.get 2))) + ) + (core instance $m (instantiate $m (with "" (instance (export "" (func $f)))))) + (func (export "f") (param (list u8)) (param u32) (canon lift (core func $m "f") + (memory $libc "memory") + (realloc (func $libc "realloc")))) + ) + + (instance $c (instantiate $c)) + (instance $c2 (instantiate $c2 (with "" (func $c "")))) + (export "f" (func $c2 "f")) +) +"# + ); + + let component = Component::new(engine, &component)?; + let mut store = Store::new(engine, ()); + + let instance = Linker::new(engine).instantiate(&mut store, &component)?; + let func = instance.get_typed_func::<(&[u8], u32), (), _>(&mut store, "f")?; + match func.call(&mut store, (bytes, len)) { + Ok(_) => Ok(None), + Err(e) => Ok(Some(e.downcast()?)), + } +} diff --git a/tests/misc_testsuite/component-model/strings.wast b/tests/misc_testsuite/component-model/strings.wast new file mode 100644 index 000000000000..398ca39563ae --- /dev/null +++ b/tests/misc_testsuite/component-model/strings.wast @@ -0,0 +1,108 @@ +;; unaligned utf16 string +(assert_trap + (component + (component $c + (core module $m + (func (export "") (param i32 i32)) + (func (export "realloc") (param i32 i32 i32 i32) (result i32) i32.const 0) + (memory (export "memory") 1) + ) + (core instance $m (instantiate $m)) + (func (export "") (param string) + (canon lift (core func $m "") (realloc (func $m "realloc")) (memory $m "memory")) + ) + ) + + (component $c2 + (import "" (func $f (param string))) + (core module $libc + (memory (export "memory") 1) + ) + (core instance $libc (instantiate $libc)) + (core func $f (canon lower (func $f) string-encoding=utf16 (memory $libc "memory"))) + (core module $m + (import "" "" (func $f (param i32 i32))) + + (func $start (call $f (i32.const 1) (i32.const 0))) + (start $start) + ) + (core instance (instantiate $m (with "" (instance (export "" (func $f)))))) + ) + + (instance $c (instantiate $c)) + (instance $c2 (instantiate $c2 (with "" (func $c "")))) + ) + "unreachable") + +;; unaligned latin1+utf16 string, even with the latin1 encoding +(assert_trap + (component + (component $c + (core module $m + (func (export "") (param i32 i32)) + (func (export "realloc") (param i32 i32 i32 i32) (result i32) i32.const 0) + (memory (export "memory") 1) + ) + (core instance $m (instantiate $m)) + (func (export "") (param string) + (canon lift (core func $m "") (realloc (func $m "realloc")) (memory $m "memory")) + ) + ) + + (component $c2 + (import "" (func $f (param string))) + (core module $libc + (memory (export "memory") 1) + ) + (core instance $libc (instantiate $libc)) + (core func $f (canon lower (func $f) string-encoding=latin1+utf16 (memory $libc "memory"))) + (core module $m + (import "" "" (func $f (param i32 i32))) + + (func $start (call $f (i32.const 1) (i32.const 0))) + (start $start) + ) + (core instance (instantiate $m (with "" (instance (export "" (func $f)))))) + ) + + (instance $c (instantiate $c)) + (instance $c2 (instantiate $c2 (with "" (func $c "")))) + ) + "unreachable") + +;; out of bounds utf8->utf8 string +(assert_trap + (component + (component $c + (core module $m + (func (export "") (param i32 i32)) + (func (export "realloc") (param i32 i32 i32 i32) (result i32) i32.const 0) + (memory (export "memory") 1) + ) + (core instance $m (instantiate $m)) + (func (export "") (param string) + (canon lift (core func $m "") (realloc (func $m "realloc")) (memory $m "memory") + string-encoding=utf8) + ) + ) + + (component $c2 + (import "" (func $f (param string))) + (core module $libc + (memory (export "memory") 1) + ) + (core instance $libc (instantiate $libc)) + (core func $f (canon lower (func $f) string-encoding=utf8 (memory $libc "memory"))) + (core module $m + (import "" "" (func $f (param i32 i32))) + + (func $start (call $f (i32.const 0x8000_0000) (i32.const 1))) + (start $start) + ) + (core instance (instantiate $m (with "" (instance (export "" (func $f)))))) + ) + + (instance $c (instantiate $c)) + (instance $c2 (instantiate $c2 (with "" (func $c "")))) + ) + "unreachable")