lifter: read the ELF through LLVM instead of bfd, libdwarf and libelf - #265
Open
o2alexanderfedin wants to merge 2 commits into
Open
lifter: read the ELF through LLVM instead of bfd, libdwarf and libelf#265o2alexanderfedin wants to merge 2 commits into
o2alexanderfedin wants to merge 2 commits into
Conversation
The lifter linked three binutils libraries to do what LLVM already does: bfd to open the binary and walk its sections and symbols, libelf to read the ehdr, the program headers and symbol sizes, and libdwarf to recover function entries from `.eh_frame` when the binary is stripped. All three are replaced by llvm::object::ELFObjectFile and llvm::DWARFContext, which the lifter already links through remill. Three dependencies are deleted, not ported -- and libiberty goes with them, since it was only ever bfd's transitive requirement. Verified byte-identical on both loader paths against the stock lifter shipped in the :arm64 image, on a static-glibc aarch64 hello: not stripped (.symtab path) 900 funcs bc ec2dc6f9... wasm b15e9a6f... 5 654 545 B stripped (.eh_frame path) 899 funcs bc 883bbf48... wasm 735ce583... 5 624 783 B The stripped arm is new. The recorded harness only ever lifted a non-stripped binary, so `GetSblFromEhFrame` -- the entire libdwarf half -- was never executed by it. THE HARNESS EARNED ITS KEEP AND THE DEFECT IS WORTH RECORDING. The first build of this port was green, linked, and emitted 900 functions, and its bitcode was WRONG: ec2dc6f9 became 1b89e054. Cause: STT_GNU_IFUNC. bfd in this binutils does not raise BSF_FUNCTION for it, so the original filter admitted none of the binary's 10 ifunc symbols; I admitted them from memory of bfd's source. That shifted every later symbol index by 10 and, because the sort is by address and std::sort is NOT stable, re-scrambled which alias won at the 96 addresses where glibc puts several names on one function. Dropping STT_GNU_IFUNC restored the digest exactly. The count was never wrong -- 900 both times -- so a function-count check would have passed the defect through. `asection *` is replaced by `SectionRange *`, a name/vma/size triple owned by the loader. Containment queries run over EVERY section, unfiltered, because bfd's did; `ELFSection` is the wrong type for that job since it drops sections based at 0x0. TraceManager loses its <bfd.h> include, and with the LLVM objects held behind a pimpl no LLVM header reaches Loader.h either, so the header's third-party surface is now zero. No LLVM component is named in CMakeLists on purpose: llvm_map_components_to_libnames resolves to the STATIC archives, whose interface demands zstd::libzstd_shared, which is absent in the build image and fails configuration outright. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nu4maFxHPncwvdP8XcSEbD
Now that the loader reads the ELF through LLVM, a function's length comes off the symbol the loader is already holding. bfd could not do that: its `asymbol` is format-neutral and carries no size, so the loader re-read the binary through libelf, built a NAME -> size map over `.symtab` and `.dynsym`, and joined on the name -- the only key bfd left it. That join is lossy, and it is wrong on a hello-world. Measured on the static-glibc aarch64 subject the harness uses: 12 names collide, and 15 function symbols are handed a length that is not their own. The worst is `free_mem` -- eight distinct static functions in eight translation units, seven of which receive 100 bytes in place of their real 36, 40, 48, 56, 104, 184 or 276, because the map keeps whichever entry came last. TraceManager prefers the symbol size over section arithmetic, so the lifter disassembled the wrong number of bytes for those seven: some truncated, some over-read. THIS CHANGES THE EMITTED BITCODE ON PURPOSE, which is why it is not folded into the dependency removal it sits on top of. That commit's entire warrant is that it changed nothing -- bitcode and wasm byte-identical on both loader paths -- and proving "three libraries removed, behaviour unchanged" requires the change to be separable from any behaviour fix. not stripped (.symtab path) ec2dc6f9... -> fba665d1... 900 funcs, unchanged stripped (.eh_frame path) 883bbf48... unchanged 899 funcs, unchanged The stripped arm does not move and cannot: `.eh_frame` carries its own address ranges and never consulted the map. STATED LIMIT: the new sizes are argued correct from the ELF, not demonstrated by execution. The build image has no wasmedge -- `elfconv.sh` line 275 exits 127 on every lift, before and after -- so no lifted artifact was run here. What is measured is that the sizes now come from the symbol that owns them, and that nothing else in the pipeline moved. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nu4maFxHPncwvdP8XcSEbD
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Reads the ELF through
llvm::object::ELFObjectFileandllvm::DWARFContextinstead ofbinutils, and fixes a function-sizing bug that bfd's API had forced.
elfliftlinked three binutils libraries to do what LLVM already does and the lifteralready links:
bfdto open the binary and walk sections and symbols,libelffor the ehdr,program headers and symbol sizes, and
libdwarfto recover function entries from.eh_framewhen the binary is stripped. All three go, and
libibertygoes with them — it was only everbfd's transitive requirement. No LLVM library is added:
LLVMObjectandLLVMDebugInfoDWARFarrive with theLLVMthatremillalready links.Two commits, deliberately separate:
1. The port — byte-identical, which is its whole warrant.
Verified against the stock lifter in the
:arm64image, both loader paths, bitcode and wasm:.symtabec2dc6f9…b15e9a6f…(5 654 545 B).eh_frame883bbf48…735ce583…(5 624 783 B)The stripped arm matters:
GetSblFromEhFrame— the entire libdwarf half — had never beenexercised by any check I could find, and LLVM's
.eh_framereader reproduces libdwarf exactly.asection *becomes a smallSectionRange(name/vma/size) owned by the loader. Containmentqueries run over every section, unfiltered, because bfd's did —
ELFSectionis the wrongtype for that job since it drops sections based at
0x0. The LLVM objects sit behind a pimpl,so no LLVM header reaches
Loader.h, andTraceManager.hloses its<bfd.h>.2. The sizing fix — this one changes output on purpose.
bfd's
asymbolcarries no size, so the loader re-read the ELF through libelf, built aNAME → size map over
.symtaband.dynsym, and joined on the name. That join is lossy.On one static-glibc hello: 12 names collide and 15 function symbols get a length that is not
their own. The worst is
free_mem— eight distinct static functions, of which seven receive100 bytes in place of their real 36, 40, 48, 56, 104, 184 or 276, because the map keeps
whichever came last.
TraceManagerprefers the symbol size over section arithmetic, so thelifter disassembled the wrong number of bytes for those seven. LLVM exposes
st_sizeon thesymbol already in hand, so the workaround dies with the abstraction that forced it.
It is a second commit precisely because the first one's claim is "nothing changed", and the two
cannot both be true in one patch.
Evidence
ninja test_dependencies && ctest: 2/2 pass, including the integration test that runs thelifted wasm under WasmEdge and matches
Hello, World!.lddon the builtelflift: nobfd,dwarf,elforiberty._startto completion, and the pair reaches the same terminal state — different bitcode, samebehaviour.
Notes
While writing this the first build was green, linked, emitted the right function count — and
produced wrong bitcode, because I admitted
STT_GNU_IFUNCsymbols on the strength of bfd'ssource. The bfd in the build image does not flag them, so the original filter admitted none of
the binary's 10 ifuncs. Admitting them shifted every later symbol index by 10 and, through an
unstable sort over tied addresses, changed which alias named each lifted function — while the
count stayed identical. Worth knowing if anyone touches that filter: a count check does not see it.
Touches
lifter/TraceManager.cpp, as does the amd64 front-end PR — if both are wanted, oneneeds a trivial rebase on the other.