LTO PPC64: InputSection too large for range extension thunk

[aik]

Trying to hack the POWERPC64 kernel to build with LTO. The config is CONFIG_LTO_CLANG_FULL=y, llvm is the latest + https://reviews.llvm.org/D96931 (fixes multiple "changed binding to global").

Now the very last call to lld fails. I added some printf at aik/llvm-project@7f5e8fe#diff-7bda54aace526c7e883a5e1c34446b3ffca171b50a4340a8795ef0ecc0d1f291R1791 , got this

[fstn1-p1 kernel-llvm-13-lto]$ ld.lld -EL -m elf64lppc -mllvm -import-instr-limit=5 -Bstatic --build-id=sha1 --orphan-handling=warn --strip-debug -o .tmp_vmlinux.kallsyms1 -T ./arch/powerpc/kernel/vmlinux.lds -error-limit=0 --whole-archive vmlinux.o --no-whole-archive
+-+-+-+ getISDThunkSec 1793: type=11 src=c00000000002cc9c osaddr=c000000000008000 off=24bac f0 [][vmlinux.o] secsize=948 secname=__ftr_alt_97
ld.lld: error: InputSection too large for range extension thunk vmlinux.o:(__ftr_alt_97+0xF0)

Which does not make much sense to me:

[fstn1-p1 kernel-llvm-13-lto]$ objdump -D vmlinux.o | grep -B3 2cc9c:

000000000002cc98 <wq_numa_init>:
   2cc98:       00 00 4c 3c     addis   r2,r12,0
   2cc9c:       00 00 42 38     addi    r2,r2,0

The __ftr_alt_97 section is a black magic which either enables chunks of asm depending on CPU features (search for END_FTR_SECTION/etc in the kernel) or fills those with noops.

[fstn1-p1 kernel-llvm-13-lto]$ readelf -S vmlinux.o  | grep -A1  __ftr_alt_97
  [19] __ftr_alt_97      PROGBITS         0000000000000000  00025cb0
       00000000000003b4  0000000000000000   A       0     0     4
  [20] .rela__ftr_alt_97 RELA             0000000000000000  0198e0e8
       0000000000000150  0000000000000018   I      134658    19     8

What does this error signal about?

My kernel with the config is here: aik@a3ba4e8

[aik]

Found some time to dig more. The problem section in the question is:

Relocation section '.rela__ftr_alt_97' at offset 0x197d8e8 contains 14 entries:           
  Offset          Info           Type           Sym. Value    Sym. Name + Addend          
000000000000  2280c0000000a R_PPC64_REL24     0000000000000000 do_page_fault + 0          
000000000008  2280c0000000a R_PPC64_REL24     0000000000000000 do_page_fault + 0          
000000000020  01370000000b R_PPC64_REL14     0000000000000000 .text + 419c                
000000000028  228470000000a R_PPC64_REL24     0000000000000000 unknown_exception + 0      
0000000000f0  01370000000b R_PPC64_REL14     0000000000000000 .text + 1bd38               
0000000000f8  01370000000a R_PPC64_REL24     0000000000000000 .text + 1bd38               
0000000000fc  01370000000a R_PPC64_REL24     0000000000000000 .text + 1bd1c               
00000000020c  2329e0000000a R_PPC64_REL24     000000000001a300 copypage_power7 + 0        
000000000210  2329c0000000a R_PPC64_REL24     0000000000019b54 __copy_tofrom_user_pow + 0 
000000000214  01370000000b R_PPC64_REL14     0000000000000000 .text + 1c94c               
000000000278  2329f0000000a R_PPC64_REL24     000000000001a4c0 memcpy_power7 + 0          
000000000288  01370000000b R_PPC64_REL14     0000000000000000 .text + 22a84               
000000000290  01370000000a R_PPC64_REL24     0000000000000000 .text + 22a10               
0000000002dc  22ea10000000a R_PPC64_REL24     0000000000008f5c isa206_idle_insn_maylo + 0 

lld looks at R_PPC64_REL14 and 1bd38, decides 1bd38 does not fit 16bit and stops even though there is another relocation right next to it which is R_PPC64_REL24. The binutils' ld handles it though, the section looks pretty much the same there. Ideas? Or I am getting it all wrong? @MaskRay ? Thanks,

[aik]

Turns out the problem with LLVM + IAS + LTO and having conditional branches (bc instruction) in the FTR_SECTION_ELSE sections (runtime code patching for various features). Despite the best effort to keep .rela__ftr_alt_97 close to .text, it fails to link. Using unconditional branches instead works, or disabling LTO. LLVM + IAS still link with no problem.

This is a workaround: aik@02ce321

How to proceed now?

[nickdesaulniers]

InputSection too large for range extension thunk

Any idea which section is too large? Perhaps add some logging to LLD?

[aik]

It is not a section, it is specific relocations within the FTR code patching section, the relocation type is shorter than the offset. I am not sure why the message is like this.

If you look at the workaround above (02ce321), one example is:

BEGIN_FTR_SECTION
	nop
FTR_SECTION_ELSE
	bne	.Ldst_unaligned
ALT_FTR_SECTION_END(CPU_FTR_UNALIGNED_LD_STD | CPU_FTR_CP_USE_DCBTZ, \
		    CPU_FTR_UNALIGNED_LD_STD)

The nop goes right to .text as normal but the alternative code for specific feature set - bne .Ldst_unaligned goes to .rela__ftr_alt_97. ld or lld link these and the assumption is that the linker script keeps this alternative section close to .text so it can be linked even if the chunks are quite apart. Which works in many cases but not LTO.

Then in runtime the kernel looks at the current feature set and either keeps the original code or patches it with the content of alt sections - and at this point the offset is going to be really small. But at the link time ld or lld do not know that and I was told that at least ld generates trampolines for these bne, and I assume lld does a similar thing because it works as is, except when I enable LTO - then it breaks.

Makes sense?

[mandlebug]

I was told that at least ld generates trampolines for these bne, and I assume lld does a similar thing because it works as is, except when I enable LTO - then it breaks.

Hi Alexey, LLD also inserts trampolines for the R_PPC64_REL14 relocation type when the offset is too large to fit the instruction, or at least it typically does. Usually we see this error when there is no section close enough to the relocated address to insert the trampoline. Can you add --reproduce to the link step to get me a reproducer to look at? I can try to figure out what's different between the LTO and non-LTO cases.

[nickdesaulniers]

This is a workaround: aik/linux@02ce321

Looking at this with @nemanjai and @mandlebug , other than the commented out assembler code that should be removed, that patch looks like it's probably worthwhile to send upstream. @mandlebug and @nemanjai mentioned that unconditional branches have 26b of range to encode while conditional branches have only 16b.