Flutter engine building

[limbouser]

I can give you my cross compilation script though, link here. I compile in WSL because compiling on-target takes ages. This needs a cloned repo of GNU binutils, LLVM 9 and LLVM 10.

I have been trying to build the flutter engine for a long time now and followed the medium article too. Nothing worked. I would like to know why you have used two versions of LLVM(9 and 10). Since it's said to use TOT Clang/LLVM, I used v11 and always get stuck at building libcxx.

From what I understood from your script, you use LLVM v9.x for setting up the toolchain, ie, Clang/LLVM and then you setup the binutils. You use LLVM v10.x to build the libcxx and libcxxabi. After that comes the setting up the engine development environment and all. Is my understanding of your script right? I use a Ubuntu-16.04 64-bit to build all the packages.

I remember you sayingt that you wouldn't provide any help towards engine building but I've been stuck at this point for a while and I assure you I wouldn't bug you a lot on this. :)

[hhk7734]

Ref: https://docs.loliot.net/books/flutter/page/flutter-engine-for-linux-arm64

[ardera]

TOT Clang/LLVM

Yeah, that's what it says in the article by Chinmay Garde, but it's actually not that accurate.

The flutter team uses tip-of-tree of their own fork of Clang/LLVM, which is not even stable Clang/LLVM 10. If you try to build the engine with clang-10/11, a lot of warnings are issued (these warnings didn't exist in clang-9, they were newly introduced to clang-10/11) and because the engine build is configured to abort on a warning (-Werror is given to clang) the build will fail. That's why I use clang-9.

For libcxx/libcxxabi I just use latest or latest stable clang. Using clang-9 would probably be fine though too.

The instructions posted by @hhk7734 may be helpful too.

I remember you sayingt that you wouldn't provide any help towards engine building but I've been stuck at this point for a while and I assure you I wouldn't bug you a lot on this.

Yeah we'll see 😄

[limbouser]

@hhk7734 Thanks for posting this. Even though this is for a different platform I tried adjusting the flags for ARM platform and here too, I had an error at libcxx building. I have mentioned it in the comments section of your page.

[limbouser]

@ardera Did you have to make any changes to the CMakeFiles.txt files of libcxxabi, like adding a rpath or something? Whichever manual I use, I'm always stuck at the same place.

/home/te_user/flutter-engine/sdk/toolchain/bin/clang++ --sysroot=/home/te_user/flutter-engine/sdk/sysroot CMakeFiles/cmTC_ea0ef.dir/testCXXCompiler.cxx.o -o cmTC_ea0ef && : /home/te_user/flutter-engine/sdk/toolchain/bin/arm-linux-gnueabihf-ld: warning: libm.so.6, needed by /home/te_user/flutter-engine/sdk/sysroot/usr/lib/gcc/arm-linux-gnueabihf/8/libstdc++.so, not found (try using -rpath or -rpath-link)

How do I add an rpath or rpath-link?

[ardera]

I remember having that error too, IIRC the cause was some dangling symlinks in my sysroot

(i.e. some of the symlinks on the Raspberry Pi inside /lib, /usr are absolute, not relative, and thus become dangling when you copy them to your host)

I think I fixed it using the symlinks utility. Execute these on the Raspberry Pi:

sudo apt install symlinks
sudo symlinks -c /lib /usr /opt

You can add the -t flag to the symlinks invocation to dry-run it, i.e. see the changes it would make without applying them.

After that's done, copy the sysroot to your host machine again

EDIT: changed /lib/usr to /usr

[limbouser]

/home/te_user/flutter-engine/sdk/toolchain/bin/clang++ --sysroot=/home/te_user/flutter-engine/sdk/sysroot CMakeFiles/cmTC_ea0ef.dir/testCXXCompiler.cxx.o -o cmTC_ea0ef && : /home/te_user/flutter-engine/sdk/toolchain/bin/arm-linux-gnueabihf-ld: warning: libm.so.6, needed by /home/te_user/flutter-engine/sdk/sysroot/usr/lib/gcc/arm-linux-gnueabihf/8/libstdc++.so, not found (try using -rpath or -rpath-link)

@ardera This error remains even after trying out symlinks utility. 😶
I used the symlinks utility, the rsynced them to my build machine.

@hhk7734 I also tried your method of avoiding the dynamic link issues, ended up having the same error.

[ardera]

try this:

• install the libc6 libc6-dev packages on your pi
• run the symlinks utility again
• rsync the files to your build machine again
• try to build the engine again

if this doesn't fix the issue, try adding the following flags to your cmake invocation:

-DCMAKE_C_FLAGS=-L/home/te_user/flutter-engine/sdk/sysroot/lib/arm-linux-gnueabihf/ -DCMAKE_CXX_FLAGS=-L/home/te_user/flutter-engine/sdk/sysroot/lib/arm-linux-gnueabihf/

[hhk7734]

@limbouser
Modified the documentation while building it on a new board. Would you like to repeat it again with the link below??
https://docs.loliot.net/link/152#bkmrk-sysroot

Mabey it is the same way as @ardera 's comment because build-essential depends on libc6-dev Etc.

[limbouser]

 
 ./flutter/tools/gn \
     --clang \
     --target-sysroot $HOME/flutter-engine/sdk/sysroot-lolit \
     --target-toolchain $HOME/flutter-engine/sdk/toolchain \
     --target-triple arm-linux-gnueabihf  \
     --linux-cpu arm \
     --runtime-mode debug \
     --embedder-for-target \
     --no-lto \
     --target-os linux \
     --arm-float-abi hard

I used the flags I showed above and when I tried ninja as the next step, got the following error. Do you have any experience dealing with this? @hhk7734 @ardera

 ninja -C out/linux_debug_arm/
ninja: Entering directory `out/linux_debug_arm/'
[1/3730] CC obj/third_party/boringssl/src/crypto/asn1/crypto.a_bitstr.o
FAILED: obj/third_party/boringssl/src/crypto/asn1/crypto.a_bitstr.o
/home/te_user/flutter-engine/sdk/toolchain/bin/clang -MD -MF obj/third_party/boringssl/src/crypto/asn1/crypto.a_bitstr.o.d --target=arm-linux-gnueabihf --sysroot /home/te_user/flutter-engine/sdk/sysroot-lolit -DUSE_OPENSSL=1 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_LIBCPP_DISABLE_AVAILABILITY=1 -D_LIBCPP_DISABLE_VISIBILITY_ANNOTATIONS -D_LIBCPP_ENABLE_THREAD_SAFETY_ANNOTATIONS -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -DBORINGSSL_ALLOW_CXX_RUNTIME -DBORINGSSL_IMPLEMENTATION -DBORINGSSL_NO_STATIC_INITIALIZER -DOPENSSL_SMALL -D_XOPEN_SOURCE=700 -I../.. -Igen -I../../third_party/libcxx/include -I../../third_party/libcxxabi/include -I../../third_party/libcxxabi/include -I../../third_party/boringssl/src/include -fno-strict-aliasing -fstack-protector --param=ssp-buffer-size=4 -march=armv7-a -mfloat-abi=hard -mtune=generic-armv7-a -mthumb -fPIC -pipe -pthread -fcolor-diagnostics -mfpu=neon -fvisibility=hidden --sysroot=/home/te_user/flutter-engine/sdk/sysroot-lolit -Wstring-conversion -Wnewline-eof -O2 -fno-ident -fdata-sections -ffunction-sections -g0 -Wendif-labels -Werror -Wno-missing-field-initializers -Wno-unused-parameter -Wno-implicit-int-float-conversion -Wno-c99-designator -Wno-non-c-typedef-for-linkage -Wno-deprecated-copy -Wno-range-loop-construct -std=c11 -c ../../third_party/boringssl/src/crypto/asn1/a_bitstr.c -o obj/third_party/boringssl/src/crypto/asn1/crypto.a_bitstr.o
error: unknown warning option '-Wno-implicit-int-float-conversion'; did you mean '-Wno-implicit-float-conversion'? [-Werror,-Wunknown-warning-option]
error: unknown warning option '-Wno-c99-designator'; did you mean '-Wno-gnu-designator'? [-Werror,-Wunknown-warning-option]
error: unknown warning option '-Wno-non-c-typedef-for-linkage'; did you mean '-Wno-return-type-c-linkage'? [-Werror,-Wunknown-warning-option]
error: unknown warning option '-Wno-deprecated-copy'; did you mean '-Wno-deprecated'? [-Werror,-Wunknown-warning-option]
error: unknown warning option '-Wno-range-loop-construct'; did you mean '-Wno-range-loop-analysis'? [-Werror,-Wunknown-warning-option]
[2/3730] CXX obj/flutter/runtime/test_font.test_font_data.o
FAILED: obj/flutter/runtime/test_font.test_font_data.o
/home/te_user/flutter-engine/sdk/toolchain/bin/clang++ -MD -MF obj/flutter/runtime/test_font.test_font_data.o.d --target=arm-linux-gnueabihf --sysroot /home/te_user/flutter-engine/sdk/sysroot-lolit -DEMBED_TEST_FONT_DATA=1 -DUSE_OPENSSL=1 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D_LIBCPP_DISABLE_AVAILABILITY=1 -D_LIBCPP_DISABLE_VISIBILITY_ANNOTATIONS -D_LIBCPP_ENABLE_THREAD_SAFETY_ANNOTATIONS -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -DSK_GL -DSK_CODEC_DECODES_JPEG -DSK_ENCODE_JPEG -DSK_CODEC_DECODES_PNG -DSK_ENCODE_PNG -DSK_CODEC_DECODES_WEBP -DSK_ENCODE_WEBP -DSK_HAS_WUFFS_LIBRARY -DSK_R32_SHIFT=16 -DSK_DISABLE_REDUCE_OPLIST_SPLITTING -DSK_ENABLE_DUMP_GPU -DSK_DISABLE_AAA -DSK_DISABLE_READBUFFER -DSK_DISABLE_EFFECT_DESERIALIZATION -DSK_DISABLE_LEGACY_SHADERCONTEXT -DSK_DISABLE_LOWP_RASTER_PIPELINE -DSK_FORCE_RASTER_PIPELINE_BLITTER -DSK_GL -I../.. -Igen -I../../third_party/libcxx/include -I../../third_party/libcxxabi/include -I../../third_party/libcxxabi/include -I../.. -I../../third_party/skia -fno-strict-aliasing -fstack-protector --param=ssp-buffer-size=4 -march=armv7-a -mfloat-abi=hard -mtune=generic-armv7-a -mthumb -fPIC -pipe -pthread -fcolor-diagnostics -mfpu=neon -Wall -Wextra -Wendif-labels -Werror -Wno-missing-field-initializers -Wno-unused-parameter -Wno-implicit-int-float-conversion -Wno-c99-designator -Wno-non-c-typedef-for-linkage -Wno-deprecated-copy -Wno-range-loop-construct -fvisibility=hidden --sysroot=/home/te_user/flutter-engine/sdk/sysroot-lolit -Wstring-conversion -Wnewline-eof -O2 -fno-ident -fdata-sections -ffunction-sections -g0 -fvisibility-inlines-hidden -std=c++17 -fno-rtti -nostdinc++ -nostdinc++ -fno-exceptions -c ../../flutter/runtime/test_font_data.cc -o obj/flutter/runtime/test_font.test_font_data.o
error: unknown warning option '-Wno-implicit-int-float-conversion'; did you mean '-Wno-implicit-float-conversion'? [-Werror,-Wunknown-warning-option]
error: unknown warning option '-Wno-c99-designator'; did you mean '-Wno-gnu-designator'? [-Werror,-Wunknown-warning-option]
error: unknown warning option '-Wno-non-c-typedef-for-linkage'; did you mean '-Wno-return-type-c-linkage'? [-Werror,-Wunknown-warning-option]
error: unknown warning option '-Wno-deprecated-copy'; did you mean '-Wno-deprecated'? [-Werror,-Wunknown-warning-option]
error: unknown warning option '-Wno-range-loop-construct'; did you mean '-Wno-range-loop-analysis'? [-Werror,-Wunknown-warning-option]
[10/3730] ASM obj/third_party/boringssl/linux-arm/crypto/fipsmodule/boringssl_asm.bsaes-armv7.o
ninja: build stopped: subcommand failed.

[ardera]

This means that your LLVM/Clang is too old for your engine sources / your engine sources are too new for your LLVM/Clang.

This is because, when you follow the steps by @hhk7734 or Chinmay Garde's blogpost, you're setting up a development environment for the flutter engine, so you're basically downloading the most recent sources in existance.

This is the correct way to do it if you want to build flutter master, and it seems like you'll have to use LLVM/Clang-10 instead if you really want to do it this way.

If you instead want to build flutter stable, you'll need some older sources. (which will compile fine with LLVM/Clang-9) Go into your engine directory and run:

gclient sync --revision=https://github.com/flutter/engine.git@<engine commit hash here>

where <engine commit hash here> is the most recent stable engine commit, which you can get from this file in the flutter SDK repo.

If you encounter issues while running the gclient sync --revision=... command, just delete your engine folder, follow the steps by @hhk7734 to initialize it again, and then, instead of running the initial gclient sync without arguments as it says in the instructions, you run gclient sync --revision=... as written above. (gclient sync doesn't seem to work that well for downgrading the sources)

[limbouser]

Thanks @ardera. I'll check. So are the engine-binaries you use based upon this stable branch?

[ardera]

So are the engine-binaries you use based upon this stable branch?

yep

[hhk7734]

https://github.com/flutter/engine/commits/master/shell/platform/linux/config/BUILD.gn
Added host-sidetarget-side compilation dependency

[ardera]

@hhk7734 Yep, just had some issues with that when I was building the engine.

In case someone finds this useful: You'll probably encounter some errors while running the gn ... command for stable engine version 1.17. This is the because it uses pkg-config to locate the gtk+ headers. Since Raspbian stores its pkg-config files in a somewhat non-standard location, you need to apply some hacks.

On your host, after every time you sync the sysroot, delete the /usr/lib/pkgconfig directory (it's empty) and create a symlink /usr/lib/pkgconfig ==> /usr/lib/arm-linux-gnueabihf/pkgconfig.

Should look something like the following in practice:

rsync ... ~/my_pi_sysroot
rm -r ~/my_pi_sysroot/usr/lib/pkgconfig
ln --symbolic --relative ~/my_pi_sysroot/usr/lib/arm-linux-gnueabihf/pkgconfig ~/my_pi_sysroot/usr/lib/pkgconfig

There is an option for setting the path in which the flutter's pkg-config will look for the package config files, but it's not that easy to set. To be more precise, it must be specified in the args.gn file, which means you can't use flutter/tools/gn tool for easy build configuration. You'd have to manually configure the engine yourself.

[hhk7734]

engine/src/build/config/linux/pkg-config.py

def SetConfigPath(options):
  """Set the PKG_CONFIG_LIBDIR environment variable.

  This takes into account any sysroot and architecture specification from the
  options on the given command line.
  """

  sysroot = options.sysroot
  assert sysroot

  # Compute the library path name based on the architecture.
  arch = options.arch
  if sysroot and not arch:
    print("You must specify an architecture via -a if using a sysroot.")
    sys.exit(1)

  libdir = sysroot + '/usr/' + options.system_libdir + '/pkgconfig'
  libdir += ':' + sysroot + '/usr/share/pkgconfig'
  os.environ['PKG_CONFIG_LIBDIR'] = libdir
  return libdir

Maybe, I fixed the code above when the error occurred.

PS. https://youtu.be/p6bzmdAJqjo
Thanks to flutter-pi, I was able to shoot this video. 👍