Cross-Compiling =============== FRR is capable of being cross-compiled to a number of different architectures. With an adequate toolchain this process is fairly straightforward, though one must exercise caution to validate this toolchain's correctness before attempting to compile FRR or its dependencies; small oversights in the construction of the build tools may lead to problems which quickly become difficult to diagnose. Toolchain Preliminary --------------------- The first step to cross-compiling any program is to identify the system which the program (FRR) will run on. From here on this will be called the "host" machine, following autotools' convention, while the machine building FRR will be called the "build" machine. The toolchain will of course be installed onto the build machine and be leveraged to build FRR for the host machine to run. .. note:: The build machine used while writing this guide was ``x86_64-pc-linux-gnu`` and the target machine was ``arm-linux-gnueabihf`` (a Raspberry Pi 3B+). Replace this with your targeted tuple below if you plan on running the commands from this guide: .. code-block:: shell export HOST_ARCH="arm-linux-gnueabihf" For your given target, the build system's OS may have some support for building cross compilers natively, or may even offer binary toolchains built upstream for the target architecture. Check your package manager or OS documentation before committing to building a toolchain from scratch. This guide will not detail *how* to build a cross-compiling toolchain but will instead assume one already exists and is installed on the build system. The methods for building the toolchain itself may differ between operating systems so consult the OS documentation for any particulars regarding cross-compilers. The OSDev wiki has a `pleasant tutorial`_ on cross-compiling in the context of operating system development which bootstraps from only the native GCC and binutils on the build machine. This may be useful if the build machine's OS does not offer existing tools to build a cross-compiler targeting the host. .. _pleasant tutorial: https://wiki.osdev.org/GCC_Cross-Compiler This guide will also not demonstrate how to build all of FRR's dependencies for the target architecture. Instead, general instructions for using a cross-compiling toolchain to compile packages using CMake, Autotools, and Makefiles are provided; these three cases apply to almost all FRR dependencies. .. _glibc mismatch: .. warning:: Ensure the versions and implementations of the C standard library (glibc or what have you) match on the host and the build toolchain. ``ldd --version`` will help you here. Upgrade one or the other if the they do not match. Testing the Toolchain --------------------- Before any cross-compilation begins it would be prudent to test the new toolchain by writing, compiling and linking a simple program. .. code-block:: shell # A small program cat > nothing.c <`` so ``-latomic`` cannot be assumed. Cross-compiling FRR Itself -------------------------- With all the necessary libraries cross-compiled and installed into the sysroot, the last thing to actually build is FRR itself: .. code-block:: shell # Clone and bootstrap the build git clone 'https://github.com/FRRouting/frr.git' # (e.g.) git checkout stable/7.5 ./bootstrap.sh # Build clippy using the native toolchain mkdir build-clippy cd build-clippy ../configure --enable-clippy-only make clippy-only cd .. # Next, configure FRR and use the clippy we just built ./configure \ CC=${HOST_ARCH}-gcc \ CXX=${HOST_ARCH}-g++ \ --host=${HOST_ARCH} \ --with-sysroot=/usr/${HOST_ARCH} \ --with-clippy=./build-clippy/lib/clippy \ --sysconfdir=/etc/frr \ --sbindir="\${prefix}/lib/frr" \ --localstatedir=/var/run/frr \ --prefix=/usr \ --enable-user=frr \ --enable-group=frr \ --enable-vty-group=frrvty \ --disable-doc \ --enable-grpc # Send it make Installation to Host Machine ---------------------------- If no errors were observed during the previous steps it is safe to ``make install`` FRR into its own directory. .. code-block:: shell # Install FRR its own "sysroot" make install DESTDIR=/some/path/to/sysroot After running the above command, FRR binaries, modules and example configuration files will be installed into some path on the build machine. The directory will have folders like ``/usr`` and ``/etc``; this "root" should now be copied to the host and installed on top of the root directory there. .. code-block:: shell # Tar this sysroot (preserving permissions) tar -C /some/path/to/sysroot -cpvf frr-${HOST_ARCH}.tar . # Transfer tar file to host machine scp frr-${HOST_ARCH}.tar me@host-machine: # Overlay the tarred sysroot on top of the host machine's root ssh me@host-machine <<-EOF # May need to elevate permissions here tar -C / -xpvf frr-${HOST_ARCH}.tar.gz . EOF Now FRR should be installed just as if ``make install`` had been run on the host machine. Create configuration files and assign permissions as needed. Lastly, ensure the correct users and groups exist for FRR on the host machine. Troubleshooting --------------- Even when every precaution has been taken some things may still go wrong! This section details some common runtime problems. Mismatched Libraries ^^^^^^^^^^^^^^^^^^^^ If you see something like this after installing on the host: .. code-block:: console /usr/lib/frr/zebra: error while loading shared libraries: libyang.so.1: cannot open shared object file: No such file or directory ... at least one of FRR's dependencies which was linked to the binary earlier is not available on the host OS. Even if it has been installed the host repository's version may lag what is needed for more recent FRR builds (this is likely to happen with YANG at the moment). If the matching library is not available from the host OS package manager it may be possible to compile them using the same toolchain used to compile FRR. The library may have already been built earlier when compiling FRR on the build machine, in which case it may be as simple as following the same workflow laid out during the `Installation to Host Machine`_. Mismatched Glibc Versions ^^^^^^^^^^^^^^^^^^^^^^^^^ The version and implementation of the C standard library must match on both the host and build toolchain. The error corresponding to this misconfiguration will look like: .. code-block:: console /usr/lib/frr/zebra: /lib/${HOST_ARCH}/libc.so.6: version `GLIBC_2.32' not found (required by /usr/lib/libfrr.so.0) See the earlier warning about preventing a `glibc mismatch`_.