Before September 2020 FreeBSD could only be built on a FreeBSD host. Alexander Richardson did a lot of work making this possible: https://wiki.freebsd.org/BuildingOnNonFreeBSD.

Get prebuilt Clang and LLD

I use prebuilt Clang and LLD from Chromium: https://chromium.googlesource.com/chromium/src/tools/clang/+/refs/heads/main/scripts/update.py. My output directory is ~/Stable.

Build compile_commands.json

I tried Bear but it could not handle such a complex build system. The process got stuck at some step, so I just gave up on it.

bmake meta mode

I learned a bit about bmake (default make on FreeBSD) and noticed a nice feature: meta mode. In meta mode bmake records rule commands into .meta files. For the next build, bmake will consult .meta files to evaluate whether the target has become out-of-date. This is more robust than just comparing file modification times. Build Systems à la Carte says such a build system is self-tracking.

For FreeBSD src, we can enable meta mode with -DWITH_META_MODE. After buildworld, we can parse these .meta files under objdir and build compile_commands.json.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

MAKEOBJDIRPREFIX=$PWD/obj/meta ./tools/build/make.py --cross-bindir=~/Stable/bin -j 20 buildworld TARGET=amd64 TARGET_ARCH=amd64 -DWITH_META_MODE -DNO_FILEMON -DWITHOUT_WERROR
(
cd obj/meta
ruby -r find -r shellwords -r stringio -e '
  cdb = StringIO.new
  cdb << "["
  Find.find "home/ray/Dev/freebsd/amd64.amd64" do |path|
    next unless FileTest.file?(path) && path.end_with?(".meta")
    args = nil
    File.readlines(path).each do |line|
      line = line.chomp
      if line =~ /^CMD.*\/clang/; args=Shellwords::split(line[4..])
      elsif line =~ /^OODATE /; file=line[7..]
        if file =~ /\.(c|cc|cpp)$/ && args
          cdb << "," if cdb.pos > 1
          cdb << %{{"directory":#{Dir.pwd.dump}, "arguments":#{args}, "file":#{file.dump}}\n}
        end
      end
    end
  end
  cdb << "]"
  print cdb.string' > compile_commands.json
)

Some notes about the other -D variables.

bmake has code dealing with filemon, which is a FreeBSD driver. On Linux we need to disable it with -DNO_FILEMON.

FreeBSD src enables -Werror by default. Our Clang is new and may have many new diagnostics. We can use -DWITHOUT_WERROR.

As of 2021-08, building on Linux still has some issues. I mostly read libexec/rtld-elf and the build process can proceed beyond libexec/rtld-elf, so I am satisfied.

ccls

With compile_commands.json, my ccls can index the repository.

Here is a screenshot browsing libexec/rtld-elf code in Emacs with (lsp-mode + emacs-ccls).

1
2
3

(setq ccls-sem-highlight-method 'font-lock)
(add-hook 'lsp-after-open-hook #'ccls-code-lens-mode)
(ccls-use-default-rainbow-sem-highlight)

Contribute to libexec/rtld-elf

I stumbled upon FreeBSD libexec/rtld-elf in 2019 to sort out how LLD should set the p_memsz field of PT_GNU_RELRO. I noticed an issue but did not get a chance to create a patch on https://reviews.freebsd.org/.

When working on some TLS issues in LLD, I noticed that rtld did not handle p_vaddr % p_align != 0 correctly. (Note: fixed for i386 and amd64.)

In 2020 I noticed a symbol resolution issue related to STB_WEAK, but did not follow up with the patch. (Note: introduced the environment variable LD_DYNAMIC_WEAK=0 to match ELF spec (glibc/musl behavior).)

Now that I have a proper setup, I can work on the aforementioned problems in a virtual machine running FreeBSD 12.2. qemu-system-x86_64 -enable-kvm -m 16384 -smp 16 -drive file=~/Images/freebsd.qcow2,if=virtio -net nic,model=virtio -net user,hostfwd=tcp::2223-:22

1
2
3
4
5

% cat /etc/src.conf
WITHOUT_CLANG=yes
WITHOUT_LLD=yes
WITHOUT_LLDB=yes
WITHOUT_CLEAN=yes

1
2
3
4
5
6
7

MAKEOBJDIRPREFIX=$PWD/obj/default make -j 12 buildworld


MAKEOBJDIRPREFIX=$PWD/obj/default make -j 12 buildenv
export CPUTYPE=  
make -C libexec/rtld-elf

The versions of rtld and libc should match if you across major versions. Simple programs may work even if you don't use a libc of the matching version.

1
2
3
4
5
6
7
8
9
10

mkdir -p /tmp/opt/lib
echo 'GROUP ( libc/libc.so.7 libc_nonshared/libc_nonshared.a libssp_nonshared/libssp_nonshared.a )' > /tmp/opt/lib/libc.so

objdir=$HOME/freebsd/obj/default/usr/home/ray/freebsd/amd64.amd64


clang a.c -Wl,--dynamic-linker=$objdir/libexec/rtld-elf/ld-elf.so.1.full -L/tmp/opt/lib -L$objdir/lib -Wl,-rpath=$objdir/lib/libc -Wl,-t


cgdb --args $objdir/libexec/rtld-elf/ld-elf.so.1.full ./a

Thanks to kib who reviewed these patches and lwhsu who added me to this contributor list: https://docs.freebsd.org/en/articles/contributors/#contrib-additional.