When can glibc be built with Clang?

In September, I wrote "So, dear glibc, will you be happy with my sending Clang patches?" in Build glibc with LLD 13. We have come to a turning point.

In Linux Plumbers Conference 2021, at the glibc Birds of a Feather session, I asked the Clang buildability question to the glibc stewards. (Interlude: I did not realize that I should attend the conference (it was a great opportunity from an outlier to meet some glibc folks). In Tuesday, Wei Wu (lazyparser) kindly gave me his account: "想去参加LPC么？我会议太多了今天参加不过来". I happily accepted it and typed the question during the glibc session.)

So I got positive responses. "Carlos: Yes, we could be happy with clang buildability." "Joseph: Patches should be split into logical changes." This is really great news! My unnesting patch had sat there for a while and I was unsure about the Clang buildability interest.

Of course, every change needs to be assessed where they belong to, LLVM/Clang or glibc.

My interest in Clang/LLD buildability did not arise suddenly. It has been multiple years. I made my first patch improving LLD's glibc buildability in June 2018. Probably since June 2019, I pushed first LLVM fix mproving glibc buildability [MC][ELF] Don't create relocations with section symbols for STB_LOCAL ifunc. I have sporadic contribution since then. Today I think there is not too much to improve on LLVM/Clang's side, but many need glibc adapatation.

As a rough estimate, we may need 30 patches to fix glibc build on x86-64 for the --disable-werror configuration. Fixing other popular architectures or all tests will take more. Adrian Ratiu mentioned that shebs/clangify exists in A tale of two toolchains and glibc. The branch in the 2018 era has about 30 commits.

Below I will introduce the main challenges (GCC nested functions, asm label after first use, Clang's less permissive warnings) and a selective set of miscellaneous changes.

GCC nested functions

glibc's dynamic loader (often called ld.so) implementation (elf/ plus arch-specific code in sysdeps/*/) made extensive use of a feature called "nested functions". https://gcc.gnu.org/onlinedocs/gcc/Nested-Functions.html says

A nested function is a function defined inside another function. Nested functions are supported as an extension in GNU C, but are not supported by GNU C++.

This is a no-go for Clang which does not support this feature (error: function definition is not allowed here). The dynamic loader usage actually impacted readability and debuggability, so people agreed that the usage should simply be removed.

As a worst example, the dynamic loader had a function like:

#ifndef RESOLVE_MAP
static
#else
auto
#endif
inline void __attribute__ ((unused, always_inline))
elf_get_dynamic_info (struct link_map *l)
{

Its file could be included twice, one with RESOLVE_MAP defined and one without. It was difficult to understand the code intention.

As of 2021-10-07, the usage in the dynamic loader has been removed by my elf: Avoid nested functions in the loader [BZ #27220]. Adhemerval Zanella fixed some fallout.

The glibc dynamic loader code is extremely tricky. I think not taking inspiration from NetBSD code in 1992 was unfortunate.

Remaining usage include:

nss/makedb.c:add_key (fixed by nss: Unnest nested function add_key)
posix/regcomp.c:{seek_collating_symbol_entry,...} (fixed by regex: Unnest nested functions in regcomp.c)

asm label after first use

In GCC and Clang, an asm label can change the symbol name for a definition or a non-definition declaration. glibc makes heavy use of this feature even in public headers. In some places it is definitely a code smell problem. Function attributes are added to one function in multiple places while adding them when the first declaration would be better.

#pragma redefine_extname is a similar feature which can change the symbol name. In Clang, #pragma redefine_extname is converted to an asm label and shares code with the asm label implementation.

Clang before 12.0.0 had an issue that a built-in function ignored the asm label. I fixed it and made some notes on https://maskray.me/blog/2020-10-15-intra-call-and-libc-symbol-renaming (TODO: translate it into English).

typedef unsigned long size_t;

#ifdef ASM_LABEL
extern void *memcpy(void *, const void *, size_t) asm("__GI_memcpy");
#else
#pragma redefine_extname memcpy __GI_memcpy 
extern void *memcpy(void *, const void *, size_t);
#endif

void *test_memcpy(void *dst, const void *src, size_t n) {
  
  
  return memcpy(dst, src, n);
}

Unfortunately, there is a limitation. If a #pragma redefine_extname is declared after the first use of the symbol, the rename operation will be silently ignored.


typedef unsigned long size_t;

extern void *memcpy(void *, const void *, size_t);

void *test_memcpy(void *dst, const void *src, size_t n) {
  return memcpy(dst, src, n);
}

#ifdef ASM_LABEL
extern void *memcpy(void *, const void *, size_t) asm("__GI_memcpy");
#else
#pragma redefine_extname memcpy __GI_memcpy 
#endif

If an asm label applies to a redeclaration after the first use, Clang will kindly give an error to notify that it cannot handle the situation:

% clang a.c -S -o - | grep callq
        callq   [email protected]
% clang a.c -S -o - -DASM_LABEL
a.c:18:14: error: cannot apply asm label to function after its first use
extern void *memcpy(void *, const void *, size_t) asm("__GI_memcpy");
             ^                                        ~~~~~~~~~~~~~
1 error generated.

glibc makes extensive usage of hidden_proto and libc_hidden_proto, where such problems may occur in many places.

There are cases where a .h file has an extern inline definition, followed by references from other inline functions. A .c file adds asm label to the declaration. Disabling __USE_EXTERN_INLINES can work around these issues, but many issues are due to other code organization difficulties.

Aliasing an asm label

nptl/pthread_create.c has


td_thr_events_t __nptl_threads_events;
libc_hidden_proto (__nptl_threads_events)
libc_hidden_data_def (__nptl_threads_events)

which desugars to

td_thr_events_t __nptl_threads_events;
extern __typeof (__nptl_threads_events) __nptl_threads_events __asm__ ("" "__GI___nptl_threads_events") __attribute__ ((visibility ("hidden")));
extern __typeof (__nptl_threads_events) __EI___nptl_threads_events __asm__("" "__nptl_threads_events"); 
extern __typeof (__nptl_threads_events) __EI___nptl_threads_events __attribute__((alias ("" "__GI___nptl_threads_events"))) __attribute__ ((__copy__ (__nptl_threads_events)));

GCC produces a hidden visibility __GI___nptl_threads_events and its default visibility alias __nptl_threads_events.

Clang reports an error:

1
2
3

pthread_create.c:49:1: error: alias must point to a defined variable or function
libc_hidden_data_def (__nptl_threads_events)
^

GCC knows that __GI___nptl_threads_events (not a C declaration) is a definition, so __attribute__((alias ("" "__GI___nptl_threads_events"))) can be used, but Clang does not know this fact.

`_Float128`

long double and float128 support in GCC is a huge mess.

In its PowerPC port, long double has 3 mangling schemes:

-mlong-double-64: e
-mlong-double-128 -mabi=ibmlongdouble: g
-mlong-double-128 -mabi=ieeelongdouble: u9__ieee128 (gcc <= 8.1: U10__float128)

In its x86 port, two different long double configurations share the same mangling code.

-mlong-double-64: e
-mlong-double-80: e
-mlong-double-128: g

(I fixed some Clang issues in https://reviews.llvm.org/D64276.)

__float128 has the same mangling code with 128-bit long double but is considered a distinct type in both C and C++. So if you have two overloads foo(__float128) and foo(long double), you will get a compiler/linker error due to conflicting symbols.

_Float128 is identical to __float128. _Float128 is available in C mode but not in C++ mode. When a future standard C++ introduces a float128 type, it has to be yet another new type to be different from the 128-bit long double.


#include <stdio.h>

int main() {
  
  puts(_Generic((_Float128)0, __float128: "same", default: "different"));
  
  puts(_Generic((_Float128)0, long double: "same", default: "different"));
}

Anyhow, glibc builds some float128 code on some PowerPC, x86, and perhaps a few other architectures. Clang doesn't support _Float128. I have a pending patch https://reviews.llvm.org/D111382 but given GCC's C++ issue I am now unsure whether Clang should support _Float128 before the C++ support case is clearer.

For now, falling back to typedef __float128 _Float128 is probably a good choice.



@@ -29,5 +29,6 @@
 #if (defined __x86_64__                                                        \
      ? __GNUC_PREREQ (4, 3)                                            \
-     : (defined __GNU__ ? __GNUC_PREREQ (4, 5) : __GNUC_PREREQ (4, 4)))
+     : (defined __GNU__ ? __GNUC_PREREQ (4, 5) : __GNUC_PREREQ (4, 4))) \
+    || defined __clang__
 # define __HAVE_FLOAT128 1
 #else
@@ -88,5 +89,5 @@ typedef __float128 _Float128;
 
 /* __builtin_huge_valf128 doesn't exist before GCC 7.0.  */
-#  if !__GNUC_PREREQ (7, 0)
+#  if !__GNUC_PREREQ (7, 0) && !defined __clang__
 #   define __builtin_huge_valf128() ((_Float128) __builtin_huge_val ())
 #  endif
@@ -97,5 +98,5 @@ typedef __float128 _Float128;
    attempts to use _Float128 sNaNs will not work properly with older
    compilers.  */
-#  if !__GNUC_PREREQ (7, 0)
+#  if !__GNUC_PREREQ (7, 0) && !defined __clang__
 #   define __builtin_copysignf128 __builtin_copysignq
 #   define __builtin_fabsf128 __builtin_fabsq
@@ -109,5 +110,5 @@ typedef __float128 _Float128;
    been a __builtin_signbitf128 in GCC and the type-generic builtin is
    only available since GCC 6.  */
-#  if !__GNUC_PREREQ (6, 0)
+#  if !__GNUC_PREREQ (6, 0) && !defined __clang__
 #   define __builtin_signbitf128 __signbitf128
 #  endif

`-fheinous-gnu-extensions`

Chris Lattner added the funny option in commit cda4d7e19619e136f2b597110b6a52ee9d641862. It has an associated comment for CheckAsmLValue:

1
2
3

GNU C has an extremely ugly extension whereby they silently ignore "noop" casts in places where an lvalue is required by an inline asm.
We emulate this behavior when -fheinous-gnu-extensions is specified, but provide a strong guidance to not use it.
This method checks to see if the argument is an acceptable l-value and returns false if it is a case we can handle.

This does look like wrong usage but also has a niche type checking feature:


void foo() {
  int i;
  long l;
  __asm ("" : "=r" ((unsigned)i));
  __asm ("" : "=r" ((long)l));
  __asm ("" : "=r" ((long long)l));
  __asm ("" : "=r" ((int)l)); 
  __asm ("" : "=r" ((long)i)); 
}


void foo() {
  int i;
  long l;
  __asm ("" : "=r" ((unsigned)i)); 
  __asm ("" : "=r" ((long)l)); 
  __asm ("" : "=r" ((long long)l)); 
  __asm ("" : "=r" ((int)l)); 
  __asm ("" : "=r" ((long)i)); 
}

I attempted a fix [PATCH] include/longlong.h: Remove incorrect lvalue to rvalue conversion from asm output constraints. If GCC/glibc folks don't accept the loss of type checking, we probably have to carry around -fheinous-gnu-extensions.



@@ -493,6 +493,6 @@ extern UDItype __umulsidi3 (USItype, USItype);
 #define add_ssaaaa(sh, sl, ah, al, bh, bl) \
   __asm__ ("add{q} {%5,%1|%1,%5}\n\tadc{q} {%3,%0|%0,%3}"              \
-          : "=r" ((UDItype) (sh)),                                     \
-            "=&r" ((UDItype) (sl))                                     \
+          : "=r" (sh),                                                 \
+            "=&r" (sl)                                                 \
           : "%0" ((UDItype) (ah)),                                     \
             "rme" ((UDItype) (bh)),                                    \

LLVM's integrated assembler

On most targets, Clang uses LLVM's integrated assembler. It lacks support for some unpopular features and may be stricter than GNU as in some cases.

% cat a.s
bndmov %bnd0,(%rsp)
bnd
jmp *%r11
% as a.s
% clang -c a.s
a.s:2:1: error: invalid instruction mnemonic 'bnd'
bnd
^~~

I attempted to improve sysdeps/x86_64/configure.ac but was asked to just remove the deprecated Intel MPX support. So I sent [PATCH] elf: Remove Intel MPX support (lazy PLT, ld.so profile, and LD_AUDIT).

As another example, sysdeps/powerpc/powerpc64/le/power10/memmove.S uses the directive .machine power9. For some other x86 projects I noticed that GNU as' .arch supports a very wide range of .arch operands.

In LLVM, *AsmParser files do not do a good job tracking what features are needed to assemble instructions. It is also tricky to recognize all GNU as supported operands. The ISA feature compatibility check, while looks nice, actually provides a lower value.

So I patched LLVM's x86 and PowerPC targets to ignore .arch and .machine operands.

A few sysdeps/i386/fpu/*.S files use .tfloat (80-bit extended precision floating point) which is unsupported by LLVM's integrated assembler.
sysdeps/x86_64/fpu/multiarch/Makefile uses -msse2avx to encode SSE instructions with VEX prefix. LLVM's integrated assembler doesn't support the option.

Inline asm parsing

While GCC does minimal parsing of an inline asm statement (https://gcc.gnu.org/onlinedocs/gcc/Size-of-an-asm.html), Clang parses everything.

glibc uses an inline asm trick to get the constant value of a macro (e.g. a Linux syscall number). @@@ is invalid asm, so Clang rejects it.

% cat a.c
#include <linux/version.h>
#include <sys/syscall.h>
#include <bits/syscall.h>

int main() {
  asm ("@@@%[email protected]@@" :: "i"((long int)(X)));
}
% gcc -S -DX=__NR_mmap a.c -o - | grep @@@
        @@@[email protected]@@
% clang -S -DX=__NR_mmap a.c -o /dev/null
a.c:6:8: error: unexpected token at start of statement
  asm ("@@@%[email protected]@@" :: "i"((long int)(X)));
       ^
<inline asm>:1:2: note: instantiated into assembly here
        @@@[email protected]@@
        ^
1 error generated.

The fix is simple. We can just surround the invalid part with a pair of /* */. I avoid # because it may have weird interpretation on some exotic architecture (https://sourceware.org/binutils/docs/as/Comments.html#Comments).

More rigid semantic analysis and finicky warnings

`_Static_assert`

The first operand of a _Static_assert declaration must be an integer constant expression. C11 defines operands which are required to make up an integer constant expression.

6 An integer constant expression shall have integer type and shall only have operands that are integer constants, enumeration constants, character constants, sizeof expressions whose results are integer constants, _Alignof expressions, and floating constants that are the immediate operands of casts. Cast operators in an integer constant expression shall only convert arithmetic types to integer types, except as part of an operand to the sizeof or _Alignof operator.

10 An implementation may accept other forms of constant expressions.

An implementation may or may not accept a const object (e.g. const size_t allocation_size = 32768;) as an operand. Clang is simple: a const object cannot be used as an operand in a constant expression. GCC is inconsistent (PR102502):

-O2: accepted
-O2 -Wpedantic: warning: expression in static assertion is not an integer constant expression [-Wpedantic]
-O0: error: expression in static assertion is not constant

% cat reduce.i
const int __alloc_dir_allocation_size = 8;
void __alloc_dir() { _Static_assert(__alloc_dir_allocation_size, ""); }

% gcc reduce.i -c -std=c11
reduce.i: In function ‘__alloc_dir’:
reduce.i:2:37: error: expression in static assertion is not constant
    2 | void __alloc_dir() { _Static_assert(__alloc_dir_allocation_size, ""); }
      |                                     ^~~~~~~~~~~~~~~~~~~~~~~~~~~
% gcc reduce.i -c -std=c11 -O1
% gcc reduce.i -c -std=c11 -O2
% gcc reduce.i -c -std=c11 -O2 -Wpedantic
reduce.i: In function ‘__alloc_dir’:
reduce.i:2:37: warning: expression in static assertion is not an integer constant expression [-Wpedantic]
    2 | void __alloc_dir() { _Static_assert(__alloc_dir_allocation_size, ""); }
      |                                     ^~~~~~~~~~~~~~~~~~~~~~~~~~~

% clang reduce.i -c -std=c11
reduce.i:2:37: error: static_assert expression is not an integral constant expression
void __alloc_dir() { _Static_assert(__alloc_dir_allocation_size, ""); }
                                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.

sysdeps/unix/sysv/linux/opendir.c has such an issue, which has been fixed by linux: Fix a possibly non-constant expression in _Static_assert.

Miscellaneous

misc/sys/cdefs.h has

1 2	# define __va_arg_pack() __builtin_va_arg_pack () # define __va_arg_pack_len() __builtin_va_arg_pack_len ()

Clang does not support __builtin_va_arg_pack.