patching libc

because it's easier than compiling rust

ive been keeping an eye on the servo browser for some time now and recently decided to give it a try

it's written in rust which i am too lazy to compile, fortunately they have prebuilt binaries on the downloads page

run servo and...

% ./servo 
mozilla::detail::MutexImpl::MutexImpl: pthread_mutexattr_settype failed: Invalid argument
Segmentation fault

great. what's going on here?

for those of you following along at home...

you may be wondering how to even launch the damn thing

sh: ./servo: not found

that's because the prebuilt servo binary is built for glibc, whose loader is named differently than musl libc

hence your shell is complaining that it can't find the loader

we could patch(elf) the binary to use musl's loader...

% cp servo servo-musl
% patchelf servo-musl --set-interpreter /lib/ld-musl-x86_64.so.1 --remove-needed ld-linux-x86-64.so.2
% ./servo-musl
Error relocating ./servo-musl: __res_init: symbol not found
...

but it turns out the loader isn't the only difference!

internally, glibc has a lot of weird non-posix symbols, and that leaks into the programs that are compiled against it

most are just aliases to existing functions (e.g. __res_init is an alias to res_init) so translating them isn't hard...

...
Error relocating ./servo-musl: __snprintf_chk: symbol not found
Error relocating ./servo-musl: __fprintf_chk: symbol not found
Error relocating ./servo-musl: __strcpy_chk: symbol not found
Error relocating ./servo-musl: __memcpy_chk: symbol not found
Error relocating ./servo-musl: __memset_chk: symbol not found
Error relocating ./servo-musl: __vfprintf_chk: symbol not found
Error relocating ./servo-musl: __realpath_chk: symbol not found
Error relocating ./servo-musl: __memmove_chk: symbol not found
Error relocating ./servo-musl: __register_atfork: symbol not found
Error relocating ./servo-musl: mallinfo: symbol not found
Error relocating ./servo-musl: gnu_get_libc_version: symbol not found
Error relocating ./servo-musl: fcntl64: symbol not found
Error relocating ./servo-musl: __libc_single_threaded: symbol not found

it's just tedious.

fortunately this is a well-established problem and there are good folks out there working on glibc compatibility layers (say that again)

one such solution is gcompat, it's essentially a collection of wrappers to translate glibc's __res_inits into musl's res_inits

install it and:

% ./servo
Error relocating /tmp/1000-runtime-dir/.cache/servo/servo: __res_init: symbol not found
Error relocating /tmp/1000-runtime-dir/.cache/servo/servo: fcntl64: symbol not found

right, alpine's gcompat is currently still at 1.1.0

you might have to clone and build it yourself, take this patch:

diff --git a/libgcompat/resolv.c b/libgcompat/resolv.c
index 0b81d0c..9305911 100644
--- a/libgcompat/resolv.c
+++ b/libgcompat/resolv.c
@@ -12,6 +12,11 @@
 
 #include "alias.h" /* weak_alias */
 
+int __res_init(void)
+{
+	return res_init();
+}
+
 int __res_ninit(res_state statp)
 {
    int rc;

and that should bring you up to speed!

okay time to do some critical thinking

analysing the error

let's start with the error message:

mozilla::detail::MutexImpl::MutexImpl: pthread_mutexattr_settype failed: Invalid argument

key points:

  • pthreads mutex impl
  • pthread_mutexattr_settype
  • mutex attr type
  • invalid argument

conclusion: it's calling pthread_mutexattr_settype with an invalid attribute type

smells like a glibc/musl discrepancy...

let's compare pthreads mutex attributes!

those should be defined in pthread.h

musl has:

  1. PTHREAD_MUTEX_NORMAL (aka PTHREAD_MUTEX_DEFAULT)
  2. PTHREAD_MUTEX_RECURSIVE
  3. PTHREAD_MUTEX_ERRORCHECK

glibc has:

  1. PTHREAD_MUTEX_NORMAL (aka PTHREAD_MUTEX_DEFAULT)
  2. PTHREAD_MUTEX_RECURSIVE
  3. PTHREAD_MUTEX_ERRORCHECK
  4. PTHREAD_MUTEX_ADAPTIVE_NP (!!)

aha!

this PTHREAD_MUTEX_ADAPTIVE_NP looks hella sus

fun fact: NP stands for non-portable

let's see what would happen if we pass it into musl's pthread_mutexattr_settype:

int pthread_mutexattr_settype(pthread_mutexattr_t *a, int type)
{
    // if type = PTHREAD_MUTEX_ADAPTIVE_NP = 3...
    if ((unsigned)type > 2) return EINVAL;  // oh.
    a->__attr = (a->__attr & ~3) | type;
    return 0;
}

we get a EINVAL!

which aligns nicely with the Invalid argument error message :D

so, that's cool and all, but...

how fix?

the upstreamable solution

ideally we would implement the "adaptive mutex" behaviour in gcompat

unfortunately i am

  1. a lazy bum
  2. who would really like to just run servo and not have to learn about mutexes and atomics and spinlocks

the acceptable solution

i could compile servo from scratch such that the mutex library recognises it's compiling against musl and knows not to use PTHREAD_MUTEX_ADAPTIVE_NP

however i do not feel like compiling rust code for the rest of this year

the quick and dirty solution

let's see what would happen if we pass it into musl's pthread_mutexattr_settype ... we get a EINVAL!

what if we simply didn't return an EINVAL?

mutexes are just a way to wait on something, the caller doesn't need to care about how it works under the hood

if the caller passes an invalid mutex type, we should be able to fall back on the default type and have it Just WorkTM (perhaps with slightly worse performance, but it's default or segfault)

rewriting the pthread_mutexattr_settype function...

int pthread_mutexattr_settype(pthread_mutexattr_t *a, int type)
{
-	if ((unsigned)type > 2) return EINVAL;
+	if ((unsigned)type > 2) type = PTHREAD_MUTEX_DEFAULT;  // return EINVAL;
    a->__attr = (a->__attr & ~3) | type;
    return 0;
}

recompile musl and reinstall...

and it launches!

servo browser displaying a html render of this blogpost! missed some styling, but still, woohoo!

conclusion

i should really get around to upstreaming my gcompat patches

not that musl hackjob though, rich felker would probably crush my skull (and for good reason)