Distributing OCaml software on opam is great (if I dare say so myself), but sometimes you need to provide your tools to an audience outside of the OCaml community, or just without recompilations or in a simpler way.
However, just distributing the locally generated binaries requires that the users have all the needed shared libraries installed, and a compatible libc. It’s not something you can assume in general, and even if you don’t need any C shared library or are confident enough it will be installed everywhere, the libc issue will arise for anyone using a distribution based on a different kind, or a little older than the one you used to build.
There is no built-in support for generating static executables in the OCaml compiler, and it may seem a bit tricky, but it’s not in fact too complex to do by hand, something you may be ready to do for a release that will be published. So here are a few tricks, recipes and advice that should enable you to generate truly portable executables with no external dependency whatsoever. Both Linux and macOS will be treated, but the examples will be based on Linux unless otherwise specified.
<h2>Example</h2>
I will take as an example a trivial HTTP file server based on <ahref="https://github.com/aantron/dream">Dream</a>.
<summary>
Sample code
</summary>
<h5>
fserv.ml
</h5>
<pre><codeclass="language-ocaml">
let () = Dream.(run @@ logger @@ static ".")
</code></pre>
<h5>
fserv.opam
</h5>
<pre><codeclass="language-python">
opam-version: "2.0"
depends: ["ocaml""dream"]
</code></pre>
<h5>
dune-project
</h5>
<pre><codeclass="language-lisp">
(lang dune 2.8)
(name fserv)
</code></pre>
The relevant part of our <code>dune</code> file is just:
<pre><codeclass="language-lisp">(executable
> Distributing OCaml software on opam is great (if I dare say so myself), but sometimes you need to provide your tools to an audience outside of the OCaml community, or just without recompilations or in a simpler way.
>
> However, just distributing the locally generated binaries requires that the users have all the needed shared libraries installed, and a compatible libc. It's not something you can assume in general, and even if you don't need any C shared library or are confident enough it will be installed everywhere, the libc issue will arise for anyone using a distribution based on a different kind, or a little older than the one you used to build.
>
> There is no built-in support for generating static executables in the OCaml compiler, and it may seem a bit tricky, but it's not in fact too complex to do by hand, something you may be ready to do for a release that will be published. So here are a few tricks, recipes and advice that should enable you to generate truly portable executables with no external dependency whatsoever. Both Linux and macOS will be treated, but the examples will be based on Linux unless otherwise specified.
## Example
I will take as an example a trivial HTTP file server based on [Dream](https://github.com/aantron/dream).
_build/default/fserv.exe: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=1991bb9f1d67807411c93f6fb6ec46b4a0ee8ed5, for GNU/Linux 3.2.0, with debug_info, not stripped
$ ldd _build/default/fserv.exe
$file _build/default/fserv.exe
_build/default/fserv.exe: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=1991bb9f1d67807411c93f6fb6ec46b4a0ee8ed5, for GNU/Linux 3.2.0, with debug_info, not stripped
(on macOS, replace <code>ldd</code> with <code>otool -L</code>; dune output is obtained with <code>(display short)</code> in <code>~/.config/dune/config</code>)
```
So let’s see how to change this result. Basically, here, <code>libev</code>, <code>libssl</code> and <code>libcrypto</code> are required shared libraries that may not be installed on every system, while all the others are part of the core system:
(on macOS, replace `ldd` with `otool -L`; dune output is obtained with `(display short)` in `~/.config/dune/config`)
<ul>
<li>
<code>linux-vdso</code>, <code>libdl</code> and <code>ld-linux</code> are concerned with the dynamic loading of shared objects ;
</li>
<li>
<code>libm</code> and <code>libpthread</code> are extensions of the core <code>libc</code> that are tightly bound to it, and always installed.
</li>
</ul>
So let's see how to change this result. Basically, here, `libev`, `libssl` and `libcrypto` are required shared libraries that may not be installed on every system, while all the others are part of the core system:
-`linux-vdso`, `libdl` and `ld-linux` are concerned with the dynamic loading of shared objects ;
-`libm` and `libpthread` are extensions of the core `libc` that are tightly bound to it, and always installed.
<h2>
Statically linking the libraries
</h2>
## Statically linking the libraries
In simple cases, static linking can be turned on as asily as passing the <code>-static</code> flag to the C compiler: through OCaml you will need to pass <code>-cclib -static</code>. We can add that to our <code>dune</code> file:
<pre><codeclass="language-lisp">
In simple cases, static linking can be turned on as easily as passing the `-static` flag to the C compiler: through OCaml you will need to pass `-cclib -static`. We can add that to our `dune` file:
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/10/../../../x86_64-linux-gnu/libcrypto.a(dso_dlfcn.o): in function `dlfcn_globallookup':
(.text+0x13): warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: ~/.opam/4.11.0/lib/ocaml/libunix.a(initgroups.o): in function `unix_initgroups':
initgroups.c:(.text.unix_initgroups+0x1f): warning: Using 'initgroups' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: /usr/lib/gcc/x86_64-linuxgnu/10/../../../x86_64-linux-gnu/libcrypto.a(dso_dlfcn.o): in function `dlfcn_globallookup':
(.text+0x13): warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: ~/.opam/4.11.0/lib/ocaml/libunix.a(initgroups.o): in function `unix_initgroups':
initgroups.c:(.text.unix_initgroups+0x1f): warning: Using 'initgroups' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
[...]
$file _build/default/fserv.exe
_build/default/fserv.exe: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=9ee3ae1c24fbc291d1f580bc7aaecba2777ee6c2, for GNU/Linux 3.2.0, with debug_info, not stripped
_build/default/fserv.exe: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=9ee3ae1c24fbc291d1f580bc7aaecba2777ee6c2, for GNU/Linux 3.2.0, with debug_info, not stripped
$ldd _build/default/fserv.exe
not a dynamic executable
</code></pre>
not a dynamic executable
```
The executable was generated… and the result <em>seems</em> OK, but we shouldn’t skip all these <code>ld</code> warnings. Basically, what <code>ld</code> is telling us is that you shouldn’t statically link <code>glibc</code> (it internally uses dynlinking, to libraries that also need <code>glibc</code> functions, and will therefore <strong>still</strong> need to dynlink a second version from the system 🤯).
Indeed here, we have been statically linking a dynamic linking engine, among other things. Don’t do it.
The executable was generated... and the result _seems_ OK, but we shouldn't skip all these `ld` warnings. Basically, what `ld` is telling us is that you shouldn't statically link `glibc` (it internally uses dynlinking, to libraries that also need `glibc` functions, and will therefore **still** need to dynlink a second version from the system 🤯).
<h3>
Linux solution: linking with musl instead of glibc
</h3>
Indeed here, we have been statically linking a dynamic linking engine, among other things. Don't do it.
The easiest workaround at this point, on Linux, is to compile with <ahref="http://musl.libc.org/">musl</a>, which is basically a glibc replacement that can be statically linked. There are some OCaml and gcc variants to automatically use musl (comments welcome if you have been successful with them!), but I have found the simplest option is to use a tiny Alpine Linux image through a Docker container. Here we’ll use OCamlPro’s <ahref="https://hub.docker.com/r/ocamlpro/ocaml">minimal Docker images</a> but anything based on musl should do.
### Linux solution: linking with musl instead of glibc
<pre><codeclass="language-shell-session">
The easiest workaround at this point, on Linux, is to compile with [musl](http://musl.libc.org/), which is basically a glibc replacement that can be statically linked. There are some OCaml and gcc variants to automatically use musl (comments welcome if you have been successful with them!), but I have found the simplest option is to use a tiny Alpine Linux image through a Docker container. Here we'll use OCamlPro's [minimal Docker images](https://hub.docker.com/r/ocamlpro/ocaml) but anything based on musl should do.
```shell-session
$docker run --rm-it ocamlpro/ocaml:4.12
[...]
~/fserv $sudo apk add openssl-libs-static
...
...
@@ -128,113 +113,97 @@ $ docker run --rm -it ocamlpro/ocaml:4.12
OK: 161 MiB in 52 packages
~/fserv $opam switch create .--deps ocaml-system
[...]
~/fserv $eval$(opam env)
~/fserv $dune build fserv.exe
~/fserv $file _build/default/fserv.exe
_build/default/fserv.exe: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, with debug_info, not stripped
~/fserv $ldd _build/default/fserv.exe
/lib/ld-musl-x86_64.so.1 (0x7ff41353f000)
</code></pre>
```
Almost there! We see that we had to install extra packages with <code>apk add</code>: the static libraries might not be already installed and in this case are in a separate package (you would get <code>bin/ld: cannot find -lssl</code>). The last remaining dynamic loader in the output of <code>ldd</code> is because static PIE executable were not supported <ahref="https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81498#c1">until recently</a>. To get rid of it, we need to either replace <code>-static</code> with <code>-static-pie</code> if the version of gcc is high enough, or add <code>-cclib -no-pie</code>:
Almost there! We see that we had to install extra packages with `apk add`: the static libraries might not be already installed and in this case are in a separate package (you would get `bin/ld: cannot find -lssl`). The last remaining dynamic loader in the output of `ldd` is because static PIE executable were not supported [until recently](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81498#c1). To get rid of it, we just need to add `-cclib -no-pie` (note: a previous revision of this blogpost mentionned `-static-pie` instead, which may work with recent compilers, but didn't seem to give reliable results):
<pre><codeclass="language-lisp">
```lisp
(executable
(public_namefserv)
(flags (:standard -cclib -static-pie))
(flags(:standard-cclib-static-cclib-no-pie))
(librariesdream))
</code></pre>
```
And we are good!
<pre><codeclass="language-shell-session">
```shell-session
~/fserv $file _build/default/fserv.exe
_build/default/fserv.exe: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, with debug_info, not stripped
_build/default/fserv.exe: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, with debug_info, not stripped
~/fserv $ldd _build/default/fserv.exe
/lib/ld-musl-x86_64.so.1: _build/default/fserv.exe: Not a valid dynamic program
</code></pre>
<strong>Trick</strong>: short script to compile through a Docker container
Passing the context to a Docker container and getting the artefacts back can be bothersome and often causes file ownership issues, so I use the following snippet to pipe them to/from it using <code>tar</code>:
} >&2 && tar c -hC _build/install/default/bin .' | \
tar vx
</code></pre>
<h3>
The other cases: turning to manual linking
</h3>
Sometimes you can’t use the above: the automatic linking options may need to be tweaked for static libraries, your app may still need dynlinking support at some point, or you may not have the musl option. On macOS, for example, the libc doesn’t have a static version at all (and the <code>-static</code> option of <code>ld</code> is explicitely “only used building the kernel”). Let’s get our hands dirty and see how to use a mixed static/dynamic linking scheme. First, we examine how OCaml does the linking usually:
The linking options are passed automatically by OCaml, using information that is embedded in the <code>cm(x)a</code> files, for example:
<pre><codeclass="language-shell-session">
/lib/ld-musl-x86_64.so.1: _build/default/fserv.exe: Not a valid dynamic program
```
> **Trick**: short script to compile through a Docker container
>
> Passing the context to a Docker container and getting the artefacts back can be bothersome and often causes file ownership issues, so I use the following snippet to pipe them to/from it using `tar`:
> } >&2 && tar c -hC _build/install/default/bin .' | \
> tar vx
> ```
### The other cases: turning to manual linking
Sometimes you can't use the above: the automatic linking options may need to be tweaked for static libraries, your app may still need dynlinking support at some point, or you may not have the musl option. On macOS, for example, the libc doesn't have a static version at all (and the `-static` option of `ld` is explicitely "only used building the kernel"). Let's get our hands dirty and see how to use a mixed static/dynamic linking scheme. First, we examine how OCaml does the linking usually:
The linking options are passed automatically by OCaml, using information that is embedded in the `cm(x)a` files, for example:
```shell-session
$ocamlobjinfo $(opam var lwt:lib)/unix/lwt_unix.cma |head
File ~/.opam/4.11.0/lib/lwt/unix/lwt_unix.cma
Force custom: no
Extra C object files: -llwt_unix_stubs -lev -lpthread
Extra C options:
Extra dynamically-loaded libraries: -llwt_unix_stubs
Unit name: Lwt_features
File ~/.opam/4.11.0/lib/lwt/unix/lwt_unix.cma
Force custom: no
Extra C object files: -llwt_unix_stubs -lev -lpthread
Extra C options:
Extra dynamically-loaded libraries: -llwt_unix_stubs
Now the linking flags, here <code>-llwt_unix_stubs -lev -lpthread</code> let the C compiler choose the best way to link; in the case of stubs, they will be static (using the <code>.a</code> files — unless you make special effort to use dynamic ones), but <code>-lev</code> will let the system linker select the shared library, because it is generally preferred. Gathering these flags by hand would be tedious: my preferred trick is to just add the <code>-verbose</code> flag to OCaml (for the lazy, you can just set — temporarily — <code>OCAMLPARAM=_,verbose=1</code>):
Now the linking flags, here `-llwt_unix_stubs -lev -lpthread` let the C compiler choose the best way to link; in the case of stubs, they will be static (using the `.a` files — unless you make special effort to use dynamic ones), but `-lev` will let the system linker select the shared library, because it is generally preferred. Gathering these flags by hand would be tedious: my preferred trick is to just add the `-verbose` flag to OCaml (for the lazy, you can just set — temporarily — `OCAMLPARAM=_,verbose=1`):
There is a lot of noise, but the interesting part is at the end, the <code>-l*</code> options before the standard <code>ocaml/libasmrun -lm -ldl</code>:
+ as -o '/tmp/build8eb7e5.dune/camlstartupc9267f.o' '/tmp/build8eb7e5.dune/camlstartup1d9915.s'
To link these statically, but the glibc dynamically:
- we disable the automatic generation of linking flags by OCaml with `-noautolink`
- we pass directives to the linker through OCaml and the C compiler, using `-cclib -Wl,xxx`. `-Bstatic` makes static linking the preferred option
- we escape the linking flags we extracted above through `-cclib`
<ul>
<li>
we disable the automatic generation of linking flags by OCaml with <code>-noautolink</code>
</li>
<li>
we pass directives to the linker through OCaml and the C compiler, using <code>-cclib -Wl,xxx</code>. <code>-Bstatic</code> makes static linking the preferred option
</li>
<li>
we escape the linking flags we extracted above through <code>-cclib</code>
</li>
</ul>
<pre><codeclass="language-lisp">
```lisp
(executable
(public_namefserv)
(flags(:standard
...
...
@@ -251,55 +220,46 @@ To link these statically, but the glibc dynamically:
-cclib-Wl,-Bdynamic
-cclib-lpthread))
(librariesdream))
</code></pre>
```
Note that <code>-lpthread</code> and <code>-lm</code> are tightly bound to the libc and can’t be static in this case, so we moved <code>-lpthread</code> to the end, outside of the static section. The part between the <code>-Bstatic</code> and the <code>-Bdynamic</code> is what will be statically linked, leaving the defaults and the libc dynamic. Result:
Note that `-lpthread` and `-lm` are tightly bound to the libc and can't be static in this case, so we moved `-lpthread` to the end, outside of the static section. The part between the `-Bstatic` and the `-Bdynamic` is what will be statically linked, leaving the defaults and the libc dynamic. Result:
_build/default/fserv.exe: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=31c93085284da5d74002218b1d6b61c0efbdefe4, for GNU/Linux 3.2.0, with debug_info, not stripped
$ ldd _build/default/fserv.ex
_build/default/fserv.exe: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=31c93085284da5d74002218b1d6b61c0efbdefe4, for GNU/Linux 3.2.0, with debug_info, not stripped
The remaining are the base of the dynamic linking / shared object systems, but we got away with <code>libssl</code>, <code>libcrypto</code> and <code>libev</code>, which were the ones possibly absent from target systems. The resulting executable should work on any glibc-based Linux distribution that is recent enough; on older ones you will likely get missing <code>GLIBC</code> symbols.
If you need to distribute that way, it’s a good idea to compile on an old release (like Debian ‘oldstable’ or ‘oldoldstable’) for maximum portability.
The remaining are the base of the dynamic linking / shared object systems, but we got away with `libssl`, `libcrypto` and `libev`, which were the ones possibly absent from target systems. The resulting executable should work on any glibc-based Linux distribution that is recent enough; on older ones you will likely get missing `GLIBC` symbols.
<h4>
Manually linking on macOS
</h4>
If you need to distribute that way, it's a good idea to compile on an old release (like Debian 'oldstable' or 'oldoldstable') for maximum portability.
Unfortunately, the linker on macOS doesn’t seem to have options to select the static versions of the libraries; the only solution is to get our hands even dirtier, and link directly to the <code>.a</code> files, instead of using <code>-l</code> arguments.
#### Manually linking on macOS
Most of the flags just link with stubs, we can keep them as is: <code>-lssl_stubs</code><code>-lcamlstr</code><code>-loverlap_stubs_stubs</code><code>-ldigestif_c_stubs</code><code>-lmtime_clock_stubs</code><code>-lmirage_crypto_rng_unix_stubs</code><code>-lmirage_crypto_stubs</code><code>-lcstruct_stubs</code><code>-llwt_unix_stubs</code><code>-lthreadsnat</code><code>-lunix</code><code>-lbigstringaf_stubs</code>
Unfortunately, the linker on macOS doesn't seem to have options to select the static versions of the libraries; the only solution is to get our hands even dirtier, and link directly to the `.a` files, instead of using `-l` arguments.
That leaves us with: <code>-lssl</code><code>-lcrypto</code><code>-lev</code><code>-lpthread</code>
Most of the flags just link with stubs, we can keep them as is: `-lssl_stubs``-lcamlstr``-loverlap_stubs_stubs``-ldigestif_c_stubs``-lmtime_clock_stubs``-lmirage_crypto_rng_unix_stubs``-lmirage_crypto_stubs``-lcstruct_stubs``-llwt_unix_stubs``-lthreadsnat``-lunix``-lbigstringaf_stubs`
<ul>
<li>
<code>lpthread</code> is built-in, we can ignore it
</li>
<li>
for the others, we need to lookup the <code>.a</code> file: I use <em>e.g.</em>
<pre><codeclass="language-shell-session">
That leaves us with: `-lssl``-lcrypto``-lev``-lpthread`
-`lpthread` is built-in, we can ignore it
- for the others, we need to lookup the `.a` file: I use _e.g._
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.60.1)
</code></pre>
_build/default/fserv.exe:
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.60.1)
```
This is as good as it will get!
<h2>
Cleaning up the build system
</h2>
We have until now been adding the linking flags manually in the <code>dune</code> file; you probably don’t want to do that and be restricted to static builds only! Not counting the non-portable link options we have been using…
## Cleaning up the build system
<h3>
The quick&dirty way
</h3>
We have until now been adding the linking flags manually in the `dune` file; you probably don't want to do that and be restricted to static builds only! Not counting the non-portable link options we have been using...
Don’t use this in your build system! But for quick testing you can conveniently pass flags to the OCaml compilers using the <code>OCAMLPARAM</code> variable. Combined with the tar/docker snippet above, we get a very simple static-binary generating command:
} >&2 && tar c -hC _build/install/default/bin .' | \
Don't use this in your build system! But for quick testing you can conveniently pass flags to the OCaml compilers using the `OCAMLPARAM` variable. Combined with the tar/docker snippet above, we get a very simple static-binary generating command:
The linking flags will depend on the chosen linking mode and on the OS. For the OS, it's easiest to generate them through a script ; for the linking mode, I use an environment variable to optionally turn static linking on.
The linking flags will depend on the chosen linking mode and on the OS. For the OS, it’s easiest to generate them through a script ; for the linking mode, I use an environment variable to optionally turn static linking on.
This will use the following <code>gen-linking-flags.sh</code> script to generate the file, passing it the value of <code>$LINKING_MODE</code> and defaulting to <code>dynamic</code>. Doing it this way also ensures that <code>dune</code> will properly recompile when the value of the environment variable changes.
This will use the following `gen-linking-flags.sh` script to generate the file, passing it the value of `$LINKING_MODE` and defaulting to `dynamic`. Doing it this way also ensures that `dune` will properly recompile when the value of the environment variable changes.
