Generating Clojure Native Images with GraalVM and clj-nix

I’ve been building a little project with Clojure to automatically switch my IKEA Trådfri lights when I start playing a video and restoring them when I pause. I’m lazy, I know. I used Clojure because it’s fun, and I like to use fun languages when not writing code for money. Also, there’s a good CoAPS library available for Java, which is a big plus, as many other languages don’t have support for CoAPS, which is CoAP + TLS.

Anyway, after I was done with implementing the basic skeleton, I wanted to run the program on my server, which is running NixOS. I had an ad-hoc Nix expression for my Clojure app that just ran java -jar <file.jar>. However, I reasoned that there should be a nicer way to do this, and found clj-nix, which allows me to do reproducible builds of my Clojure project. This then allowed me to generate a native image, which is a bit easier to run and faster to start than the “regular” way.

Let’s see how it works!

Project setup: Know your Dependencies

You can see the code here (excuse the missing readme, it’s strictly a me project). The deps.edn file is very basic, just including the necessary dependencies and some boilerplate for generating an uberjar using some code in build.clj¹.

How do I get this into a Nix expression?

Broadly speaking, the same way that other languages do this (like haskell.nix, cargo2nix or poetry2nix): You need a way to translate from your “native” package specification into a Nix expression. Then, you take the full set of packages that your package manager generated (for example, reading Cargo’s cargo.lock package lock file), and translate every dependency into a Nix expression recursively, with a depth first traversal.

In this example, you’d generate a Nix expression for Dep C, generate a Nix expression for Dep B having Dep C as a build input, generate a Nix expression for Dep A, and finally generate a Nix expression for the main program having Dep A and Dep B as inputs.

Getting the sets of dependencies from the package manager makes sure that we are building the exact same code in the package manager and the Nix expression. Some package managers have better support for this, and some have worse. Unfortunately for us, both Leiningen and the “builtin” deps.edn package management can’t generate a lockfile for us. Luckily, clj-nix can do this for us. Note that you need to have Nix installed, which is always helpful when you’re trying to write a Nix expression. Here’s how it looks:

nix run github:jlesquembre/clj-nix#deps-lock

This will generate a deps-lock.json file from your deps.edn file. If you have multiple aliases, and want to cache the deps for only one alias, use the --alias-include argument. In my case, I used --alias-include run, as that’s the alias I care about. Add the file to your Git repository, and that’s the first step done.

Writing a Nix Flake

I must confess I have a love-hate relationship with Nix flakes. On the one hand, they’re atrociously documented, and everybody just seems to be copying templates around. On the other hand, they mostly seem to work well. Anyway, clj-nix also works using flakes. A Nix flake is a Nix concept that maps from some inputs to some outputs. Flakes can consume other flakes as inputs (such as nixpkgs or flakes from Github), and generate outputs, such as programs and libraries. From my understanding, Flakes allow specifying dependencies in a much simpler manner than mucking about with Nixpkgs versions, overlays and overrides.

Following my tried-and-true flake approach, I copied the flake template into my repository, and updated the description, main-ns and name. Make sure that you haven’t enabled the nativeImage configuration, as this will likely fail. Then, you should be able to run nix build . in the root directory to build your flake. This will create a “result” symlink to the Nix store containing your packaged application.

Making a Native Image

To build a native image, enable native image support in your flake. Make sure that you’ve staged or committed your changes (or nix will use the old version of your files), and run nix build . again. This will take some time, and likely fail due to two causes: Class initialisation and reflection. Let’s deal with class initialization first.

Class Initialisation

The problem with class initialization is described here: The Java specification states that classes are initialized when they are first accessed. This isn’t great for a native image: You have to either check whether your class was already initialized with every access, which is expensive, or modify your own code after the first access to remove the checks. A native image can’t modify it’s own code with the same reliability as a JIT-compiling VM does. To solve this dilemma, you can tell GraalVM that certain classes are okay to initialize at build time.

Californium, the Java dependency in my project, uses SLF4J for logging. SLF4J uses the ServiceLoader mechanism to discover available logging frameworks, and select one of them. We can tell GraalVM that we’re okay with SLF4J doing this at build time by providing the --initialize-at-build-time argument to the native image builder. This works via extraNativeImageBuildArgs, as shown here.

If you have a larger project, you’ll most probably have to extend this list further. Just read the error messages you get from a failed nix build . and add packages as needed.

Reflection

Now for reason number two, which is reflection: GraalVM is an optimizing compiler doing dead-code elimination. To eliminate correctly, it needs to find out what code is and isn’t used. Due to its focus on “do what I mean”, Clojure often uses reflection to find out what method to call, when it can’t be discovered statically. A simple example:

(defn test [val]
  (.getBytes val))

Without any extra information about the type of val, Clojure has to use reflection to find the .getBytes method, which is A) slow and B) impossible to follow for GraalVM. Native-compiling this code an then running it will likely lead to a “NoSuchMethodError: java.lang.String.getBytes()”. This is because GraalVM did not see any access to the .getBytes method, and removed it from the resulting native image. To detect this early, set the *warn-on-reflection* variable to true, and add type hints to your Clojure code. I’m enabling *warn-on-reflection* here. You can also do it by calling (set! *warn-on-reflection* true), but this needs to be repeated for every namespace. You can see an example for type hints here. In this case, I’ve annotated the return type of the function. Liberally sprinkle type hints through your code until you don’t get any reflection warnings when running via clj -M:run, and your native image works.

Congratulations, you’re done!