Production-Ready Clojure: show me the libraries!

In this blog post, I’ll try to make an overview of the most used libraries in real production-grade Clojure applications. I hope this guide helps you navigate the rich Clojure ecosystem.

Dec 20, 2023

Hello, Clojure enthusiasts!

I'm Andrey, and in today's blog post, I want to share my thoughts on the libraries and tools we use in production when working with Clojure.

Whether you're a seasoned developer or just starting your journey with Clojure, I hope this information proves valuable.

Libraries vs Frameworks

Before delving into specifics, let's address a crucial point. While there are web frameworks in the Clojure ecosystem, I recommend, especially for beginners, to start by combining libraries instead. This approach provides a deeper understanding of how different pieces work together. Later, if you find frameworks appealing, feel free to explore them. However, in my experience with real production code, I've rarely (never) seen frameworks used.

As a framework - `biff` (https://biffweb.com) looks interesting, maybe for beginners, it will be a quick way to get up and running.

Anyway in this blog post, we will focus on libraries!

Managing Dependencies

Now, let's talk about managing dependencies. Traditionally, leiningen has been the go-to tool, but `deps.edn` is gaining traction. While Leningen still works fine, `deps.edn` is becoming a standard in new projects.

Managing Components

Handling components in a Clojure system is crucial. While you can roll your solution, using a library can add structure and make it easier to manage start and stop logic. Two solid options that I can recommend here are:

`component` - https://github.com/stuartsierra/component
`integrant` - https://github.com/weavejester/integrant

System creation with `component` will look like this:

(defn example-system [config-options]
  (let [{:keys [host port]} config-options]
    (component/system-map
      :db (new-database host port)
      :scheduler (new-scheduler)
      :app (component/using
             (example-component config-options)
             {:database  :db
              :scheduler :scheduler}))))

With `integrant` you’ll define your system as a plain Clojure map (or an EDN file) with custom tags:

(def config
  {:adapter/jetty {:port 8080, :handler (ig/ref :handler/greet)}
   :handler/greet {:name "Alice"}})

(defmethod ig/init-key :adapter/jetty [_ {:keys [handler] :as opts}]
  (jetty/run-jetty handler (-> opts (dissoc :handler) (assoc :join? false))))

(defmethod ig/init-key :handler/greet [_ {:keys [name]}]
  (fn [_] (resp/response (str "Hello " name))))

Integrant could work quite well with the `aero` library, but again there are two possible approaches, so I recommend reading both links to learn more:

I'd advise against using `mount` (https://github.com/tolitius/mount) due to its potential to encourage bad coding practices (IMO) - sharing global state and making functions impure but using global components directly (not passed through input arguments). That could be overcome with some best practices (not allowing this pattern) - the issue that it is almost impossible to enforce.

Managing configuration

This is pretty simple for me, a good option is `aero` (https://github.com/juxt/aero), the configuration file is defined in EDN and it’s really easy to read:

{:input-topic #env KAFKA_INPUT_TOPIC
 :output-topic #env KAFKA_OUTPUT_TOPIC
 :kafka {:bootstrap.servers #env KAFKA_BOOTSTRAP_SERVERS
                 :application.id #env KAFKA_APPLICATION_ID
                 :security.protocol "SSL"
                 :ssl.keystore.type "PKCS12"
                 :ssl.truststore.type "JKS"
                 :auto.offset.reset #env KAFKA_AUTO_OFFSET_RESET
                 :producer.compression.type #or [#env KAFKA_PRODUCER_COMPRESSION_TYPE "zstd"]
                 :processing.guarantee #or [#env KAFKA_PROCESSING_GUARANTEE "exactly_once_v2"]
                 :producer.acks "all"}
 :server {:host "0.0.0.0"
          :port 10124}
 :service-version #env SERVICE_VERSION}

So on startup of your application, you just render it (so it will fill the placeholders with values from environment variables) and pass it to your application system. A smart idea would be to use one of the schema validation libraries and create a schema in your application configuration - so the application won’t start if it’s not valid (that will usually mean that you forgot to pass the required environment variable).

If you want to keep it simple use `(System/getenv "ENV_VAR")`.

Schemas

For schemas and validation, there are three solid options:

`schema` - https://github.com/plumatic/schema
`clojure.spec` - https://github.com/clojure/spec.alpha
`malli` - https://github.com/metosin/malli

I prefer to use `malli` most of the time, an example of schema will look like this:

(def kafka-config-base-schema
  (m/schema
    [:map
     [:bootstrap.servers :string]
     [:application.id :string]
     [:auto.offset.reset [:enum "earliest" "latest" "none"]]
     [:producer.compression.type [:enum "none" "zstd"]]
     [:processing.guarantee [:enum "at_least_once" "exactly_once_v2"]]
     [:producer.acks [:enum "0" "1" "all"]]]))

HTTP Server

For spinning up HTTP servers, we've primarily used Jetty as the base. On top of Jetty, you can use Ring or Pedestal - both are fine!

There was a period when `http-kit` was popular, I’m not a huge fan TBH, mainly because there is a lack of support if you will finally want some advanced monitoring for your application: APM and distributed tracing from the OpenTelementry and auto-instrumentation (Datadog, NewRelic, etc).

HTTP Router

In addition to the base HTTP server and HTTP abstraction, you’ll need a way to route your request to correct handlers, one of the oldest options is Compojure, it’s simple to learn and use but has its limitation being macros-based. People tend to use data-based approaches, so there are libraries on the plate like:

If you’ve decided to use Pedestal not Ring it comes with its own data-based way to define routes - and it’s good enough for most use cases (also Reitit can work on top of Pedestal as well).

HTTP Requests

When it comes to handling HTTP requests, `clj-http` is a reliable choice, utilizing the Apache HTTP library.

If you're on Java 11 or higher, the built-in Java HTTP client is also worth exploring, for example, the https://github.com/babashka/http-client

Working with JSON

Using JSON is popular nowadays (I don’t even remember working with anything else), a couple of libraries to work with JSON in Clojure:

Check this link - it’s a good overview of the JSON libraries available: https://www.juxt.pro/blog/json-in-clojure/

Working with Relational Databases

For working with relational databases, `next.jdbc` (https://github.com/seancorfield/next-jdbc) is the current standard, and HikariCP works well for connection pools.

Regarding SQL query libraries, consider either HoneySQL or HugSQL, both of which we've used in production.

Other Libraries

`core.cache` or `core.memoize` if you’d like a bit more advanced caching library
`core.async` if you are big on the idea of building non-blocking code (be careful it’s really easy to overengineer here)
`claypoole` - a great library to work with Java thread pools in Clojure: https://github.com/clj-commons/claypoole

Other tools

`babashka` (https://babashka.org) - great tool to write scripts in Clojure, could sound crazy at first, but give it a try if you haven’t before! We have all our scripting for CI written in babashka now!
`cljfmt` - my go-to code formatter
`clj-kondo` - great static linter for Clojure code

We use both `cljfmt` and `clj-kondo` as pipeline steps in the CI to keep the Clojure code tidy!

Tests

Although Clojure developers rely on REPL a lot to debug and write code, it’s still crucial to eventually produce tests for your code. There are different levels of testing available, the most common groups I’ve seen so far are:

`unit` - simple tests that do not require any complex setup or starting your system (or parts of the system), call your functions or chains of functions and assert the result
`persistence` - it is slightly more high-level, in those tests we are talking to the relational database (or other type of persistent storage) and checking for side-effects (e.g. a database row was created as the result of the test), but we still don’t usually start our application system
`component` - it is testing the full flow that requires spinning your application system, here are a couple of examples: call the HTTP endpoint of your test system and expect correct side-effects (change in the database, event published to the queue, a call to the third party system was made, etc)

Regarding writing tests, I just suggest you stick with the simplest option: `clojure.test` that comes out-of-the-box with Clojure.

The slightly more interesting topic is which test runner to use. Here we have a bunch of options:

`lein test` - in case you are using Leininen, that comes out-of-the-box
`eftest` - has more configuration options and better reporting, and could be used for both `leiningen` and `deps.edn`: https://github.com/weavejester/eftest
`cognitect-labs/test-runner` - a simple (but good enough) option if you are using `deps.edn` and missing a built-in option to run tests: https://github.com/cognitect-labs/test-runner
`kaocha` - a test runner with a lot of features and great docs, for both Leiningen and `deps.edn`: https://github.com/lambdaisland/kaocha

Although there are a bunch of options, I don’t suggest you spend too much time picking one, test runner is doing a simple job: search and run your tests - so you will be fine in any case. Better focus on writing more robust and reliable tests so you worry less after refactoring your code!

Conclusion

And there you have it – a comprehensive overview of the libraries and tools we use in production Clojure code (or at least I’ve tried). I hope this guide helps you navigate the rich Clojure ecosystem.

If you prefer the video content, you can find almost the same as the video on my channel:

Andrey Fadeev

Discussion about this post