Exploring Java Virtual Threads in Clojure Applications

Aug 12, 2024

It feels like the right time to make yourself familiar with Java Virtual Threads and here are a couple of reasons:

In JDK 21, the Virtual Threads feature is out of preview and enabled by default
Looks like we will have a Clojure 1.12 release soon (let’s see why it is important later in this article)
Ring release supports Jetty 12 (which has support for Virtual Threads)

Virtual Threads vs Platform Threads

If you want to learn more about virtual threads and all their whys and hows, I highly recommend this article: https://blog.rockthejvm.com/ultimate-guide-to-java-virtual-threads/

Virtual threads allow us to follow a simple pattern: to create a new thread for every concurrent task. This model is called one task per thread.

But why it’s not desired with old Java threads (platform threads) — the answer is that they are expensive in many ways. One way to see it is to try to create multiple platform threads in a loop:

(defn with-platform-threads
  []
  (dotimes [i 100000]
    (.start (Thread. (fn []
                       (Thread/sleep 1000)))))
  :done)

(with-platform-threads)

It’s quite easy to reach a message like this:

Execution error (OutOfMemoryError) at java.lang.Thread/start0 (Thread.java:-2).
unable to create native thread: possibly out of memory or process/resource limits reached

On JDK 21 (in previous releases we need to enable the preview feature) we can use virtual threads and try the same:

(defn with-virtual-threads
  []
  (dotimes [i 100000]
    (.start (.name (Thread/ofVirtual) "virtual-thread-")
            (fn [] (Thread/sleep 1000))))
  :done)

(with-virtual-threads)
:done

The other benefit is that if a virtual thread is blocked on IO, it will be unmounted from the platform (carrier) thread — and a pending virtual thread could reuse the same platform thread. As you can see we can efficiently reuse a limited pool of platform threads while keeping the virtual threads number high!

Ring and Jetty 12

Jetty 12 could be configured with a pool of virtual threads to handle requests, and it can significantly improve the server performance in case of lots of IO operations in handlers.

;; project.clj 

(defproject jetty-ring-virtual-threads "0.1.0-SNAPSHOT"
  :dependencies [[org.clojure/clojure "1.11.1"]
                 [ring/ring "1.12.2"]]
  :main ^:skip-aot jetty-ring-virtual-threads.core
  :target-path "target/%s")

;; jetty-ring-virtual-threads.core

(defn handler [_]
  (Thread/sleep 50)
  {:status 200
   :headers {"Content-Type" "text/plain"}
   :body "Hello World"})

(defn -main []
  (println "Starting Jetty Server")
  (jetty/run-jetty handler {:port 3000}))

;; and let's run it!
(-main)

To do a simple benchmarking of the server I’ll use `wrk` tool https://github.com/wg/wrk:

> wrk -t12 -c400 -d30s http://localhost:3000

Running 30s test @ http://localhost:3000
12 threads and 400 connections
25625 requests in 30.06s, 3.79MB read
Socket errors: connect 0, read 1303, write 0, timeout 0
Requests/sec: 852.39
Transfer/sec: 129.02KB

We got 850 Request per second with Jetty 12 on platform threads, now let’s configure it to use virtual threads:

(defn -main
  []
  (println "Starting Jetty Server (Virtual Threads)")
  (let [thread-pool (QueuedThreadPool.)]
    (.setVirtualThreadsExecutor
      thread-pool (Executors/newVirtualThreadPerTaskExecutor))
    (jetty/run-jetty handler {:port 3000
                              :thread-pool thread-pool})))

Everything else stays the same — time to rerun the benchmark:

> wrk -t12 -c400 -d30s http://localhost:3000

Running 30s test @ http://localhost:3000
12 threads and 400 connections
214960 requests in 30.10s, 31.78MB read
Socket errors: connect 0, read 1251, write 0, timeout 0
Requests/sec: 7140.46
Transfer/sec: 1.06MB

So we have almost 10x improvement — 7100 RPS is impressive!

Pinned virtual threads

So now let’s understand why future Clojure 1.12 release is important — and the reason is to avoid virtual threads pinning.

There are some cases where a blocking operation doesn’t unmount the virtual thread from the carrier thread, blocking the underlying carrier thread. In such cases, we say the virtual is pinned to the carrier thread. It’s not an error but a behavior that limits the application’s scalability. Note that if a carrier thread is pinned, the JVM can always add a new platform thread to the carrier pool if the configurations of the carrier pool allow it.
Fortunately, there are only two cases in which a virtual thread is pinned to the carrier thread:
When it executes code inside a synchronized block or method;
When it calls a native method or a foreign function (i.e., a call to a native library using JNI).

In Clojure delay and lazy-seq are using synchronized blocks, let’s check clojure.lang.Delay internals:

  public Object deref() {
    if (this.fn != null) {
      // this is the problem
      synchronized(this) {
        if (this.fn != null) {
          try {
            this.val = this.fn.invoke();
          } catch (Throwable var4) {
            Throwable t = var4;
            this.exception = t;
          }

          this.fn = null;
        }
      }
    }

    if (this.exception != null) {
      throw Util.sneakyThrow(this.exception);
    } else {
      return this.val;
    }
  }

Let’s change our handler to demonstrate the problem:

(defn handler [_]
  @(delay
    (Thread/sleep 50))
  {:status 200
   :headers {"Content-Type" "text/plain"}
   :body "Hello World"})

Technically nothing changed, we still sleep for 50ms but it’s wrapped in a delay that immediately derefed, but now we have a blocking call inside synchronized block — let’s run the benchmark:

> wrk -t12 -c400 -d30s http://localhost:3000

Running 30s test @ http://localhost:3000
12 threads and 400 connections
12534 requests in 30.06s, 1.85MB read
Socket errors: connect 0, read 1272, write 0, timeout 2844
Requests/sec: 416.90
Transfer/sec: 63.10KB

As you can see the result is much worse, but it’s on a first run after the server start, the sequential result is giving similar results (7000 RPS). I think it’s because the internal platform pool is scaled to max allowed size during the first test run — but I need to investigate more to be sure.

Also by enabling -Djdk.tracePinnedThreads=full I was able to get some errors about virtual thread pinning:

Thread[#148,ForkJoinPool-1-worker-3,5,CarrierThreads]
    java.base/java.lang.VirtualThread$VThreadContinuation.onPinned(VirtualThread.java:183)
    java.base/jdk.internal.vm.Continuation.onPinned0(Continuation.java:393)
    java.base/java.lang.VirtualThread.parkNanos(VirtualThread.java:621)
    java.base/java.lang.VirtualThread.sleepNanos(VirtualThread.java:791)
    java.base/java.lang.Thread.sleep(Thread.java:507)

Clojure 1.12

In future Clojure release there will be changes related to delays and lazy-seqs: https://github.com/clojure/clojure/blob/master/changes.md

1.2 Java 21 - Virtual thread pinning from user code under synchronized
Clojure users want to use virtual threads on JDK 21. Prior to 1.12, Clojure lazy-seqs and delays, in order to enforce run-once behavior, ran user code under synchronized blocks, which as of JDK 21 don't yet participate in cooperative blocking. Thus if that code did e.g. blocking I/O it would pin a real thread. JDK 21 may emit warnings for this when using -Djdk.tracePinnedThreads=full.
To avoid this pinning, in 1.12 lazy-seq and delay use locks instead of synchronized blocks.

Let’s update the version in the project.clj:

[org.clojure/clojure "1.12.0-rc1"]

Now if we go to Delay implementation — we will see the usage of locks instead of synchronized blocks:

this.lock = new ReentrantLock();

And let’s rerun our benchmark again:

> wrk -t12 -c400 -d30s http://localhost:3000

Running 30s test @ http://localhost:3000
12 threads and 400 connections
214040 requests in 30.11s, 31.64MB read
Socket errors: connect 0, read 1240, write 0, timeout 0
Requests/sec: 7109.61
Transfer/sec: 1.05MB

Alghtouth I still have some concerns about the benchmarking results (especially in the case when pinning should occur), clearly virtual threads in Jetty are definitely worth exploring — I’m looking forward to try that on some real applications soon!

Andrey Fadeev

Discussion about this post