An 8-year-old Clojure bug, resorting to Java

I loath private, protected, and their ilk, much preferring to make everything public 99% of the time. Power to the consumer! Here's an (abbreviated) example from the Java library. A protected base class which provided the subclass with a public method.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

protected class OptionsBase<T> {

    public T timeouts (Timeouts timeouts) {
        this.timeouts = timeouts;
        return (T)this;
    }
}

public class ReadAllOptions extends OptionsBase<ReadAllOptions> {
    public static ReadAllOptions get() {
        return new ReadAllOptions();
    }
}

In Java, this is perfectly legal. In Clojure:

1
2

(-> (ReadAllOptions/get)
    (.timeouts timeouts))

1
2
3

Error in ->ReadAllOptions-test
Uncaught exception, not in assertion
   error: java.lang.IllegalArgumentException: No matching method timeouts found taking 1 args for class com.eventstore.dbclient.ReadAllOptions

I get exceptions like this all the time working with Clojure–Java interop, either I've:

• called the right method on the wrong class
• called the wrong method on the right class
• called an inaccessible method

Let's find out which methods ReadAllOptions provides:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

(->> ReadAllOptions
     .getMethods
     (map str))
; => ...
;  17. "public java.lang.Object com.eventstore.dbclient.OptionsBase.requiresLeader()"
;  18. "public java.lang.Object com.eventstore.dbclient.OptionsBase.requiresLeader(boolean)"
;  19. "public java.lang.Object com.eventstore.dbclient.OptionsBase.timeouts(com.eventstore.dbclient.Timeouts)"
;  20. "public java.lang.Object com.eventstore.dbclient.OptionsBase.notRequireLeader()"
;  21. "public final void java.lang.Object.wait(long,int) throws java.lang.InterruptedException"
; ...

Look at #19, it has a public method called timeouts, and after checking (.getClass timeout) we can be sure that this call should work.

It turns out that this is a bug that goes back to at least 2013, when it was first tracked in the Clojure Jira (#CLJ–1243) and I suspect it might be related to a 22-year-old JVM bug (#JDK–4283544).

Well, what to do? I don't think there's much use in waiting for this to get fixed. Let's do as hackers do and hack. Given that I'm fairly certain that what I want to do is legal in Java, let's look at the Java interop options of which there are three levels:

1. class / member accesses and the dot special form
2. proxy, gen-class, reify
3. writing Java

In this rare case, it looks like writing Java is ones only option. But I, like many other Clojure developers, was once a Java developer, and the Java needed is very simple, it is a wrapper for ReadAllOptions, so the offending methods will be called inside our Java code.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

import com.eventstore.dbclient.ReadAllOptions;
import com.eventstore.dbclient.Timeouts;

public class ReadAllOptionsClj {
    public ReadAllOptions foreign;

    public ReadAllOptionsClj () {
        this.foreign = ReadAllOptions.get();
    }

    public ReadAllOptionsClj timeouts (Timeouts timeouts) {
        this.foreign.timeouts(timeouts);
        return this;
    }

    public ReadAllOptions build () {
        return this.foreign;
    }
}

I've used a builder pattern here. This is good to call using Clojure's cond-> threading macro. It works by evaluating a predicate, and treating the next as if in a normal thread-first macro. In the following code this makes sure that methods are only called if the argument is non-nil, the builder's default value is used otherwise.

1
2
3
4
5

(cond-> (new ReadAllOptionsClj)
  (some? timeouts) (.timeouts timeouts)
  ;; add many other fields to the builder
  ;; (some? host) (.addHost host)
  true (.build))

As always when calling Java from Clojure, one must import the class:

1
2
3

(ns ...
  (:require [...])
  (:import [ae.eventstore.j ReadAllOptionsClj]))

But before it can be used, the Java needs to be compiled.

Building this in Leiningen should be really simple. Lein's defproject has the key :java-source-paths which can be compiled using lein javac, but this is usually unnecessary as all the usual Lein tasks will do it for you.

I made the switch to deps.edn last year, unfortunately it won't do compile Java code automatically. But this does provide one with an opportunity to learn how to use tools.build, written by Alex Miller (@puredanger). Asserting that builds are programs, we must do for ourselves what Lein would be doing.

To deps.edn one must add an alias for the task to be performed, add any dependencies needed for that task, and the namespace containing the functions to use. Our tasks will be build, our only dependency tools.build, and our namespace build.

1
2

{:alias {:build {io.github.clojure/tools.build {:git/tag "v0.6.8" :git/sha "d79ae84"}
                 :ns-default build}}}

One then creates build.clj in the project's root where we will start defining our build task:

1
2

(ns build
  (:require [clojure.tools.build.api :as b]))

The most important subtask will be compiling our Java code, for which tools.build provides the javac function.

1
2
3
4
5
6
7

(def class-dir "target/classes")
(def basis (b/create-basis {:project "deps.edn"}))
(defn compile [_]
  (b/javac {:src-dirs ["java"]
            :class-dir class-dir
            :basis basis
            :javac-opts ["-source" "8" "-target" "8"]}))

I assume here that the Java code is within java but it might be possible to keep it as a subdirectory or src as one would with clj, cljc, and cljs.

Running clojure -T:build compile will compile the Java source files as can be seen by looking in target/classes. Once target/classes is added to the classpath you will be ready to call the class from Clojure.

1

{:paths ["src" "resources" "target/classes"]}

In attempting to use gen-class to circumvent this bug, I found an even older Clojure bug! Using the builder pattern, the builder's methods return itself. This allows one to use method chaining in Java, or threading macros in Clojure.

1
2
3
4
5
6
7

Foo.newBuilder()
    .red()
    .mirrored()
    .short()
    .wearing(jacket)
    .build()
// => an instance of Foo

1
2
3
4
5
6
7
8

(-> Foo
    .newBuilder
    .red
    .mirrored
    .short
    (.wearing jacket)
    .build)
;; => an instance of Foo

One would write the method like so, so the function makes the change to the internal state, and then returns itself.

1
2
3
4
5
6
7
8

(gen-class :name "ae.ReadAllOptionsBuilder"
           :methods [[timeouts
                      [com.eventstore.dbclient.Timeouts]
                      ae.ReadAllOptionsBuilder]])

(defn -timeouts [this timeouts]
  (-> this .state (.timeouts timeouts))
  this)

If it only were so. Ticket #CLJ-84, compile gen-class fail when class returns self, makes it clear that it is not possible for a gen-class method to return an instance of the same class. Created on 17th June 2009 by Rick Hickey, this ticket is as old as Clojure's ticketing system itself. What happens if we return void instead?

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

(ns ae.read-all-options
  (:import [com.eventstore.dbclient
            Direction
            Position
            ReadAllOptions
            Timeouts
            UserCredentials]))

(gen-class
 :name "ae.eventstore.ReadAllOptionsCljTwo"
 :state state
 :init init
 :constructors {[] []}
 :methods [[getForeign [] com.eventstore.dbclient.ReadAllOptions]
           [authenticated [com.eventstore.dbclient.UserCredentials] void]
           [requiresLeader [boolean] void]
           [timeouts [com.eventstore.dbclient.Timeouts] void]
           [resolveLinkTos [boolean] void]
           [fromPosition [com.eventstore.dbclient.Position] void]
           [direction [com.eventstore.dbclient.Direction] void]
           [build [] com.eventstore.dbclient.ReadAllOptions]])

(defn -init []
  [[] (ReadAllOptions/get)])

(defn -getForeign [this]
  (.state this))

(defn -authenticated [this credentials]
  (-> this .state (.authenticated credentials)))

(defn -requiresLeader [this value]
  (-> this .state (.requiresLeader value)))

(defn -timeouts [this timeouts]
  (-> this .state (.timeouts timeouts)))

Given that the methods are returning void, we have to use an abomination like this with an implicit do and explicit altering of state:

1
2
3
4
5

(defn apply-base-options [builder {::options/keys [timeouts requires-leader?]
                                   ::keys [credentials]}]
  (if (some? timeouts) (.timeouts builder (->Timeouts timeouts)) nil)
  (if (some? requires-leader?) (.requiresLeader builder requires-leader?) nil)
  (if (some? credentials) (.authenticated builder (->UserCredentials credentials)) nil))

And AOT compile to make use of gen-class by adding a new subtask to build.clj:

1
2
3
4
5

(defn compile-2 [_]
  (compile nil) ;; only needed if you haven't removed the Java code
  (b/compile-clj {:basis basis
                  :src-dirs ["src"]
                  :class-dir class-dir}))

And the best thing? It doesn't even work.

1
2
3
4
5
6
7
8

java.lang.IllegalArgumentException: No matching method timeouts found taking 1 args for class com.eventstore.dbclient.ReadAllOptions
   No matching method timeouts found taking 1 args for class
   com.eventstore.dbclient.ReadAllOptions
            Reflector.java:  127  clojure.lang.Reflector/invokeMatchingMethod
            Reflector.java:  102  clojure.lang.Reflector/invokeInstanceMethod
      read_all_options.clj:   36  ae.read-all-options/-timeouts
      read_all_options.clj:   35  ae.read-all-options/-timeouts
                       nil:   -1  ae.eventstore.ReadAllOptionsCljTwo/timeouts

I can't imagine that such a setup will be useful too often, but here are a couple that come to mind:

• Java is more performant (~2×?)
• an existing codebase migrating to Clojure (à la strangler architecture)
• providing a reliable Java API
• using tools that require Java magic