REPL vs CLI: IDE wars

I’ve been thinking recently that Clojure REPL and CLI are both IDEs, and one might be better than another at being an IDE, so I decided to collect some scattered thoughts about the subject because I think this would be interesting to discuss.

The target audience of this post is software developers writing projects in Clojure.

Intro

Let’s start with what I mean by abbreviations used throughout the post:

I will not be the first to argue that CLIs are IDEs. You can use CLI for file management, text editing, building executables, debugging, version control, and more.

It might be a bit of a stretch to say that REPL is an IDE — especially when my definition of REPL for this post already includes some other text editor of choice 😄. In this post, I want to mostly focus on the “building executables” area of IDEs, e.g. helper tasks (like building/testing/deploying/configuring environment) that programmers usually use CLI for when developing Clojure projects. I would argue that even today REPL can be used as IDE, there is also seems to be a movement towards making more tooling a first-class citizen at the REPL, and REPL — when used as an IDE — might work much better than CLIs. Let’s compare them to see why I think so.

Tool installation and project setup

brew install foo or apt install bar are easy but complect dependency installation. If your project setup depends on foo and bar being in your IDE of choice, they should be automatically installed when developing the project instead of being mentioned in the readme. CLI dependency management is implicit and not reproducible: you can’t (probably?) make your shell auto-install some tools when you work in the context of your project.

Required tools will be forgotten to be mentioned in the readme. Installation instructions will differ for different operating systems. Default versions of tools provided by the OS will be incompatible with versions required by the project. All these issues happened to me while I was writing this post!

REPL tools are specified in deps.edn and automatically pulled by clj. Explicit, unforgotten, local to the project, reproducible.

For example, let’s have a look at a project that is available both on the command line and in the REPL: borkdude/jet.

Here is how you install it for your project using CLI tooling:

$ bash <(curl -s https://raw.githubusercontent.com/borkdude/jet/master/install)

or using brew:

$ brew install borkdude/brew/jet

Windows instructions will be different:

scoop bucket add scoop-clojure https://github.com/littleli/scoop-clojure
scoop bucket add extras
scoop install jet

If you use it in a project at work, your colleagues now have to repeat your jet installation steps. If there is a new version that your project requires, you now need to update it using another CLI command and remind everyone on your team to do the same.

Here is how you install it when your IDE is REPL:

;; deps.edn
{:deps {borkdude/jet {:mvn/version "0.0.12"}}}

Your colleagues can start using this tool as soon as you commit it. If there is an upgrade, you commit the new version and your colleagues will get it automatically when they pull the latest changes from the repo. Unlike with command-line tools, installation and updates are the same on Windows, Linux, and macOS.

Dynamic runtime

In the command line, I can install tools from the internet and start using them in the same shell session. When using Clojure REPL, I need to add the dependency to deps.edn and then restart the REPL.

Except I don’t: I use add-lib branch of tools.deps.alpha that allows me to add dependencies dynamically at the REPL and then start using them immediately, just like in the shell.

Scripting

Shell scripts are error-prone by default: any failure is ignored and the execution continues. Don’t forget the set -euo pipefail at the beginning of your script. What does -euo means? I don’t know, I copy-pasted it and man set said there is no manual entry for set.

REPL scripting is fantastic because Clojure is a fantastic language even when used for imperative do-this do-that pipelines. If you get an error, the execution stops by default and you get a stack trace. You have the power of the REPL to debug, profile and develop your script.

If I’m writing Clojure for the main program, why use different inferior languages for scripts around the main program? After all, it increases cognitive complexity and I’d rather spend my complexity budget on solving problems…

Is it because there are simply too few tools available in the REPL? With modern tooling embracing clj-exec invocation that works both at the command line and in the REPL, there are more and more tasks I can do in the REPL. Running tests — io.github.cognitect-labs/test-runner. Building jars — com.github.seancorfield/depstar. If there is no tool for the JVM, you can always shell out from the REPL using clojure.java.shell ns.

Maybe it’s because of the inertia? Users of verbose heavy-weight programming languages like Java probably would never consider writing smaller scripts in the main program language, but Clojure is definitely as good for writing small programs as it is for writing big programs.

Or maybe it’s the startup time?

The startup time

This is where shell scripts shine, right? Clojure tools take seconds to start while command line tools take millis. Except this is a false comparison: when I work in the REPL, command invocation is a function call, not starting up the whole JVM. Let’s compare how fast the startup time of a CLI jet vs REPL jet!

CLI:

time for i in {1..1000}; do echo '{:a 1}' | jet --to json > /dev/null ; done
real    0m6.337s
user    0m4.107s
sys     0m2.838s

REPL:

$ clj -Sdeps '{:deps {borkdude/jet {:mvn/version "0.0.12"}}}'
Clojure 1.10.3
user=> (require 'jet.formats)
nil
user=> (time (dotimes [_ 1000] (jet.formats/generate-json {:a 1} false)))
"Elapsed time: 37.1361 msecs"
nil

6 seconds vs 37 milliseconds! Function startup time and JIT FTW 🚀🚀🚀

Command editing

When typing command-line invocations, the “editor”, a.k.a. the command line input is clunky: pressing up in multi-line command will go through history instead of moving the cursor up, most common modern shortcuts for editing and selecting don’t work or inconsistent. Ctrl+Left to jump words and holding Shift to extend selection is shell-dependent (i.e. it works in PowerShell and does not work in Bash).

In Clojure REPL, command “editor” is your favorite text editor that works as you want instead of working as compatible with terminals from the eighties. Completion, documentation, syntax highlighting is there. History — contextual! — is there if you use rich comment forms.

Program arguments

CLI args accept only one data type as an argument: an array of strings. Each command-line tool parses this array in its own way. There are GNU Program Argument Syntax Guidelines, but they are not enforced in any way and modern command-line tools don’t always follow them either. Figuring out what are the expected arguments is not always trivial.

Clojure functions accept a big variety of data types, which is more expressive. Function invocation is regular — there is only one way to provide args to a function.

Command composition

You can glue shell commands together with pipes and $(command interpolation).

Clojure has a function composition that is similar to both interpolation and piping. Another benefit in terms of composition at the REPL is that objects in the VM are more interactive than commands in the shell.

Let me illustrate this with another example: Datomic command line tools. These tools are distributed as a command-line executable. When I’m developing an application that connects to the Datomic cloud, before starting the REPL I need to start a proxy that allows the Datomic client on my machine to connect to the Datomic cluster in the cloud.

If you look at the executable, you will see that the entire source of it is a clj main function invocation:

#!/usr/bin/env bash
clojure \
-Sdeps "{:deps {com.datomic/tools.ops {:mvn/version \"0.10.82\"}}}" \
-m datomic.tools.ops "$@"

So this shell script is also available as Apache 2.0 licensed Clojure library. When using it as a library, I can start the proxy from the REPL, e.g.

(datomic.tools.ops.ssh/access 
  {:port 8182
   :ssho ["IdentitiesOnly=yes"]
   :type :client
   :system "my-datomic-system"})

The current implementation will block the calling thread until the proxy disconnects, but with a small modification it will be able to return an instance of java.lang.Process — a process handle that allows me to query it, stop it, add exit callbacks — all from the REPL. This makes it possible to inject the proxy startup in a code that initializes the Datomic client so I won’t need to think about proxying at all — no more “run this command in a separate terminal before launching the REPL”!

Wouldn’t it be nice if datomic.tools.ops.ssh/access didn’t block by default? 😛

Instead of a summary

When I was writing this post, I considered converting all my Bash and PowerShell scripts to pure Clojure to prove my point, but I haven’t done it because they work just fine as they are. CLI tools are neither broken nor bad. I’m not proposing to Rewrite It In Clojure™ — there will always be CLI, and there will always be command-line tools that are required for the development, but minimizing the amount of these tools is good for the project and developer. Maybe one day git, java, clj and ssh will be enough for every Clojure project…

Right now using REPL as IDE might be rough around the edges. I need rich comment forms for history and convenience. I need an unofficial tools-deps branch to add dependencies dynamically. The amount of clj-exec-friendly tooling is small.

Bit it’s growing. Cognitect will soon release tools-build that expands the REPL toolbelt. There are rumors add-lib branch will end up in some future version of Clojure. When that happens, I think the local maxima for the project tooling will be much higher.

What do you think? Discuss here, on Reddit or Hacker News.