About & Benchmarks
Where Clojask comes from, how fast it is, and how it compares with other big-data tools.
On this page
The project
Clojask is an open-source Clojure library developed at HKU Business School, as part of clojure-finance. It aims to make data that doesn't fit in memory as easy to work with as a small dataframe, on a single machine.
It is made of three parts:
- clojask: the dataframe API and the engine that runs it.
- clojask-io: reading and writing text and Excel files, installed together with clojask.
- clojask-examples: runnable example programs, shown on the Examples page.
Talk at the Clojure data-recur meeting
The general state of the project as of October 2022, presented at a Clojure data-recur meeting (from 9:09 to 58:02):
Benchmarks
Clojask against Dask, the Python library for larger-than-memory dataframes, with 4 workers each. Times are in seconds; N is the number of rows in the CSV file.
| Operation | Dask N = 1.8M | Dask N = 3.6M | Dask N = 80M | Clojask N = 1.8M | Clojask N = 3.6M | Clojask N = 80M |
|---|---|---|---|---|---|---|
| Element-wise operation | 119.3 | 261.3 | N/A* | 72.3 | 133.3 | 1836.6 |
| Row-wise selection | 115.0 | 232.0 | N/A* | 67.9 | 145.6 | 1757.5 |
| Aggregation | 116.0 | 226.7 | N/A* | 58.6 | 112.1 | 1236.9 |
| Group-by aggregate | 116.7 | 229.3 | N/A* | 459.4 | 803.1 | 25860.0 |
| Left join | 114.7 | 248.7 | N/A* | 1174.4 | 2310.2 | 14007.9 |
| Inner join | 116.7 | 242.0 | N/A* | 1138.8 | 2768.5 | 21609.3 |
| Rolling join | – | – | – | 2812.1 | 3943.1 | > 28800 |
The faster time in each comparable pair is in bold. * Dask could not finish any operation on 80M rows within 7 hours. The rolling join was not benchmarked for Dask.
How to read these numbers:
- Clojask was faster at element-wise operations, filtering and whole-file aggregation, and it still finished at 80M rows, where Dask did not.
- Dask was faster at group-by and joins on the smaller files.
- Every time includes loading the libraries, reading the CSV file, and writing the result to a single CSV file.
Warning These numbers were measured in 2021 with Clojask 1.x, which ran on Onyx. Since 2.1.0, Clojask uses its own thread pool, which has been faster in every operation tested so far; the benchmarks have not been rerun yet.
The benchmarks ran on macOS (Darwin 20.4.0) on an x86_64 machine with 8 GB of RAM. The code is in the clojask repository:
Compared with Hadoop and Spark
Clojask borrows ideas from the big distributed systems, but runs inside one JVM.
Hadoop MapReduce
| Feature | Clojask | Hadoop MapReduce |
|---|---|---|
| Larger-than-memory source files | ✅ | ✅ |
| Writes intermediate results to temporary files | ✅ | ✅ |
| MapReduce paradigm | ✅ | ✅ |
| Join, filter, aggregate, etc. on large files out of the box | ✅ | ❌ |
Spark
| Feature | Clojask | Spark |
|---|---|---|
| Builds a DAG of operations and optimizes it | ❌ | ✅ |
| Join, filter, aggregate, etc. | ✅ | ✅ |
| Caches intermediate results in memory between stages | ❌ | ✅ |
| Minimal memory use | ✅ | ❌ |
Design history
Up to version 2.0.x, Clojask ran its pipelines on Onyx, a distributed computation framework, inside one process. Version 2.1.0 replaced Onyx with a small thread pool of its own: one thread reads the input in batches, a pool of worker threads runs the operations, and the calling thread writes the output. That removed a large dependency and the JVM flags it needed, and made every operation tested so far faster. Core Concepts describes the current design.
The diagram below shows the original design. The steps inside each stage are the same today, but the "nodes" are now threads, and add-parser and add-formatter have become set-parser and set-formatter. Click it to open the full-size image.
