Clojure dataframes · v2.1.3
Process data that doesn’t fit in memory.
Clojask reads a large CSV file in chunks, runs your filters, transformations, group-bys and joins on all CPU cores, and writes the result straight to a new file. You write ordinary Clojure functions; Clojask handles the streaming.
com.github.clojure-finance/clojask {:mvn/version "2.1.3"}[com.github.clojure-finance/clojask "2.1.3"](require '[clojask.dataframe :as ck])
;; 1. point at a file (only the header is read)
(def df (ck/dataframe "employees.csv"))
;; 2. describe the work
(ck/set-type df "Salary" "int")
(ck/filter df "Salary" #(< % 10000))
(ck/operate df #(quot % 10) "Salary" "Bonus")
;; 3. run it: 4 worker threads, file to file
(ck/compute df 4 "with-bonus.csv")Why Clojask?
Larger than memory
Stream a large CSV through filters and transformations in small batches, then write the result straight to a file.
Parallel on one machine
Use up to eight worker threads in one JVM. There is no cluster to set up or manage.
Plain Clojure functions
Clean, filter, group and join your data using ordinary functions on Java and Clojure values.
Lazy pipeline
Steps are recorded, checked on a sample, and run together in a single pass over the file.
Relational operations
Filter, group-by, aggregate, inner, left, right and outer joins, plus rolling joins for time series.
File in, file out
Reads CSV, TSV and other text files (and Excel through clojask-io), and writes text or .xlsx.
Is Clojask the right tool?
A good fit
- CSV files too large to load, or close to the size of your RAM
- Batch jobs: clean, filter, enrich, group and join, then write a file
- Logic you want to write as plain Clojure functions
- One machine, no cluster to run
Look elsewhere when
- The data fits in memory and you want to explore it interactively: try tech.ml.dataset or Datajure
- You need to sort a large result, or look up individual rows
- You run on Windows (Clojask supports macOS and Linux)
Explore the docs
- Get Started — download a runnable example, then follow the tour.
- Core Concepts — understand how a pipeline runs.
- API Docs · Cheat Sheet — find a function or check its arguments.
- Examples — complete programs with their inputs and outputs.
Watch the demo
A short introduction to Clojask and some of its uses, including an inner join and a group-by aggregation:
Get involved
Clojask is open source under the MIT license and developed at HKU Business School. If something doesn't work as documented, please open an issue; bug reports and pull requests are welcome.