API Docs

• Basic Information
• dataframe
• print-df
• preview
• get-col-names
• rename-col
• filter
• set-type
• set-parser
• set-formatter
• operate (In-place modification)
• operate (Column generating)
• group-by
• aggregate
• sort (Deprecated)
• inner-join / left-join / right-join / outer-join
• rolling-join-forward / rolling-join-backward
• compute

See also extensions for more native Clojask functions.

Basic Information

• The APIs below are defined in namespace clojask.dataframe.

• Most dataframe manipulation operations are performed lazily (except for sort and join). They will be executed all at once when compute is called.

• By default (except for Excel input), all columns are assigned with the data type string when the dataframe is initialized.

• [ ] surrounding the argument indicates an optional operation.

• Without further specification, the return of all these functions is the resultant Clojask dataframe with type clojask.dataframe.classes.DataFrame . Therefore, you can pipeline these functions with -> macros.

  (-> (dataframe "xxx.csv")
      (set-type "Colx" "int")
      (operate inc "Colx")
      ...
      (compute 8 "xxx-modified.csv"))
  

dataframe

Defines the dataframe and returns clojask.dataframe.classes.DataFrame

;; defines df as a dataframe from dataframe.csv file
(def df (dataframe "resources/Employee.csv"))

;; define df to be a dataframe with customized seperator
(require '[clojask-io.input :as input])
(def df (dataframe (fn [] (input/read-file "resources/Employees.csv" :sep "," :output true :stat true))))

;; define df with a lazy sequence
(def df (dataframe (fn [] (map vector (take 1000 (iterate inc 1)) (take 1000 (repeat 1)))) :if-header false))

print-df

Provides a preview of the resulting data (column headings, datatype, and data) by performing a sample based compute on the current dataframe manipulation operations to be performed by compute. Print the result to the nice-formatted table.

(print-df x 1000 10)
;; prints 10 rows of data based on 1000 sample data entries with the current operations 

preview

Provides a preview of the resulting data (column headings, datatype, and data) by performing a sample based compute on the current dataframe manipulation operations to be performed by compute. Return the result as a vector of maps.

(preview x 1000 10)
;; [{"Employee" "1" "EmployeeName" "Alice" "Department" "11" "Salary" "300"} {...} ...]

get-col-names

Get the column names of the dataframe

Return

get-col-names returns a clojure.lang.PersistentVector

(get-col-names x)
;; columns: ["Employee" "EmployeeName" "Department" "Salary"]

rename-col

Rename the column names in the dataframe

;; columns: ["Employee" "EmployeeName" "Department" "Salary"]
(rename-col x "Department" "new-Department")

filter

Filter the dataframe by rows.

Example

(filter x "Salary" (fn [salary] (<= salary 800)))
;; this statement deletes all the rows that have a salary larger than 800
(filter x ["Salary" "Department"] (fn [salary dept] (and (<= salary 800) (= dept "computer science"))))
;; keeps only people from computer science department with salary not larger than 800

set-type

Set the data type of a column. As a result, the value will be parsed as the assigned data type when it is used in any subsequent operations.

Example

;; set data type of the column "Salary" to be double 
(set-type x "Salary" "double")

set-parser

A more flexible way to set type by specifying the customized parser.

Example

;; parse all the values in "Salary" with this function
(set-parser x "Salary" #(Double/parseDouble %))

set-formatter

A more flexible way to set type by specifying the customized formatter.

Example

;; parse all the values in "Salary" with this function
(set-parser x "Salary" #(Double/parseDouble %))

operate (In-place modification)

In-place modification on a single column

Example

;; set data type as double
(set-type x "Salary" "double")
;; take the negative of the column "Salary"
(operate x - "Salary")

operate (Column generating)

Calculate the result and store in a new column

Example

(operate x str ["Employee" "EmployeeName"] "new")
;; concats the two columns into the "new" column

group-by

Group the dataframe by some specific columns (always used together with aggregate), or group the dataframe by function output(s)

Example

;; group by one or more columns
(group-by x ["Department"])
(group-by x ["Department" "DepartmentName"])

Group-by Keys Specification

Group-by Functions Specification

• Take one argument
• Return type: int / double / string

Pair group-by functions with group-by keys

One general rule is to put the group-by function and its corresponding column name together.

Examples

(defn rem10
  "Get the reminder of the num by 10"
  [num]
  (rem num 10))

(group-by x [rem10 "Salary"])
;; or
(group-by x [[rem10 "Salary"]])

If no group-by function, the column name can be alone.

(group-by x "Salary")
;; or
(group-by x ["Salary"])

You can also group by the combination of keys. (Use the above two rules together)

(group-by x [[rem10 "Salary"] "Department"])
;; or
(group-by x [[rem10 "Salary"] ["Department"]])

aggregate

Aggregate the dataframe(s) by applying some functions. The aggregation function will be applied to every column registered in sequence.

Example

(require '[clojask.api.gb-aggregate :as gb-agg])
(require '[clojask.api.aggregate :as agg])
;; aggregate without group, apply aggregation to the whole dataframe
(aggregate x agg/max "Salary")
(group-by x "Department")
;; get the max/min of the selected column(s) of each group
(aggregate x gb-agg/max ["Salary"] ["Salary-max"])
(aggregate x gb-agg/min ["Employee" "EmployeeName"] ["Employee-min" "EmployeeName-min"])

Custom functions can be made for aggregation. Please refer to Aggregation Function API for additional details

The keys used in specifying the aggregate operation are identical to the group-by function

sort (Deprecated)

Immediately sort the dataframe

Example

;; sort by "Salary" in ascending order
(sort x ["+" "Salary"] "path/output.csv")

inner-join / left-join / right-join / outer-join

Inner / left / right join two dataframes on specific columns

Remarks:

The registered operations and filters (like compute) will be automatically pipelined. You could think of join as as an operation that first computes the two dataframes then joins them together.

Only compute will be able to be performed after joing functions

Example

(def x (dataframe "path/to/a"))
(def y (dataframe "path/to/b"))

(def z (inner-join x y ["col a 1" "col a 2"] ["col b 1" "col b 2"]))
(compute z 8 "path/to/output")
;; inner join x and y

(def z (left-join x y ["col a 1" "col a 2"] ["col b 1" "col b 2"]))
(compute z 8 "path/to/output")
;; left join x and y

(def z (right-join x y ["col a 1" "col a 2"] ["col b 1" "col b 2"]))
(compute z 8 "path/to/output")
;; right join x and y

Return

A Clojask.JoinedDataFrame

Unlike clojask.classes.DataFrame.DataFrame, it only supports three operations:

• print-df
• get-col-names
• compute

This means you cannot further apply complicated operations to a joined dataframe. An alternative is to first compute the result, then read it in as a new dataframe.

rolling-join-forward / rolling-join-backward

Rolling join two dataframes on columns. Forward will find the largest of the smaller in b while backward, while backward find the smallest of the larger in b.

You can refer to here for more details about rolling joins.

Example

(def x (dataframe "path/to/a"))
(def y (dataframe "path/to/b"))

(rolling-join-forward x y ["Employee"] ["Employee"] "Salary" "Salary")
(rolling-join-forward x y ["Employee"] ["Employee"] "Salary" "Salary" :limit (fn [a b] (if (> (- a b) 10) false true)))
;; if the salary of a - b exceeds 10, we will not join the two rows

Return (Same as inner-join)

A Clojask.JoinedDataFrame

Unlike clojask.classes.DataFrame.DataFrame, it only supports three operations:

• print-df
• get-col-names
• compute

This means you cannot further apply complicated operations to a joined dataframe. If you need so, an alternative is to first compute, then read the result in as a new dataframe.

compute

Compute the result. The pre-defined lazy operations will be executed in pipeline, ie the result of the previous operation becomes the argument of the next operation.

Return

A clojask.classes.DataFrame.DataFrame, which is the resultant dataframe. / A vector of vectors, which represent each row, if output path = nil.

Example

(compute x 8 "output.csv" :exception true)
;; computes all the pre-registered operations

(compute x 8 "output.csv" :select "col a")
;; only select column a

(compute x 8 "output.csv" :order true)
;; make sure the order of the output is the same of the input

(compute x 8 nil :in-memory true)
;; compute the dataframe in memory and store the dataframe also in memory

(compute x 8 "output.csv" :select ["col b" "col a"])
;; select two columns, column b and column a in order

(compute x 8 "output.csv" :exclude ["col b" "col a"])
;; select all columns except column b and column a, other columns are in order

(compute x 3 "output.csv" :output (fn [wtr rows] (doseq [row rows] (.write wtr (str (str/join ", " row) "\n")))))
;; seperate each value in the row with ", "; seperate each row by "\n"

(compute x 8 "output.csv" :melt (fn [row] (map concat (repeat (take 2 x)) (take-last 2 x))))
;; each result row becomes two rows
;; [a b c d] => [[a b c] [a b d]]