Kezdi.jl Documentation

Kezdi.jl is a Julia package that provides a Stata-like interface for data manipulation and analysis. It is designed to be easy to use for Stata users who are transitioning to Julia.^[stata]

It imports and reexports CSV, DataFrames, FixedEffectModels, FreqTables, ReadStatTables, Statistics, and StatsBase. These packages are not covered in this documentation, but you can find more information by following the links.

Getting started

Installation

To install the package, run the following command in Julia's REPL:

using Pkg; Pkg.add(url="https://github.com/codedthinking/Kezdi.jl")

Every Kezdi.jl command is a macro that begins with @. These commands operate on a global DataFrame that is set using the setdf function. Alternatively, commands can be executed within a @with block that sets the DataFrame for the duration of the block.

Example

using Kezdi
using RDatasets

setdf(dataset("datasets", "mtcars"))

@rename HP Horsepower
@rename Disp Displacement
@rename WT Weight
@rename Cyl Cylinders

@tabulate Gear
@keep @if Gear == 4
@keep MPG Horsepower Weight Displacement Cylinders
@summarize MPG
@regress log(MPG) log(Horsepower) log(Weight) log(Displacement) fe(Cylinders), robust 

Alternatively, you can use the @with block to avoid writing to a global DataFrame:

using Kezdi
using RDatasets

df = dataset("datasets", "mtcars")

renamed_df = @with df begin
    @rename HP Horsepower
    @rename Disp Displacement
    @rename WT Weight
    @rename Cyl Cylinders
end

@with renamed_df begin
    @tabulate Gear
    @keep @if Gear == 4
    @keep MPG Horsepower Weight Displacement Cylinders
    @summarize MPG
    @regress log(MPG) log(Horsepower) log(Weight) log(Displacement) fe(Cylinders), robust 
end

Benefits of using Kezdi.jl

Free and open-source

Speed

Use any Julia function

@generate logHP = log(Horsepower)

Easily extendable with user-defined functions

The function can operate on individual elements,

get_make(text) = split(text, " ")[1]
@generate Make = Main.get_make(Model)

or on the entire column:

function geometric_mean(x::AbstractVector)
    n = length(x)
    return exp(sum(log.(x)) / n)
end
@collapse geom_NPG = Main.geometric_mean(MPG), by(Cylinders)

Commands

Setting and inspecting the global DataFrame

setdf(df::Union{AbstractDataFrame, Nothing})

getdf() -> AbstractDataFrame

@names

@list

@head [n]

@tail [n]

Filtering columns and rows

@keep y1 y2 ... [@if condition]

@drop y1 y2 ...

Modifying the data

@rename oldname newname

@generate y = expr [@if condition]

@replace y = expr [@if condition]

@egen y1 = expr1 y2 = expr2 ... [@if condition], [by(group1, group2, ...)]

@collapse y1 = expr1 y2 = expr2 ... [@if condition], [by(group1, group2, ...)]

@sort y1 y2 ...

Summarizing and analyzing data

@count if condition]

@tabulate y1 y2 ... [@if condition]

@summarize y [@if condition]

@regress y x1 x2 ... [@if condition], [robust] [cluster(var1, var2, ...)]

Use on another DataFrame

@with df begin
    # do something with df
end

@with! df begin
    # do something with df
end

Differences to standard Julia and DataFrames syntax

To maximize convenience for Stata users, Kezdi.jl has a number of differences to standard Julia and DataFrames syntax.

Everything is a macro

While there are a few convenience functions, most Kezdi.jl commands are macros that begin with @.

@tabulate Gear

Comma is used for options

Due to this non-standard syntax, Kezdi.jl uses the comma to separate options.

@regress log(MPG) log(Horsepower), robust

Here log(MPG) and log(Horsepower) are the dependent and independent variables, respectively, and robust is an option. Options may also have arguments, like

@regress log(MPG) log(Horsepower), cluster(Cylinders)

Automatic variable name substitution

Column names of the data frame can be used directly in the commands without the need to prefix them with the data frame name or using a Symbol.

@generate logHP = log(Horsepower)

Automatic vectorization

All functions are automatically vectorized, so there is no need to use the . operator to broadcast functions over elements of a column.

@generate logHP = log(Horsepower)

If you want to turn off automatic vectorization, use the convenience function DNV ("do not vectorize").

@generate logHP = DNV(log(Horsepower))

The exception is when the function operates on Vectors, in which case Kezdi.jl understands you want to apply the function to the entire column.

@collapse mean_HP = mean(Horsepower), by(Cylinders)

If you need to apply a function to individual elements of a column, you need to vectorize it with adding . after the function name:

@generate words = split(Model, " ")
@generate n_words = length.(words)

The @if condition

Almost every command can be followed by an @if condition that filters the data frame. The command will only be executed on the subset of rows for which the condition evaluates to true. The condition can use any combination of column names and functions.

@summarize MPG @if Horsepower > median(Horsepower)

@generate HP_p75 = median(Horsepower) @if Horsepower > median(Horsepower)

s = @summarize Horsepower
s.p75

Handling missing values

Kezdi.jl ignores missing values when aggregating over entire columns.

@with DataFrame(A = [1, 2, missing, 4]) begin
    @collapse mean_A = mean(A)
end

returns mean_A = 2.33.

Row-count variables

The variable _n refers to the row number in the data frame, _N denotes the total number of rows. These can be used in @if conditions, as well.

@with DataFrame(A = [1, 2, 3, 4]) begin
    @keep @if _n < 3
end

Differences to Stata syntax

All commands begin with @

To allow for Stata-like syntax, all commands begin with @. These are macros that rewrite your Kezdi.jl code to DataFrames.jl commands.

@tabulate Gear
@keep @if Gear == 4
@keep Model MPG Horsepower Weight Displacement Cylinders

@if condition also begins with @

The @if condition is non-standard behavior in Julia, so it is also implemented as a macro.

@collapse has same syntax as @egen

Unlike Stata, where egen and collapse have different syntax, Kezdi.jl uses the same syntax for both commands.

@egen mean_HP = mean(Horsepower), by(Cylinders)
@collapse mean_HP = mean(Horsepower), by(Cylinders)

Different function names

To maintain compatibility with Julia, we had to rename some functions. For example, count is called rowcount, missing is called ismissing in Kezdi.jl.

Convenience functions

distinct(x::AbstractVector) = unique(x)

rowcount(x::AbstractVector) = length(collect(skipmissing(x)))

DNV(f(x))

Acknowledgements

Inspiration for the package came from Tidier.jl, a similar package launched by Karandeep Singh that provides a dplyr-like interface for Julia. Johannes Boehm has also developed a similar package, Douglass.jl.

The package is built on top of DataFrames.jl, FreqTables.jl and FixedEffectModels.jl. The @with function relies on Chain.jl by Julius Krumbiegel.

The package is named after Gabor Kezdi, a Hungarian economist who has made significant contributions to teaching data analysis.