jarmitage / DataFrame

C++ DataFrame -- R's and Pandas DataFrame in modern C++ using native types, continuous memory storage, and no virtual functions

DataFrame

This is a C++ statistical library that provides an interface similar to Pandas package in Python.
A DataFrame can have one index column and many data columns of any built-in or user-defined type.
You could do almost everything you could do with Pandas within the C++ syntax and type checking framework. You can add/delete any data column type, slice, run summarization functors, transpose, etc. like Pandas.

Views

• You can slice the data frame and instead of getting another data frame you can opt to get a view. A view is a data frame that is a reference to a slice of the original data frame. So if you change the data in the view the corresponding data in the original data frame will also be changed.
  

Multithreading

1. DataFrame uses static containers to achieve type heterogeneity. By default, these static containers are unprotected. This is done by design. So by default, there is no locking overhead. If you use DataFrame in a multithreaded program you must provide a SpinLock defined in ThreadGranularity.h file. DataFrame will use your SpinLock to protect the containers.
   Please see documentation, set_lock(), remove_lock(), and dataframe_tester.cc for code example.
2. In addition, instances of DataFrame are not multithreaded safe either. In other words, a single instance of DataFrame must not be used in multiple threads without protection.
3. In the meantime, DataFrame utilizes multithreading in two different ways internally:
   1. There are asynchronous versions of some methods. For example, you have both sort() and sort_async(). The latter returns a std::future that could execute in parallel.
   2. DataFrame uses multiple threads, internally and unbeknown to the user, in some of its algorithms when appropriate. User can control (or turn off) the multithreading by calling set_thread_level() which sets the max number of threads to be used. The default is 0. The optimal number of threads is a function of users hardware/software environment and usually obtained by trail and error. set_thread_level() and threading level in general is a static property and once set, it applies to all instances.
      

DateTime

• DateTime class included in this library is a very cool and handy object to manipulate date/time with nanosecond precision.

---

Example Code

using namespace hmdf;

// Define a DataFrame with unsigned long index type
typedef StdDataFrame<unsigned long> MyDataFrame;

MyDataFrame                df;
std::vector<int>           intvec = { 1, 2, 3, 4, 5 };
std::vector<double>        dblvec = { 1.2345, 2.2345, 3.2345, 4.2345, 5.2345 };
std::vector<double>        dblvec2 = { 0.998, 0.3456, 0.056, 0.15678, 0.00345,
                                       0.923, 0.06743, 0.1 };
std::vector<std::string>   strvec = { "Insight", "John Dow", "Alakazam",
                                      "Persian Prince", "Bugs Bunny" };
std::vector<unsigned long> ulgvec = { 1UL, 2UL, 3UL, 4UL, 5UL, 8UL, 7UL, 6UL }
std::vector<unsigned long> xulgvec = ulgvec;

// This is only one way of loading data into a DataFrame instance. There are
// many different ways of doing it. Please see the documentation,
// or dataframe_tester.cc
int rc = df.load_data(std::move(ulgvec),  // Index column
                      std::make_pair("int_col", intvec),
                      std::make_pair("dbl_col", dblvec),
                      std::make_pair("dbl_col_2", dblvec2),
                      std::make_pair("str_col", strvec),
                      std::make_pair("ul_col", xulgvec));

// This is another way of loading a DataFrame
MyDataFrame       df2;
std::future<bool> fut = df2.read_async("sample_data.csv", io_format::csv);

fut.get();

// Sort the Frame by index
df.sort<MyDataFrame::TimeStamp, int, double, std::string>("INDEX", sort_spec::ascen);
// Sort the Frame by column “dbl_col_2” in descending order
df.sort<double, int, double, std::string>("dbl_col_2", sort_spec::desce);

// A functor to calculate mean, variance, skew, kurtosis, defined in
// DataFrameStatsVisitors.h file.
// You can implement your own algorithms and extend the DataFrame easily
StatsVisitor<double>  stats_visitor;
// Calculate the stats on column “dbl_col”
df.visit<double>("dbl_col", stats_visitor);

---

DataFrame Documentation

DateTime Documentation

---

DataFrame Test File
Heterogeneous Vectors Test File
Date/Time Test File

---

Contributions
License

---

Installing using CMake

mkdir [Debug | Release]
cd [Debug | Release]
cmake -DCMAKE_BUILD_TYPE=[Debug | Release] ..
make
make install

Uninstalling

cd [Debug | Release]
make uninstall

---

Performance

There is a test program dataframe_performance that should give you some sense of how this library performs. As a comparison, there is also a Pandas Python pandas_performance script that does exactly the same thing.
dataframe_performance.cc is compiled with gcc compiler with -O3 flag.
pandas_performance.py is ran with Python 3.7.
I ran both on my mac-book, doing the following:

1. Generate ~1.6 billion second resolution timestamps and load it into the DataFrame/Pandas as index.
2. Generate ~1.6 billion random numbers each for 3 columns with normal, log normal, and exponential distributions and load them into the DataFrame/Pandas.
3. Calculate the mean of each of the 3 columns.

Result:

MacBook> time python pandas_performance.py
All memory allocations are done. Calculating means ...

real  17m18.916s
user  4m47.113s
sys   5m31.901s
MacBook>
MacBook>
MacBook> time ../bin/Linux.GCC64/dataframe_performance
All memory allocations are done. Calculating means ...

real  6m40.222s
user  2m54.362s
sys   2m14.951s

The interesting part:

1. Pandas script, I believe, is entirely implemented in Numpy which is in C.
2. In case of Pandas, allocating memory + random number generation and calculating means take almost the same time.
3. In case of DataFrame 85% of the time is spent in allocating memory + random number generation.
4. You load data once, but calculate statistics many times. So DataFrame, in general, is about 8x faster than parts of Pandas that are implemented in Numpy. I leave parts of Pandas that are purely in Python to imagination.