abs-tudelft / fast-p2a

An FPGA-based Parquet-to-Arrow converter

fast-p2a

Status

fast-p2a supports plain and delta encoding for primitives, and delta-len encoding for strings. Many changes were performed only on the oc-accel platform. Should check whether everything still works on the other platform(s).

Decompression

There are now 2 snappy decompressors available:

1. vhsnunzip has a 64 bits wide datapath
2. FPGA-Snappy-Decompressor has a 512 bits wide datapath vhsnunzip is a smaller circuit and uses 1 URAM. FPGA-Snappy-Decompressor is larger, and uses 16 BRAMs to achieve the higher performance. It seems to work in simulation, but does not do so on the board. It has some challenges such as needing to clear valid bits in BRAMs after reset and in between pages.

Usage

Modelsim

For any testbench:

1. Source env.sh in fletcher repo
2. vsim
3. do /path/to/fletcher.tcl
4. add_fletcher (if axi_top is also included don't forget to add_sources on the fletcher axi files)
5. compile_sources
6. compile
7. simulate Many of the testbenches have a corresponding Python script for generating input and validation data

OpenCAPI (oc-accel)

1. point ACTION_ROOT to one of the oc-accel platform examples in the snap_env.sh file
2. run the oc-accel flow using ./ocaccel_workflow.py
3. when the simulation terminal window opens, run the host code (which is automatically compiled in the flow) in the sw subdir
4. To view the simulation waveform after running your tests, close the terminal and your the original terminal run ./display_traces

AWS

Project Setup

1. Source hdk_setup.sh in aws-fpga repo
2. Copy any of the existing projects in platforms/aws-f1
3. Change start_build_on_cluster.sh for the correct CL_DIR
4. Change build/scripts/encrypt.tcl to include any additional VHDL files
5. Change verif/scripts/top_vhdl.vivado.f to include any additional VHDL files
6. Run generate_ip.sh

Simulate for AWS

For testbenches in verif/tests:

1. Source env.sh in fletcher repo and fast-p2a repo
2. source hdk_setup.sh in aws-fpga repo
3. In verif/scripts make TEST={name_of_sv_testbench}
4. For gui (in verif/sim/vivado/{test_dir}) run xsim tb --gui

Synthesize for AWS

1. Source start_build_on_cluster.sh (with or withour -foreground switch in script)
2. Install aws cli tools
3. Run aws s3 cp {design_file} {s3://your_bucket/your_dir} (design_file in build/checkpoints/to_aws (.tar))
4. Run aws ec2 create-fpga-image --name "hw_name" --description "hw_desc" --input_storage_location "Bucket=your_bucket,Key=your_dir/your_designfile" --logs-storage-location "Bucket=your_bucket,Key=your_dir"