andrioni / nessie

Nessie provides Git-like capabilities for your Data Lake

Project Nessie

Project Nessie is a system to provide Git like capability for Iceberg Tables, Delta Lake Tables, Hive Tables and Sql Views.

More information can be found at projectnessie.org.

Using Nessie

You can quickly get started with Nessie by using our small, fast docker image.

docker pull projectnessie/nessie
docker run -p 19120:19120 projectnessie/nessie

Then install the Nessie CLI tool

pip install pynessie

From there, you can use one of our technology integrations such those for

• Spark
• Hive

Have fun! We have a Google Group and a Slack channel we use for both developers and users. Check them out here.

Building and Developing Nessie

Requirements

• JDK 11 or higher: JDK11 or higher is needed to build Nessie (artifacts are built for with Java 8)

Installation

Clone this repository and run maven:

git clone https://github.com/projectnessie/nessie
cd nessie
./mvnw clean install

Delta Lake artifacts

Nessie required some minor changes to Delta for full support of branching and history. These changes are currently being integrated into the mainline repo. Until these have been merged we have provided custom builds in our fork which can be downloaded from a separate maven repository.

Distribution

To run:

1. configuration in servers/quarkus-server/src/main/resources/application.properties
2. execute ./mvnw quarkus:dev
3. go to http://localhost:19120

UI

To run the ui (from ui directory):

1. If you are running in test ensure that setupProxy.js points to the correct api instance. This ensures we avoid CORS issues in testing
2. npm install will install dependencies
3. npm run start to start the ui in development mode via node

To deploy the ui (from ui directory):

1. npm install will install dependencies
2. npm build will minify and collect the package for deployment in build
3. the build directory can be deployed to any static hosting environment or run locally as serve -s build

Docker image

When running mvn clean install a docker image will be created at projectnessie/nessie which can be started with docker run -p 19120:19120 projectnessie/nessie and the relevant environment variables. Environment variables are specified as per https://github.com/eclipse/microprofile-config/blob/master/spec/src/main/asciidoc/configsources.asciidoc#default-configsources

AWS Lambda

You can also deploy to AWS lambda function by following the steps in servers/lambda/README.md