cometta / lighter

REST API for Apache Spark on K8S or YARN

Lighter

Lighter is an opensource application for interacting with Apache Spark on Kubernetes or Apache Hadoop YARN. It is heavily inspired by Apache Livy and has some overlapping features.

Lighter supports:

• Interactive Python Sessions through Sparkmagic kernel
• Batch application submissions through the REST API

You can read a breaf description on how Lighter works here.

Using Lighter

• Quickstart with Minikube
• Installation on Kubernetes
• Installation on Docker
• Configuration Properties
• Configuring Sparkmagic
• Using REST API

Developing Lighter

Building

To build Docker image, containing application, you need to run

docker build -t lighter .

Spark versions

Lighter will always try to support the latest patch version for all officially recommended Apache Spark releases i.e.: 3.4.2 and 3.5.1.

Contributing

See Contribution guide

License

Lighter is MIT licensed.