jvalue / jayvee

Jayvee is a domain-specific language and runtime for automated processing of data pipelines

Jayvee

Jayvee is a domain-specific language (DSL) for automated processing of data pipelines. The Jayvee interpreter allows executing such data pipelines on local machines. Data engineers can use Jayvee and its interpreter to clean and preprocess data for later activities like data science or machine learning.

Roadmap

Explore a glimpse of our upcoming features in the following list. This overview is broad and subject to evolution. We're excited to share our vision of the exciting journey ahead, and we invite you to accompany us on this adventure!

• ✅ Blocks and pipes
• ✅ Simple value types and constraints
• ✅ Natively support table-based data
• ✅ Column-based transformations
• ✅ Describe blocks via builtin block types in Jayvee
• ✅ Compose logic of multiple blocks via composite block types
• ✅ Multi-file Jayvee to distribute programs over multiple files
• ✅ Jayvee formatter
• ⌛ Improve the syntax of value types (see RFC 0014)
• ⌛ Reusable libraries (with a package manager)
• ⌛ Further extractors and sinks
• 🤔 Composite value types (with multiple fields)
• 🤔 Natively support tree data (XML, JSON)
• 🤔 Valuetypes parsers (to read and write different formats)
• 🤔 Customizable invalid value handling (default value, average, median, interpolation, ...)
• 🤔 VSCode Debugger
• 🤔 Block types with multiple ports (e.g., for merging different data)

Anything missing, or you have ideas how some of the items on the list could be approached? Feel free to create and issue and share your thoughts with us!

You like the project and our vision? Then we'd appreciate your star! ⭐

Contribute

In case you would like to contribute to Jayvee, please have a look at our contribution guide.

Development Quickstart

1. Run npm ci to install the dependencies.
2. Run npm run generate to generate TypeScript code from the Jayvee grammar definition and the standard library.
3. Run npm run build to compile all projects.
4. In Visual Studio Code, press F5 to open a new window with the Jayvee extension loaded.
5. Create a new file with a .jv file name suffix or open an existing file in the directory example.
6. Verify that syntax highlighting, validation, completion etc. are working as expected.
7. Run node dist/apps/interpreter/main.js to see options for the CLI of the interpreter; node dist/apps/interpreter/main.js <file> interprets a given .jv file.

In case you run into problems, make sure to use the current LTS version of Node.js and npm.

Projects overview

Name	Description	NPM package
language-server	Jayvee language definition and language server implementation	@jvalue/jayvee-language-server
interpreter	Command line tool for interpreting Jayvee files	@jvalue/jayvee-interpreter
language-server-web-worker	Ready-to-use Jayvee language server, bundled as a Web Worker	@jvalue/jayvee-language-server-web-worker
vs-code-extension	Visual Studio Code extension for editing Jayvee files	-
docs	Website for Jayvee user documentation	-
monaco-editor	React component for editing Jayvee files	@jvalue/jayvee-monaco
execution	Shared code for Jayvee extensions and the interpreter	-
extensions/std	Standard Jayvee extension consisting of the extensions below	-
extensions/rdbms	Jayvee extension for relational databases	-
extensions/tabular	Jayvee extension for tabular data	-

Scripts

Building all projects

npm run build

Linting all projects

npm run lint

Formatting project files via Nx

npm run format

Testing all projects

npm run test

Generating TypeScript code from the grammar definition

npm run generate

Examples

Load data about cars into a local SQLite DB

npm run example:cars

Load GTFS data into a local SQLite DB

npm run example:gtfs

Load data about electric vehicles into local Postgres and SQLite DBs

1. Start the Postgres database locally

docker compose -f ./example/docker-compose.example.yml up

2. Run the Jayvee model

npm run example:vehicles