This is the main repo for the Streamdal project.
Streamdal is a combination of open source services and SDKs that enable real-time:
- Data observability (think
tail -f
for your data!) - Data governance & enforcement
- Data quality monitoring
- Data transformation + obfuscation + masking + stripping
.. and a bunch of other things.
Live Demo
You don't have to install the server, the console or instrument any of your apps to see Streamdal in action. We've got a live demo :)
DEMO.STREAMDAL.COM
The demo will showcase:
- Tail
- Select any Component and start
Tail
request to watch live output
- Select any Component and start
- Welcome Service/Emailer
- View how attached
Mask dot com emails
pipeline obfuscatesrecipient
field if field contains.com
- View how attached
- Signup Service/Recorder
- Observe attached
Scrub ICMP
pipeline changingobject.ipv4_address
ifobject.protocol
equals "ICMP"
- Observe attached
- Observe Throughput
- Watch the read/write-per-second throughput for any component
- Data Graph
- Use the service map to see all services, operations and components and how they relate to each other
- The data graph is the "node map" of all services and components
- Schema Inference
- View the inferred JSON Schema for all consumers and producers
Each one of these actions is occurring in real-time!
You can read more about how this is achieved in the "how does it work?" docs.
Getting Started
Getting started consists of two steps:
- Installing the server, console and their dependencies
- Instrumenting your code with one of our SDKs
Install
The easiest way to get Streamdal running is via curl | bash
:
curl -sSL https://sh.streamdal.com | bash
- The install script will verify that you have
git
,docker
anddocker-compose
installed - The install script will clone this repo to
~/streamdal
- The install script will bring up all components via
docker-compose
Once done:
🎉 Openhttp://localhost:8080
in your browser! 🎉
You should be presented with a beautiful (but empty) UI! To populate it, we will need to instrument some code. Onto the next section!
For alternative installation methods, check the install dir.
Instrument
Once you've installed the server and console, you can instrument your code using one of our SDKs.
To see an example of a complete instrumentation, take a look at the Go demo client that is bundled with the server.
package main
import (
"context"
"github.com/streamdal/go-sdk"
)
func main() {
sc, _ := streamdal.New(&streamdal.Config{
ServerURL: "streamdal-server.svc.cluster.local:8082",
ServerToken: "1234",
ServiceName: "billing-svc",
ShutdownCtx: context.Background(),
})
resp, _ := sc.Process(context.Background(), &streamdal.ProcessRequest{
OperationType: streamdal.OperationTypeConsumer,
OperationName: "new-order-topic",
ComponentName: "kafka",
Data: []byte(`{"object": {"field": true}}`),
})
}
from streamdal import (OPERATION_TYPE_CONSUMER, ProcessRequest, StreamdalClient, StreamdalConfig)
client = StreamdalClient(
cfg=StreamdalConfig(
service_name="order-ingest",
streamdal_url="streamdal-server.svc.cluster.local:8082",
streamdal_token="1234",
)
)
res = client.process(
ProcessRequest(
operation_type=OPERATION_TYPE_CONSUMER,
operation_name="new-order-topic",
component_name="kafka",
data=b'{"object": {"field": true}}',
)
)
import { OperationType, Streamdal } from "@streamdal/node-sdk";
export const example = async () => {
const streamdal = new Streamdal({
streamdalUrl: "localhost:8082",
streamdalToken: "1234",
serviceName: "test-service-name",
pipelineTimeout: "100",
stepTimeout: "10",
});
const result = await streamdal.processPipeline({
audience: {
serviceName: "test-service",
componentName: "kafka",
operationType: OperationType.PRODUCER,
operationName: "kafka-producer",
},
data: new TextEncoder().encode(JSON.stringify({ key: "value" })),
});
};
Important
These are basic, minimal examples and should NOT be used in production code.
Refer to the instrumentation docs for more thorough directions.
How Does It Work?
Streamdal consists of three main components:
The basic flow is that you install the server and console and wrap any reads or writes in your app with our SDK. Once that's done, you will be able to see the data your app is reading or writing in the console (or use the CLI).
You will also be able to enforce rules on your data (such as "this should be
valid JSON", "message should contain a field called foo
", "strip all email
addresses" and so on).
Important
For a more in-depth explanation of the flow and the various components, visit our docs.
Community
We're building Streamdal in the open and we'd love for you to join us!
Join our Discord!
Resources
Getting Help
Stuck? Something not working right? Have questions?
- First and easiest way to get help is to join our Discord
- If you're not in the mood to chat - there's docs
- If all else fails, open an issue!
Manifesto
To get a better understanding of what we're trying to do, our ethos, principles, and really, our state of mind - check out our manifesto.
Roadmap
You have control over what we're building - our roadmap is 100% public!
Feel free to stop by to discuss features, suggest new ones or just keep an eye on what's in the pipeline.
Contributing
We ❤️ contributions! But... before you craft an amazing PR, please read through our contributing docs.
License
This project is licensed under the Apache-2.0
license.
See the LICENSE file for more info.