This repo houses all of the assets for the Streamlio sandbox, a version of the Streamlio real-time messaging, compute, and stream storage platform tailored to local experimentation and development. The Streamlio sandbox combines three platforms---Apache Pulsar (incubating), and Apache BookKeeper---into a single Docker image, along with an example word count Pulsar Function. That function does the following:
- consumes randomly chosen sentences published to a Pulsar topic by a Pulsar producer
- splits incoming sentences into individual words
- counts each word into an aggregated time interval
- periodically publishes those counts to a Pulsar topic that is then read by a Pulsar consumer.
The producer.py and consumer.py scripts are processes that you'll run outside the Docker container; the gray section in the middle shows everything inside the container.
The consumer.py and producer.py Python scripts require the pulsar-client library to run. You can install it using pip:
pip install pulsar-client --upgradeYou have two options for running the Streamlio sandbox. You can run it using Docker or on a Kubernetes cluster.
In order to run the Streamlio sandbox, you'll need to have a Docker engine running locally. You can find installation instructions for your platform below:
MacOS: Download Link
Linux: Download Link
## Run the sandbox image using Docker
The Docker image for the Streamlio sandbox is available via [Docker Hub](https://hub.docker.com/r/streamlio/sandbox/). You can run it using this command:
```bash
$ docker run -d \
--name streamlio-sandbox \
-p 6650:6650 \
-p 8080:8080 \
-p 8000:8000 \
streamlio/sandbox
If you'd prefer to build the Docker image from source rather than pulling from Docker Hub, see the instructions below.
You can check to make sure the image is running using docker ps, which should output something like this:
CONTAINER ID IMAGE ...
c90100be5ea8 streamlio/sandbox ...
You can run the Streamlio sandbox on a running Kubernetes cluster using just a few kubectl commands. First, fetch the YAML configuration:
$ wget https://raw.githubusercontent.com/streamlio/sandbox/master/kubernetes/streamlio-sandbox.yamlThen apply the configuration to your cluster:
$ kubectl apply -f streamlio-sandbox.yamlThe streamlio/sandbox Docker image is fairly large, so it may take a minute or more to pull the image and start it up. You can check on the progress of the installation at any time:
$ kubectl get pods -l app=streamlio-sandboxOnce the STATUS changes to RUNNING, you can connect to the running pod using kubectl's port-forward command. First, get the name of the running pod by running the kubectl get command from above. Then open up the ports to that pod:
$ kubectl port-forward <pod name> 6650:6650 8080:8080 8000:8000As you can see, the image requires several open ports. The table below examples what each port is used for.
| Component | Host Port | Docker Container Port |
|---|---|---|
| Pulsar Broker | 6650 | 6650 |
| Pulsar Admin | 8080 | 8080 |
| Pulsar UI | 8000 | 8000 |
There are two Python scripts in the sandbox that act as a Pulsar producer and consumer, respectively. You can fetch them like this:
$ wget https://raw.githubusercontent.com/streamlio/sandbox/master/producer.py
$ wget https://raw.githubusercontent.com/streamlio/sandbox/master/consumer.pyIf the Docker image is currently running, start up the consumer (just make sure to wait a few seconds after you've started up the Docker image):
$ python consumer.pyIf you get an error along the lines of
Exception: Pulsar error: ConnectError, try waiting a few seconds and retrying. If that doesn't work, rundocker psto check on the status of the running image.
Initially, no messages will be published to the topic that the consumer is listening on. This will change when you start up the producer:
$ python producer.pyOnce you start up the producer, you should begin to see messages like this via the consumer:
Received message: {"a": 273,"ago": 273,"am": 273,"an": 273,"and": 547,"apple": 273,"at": 273,"away": 273,"cow": 274,"day": 273,"doctor": 273,"dwarfs": 274,"four": 273,"i": 273,"jumped": 274,"keeps": 273,"moon": 274,"nature": 273,"over": 274,"score": 273,"seven": 547,"snow": 274,"the": 1095,"two": 273,"white": 274,"with": 273,"years": 273}
Received message: {"a": 284,"ago": 284,"am": 283,"an": 284,"and": 568,"apple": 284,"at": 283,"away": 284,"cow": 283,"day": 284,"doctor": 284,"dwarfs": 284,"four": 284,"i": 283,"jumped": 283,"keeps": 284,"moon": 283,"nature": 283,"over": 283,"score": 284,"seven": 568,"snow": 284,"the": 1134,"two": 283,"white": 284,"with": 283,"years": 284}
Received message: {"a": 294,"ago": 294,"am": 293,"an": 294,"and": 588,"apple": 294,"at": 293,"away": 294,"cow": 294,"day": 294,"doctor": 294,"dwarfs": 294,"four": 294,"i": 293,"jumped": 294,"keeps": 294,"moon": 294,"nature": 293,"over": 294,"score": 294,"seven": 588,"snow": 294,"the": 1176,"two": 293,"white": 294,"with": 293,"years": 294}
Received message: {"a": 304,"ago": 304,"am": 303,"an": 304,"and": 608,"apple": 304,"at": 303,"away": 304,"cow": 305,"day": 304,"doctor": 304,"dwarfs": 304,"four": 304,"i": 303,"jumped": 305,"keeps": 304,"moon": 305,"nature": 303,"over": 305,"score": 304,"seven": 608,"snow": 304,"the": 1218,"two": 303,"white": 304,"with": 303,"years": 304}
The producer, in turn, should be producing output like this:
Sending message - four score and seven years ago
Sending message - i am at two with nature
Sending message - i am at two with nature
Sending message - four score and seven years ago
Sending message - an apple a day keeps the doctor away
Sending message - the cow jumped over the moon
Sending message - snow white and the seven dwarfs
If your output looks something like that, then the sandbox is working! That means that you now have an end-to-end, real-time, stateful processing platform powered by Apache Pulsar (incubating), Heron, and Apache BookKeeper running on your laptop.
curl http://localhost:8080/admin/v2/functions/public/default/wordcount/status
{
"functionStatusList": [{
"running": true,
"numProcessed": "2347",
"numSuccessfullyProcessed": "2347",
"lastInvocationTime": "1530237837516",
"instanceId": "0"
}]
}You can get insight into Pulsar topics using the Pulsar Dashboard. The sandbox uses two topics: sentences and wordcount. You can get info on those topics by navigating to http://localhost:8000/stats/namespace/sample/standalone/ns1 in your browser.
The Pulsar Dashboard updates once every minute.
You can see the input and output topics in Pulsar:
You can also drill down into the stats of the input topic queue (named sentences):
We can also take a look at the wordcount topic, which contains word count results:
Once you're finished experimenting with the Streamlio sandbox, you can kill the running container:
$ docker kill streamlio-sandboxYou can also remove the container at any time:
$ docker rm streamlio-sandboxIf you prefer to build the Docker image from source rather than pulling from Docker Hub, follow the instructions below.
Maven is required to build the Heron topology from source (the topology is written in Java). Please follow the instructions here if not present on your system.
$ git clone https://github.com/streamlio/sandbox.git
$ cd sandbox$ mvn package
$ cp target/streamlio-starter-latest.jar ./docker/$ docker build docker/ -t streamlio-sandbox:latest
# Check to make sure that the image has been installed
$ docker images$ docker run -d \
--name streamlio-sandbox \
-p 6650:6650 \
-p 8080:8080 \
-p 8000:8000 \
streamlio-sandbox:latestCheck that the image is running using docker ps.