nitisht / parseable

Cloud Native Log Storage and Management Platform

Cloud native log observability

Parseable is a lightweight, cloud native log observability engine. It can use either a local drive or S3 (and compatible stores) for backend data storage.

Parseable is written in Rust and uses Apache Arrow and Parquet as underlying data structures. Additionally, it uses a simple, index-free mechanism to organize and query data allowing low latency, and high throughput ingestion and query.

Parseable consumes up to ~80% lower memory and ~50% lower CPU than Elastic for similar ingestion throughput.

• Parseable UI Demo (Credentials: parseable,parseable) ↗︎
• Grafana Dashboard Demo ↗︎

🚀 Features

• Choose your own storage backend - local drive or S3 (or compatible) object store.
• Ingestion API compatible with HTTP + JSON output of log agents - Fluentbit ↗︎, Vector ↗︎, Logstash ↗︎ and others.
• Query log data with PostgreSQL compatible SQL.
• Grafana ↗︎ for visualization.
• Auto schema inference (schema evolution coming soon ↗︎).
• Send alerts ↗︎ to webhook targets including Slack.
• Stats API ↗︎ to track ingestion and compressed data.
• Single binary includes all components - ingestion, store and query. Built-in UI.

✅ Getting Started

Run the below command to deploy Parseable in local storage mode with Docker.

mkdir -p /tmp/parseable/data
mkdir -p /tmp/parseable/staging

docker run -p 8000:8000 \
  -v /tmp/parseable/data:/parseable/data \
  -v /tmp/parseable/staging:/parseable/staging \
  -e P_FS_DIR=/parseable/data \
  -e P_STAGING_DIR=/parseable/staging \
  parseable/parseable:latest \
  parseable local-store

Once this runs successfully, you'll see dashboard at http://localhost:8000. You can login to the dashboard default credentials admin, admin.

Create a stream

curl --location --request PUT 'http://localhost:8000/api/v1/logstream/demo' \
--header 'Authorization: Basic YWRtaW46YWRtaW4='

Send events to the stream

curl --location --request POST 'http://localhost:8000/api/v1/logstream/demo' \
--header 'X-P-META-meta1: value1' \
--header 'X-P-TAG-tag1: value1' \
--header 'Authorization: Basic YWRtaW46YWRtaW4=' \
--header 'Content-Type: application/json' \
--data-raw '[
    {
        "id": "434a5f5e-2f5f-11ed-a261-0242ac120002",
        "datetime": "24/Jun/2022:14:12:15 +0000",
        "host": "153.10.110.81", 
        "user-identifier": "Mozilla/5.0 Gecko/20100101 Firefox/64.0", 
        "method": "PUT", 
        "status": 500, 
        "referrer": "http://www.google.com/"
    }
]'

Query the stream

You can see the events in Parseable UI, or use the below curl command to see the query response on CLI.

NOTE: Please change the startTime and endTime to the time range corresponding to the event you sent in the previous step.

curl --location --request POST 'http://localhost:8000/api/v1/query' \
--header 'Authorization: Basic YWRtaW46YWRtaW4=' \
--header 'Content-Type: application/json' \
--data-raw '{
    "query":"select * from demo",
    "startTime":"2023-01-09T00:00:00+00:00",
    "endTime":"2023-01-09T23:59:00+00:00"
}'

📚 Documentation

• Complete documentation ↗︎
• Roadmap ↗︎
• FAQ ↗︎

🎯 Motivation

Traditionally, logging has been seen as a text search problem. Log volumes were not high, and data ingestion or storage were not really issues. This led us to today, where all the logging platforms are primarily text search engines.

But with log data growing exponentially, today's log data challenges involve whole lot more – Data ingestion, storage, and observation, all at scale. We are building Parseable to address these challenges.

🩺 Support

• For questions and feedback please feel free to reach out to us on Slack ↗︎.
• For bugs, please create issue on GitHub ↗︎.
• For commercial support and consultation, please reach out to us at [e-mail] ↗︎.

🏆 Contributing

Refer to the contributing guide here ↗︎.

Contributors

Supported by