Databend Roadmap for 2024 (Discussion)
BohuTANG opened this issue ยท comments
Databend Roadmap for 2024 (Discussion)
Explore our ongoing journey and future plans for Databend. Join the discussion and contribute your ideas!
2024: Compute Where Data Lives: Swift, Smart, Seamless.
Review of 2023
In 2023, Databend scaled significantly.
The largest single table in Databend managed to handle hundreds of thousands of segments, several ten million blocks, tens of trillions of records, encompassing 7PB of raw data and over 300TB of index data.
Main Tasks for 2024
| Task | Status | Comments |
|---|---|---|
| Concurrency and Scheduler | In Progress | Aiming for faster, more efficient task handling and improved system responsiveness. |
| GEOMETRY Data Type | In Progress | |
| TPC-DS Performance | In Progress | Continuously optimizing for better performance benchmarks. |
| Full-Text Indexes | Done | |
| Multi-Statement Transactions | Done | |
| Stored Procedures(Python) | In Progress | Adding Python support for versatile data analysis alongside SQL. |
| Storage + Compute + Inference | Not Specified | Creating a cohesive data platform for AI and cloud computing, provisioning CPU & GPU resources. |
Previous Roadmaps for Reference:
- 2023 Roadmap: Issue #9448
- 2022 Roadmap: Issue #3706
- 2021 Roadmap: Issue #746
Congratulations on what Databend has achieved in such a short time. Looking forward to 2024!
What exactly is the Support Python Worksheet? Does it enable running Python in Databend?
Any plan for SQL transaction and stored procedures?
besides, I think we could also support query queueing, warehouse automatica scaling based on pending queue and separate another coordinator component for dispatching physical plan to warehouse compute node.
besides, I think we could also support query queueing, warehouse automatica scaling based on pending queue and separate another coordinator component for dispatching physical plan to warehouse compute node.
This is a part of Enhancements to Concurrency and Scheduler.
What exactly is the
Support Python Worksheet? Does it enable running Python in Databend?
The goal is to make the Hugging Face Model + Python + GPU( or CPU) + Data in Databend is possible.
all I want is to be able to read a delta table from a local path :)
How to understand Inference ๏ผWhich abilities does it refer to?
How to understand Inference ๏ผWhich abilities does it refer to?
Move the models(huggingface models) to the database, the database can load and run them.
Thanks for making this available to everyone. I'm currently interested to anything that can deal with Geospatial stuff so adding Geometry support is very nice.
Thanks for making this available to everyone. I'm currently interested to anything that can deal with Geospatial stuff so adding Geometry support is very nice.
Working on it: #14470
Thanks for making this available to everyone. I'm currently interested to anything that can deal with Geospatial stuff so adding Geometry support is very nice.
@keltia Databend uses H3 for geospatial operations. Is that what you are referring to? https://docs.databend.com/sql/sql-functions/geo-functions/