influxdata / docs-v2

InfluxData Documentation that covers InfluxDB Cloud, InfluxDB OSS 2.x, InfluxDB OSS 1.x, InfluxDB Enterprise, Telegraf, Chronograf, Kapacitor, and Flux.

Home Page:https://docs.influxdata.com

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Add information about partition limit on write

sanderson opened this issue · comments

InfluxDB Cloud Serverless limits the number of partitions per database that can be written to in ~15 minute periods to ensure availability and consistent service for other tenants. Serverless partitions data by day, so the limit will be hit when a user tries to write data with timestamps from multiple days within a 15 minute period.

We need to add this limit to the Global limits section in the docs.

@domodwyer What is the specific number of partitions that we limit to?

Relevant URLs
commented

Err 5,000 - so that works out to be more than 13 years of data in 15m!

Note it only creates partitions / counts towards the 15m limit when a user writes to a date, not the interval, so writing a value for monday and wednesday doesn't turn into [monday, tuesday, wednesday], it's just [monday, wednesday].

This must be 5,000 days of writes!

commented

Thanks for this @sanderson 🙇

Thanks @domodwyer! Question:

Note it only creates partitions / counts towards the 15m limit when a user writes to a date, not the interval, so writing a value for monday and wednesday doesn't turn into [monday, tuesday, wednesday], it's just [monday, wednesday].

Is the limit on the number of new partitions created, or the number of distinct partitions (existing or new) that data is written to?

commented

Distinct, in a 15m window.

You can write to 5,000 partitions, wait 20 minutes, and then write to those 5,000 partitions again, but you can't wait and the write to 5,001 - at any one time, the system will allow at most 5,000 non-empty partitions in a 15m period.

Note the period can be changed at a service / platform level for various operational reasons, but it's been 15m for a long time historically speaking.