v4.7.3: CALL partman.run_maintenance_proc() - ERROR: updated partition constraint for default partition "consumption_default" would be violated by some row

Question

v4.7.3: CALL partman.run_maintenance_proc() - ERROR: updated partition constraint for default partition "consumption_default" would be violated by some row

braquino opened this issue 5 months ago · comments

Using version v4.7.3 because it seams to be the Goocle Cloud SQL latest supported version.

Following this script on a Google Cloud SQL instance of PostgreSQL 15.4:

CREATE SCHEMA partman;
-----------------------------
CREATE EXTENSION pg_partman SCHEMA "partman";
-----------------------------

CREATE TABLE public.consumption (
	device_join_key varchar NOT NULL,
	measure_type varchar(3) NOT NULL,
	arrival_time timestamp NOT NULL,
	measure_time timestamp NOT NULL,
	measure numeric NOT NULL,
	prev_date timestamp NOT NULL,
	prev_measure numeric NOT NULL,
	hours_diff_measures numeric NOT NULL,
	measure_diff numeric NOT NULL,
	hourly_consumption numeric NOT NULL,
	msg_technology varchar NULL
) PARTITION BY RANGE(measure_time);

CREATE INDEX consumption_idx ON public.consumption(arrival_time);

ALTER TABLE consumption ADD UNIQUE (device_join_key, measure_time, measure_type);

------------------------

SELECT partman.create_parent(
	'public.consumption',
	'measure_time',
	'native',
	'monthly'
);

--------------------------

-- it goes to the already created partition: consumption_p2024_01
INSERT INTO public.consumption
(device_join_key, measure_type, arrival_time, measure_time, measure, prev_date, prev_measure, hours_diff_measures, measure_diff, hourly_consumption, msg_technology)
VALUES('dev-key-1', '4', '2024-01-26 17:16:00', '2024-01-26 00:00:00', 50, '2024-01-26 15:15:00', 40, 10, 0, 0, '');

-- it goes to the already created partition:  consumption_p2024_05
INSERT INTO public.consumption
(device_join_key, measure_type, arrival_time, measure_time, measure, prev_date, prev_measure, hours_diff_measures, measure_diff, hourly_consumption, msg_technology)
VALUES('dev-key-1', '4', '2024-01-26 17:16:00', '2024-05-15 00:00:00', 50, '2024-01-26 15:15:00', 40, 10, 0, 0, '');

-- it goes to the already created partition:  consumption_default
INSERT INTO public.consumption
(device_join_key, measure_type, arrival_time, measure_time, measure, prev_date, prev_measure, hours_diff_measures, measure_diff, hourly_consumption, msg_technology)
VALUES('dev-key-1', '4', '2024-01-26 17:16:00', '2024-07-03 00:00:00', 50, '2024-01-26 15:15:00', 40, 10, 0, 0, '');

---------------------------

-- should creare partition consumption_p2024_07 from the row in default
CALL partman.run_maintenance_proc();

The run_maintenance_proc call throws the following error:

Errore SQL [P0001]: ERROR: updated partition constraint for default partition "consumption_default" would be violated by some row
CONTEXT: SQL statement "ALTER TABLE public.consumption ATTACH PARTITION public.consumption_p2024_07 FOR VALUES FROM ('2024-07-01 00:00:00+02') TO ('2024-08-01 00:00:00+02')"
PL/pgSQL function create_partition_time(text,timestamp with time zone[],boolean,text) line 249 at EXECUTE
PL/pgSQL function partman.run_maintenance(text,boolean,boolean) line 284 at assignment
SQL statement "SELECT partman.run_maintenance('public.consumption', p_jobmon := 't')"
PL/pgSQL function partman.run_maintenance_proc(integer,boolean,boolean) line 42 at EXECUTE
DETAIL: 
HINT: 
CONTEXT: PL/pgSQL function create_partition_time(text,timestamp with time zone[],boolean,text) line 511 at RAISE
PL/pgSQL function partman.run_maintenance(text,boolean,boolean) line 284 at assignment
SQL statement "SELECT partman.run_maintenance('public.consumption', p_jobmon := 't')"
PL/pgSQL function partman.run_maintenance_proc(integer,boolean,boolean) line 42 at EXECUTE
DETAIL: 
HINT: 
  Dove: PL/pgSQL function partman.run_maintenance(text,boolean,boolean) line 413 at RAISE
SQL statement "SELECT partman.run_maintenance('public.consumption', p_jobmon := 't')"
PL/pgSQL function partman.run_maintenance_proc(integer,boolean,boolean) line 42 at EXECUTE

braquino · Answer 1 · Mon Jan 29 2024 00:44:47 GMT+0800 (China Standard Time)

Adding another test, now on version v5.0.1 runnig on Docker.
And now, withou any UNIQUE constraint.

CREATE SCHEMA partman;
-----------------------------
CREATE EXTENSION pg_partman SCHEMA "partman";
-----------------------------

CREATE TABLE public.consumption (
	device_join_key varchar NOT NULL,
	measure_type varchar(3) NOT NULL,
	arrival_time timestamp NOT NULL,
	measure_time timestamp NOT NULL,
	measure numeric NOT NULL,
	prev_date timestamp NOT NULL,
	prev_measure numeric NOT NULL,
	hours_diff_measures numeric NOT NULL,
	measure_diff numeric NOT NULL,
	hourly_consumption numeric NOT NULL,
	msg_technology varchar NULL
) PARTITION BY RANGE(measure_time);

------------------------

SELECT partman.create_parent(
	'public.consumption',
	'measure_time',
	'1 month'
);

--------------------------

-- it goes to the already created partition: consumption_p2024_01
INSERT INTO public.consumption
(device_join_key, measure_type, arrival_time, measure_time, measure, prev_date, prev_measure, hours_diff_measures, measure_diff, hourly_consumption, msg_technology)
VALUES('dev-key-1', '4', '2024-01-26 17:16:00', '2024-01-26 00:00:00', 50, '2024-01-26 15:15:00', 40, 10, 0, 0, '');

-- it goes to the already created partition:  consumption_p2024_05
INSERT INTO public.consumption
(device_join_key, measure_type, arrival_time, measure_time, measure, prev_date, prev_measure, hours_diff_measures, measure_diff, hourly_consumption, msg_technology)
VALUES('dev-key-1', '4', '2024-01-26 17:16:00', '2024-05-15 00:00:00', 50, '2024-01-26 15:15:00', 40, 10, 0, 0, '');

-- it goes to the already created partition:  consumption_default
INSERT INTO public.consumption
(device_join_key, measure_type, arrival_time, measure_time, measure, prev_date, prev_measure, hours_diff_measures, measure_diff, hourly_consumption, msg_technology)
VALUES('dev-key-1', '4', '2024-01-26 17:16:00', '2024-07-03 00:00:00', 50, '2024-01-26 15:15:00', 40, 10, 0, 0, '');

---------------------------
-- should creare partition consumption_p2024_07 from the row in default
select partman.run_maintenance('public.consumption');

Similar error:

Errore SQL [P0001]: ERROR: updated partition constraint for default partition "consumption_default" would be violated by some row
CONTEXT: SQL statement "ALTER TABLE public.consumption ATTACH PARTITION public.consumption_p20240701 FOR VALUES FROM ('2024-07-01 00:00:00+02') TO ('2024-08-01 00:00:00+02')"
PL/pgSQL function create_partition_time(text,timestamp with time zone[],text) line 197 at EXECUTE
PL/pgSQL function partman.run_maintenance(text,boolean,boolean) line 265 at assignment
DETAIL: 
HINT: 
CONTEXT: PL/pgSQL function create_partition_time(text,timestamp with time zone[],text) line 366 at RAISE
PL/pgSQL function partman.run_maintenance(text,boolean,boolean) line 265 at assignment
DETAIL: 
HINT: 
  Dove: PL/pgSQL function partman.run_maintenance(text,boolean,boolean) line 389 at RAISE

Keith Fiske · Answer 2 · Mon Jan 29 2024 05:46:55 GMT+0800 (China Standard Time)

pg_partman's default premake value (partitions created ahead of "now" for time) is 4. Therefore, when partitioning monthly, it creates partitions up until May when initially created in January as you've done.

github620=# \d+ consumption
                                                  Partitioned table "public.consumption"
       Column        |            Type             | Collation | Nullable | Default | Storage  | Compression | Stats target | Description 
---------------------+-----------------------------+-----------+----------+---------+----------+-------------+--------------+-------------
 device_join_key     | character varying           |           | not null |         | extended |             |              | 
 measure_type        | character varying(3)        |           | not null |         | extended |             |              | 
 arrival_time        | timestamp without time zone |           | not null |         | plain    |             |              | 
 measure_time        | timestamp without time zone |           | not null |         | plain    |             |              | 
 measure             | numeric                     |           | not null |         | main     |             |              | 
 prev_date           | timestamp without time zone |           | not null |         | plain    |             |              | 
 prev_measure        | numeric                     |           | not null |         | main     |             |              | 
 hours_diff_measures | numeric                     |           | not null |         | main     |             |              | 
 measure_diff        | numeric                     |           | not null |         | main     |             |              | 
 hourly_consumption  | numeric                     |           | not null |         | main     |             |              | 
 msg_technology      | character varying           |           |          |         | extended |             |              | 
Partition key: RANGE (measure_time)
Partitions: consumption_p20230901 FOR VALUES FROM ('2023-09-01 00:00:00') TO ('2023-10-01 00:00:00'),
            consumption_p20231001 FOR VALUES FROM ('2023-10-01 00:00:00') TO ('2023-11-01 00:00:00'),
            consumption_p20231101 FOR VALUES FROM ('2023-11-01 00:00:00') TO ('2023-12-01 00:00:00'),
            consumption_p20231201 FOR VALUES FROM ('2023-12-01 00:00:00') TO ('2024-01-01 00:00:00'),
            consumption_p20240101 FOR VALUES FROM ('2024-01-01 00:00:00') TO ('2024-02-01 00:00:00'),
            consumption_p20240201 FOR VALUES FROM ('2024-02-01 00:00:00') TO ('2024-03-01 00:00:00'),
            consumption_p20240301 FOR VALUES FROM ('2024-03-01 00:00:00') TO ('2024-04-01 00:00:00'),
            consumption_p20240401 FOR VALUES FROM ('2024-04-01 00:00:00') TO ('2024-05-01 00:00:00'),
            consumption_p20240501 FOR VALUES FROM ('2024-05-01 00:00:00') TO ('2024-06-01 00:00:00'),
            consumption_default DEFAULT

Your last insert statement has a measure_type value for July, which doesn't fit into any existing child tables except the default

github620=# select * from consumption_default ;
-[ RECORD 1 ]-------+--------------------
device_join_key     | dev-key-1
measure_type        | 4
arrival_time        | 2024-01-26 17:16:00
measure_time        | 2024-07-03 00:00:00
measure             | 50
prev_date           | 2024-01-26 15:15:00
prev_measure        | 40
hours_diff_measures | 10
measure_diff        | 0
hourly_consumption  | 0
msg_technology      |

Therefore, when partman tries to run, it cannot create the next child table in sequence (July) because there is matching data in the default. There cannot be two destinations for the same data in a single partition set, so it fails to create.

You can either remove the data from the default completely, or run the partition_data_proc() procedure to move that data to the proper child table, creating it if needed. Beware doing this with data far outside the existing child tables in the future since that will then create a gap in the child tables and partman will only know to create new child tables from the "latest" one.

Keith Fiske · Answer 3 · Mon Jan 29 2024 05:50:17 GMT+0800 (China Standard Time)

If you expect data further in the future than the default (4 partitions), then you can set the premake value higher.

braquino · Answer 4 · Mon Jan 29 2024 15:35:17 GMT+0800 (China Standard Time)

Thank you for the explanation Keith, I have misunderstood the run_maintenance function, thought it was responsable for move data from default. It's clear now.

Keith Fiske · Answer 5 · Tue Jan 30 2024 11:52:53 GMT+0800 (China Standard Time)

Thank you for the explanation Keith, I have misunderstood the run_maintenance function, thought it was responsable for move data from default. It's clear now.

I split that out to its own distinct operation from normal maintenance to avoid problems of a massive amount of data going into the default causing normally fast running maintenance operations to suddenly become massive, long write transactions.