dgapitts / pg-ora-demo-scripts

pg-scripts for monitoring, basic testing and demo exercises

pg-ora-demo-scripts overview

This pg-ora-demo-scripts project/repro is for basic postgres monitoring, simple load tests, interesting Postgres DBA edge cases and other demo exercises. It is especially aimed at Oracle DBA with comparisons to how Oracle handles such edge cases and some gotchas around migrating/moving from Postgres to Oracle.

note:

• each subfolder has it own README.md with setup instructions and details of various perf/monitoring issues
• the dgapitts/vagrant-postgres9.6 can be used to quickly build a postgres-on-linux VM
• I will also work on a simple RDS/EC2 setup instructions, so you can run these tests in the AWS Cloud

pg-ora-demo-scripts details

Already covered

• Rewriting this query with a NOT EXIST clause instead of NOT IN (Postgres specific gotcha and costs grow exponentially - demos with scale factors 1,2,3,5,10 and 20)
• UNION vs UNION ALL this a classic developer gotcha and similar perf issues on BOTH Postgres and Oracle, but watchout for minor manual pg-2-ora SQL conversions e.g. 'existence checks' with LIMIT=1 vs ROWNUM<2 and bracketing

To Do

• Extreme Postgres Dead Row around Idle in Transaction
• Using CTE to get around Postgres hinting limitations (possibly with DBLinks) ?
• Issues with postgres blocking and waiting scripts i.e. https://wiki.postgresql.org/wiki/Lock_Monitoring (latest version of this pages is looking good)
• pgsql functions STABLE or VOLATILE (default) ?

Going deeper into the Postgres optimizer

Reading Postgres Execution plans isn't too tricky, I've written some simple scripts and made some notes to demo this:

• Demo-01 Three key execution plan join operations Nested Loop, [Sort] Merge Join and Hash Join Operations
• Demo-02 Using EXPLAIN with ANALYZE BUFFERS to trace sql execution actual performance details
• Demo-03 postgres WORK_MEM parameter challenge and why you might see more Sorts-to-Disk
• Demo-04 Anti Join (with nested loops)
• Demo-05 Materialize SubPlan
• Demo-06 Postgres optimizer NOT IN gotcha - still in pg12
• Demo-07 Index Scan Backward
• Demo-08 Introducing Columnar Projected Columns
• Demo-09 More Columnar Projected Columns
• Demo-10 Prepared statements, partitioning pruning and plan_cache_mode workaround (use )
• Demo-11 Reviewing plan_cache_mode options - the difference between auto, force_generic_plan and force_custom_plan
• Demo-12 Private Process Memory Management - work_mem and hash_mem_multiplier

Transaction isolation levels

Exploring how different DB Engines implement transaction isolation levels - exploring edge cases!

• Default READ-COMMITED bahaviour in Postgres - some issues with overlapping transaction
• Default SERIALIZATION in CockroachDB update single row in BLOCKS other SELECT transactions - ouch
• Using SERIALIZATION in Postgres transactions start failing - due to read/write dependencies among transactions

Useful bits and pieces

• Quick setup notes for Mac (brew install postgresql)
• Quick setup notes for Mac (brew install postgresql@15)
• Useful bits and pieces (all)
• pg startup and running time
• WAL Location and Sizing
• plpgsql random_json from ryanbooz's FOSDEM 2022 presenation
• psql variables and running in parallel schemas
• psql os-host commands
• Install citus RPMs and pg extension
• Introduction to Columnar Compression
• auto_explain - setup notes and sample usage
• tracking replicalag
• Loading large datasets and exploring text analysis with Ulysses
• Starting gin_trgm_ops indexing with Ulysses
• Laterial joins intro - more like a correlated subquery
• Extra steps to setup local replica (for testing)
• Exceeding work mem - individual (private) process memory usage can get out-of-control

Unindexed foreign keys

• Demo01 Unindexed_foreign_keys postgres check query
• Demo02 ERROR: there is no unique constraint matching given keys for referenced table
• Demo03 Unindexing FK example fixed by check script

pgbench

• Quick Intro
• Intro Postgres Partitioning --partition-method=range
• Intro Postgres Partitioning --partition-method=hash
• Prepared statements can be a lot faster but issues around partitioning
• Prepared statements around partitioning faster with plan_cache_mode = force_generic_plan

Postrgres FillFactor and HOT (Heap Only Tuple) updates

Background

I want to write up some notes on FF and HOT updates

• this is a good start point - nice summary https://www.cybertec-postgresql.com/en/hot-updates-in-postgresql-for-better-performance/
• I have also written a set of demo script under pg-ora-demo-scripts/demo/generic-fillfactor with an Intro

Examples

• FF and HOT updates - part 01 - simple example with fillfactor 9 - 3 updates and 3 out of 3 are HOT
• FF and HOT updates - part 02 - simple example with fillfactor 100 - 3 updates and only 2 out of 3 are HOT
• FF and HOT updates - part 03 - added explain (analyze,wal)