jackc / pgx

PostgreSQL driver and toolkit for Go

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Using pgxpool connection to query by joining multiple tables does not return all the rows.

padkrish opened this issue · comments

Describe the bug
Using pgxpool connection to query by joining multiple tables does not return all the rows.
The query does a Join of. multiple tables (5 tables), please see below for a sample query. This was working fine when the number of entries returned was less than 50. But, when the number of entries went above 60, it was returning only 56 entries. Most of the times it returned 56, but sometimes when i change the number of connections, i have see it return 51 entries as well.
Running the exact same query in the DB console returns the right number of entries.

Also, if i use pgx.Connect and do a query based on this connection, it returns the correct number of entries. Only pgxpool has the problem.

I also followed the sample code given in a similar issue:
#639

I still see the same issue.

I am using version 4. I also upgraded to V5 and that also did not solve the problem. I dumped the cdbpool stats, hoping to see some counts going up, but the values were 0 for stats.CanceledAcquireCount(), stats.MaxIdleDestroyCount(), stats.MaxLifetimeDestroyCount().

To Reproduce
Steps to reproduce the behavior:

package main

import (
	"context"
	"log"
	"os"

	"github.com/jackc/pgx/v4"
)

func main() {
       config, err := pgxpool.ParseConfig(url)
        if err != nil {
               ...
                return nil, err
        }

        // Even tried changing the below to values like 8 or 16
         numThreads = runtime.GOMAXPROCS(0)
        config.MaxConns = int32(numThreads)
        pool, err := pgxpool.ConnectConfig(ctx, config)
        ....
}

func query(ctx context.Context, tbl string, args ...interface{}) {
    conn, err := cdbpool.Acquire(ctx)
    // Check for errors
    rows, err := conn.Query(ctx, q, args...)
    // check for errors
}

The args passed is a join of multiple tables that looks something like
SELECT f1, f2, entry.*, d AS f, tbl1 INNER JOIN tbl2 ON conditions INNER JOIN tbl3 ON cond INNER JOIN tbl4 ON cond INNER JOIN tbl5  cond INNER JOIN cond;

Please note that the above query is just to give an idea. Pasting the exact query in the DB console gives the right results.

Version

  • Go: 1.19
  • PostgreSQL: CockroachDB CCL v21.2.17
  • pgx: github.com/jackc/pgx/v4 v4.18.1

So it works with a single pgx connection, but doesn't work through the pool? That is very surprising. The pool is a very thin wrapper. There shouldn't be much that can go wrong there.

I noticed you are manually acquiring a conn and then calling Query on that conn instead of calling Query on the pool itself. Does it work on the pool? Are you closing the rows before releasing the conn to the pool?

Also, see if the race detector notices any issues.