Weird behavior: Updates/inserts being routed to slave

[Ashu Pachauri]

We configured distributed reads in our PostgresQL master slave setup with Ruby on Rails app and PGBouncer sitting on top of each of the Postgres instances. We are using the following settings in our database.yml

production:
  adapter: postgresql_makara
  encoding: utf8
  database: <db_name>
  pool: 500
  username: <user_name>
  prepared_statements: false
  reaping_frequency: 10
  connect_timeout: 5
  checkout_timeout: 5
  variables:
    statement_timeout: 600000
  makara:
    sticky: true
    connections:
      - role: master
        name: postgres-prod-1
        host: postgres-prod-1
        port: <pg_bouncer_port>
      - role: slave
        name: postgres-prod-2
        host: postgres-prod-2
        port: <pg_bouncer_port>

We have put exactly one method under distribute_reads {} block which we are sure does not do any SET/UPDATE/INSERT operations. However, we are constantly seeing error like ERROR: cannot execute UPDATE in a read-only transaction in our slave's postgresql logs. Also, Postgres spits out the statement that was being executed in the logs and the statement in this error is something that's executed on a code path that's not even part of distribute_reads {} block.

Are we missing something in the config?

[Andrew Kane]

Hey @ashu210890, a few ideas:

1. If you have a DATABASE_URL environment variable set, it could interfere with that config. Follow the config in the readme to avoid this.
2. One of your PgBouncers could be misconfigured.
3. If you use hostnames to connect, the DNS could be misconfigured.

[Ashu Pachauri]

@ankane Thanks for the suggestions.

I re-verified all the above points but no luck. There is not DATABASE_URL env variable being set anywhere, PgBouncers are talking only on 127.0.0.1 and the hostname problem would happen even without makara, but no such issues. But, just to be sure, I tried resolving the hostnames from bash and the Rails app too, but they were all correct.

I am digging through makara configs now, please let me know if any other pointers come to you mind.

[Andrew Kane]

Let me know what you find.

[Andrew Kane]

Hey @ashu210890, were you able to figure it out?