Local development postgres is configured differently to production

[Bernie Sumption]

Bug report

Describe the bug

I mentioned this in #198 but I think it might deserve its own issue. I discovered that ICU is enabled in local development and tests (supabase start) but not in production.

The issue is not that ICU is disabled in production - it's the deviation between the two environments, which damages my confidence that a build that passes tests will be bug-free in production.

Now that I've discovered one difference, I'm wondering whether there are more and if I need to set up a "staging" project to run integration tests before deploying to production.

To Reproduce

This code fails in production and works locally:

    CREATE COLLATION test_icu (
      provider = icu,
      locale = 'und-u-ks-level2',
      deterministic = false
    );

Expected behavior

The two environments should produce the same result for this and all other queries.

System information

• OS: macOS 12.5.1
• Browser: n/a
• Version of supabase-js: n/a
• Version of Node.js: v16.13.0

[Pavel Borisov]

#489 was merged just recently and will most likely be available on prod for newly created and paused/unpaused projects next week on Tuesday

[Bernie Sumption]

@pashkinelfe Thanks, but does this resolve the only known difference between production and local? The bug I was trying to report was not so much that ICU was disabled in production, but that it is possible for the two to be configured differently and so I don't know how much confidence I should have that my app will work in production just because it passed tests locally.

Are the development and production binaries now compiled with identical configurations, or may there be more differences to be discovered?

[Pavel Borisov]

Production and local have identical binaries and configuration.

For local environments, the Docker image is used, which is built immediately when something is merged in the develop branch.

For prod (the platform), we have to create a migration containing the new AWS AMI entry and then merge that into master so it’d run agains the prod database. Then the platform selects the latest PG AMI to launch new projects (or newly unpaused projects) on. So, for production support, it’s delayed based on our prod deploy cycles, which happen on Tuesdays and Thursdays.

[Bernie Sumption]

@pashkinelfe Thanks for that, I really appreciate it the detail. It's great to know that the intention is that local and production binaries are the same.

But it would seem that something slipped through the process, because ICU has been enabled on local development for a long time - I just tested 1.25.0 released in early December 2022 and still on Postgres 14, and it has ICU enabled. Whereas ICU support in production is only just being enabled in PR #491 now. So it would seem that there's some way in which it's possible for differences in capability between production and local to exist for much longer than the week or so that would be implied by your comment above.

Do we know how ICU came to be enabled in local before the --with-icu flag was added in PR #491, and whether that could apply to other capabilities?

[Paul Cioanca]

Hi @BernieSumption,

I think this is due to us basing our Docker builds (used locally) off the official Postgres Docker images, which come with a higher number of features enabled than we ourselves enable it for the Supabase platform Postgres AMI.

The official Docker image enables these features/flags with the goal of being as flexible as possible - we'll need to research the impact of equivalating the AMI to the Docker build - I've created an issue regarding this here: #511.

As for ICU support, it's been rolled out to the Supabase platform today for newly created projects, or project which are paused and then unpaused (thus launching using the updated AMI).
If your project is on a paid tier and can't be paused, let us know through support and link this issue to run the operation on your behalf.