Allow write access to `pg_attributes`

[Andrew Powell]

Apologies if this isn't the correct issue template. It seemed to be the closet to fit the situation. @kiwicopple asked that I take the discussion from Twitter to here.

Chore

Describe the chore

We have the need to update pg_attributes. With the security changes in '22 we're no longer able to do so, and we're not able to give the postgres user superuser rights.

Additional context

Our system/product relies on introspection of database objects which includes views and materialized views. Our particular use-case (and production need) involves update pg_attributes and the attrnotnull for a view against a target table. e.g.

CREATE OR REPLACE VIEW public.boards AS SELECT * FROM private.boards;

And because Postgres does not convey nullability to views (for reasons) by default, we're doing this:

WITH not_null as (
  SELECT attname, attnotnull from pg_attribute WHERE attrelid = 'private.boards'::regclass::oid AND attnotnull = TRUE
)
UPDATE pg_attribute SET attnotnull = TRUE WHERE attrelid = 'public.boards'::regclass::oid AND attname IN (select attname from not_null);

As all of our views are in some part based on a base table in the private schema, this works very well for us, and enables one of our product's core features.

It's worth noting that this is not an issue with RDS et al, but those are much closer to the metal. We're also not first-time or ever first-year Postgres users. The majority of our team are seasoned veterans that have nuked many a prod database in their younger years, so we know what we're getting into on this one.

Possible solutions to this may be:

• Allowing us to just have superuser and allowing us to own liability for the supabase schemas
• Creating a role that allows postgres to r/w to pg_attributes
• Some form of RLS black magic that allows updating pg_attributes rows that pertain only to customer objects

[Copple]

Link to twitter discussion: https://twitter.com/shellscape/status/1689852905853972480?s=20

(edit: I accidentally closed the issue with this comment, so I reopened)

[Andrew Powell]

No worries. Would love to get us unblocked while we work through a better solution here.

[Steve Chavez]

Looks related to https://stackoverflow.com/a/17306969/4692662

It's worth noting that this is not an issue with RDS et al,

Does RDS allow updating pg_attributes?

[Steve Chavez]

As an alternative, how about setting a default on the views columns ALTER VIEW [ IF EXISTS ] name ALTER [ COLUMN ] column_name SET DEFAULT expression

https://www.postgresql.org/docs/current/sql-alterview.html

[Andrew Powell]

Does RDS allow updating pg_attributes?

Yes. On RDS you have full control of the db (with some notable guardrails with pg plugins, etc)

As an alternative

We're not really interested in default values. It's not about consuming, it's about accurate introspection, and that comes with being able to update pg_attributes for views due to Postgres defaults for views.

Is there any way we could enable our account to update pg_attributes while we work through this so the team is unblocked?

[Steve Chavez]

Yes. On RDS you have full control of the db

You have no superuser on RDS (or any cloud hosted db), so no full control really.

Also a quick search tells me that modifying pg_attribute doesn't work on RDS: https://dba.stackexchange.com/questions/182307/amazon-postgres-rds-update-pg-attribute-to-update-my-custom-attribute

That might have changed though. Can you share the steps you use to update pg_attribute on RDS?

[Andrew Powell]

Is there any way we could enable our account to update pg_attributes while we work through this so the team is unblocked?

I'm a little miffed you're more interested in talking about RDS than helping our team at the moment. Could we focus on that first and discuss other platforms after?

[Steve Chavez]

I suggest you email [e-mail] with your request.

[Andrew Powell]

I did last night. They asked me if I opened an issue here. #1826346591 @kiwicopple also requested an issue here in our Twitter exchange.

Kind of going in circles here.

[Andrew Powell]

wrt RDS I just confirmed with my former colleague that we had made a support request to AWS to grant write to the system catalog for our target user, which was granted. Access is not granted by default, but was allowed on request.

[Copple]

I'm a little miffed you're more interested in talking about RDS than helping our team at the moment.

copying my comment here from twitter:

We use RDS as a proxy for what is considered secure, so the more accurate/precise you can be the more helpful we can be in return.

Again I’m sorry, but we can’t play around with platform security. Everything needs to be verified before relaxing any restrictions. I’m sure one day you’ll be grateful for this approach, since it’s also designed to protect your own database from others.

I just confirmed with my former colleague that we had made a support request to AWS to grant write to the system catalog for our target user, which was granted. Access is not granted by default, but was allowed on request.

We can make the change for everyone, or not at all (we need to see if there are second-order effects of any changes first). I'm sure as an open source maintainer yourself you can appreciate the difficulty of managing special exceptions in a small team. That's probably an unsatisfying answer for you, but we'd rather be safe than sorry.

[Andrew Powell]

That's probably an unsatisfying answer for you, but we'd rather be safe than sorry.

Not at all. It's quite thorough and appreciated.

Copying from twitter as well, as it adds context (but isn't intended to change the direction here):

I think where Steve got lost was that it was to say "hey we've done this before, we're not new here" rather than "you need to do this too" - I was utterly confused why that was a path he wanted to go down. and it wasn't glamorous, we'd just emailed AWS. maybe bc we were a YC co (Openbase) they let us have r/w access, I'm not sure the why.

The additional details make a lot more sense, and explains why @steve-chavez was curious and focusing on that.

Looking forward to seeing where this heads.

[Copple]

OK final conclusion - we couldn't figure out how the AWS team were giving you this level of access. They confirmed that they generally don't make exceptions because it creates an upgrade-burden (something we also want to avoid).

(Note: I'm not saying you didn't get this from them, it's just that we can't figure out how to support this on our platform).

For now I will close this issue as a "cannot support" - sorry @shellscape

---

As an aside, here is some investigation we did into the issue. It works for the vast majority of cases, but it doesn't work if the view column has a different name to the table column, since postgres tracks the dependency on the view as a whole, not on each view column. It might be helpful -

You can create a view that looks like pg_attribute, but "correctly" joins in the right nullability information that you want:

postgres=# create table foo (id bigserial, bar text not null, bing text);
CREATE TABLE
postgres=# create view foov as select * from foo;
CREATE VIEW
postgres=# SELECT 
        dependent_ns.nspname as dependent_schema
        , dependent_view.relname as dependent_view 
        , dependent_view.oid as dependent_oid
        , source_ns.nspname as source_schema
        , source_table.relname as source_table
        , source_table.oid as source_oid
        , source_attr.attname as column_name
        , source_attr.attnotnull as source_attr_notnull
        , dependent_attr.attnotnull as dependent_attr_notnull
        FROM pg_depend 
        JOIN pg_rewrite ON pg_depend.objid = pg_rewrite.oid 
        JOIN pg_class as dependent_view ON pg_rewrite.ev_class = dependent_view.oid 
        JOIN pg_class as source_table ON pg_depend.refobjid = source_table.oid 
        JOIN pg_attribute source_attr ON source_table.oid = source_attr.attrelid 
                AND pg_depend.refobjsubid = source_attr.attnum 
        JOIN pg_attribute dependent_attr ON dependent_view.oid = dependent_attr.attrelid
                AND source_attr.attname = dependent_attr.attname
        JOIN pg_namespace dependent_ns ON dependent_ns.oid = dependent_view.relnamespace
        JOIN pg_namespace source_ns ON source_ns.oid = source_table.relnamespace
        WHERE 
                source_ns.nspname = 'public'
                AND source_attr.attnum > 0
        ORDER BY 1,2

;
 dependent_schema | dependent_view | dependent_oid | source_schema | source_table | source_oid | column_name | source_attr_notnull | dependent_attr_notnull 
------------------+----------------+---------------+---------------+--------------+------------+-------------+---------------------+------------------------
 public           | foov           |         17277 | public        | foo          |      17271 | id          | t                   | f
 public           | foov           |         17277 | public        | foo          |      17271 | bar         | t                   | f
 public           | foov           |         17277 | public        | foo          |      17271 | bing        | f                   | f
(5 rows)

[Andrew Powell]

Alright and understood. Really appreciate all the effort that went into this.

The same issue just popped up on the Drizzle repo and was able to point the user here for context. That's also an elegant side solution that I think will be adaptable.