Allow :colon variables

[Andrew]

PostgreSQL uses a :colon syntax for parameterizing SQL commands with command line variables.

create-database.sql:

CREATE DATABASE :db;

Usage:

$ psql -f create-database.sql -v db=test

However, pgsanity/ecpg rejects these.

$ pgsanity create-database.sql
line 1: ERROR: syntax error at or near ":db"

[Mark Drago]

Andrew,

Interesting. I wasn't familiar with the colon syntax until now. I did some research and it seems that the colon syntax is primarily handled outside of the postgresql server (either in psql or in other client-side apps). It is also supported by ecpg, which is what pgsanity leverages for checking syntax. So, it may be possible to get pgsanity to do what you want. I think it should be behind some sort of command-line flag though, perhaps --allow-colon-variables or something similar. This way people can opt-in to having the syntax check work that way.

I'm not exactly sure how we would handle this situation though. We would need to either swap out the variable for something else or give ecpg more information about the variables.

There's more information about how ecpg handles variables here: http://www.postgresql.org/docs/9.1/static/ecpg-variables.html . It may be nontrivial to get the types to match up. This might be an interesting problem. 😄

For now you may be able to use pgsanity by doing the following:

cat create-database.sql | sed -e 's/:db/test' | pgsanity

You would have to add a -e expression to the sed call for each of your variables, but that might work.

[Andrew]

I don't see an option for this in the ecpg man page.

Yeah, there might be a sed regex general enough to work for most input parameter names.

In the meantime, I've reached out to the postgres mailing list for more help.

http://www.postgresql.org/message-id/CAHXt_SV=FaYpTr+scFFROo6MePOqFg0wrj6=[e-mail]

[Andrew]

A response from the mailing list:

I'm not sure why you think that should work. psql and ecpg have quite
distinct input languages. Both are extensions of SQL, but the key word
there is "extension". ecpg certainly isn't going to accept psql's
backslash commands for instance, any more than psql would accept ecpg's
C code portions. And I doubt it would be useful for ecpg to simply ignore
the variable-interpolation symbols; but it has no way to know what's going
to be substituted for those symbols.

It would be more interesting to consider giving psql a syntax-check-only
mode

Seems that ecpg isn't meant to check PostgreSQL syntax, just SQL syntax.

[Mark Drago]

I think it's more accurate to say that the colon variables are neither postgresql syntax nor sql syntax. They just happen to be used by embedded sql in C and psql. The postgresql server knows nothing about them.

I fully acknowledge that pgsanity is kind of a hack in that it leverages ecpg for syntax checking. I was examining the postgresql source code with the intention of creating a syntax checker when I stumbled upon ecpg. It would be much better to have a dedicated syntax checker, but pgsanity is capable of filling that void until we have one.

Regardless, the colon syntax would still be unsupported by that syntax checker since the colon variables are not valid postgresql syntax. psql and ecpg replace them long before they get to the server. You won't find mention of it in the ecpg man page because the variable substitutions are defined within the C file in the ecpg case. So, while it will do variable substitution in a similar fashion to psql, they are not triggered the same way.

I would suggest you go with the sed route, or do the templating outside of psql and check the output of that tool (essentially what sed is doing anyway).

[Andrew]

What if pgsanity automatically dropped the colons before handing the data to ecpg?

[Mark Drago]

I think I would prefer if pgsanity didn't know about the colon variable syntax. I think it's best if you preprocess the sql before passing it in to pgsanity. That way you can decide how best to deal with the variables in your situation. So, if you want to remove the colons you can do this:

cat file.sql | sed -e 's/://' | pgsanity

The downside to that approach is you're not checking if your variable substitutions will result in valid syntax.

I guess the main reason why I'm objecting to adding this to pgsanity is that it adds support for something that is decidedly outside of postgresql's standard (server-side) syntax and pgsanity is not in a position to be excellent at providing that support.

[Andrew]

Perfectly reasonable.