Vendoring with go mod vendor leads to "fatal error: 'pg_query.h' file not found"

[Manit Gupta]

Hi,

We are using pg_query_go as a dependency for harbourbridge.

When I vendor in pg_query_go using go mod vendor, I get a missing C header error:

⇪manitgupta:🏠hb_main/harbourbridge#>go mod vendor
⇪manitgupta:🏠hb_main/harbourbridge#>make build-static
go build -a -tags osusergo,netgo -ldflags '-w -extldflags "-static"' -o harbourbridge main.go
# github.com/pganalyze/pg_query_go/v2/parser
vendor/github.com/pganalyze/pg_query_go/v2/parser/parser.go:6:10: fatal error: 'pg_query.h' file not found
#include "pg_query.h"
         ^~~~~~~~~~~~
1 error generated.
make: *** [Makefile:6: build-static] Error 2

I am on pg_query_go v2.0.5.

This is what gets vendored in:

⇪manitgupta:🏠hb_main/harbourbridge#>tree vendor/github.com/pganalyze/pg_query_go/
vendor/github.com/pganalyze/pg_query_go/
└── v2
    ├── CHANGELOG.md
    ├── go.mod
    ├── go.sum
    ├── LICENSE
    ├── LICENSE.POSTGRESQL
    ├── Makefile
    ├── makefuncs.go
    ├── parser
    │   ├── guc-file.c
    │   ├── parser.go
    │   ├── pg_query.c
    │   ├── pg_query_deparse.c
    │   ├── pg_query_fingerprint.c
    │   ├── pg_query_fingerprint.h
    │   ├── pg_query_internal.h
    │   ├── pg_query_json_plpgsql.c
    │   ├── pg_query_json_plpgsql.h
    │   ├── pg_query_normalize.c
    │   ├── pg_query_outfuncs.h
    │   ├── pg_query_outfuncs_json.c
    │   ├── pg_query_outfuncs_protobuf.c
    │   ├── pg_query_parse.c
    │   ├── pg_query_parse_plpgsql.c
    │   ├── pg_query.pb-c.c
    │   ├── pg_query_readfuncs.h
    │   ├── pg_query_readfuncs_protobuf.c
    │   ├── pg_query_scan.c
    │   ├── pg_query_split.c
    │   ├── protobuf-c.c
    │   ├── src_backend_catalog_namespace.c
    │   ├── src_backend_catalog_pg_proc.c
    │   ├── src_backend_commands_define.c
    │   ├── src_backend_libpq_pqcomm.c
    │   ├── src_backend_nodes_bitmapset.c
    │   ├── src_backend_nodes_copyfuncs.c
    │   ├── src_backend_nodes_equalfuncs.c
    │   ├── src_backend_nodes_extensible.c
    │   ├── src_backend_nodes_list.c
    │   ├── src_backend_nodes_makefuncs.c
    │   ├── src_backend_nodes_nodeFuncs.c
    │   ├── src_backend_nodes_value.c
    │   ├── src_backend_parser_gram.c
    │   ├── src_backend_parser_parse_expr.c
    │   ├── src_backend_parser_parser.c
    │   ├── src_backend_parser_scan.c
    │   ├── src_backend_parser_scansup.c
    │   ├── src_backend_postmaster_postmaster.c
    │   ├── src_backend_storage_ipc_ipc.c
    │   ├── src_backend_storage_lmgr_s_lock.c
    │   ├── src_backend_tcop_postgres.c
    │   ├── src_backend_utils_adt_datum.c
    │   ├── src_backend_utils_adt_expandeddatum.c
    │   ├── src_backend_utils_adt_format_type.c
    │   ├── src_backend_utils_adt_ruleutils.c
    │   ├── src_backend_utils_error_assert.c
    │   ├── src_backend_utils_error_elog.c
    │   ├── src_backend_utils_fmgr_fmgr.c
    │   ├── src_backend_utils_hash_dynahash.c
    │   ├── src_backend_utils_init_globals.c
    │   ├── src_backend_utils_mb_mbutils.c
    │   ├── src_backend_utils_misc_guc.c
    │   ├── src_backend_utils_mmgr_aset.c
    │   ├── src_backend_utils_mmgr_mcxt.c
    │   ├── src_common_encnames.c
    │   ├── src_common_hashfn.c
    │   ├── src_common_keywords.c
    │   ├── src_common_kwlist_d.h
    │   ├── src_common_kwlookup.c
    │   ├── src_common_psprintf.c
    │   ├── src_common_string.c
    │   ├── src_common_stringinfo.c
    │   ├── src_common_wchar.c
    │   ├── src_pl_plpgsql_src_pl_comp.c
    │   ├── src_pl_plpgsql_src_pl_funcs.c
    │   ├── src_pl_plpgsql_src_pl_gram.c
    │   ├── src_pl_plpgsql_src_pl_handler.c
    │   ├── src_pl_plpgsql_src_pl_reserved_kwlist_d.h
    │   ├── src_pl_plpgsql_src_pl_scanner.c
    │   ├── src_pl_plpgsql_src_pl_unreserved_kwlist_d.h
    │   ├── src_port_erand48.c
    │   ├── src_port_pg_bitutils.c
    │   ├── src_port_pgsleep.c
    │   ├── src_port_pgstrcasecmp.c
    │   ├── src_port_qsort.c
    │   ├── src_port_random.c
    │   ├── src_port_snprintf.c
    │   ├── src_port_strerror.c
    │   ├── src_port_strnlen.c
    │   └── xxhash.c
    ├── pg_query.go
    ├── pg_query.pb.go
    └── README.md

2 directories, 91 files

When I just rm -r vendor, this problem goes away and everything works fine.

I saw a similar issue in another repo here: goccy/go-graphviz#28

Could you please suggest a path to vendoring in this library?

[Lukas Fittl]

@manitgupta Unfortunately I'm not sure what is causing this issue - we have seem similar problems ourselves, and are using modvendor to solve them for our own internal use case with the pganalyze collector:

https://github.com/pganalyze/collector/blob/main/Makefile#L32

(possibly you can ask someone internally on the Go team at Google, how go mod vendor should work for .c/.h files?)

[Manit Gupta]

@lfittl I looked into this issue and the reason seems to be that go mod vendor does not correctly include source files from a directory if there are no .go file present in it. The solution seems to be to include a dummy.go file in the directories.
This has been done go-graphviz repo previously and seems to be work.
Here is the PR for it: https://github.com/goccy/go-graphviz/pull/37/files

I am willing to do a similar PR for pg_query_go if you'd be willing to review it.

[Manit Gupta]

This modvendor solution is also interesting. As a workaround, I had earlier written a script in our Makefile to get the source at the needed version for pg_query_go package.

build-vendor:
# vendor the dependencies
	go mod vendor
# git clone the repository and switch to the tag present in harbourbridge's go.mod file (currently: v2.0.5)
	rm -rf vendor/github.com/pganalyze/pg_query_go/v2
	cd vendor/github.com/pganalyze/pg_query_go/ && git clone https://github.com/pganalyze/pg_query_go 
	cd vendor/github.com/pganalyze/pg_query_go/pg_query_go/ && git checkout tags/v2.0.5
	mv vendor/github.com/pganalyze/pg_query_go/pg_query_go/ vendor/github.com/pganalyze/pg_query_go/v2/
# revert to the home directory and build
	go build

[graywolf-at-work]

I've just encountered this as well, upstream ticket is golang/go#26366 ("resolved", yeah, great).