Error when using extension in a Lambda function

Question

Error when using extension in a Lambda function

tobilg opened this issue a year ago · comments

I was trying to build the extension so that I can use it in a Lambda function (see https://github.com/tobilg/duckdb-nodejs-layer/blob/main/Dockerfile.prql). Once I try to load in, I get the following error:

"IO Error: Extension \"/opt/nodejs/node_modules/duckdb/extensions/prql.duckdb_extension\" could not be loaded: /opt/nodejs/node_modules/duckdb/extensions/prql.duckdb_extension: undefined symbol: _ZTIN6duckdb9ExceptionE"

Not sure how I can get more info on this error though. It's compiled for the latest master branch of DuckDB. Thanks for any hint!

Yannick Welsch · Answer 1 · Thu Apr 06 2023 17:33:43 GMT+0800 (China Standard Time)

What if you remove the line that strips the extension (i.e. https://github.com/tobilg/duckdb-nodejs-layer/blob/2b367354e9c2d1d0494872caf029f81481ea5d69/Dockerfile.prql#L20)?

Tobi · Answer 2 · Thu Apr 06 2023 17:44:48 GMT+0800 (China Standard Time)

I'm fearing that this will eventually strongly increase the layer size, but I'll check. Stripping with other extensions (e.g. https://github.com/tobilg/duckdb-nodejs-layer/blob/main/Dockerfile.geo#L17) worked fine though. Will report back

Tobi · Answer 3 · Thu Apr 06 2023 18:48:17 GMT+0800 (China Standard Time)

Build fails with the following error:

#9 132.7 [ 36%] Building CXX object src/main/settings/CMakeFiles/duckdb_main_settings.dir/ub_duckdb_main_settings.cpp.o
#9 134.7 In file included from /tmp/from-git/duckdb-prql/build/release/include/extension_oote_headers.hpp:4:0,
#9 134.7                  from /tmp/from-git/duckdb-prql/build/release/include/extension_oote_loader.hpp:2,
#9 134.7                  from /tmp/from-git/duckdb-prql/duckdb/src/main/extension/extension_helper.cpp:83,
#9 134.7                  from /tmp/from-git/duckdb-prql/build/release/src/main/extension/ub_duckdb_main_extension.cpp:3:
#9 134.7 /tmp/from-git/duckdb-prql/src/include/prql_extension.hpp:22:3: error: invalid covariant return type for ‘virtual std::unique_ptr<duckdb::LogicalExtensionOperator> duckdb::PrqlOperatorExtension::Deserialize(duckdb::LogicalDeserializationState&, duckdb::FieldReader&)’
#9 134.7    Deserialize(LogicalDeserializationState &state,
#9 134.7    ^~~~~~~~~~~
#9 134.7 In file included from /tmp/from-git/duckdb-prql/duckdb/src/include/duckdb/main/config.hpp:30:0,
#9 134.7                  from /tmp/from-git/duckdb-prql/duckdb/src/include/duckdb/main/database.hpp:11,
#9 134.7                  from /tmp/from-git/duckdb-prql/duckdb/src/include/duckdb.hpp:12,
#9 134.7                  from /tmp/from-git/duckdb-prql/duckdb/src/include/duckdb/main/extension_helper.hpp:12,
#9 134.7                  from /tmp/from-git/duckdb-prql/duckdb/src/main/extension/extension_alias.cpp:1,
#9 134.7                  from /tmp/from-git/duckdb-prql/build/release/src/main/extension/ub_duckdb_main_extension.cpp:2:
#9 134.7 /tmp/from-git/duckdb-prql/duckdb/src/include/duckdb/planner/operator_extension.hpp:37:47: error:   overriding ‘virtual duckdb::unique_ptr<duckdb::LogicalExtensionOperator> duckdb::OperatorExtension::Deserialize(duckdb::LogicalDeserializationState&, duckdb::FieldReader&)’
#9 134.7   virtual unique_ptr<LogicalExtensionOperator> Deserialize(LogicalDeserializationState &state,
#9 134.7                                                ^~~~~~~~~~~
#9 134.7 In file included from /tmp/from-git/duckdb-prql/build/release/include/extension_oote_headers.hpp:4:0,
#9 134.7                  from /tmp/from-git/duckdb-prql/build/release/include/extension_oote_loader.hpp:2,
#9 134.7                  from /tmp/from-git/duckdb-prql/duckdb/src/main/extension/extension_helper.cpp:83,
#9 134.7                  from /tmp/from-git/duckdb-prql/build/release/src/main/extension/ub_duckdb_main_extension.cpp:3:
#9 134.7 /tmp/from-git/duckdb-prql/src/include/prql_extension.hpp: In member function ‘virtual duckdb::unique_ptr<duckdb::ParserExtensionParseData> duckdb::PrqlParseData::Copy() const’:
#9 134.7 /tmp/from-git/duckdb-prql/src/include/prql_extension.hpp:45:12: error: ‘make_unique_base’ was not declared in this scope
#9 134.7      return make_unique_base<ParserExtensionParseData, PrqlParseData>(
#9 134.7             ^~~~~~~~~~~~~~~~
#9 134.8 /tmp/from-git/duckdb-prql/src/include/prql_extension.hpp:45:12: note: suggested alternative: ‘make_uniq_base’
#9 134.8      return make_unique_base<ParserExtensionParseData, PrqlParseData>(
#9 134.8             ^~~~~~~~~~~~~~~~
#9 134.8             make_uniq_base
#9 134.8 /tmp/from-git/duckdb-prql/src/include/prql_extension.hpp:45:53: error: expected primary-expression before ‘,’ token
#9 134.8      return make_unique_base<ParserExtensionParseData, PrqlParseData>(
#9 134.8                                                      ^
#9 134.8 /tmp/from-git/duckdb-prql/src/include/prql_extension.hpp:45:68: error: expected primary-expression before ‘>’ token
#9 134.8      return make_unique_base<ParserExtensionParseData, PrqlParseData>(
#9 134.8                                                                     ^
#9 136.6 gmake[3]: Leaving directory `/tmp/from-git/duckdb-prql/build/release'
#9 136.6 [ 36%] Built target duckdb_main_capi
#9 136.6 gmake[3]: Entering directory `/tmp/from-git/duckdb-prql/build/release'
#9 136.6 gmake[3]: Leaving directory `/tmp/from-git/duckdb-prql/build/release'
#9 136.7 gmake[3]: Entering directory `/tmp/from-git/duckdb-prql/build/release'
#9 136.7 [ 36%] Building CXX object src/parallel/CMakeFiles/duckdb_parallel.dir/ub_duckdb_parallel.cpp.o
#9 137.4 gmake[3]: *** [src/main/extension/CMakeFiles/duckdb_main_extension.dir/ub_duckdb_main_extension.cpp.o] Error 1
#9 137.4 gmake[3]: Leaving directory `/tmp/from-git/duckdb-prql/build/release'
#9 137.4 gmake[2]: *** [src/main/extension/CMakeFiles/duckdb_main_extension.dir/all] Error 2

Yannick Welsch · Answer 4 · Thu Apr 06 2023 20:23:35 GMT+0800 (China Standard Time)

OK, I think the issue is the make pull command. Instead of using the matching checkouts of DuckDB etc., that command is updating the git submodules to whatever latest version of code that is on the branch (and we want to build against DuckDB 0.7.1, rather than latest DuckDB master branch). I've pushed a change that adapts the behavior of make pull (see b437c90). Can you give it another try?

FWIW, this make pull comes from the DuckDB extension template, and I had never used it.

Tobi · Answer 5 · Thu Apr 06 2023 20:34:37 GMT+0800 (China Standard Time)

Thanks again. I tried without it (just make) and it won’t work due to the missing submodules. Will try again later

Yannick Welsch · Answer 6 · Thu Apr 06 2023 20:37:26 GMT+0800 (China Standard Time)

Yeah, without it you would have to pass --recurse-submodules to the git clone command (to make sure those get checked out as well).

Tobi · Answer 7 · Sat Apr 08 2023 00:02:38 GMT+0800 (China Standard Time)

It works now, thanks a lot!

Tobi · Answer 8 · Sat Apr 08 2023 01:33:22 GMT+0800 (China Standard Time)

Build works, but when loading I see the same error as before... I deactivated the stripping of the binary though.

Yannick Welsch · Answer 9 · Wed Apr 12 2023 17:13:35 GMT+0800 (China Standard Time)

Just wondering: Why build the binaries yourself rather than use the prebuilt ones? If you want to build them yourself, I would try to mimick the build setup that I've used as close as possible: https://github.com/ywelsch/duckdb-prql/blob/main/.github/workflows/Linux.yml

In particular, try setting STATIC_LIBCPP=1 and EXTENSION_STATIC_BUILD=1 as environment variables.

Tobi · Answer 10 · Wed Apr 12 2023 17:18:59 GMT+0800 (China Standard Time)

Because I want to use it in AWS Lambda, which isn’t compatible with the used GLIBC version