Conan package

Question

Conan package

Myroendan opened this issue 8 months ago · comments

It would be nice to have a conan package available for this project. I've created a conan2 recipe, let me know if you would like to review it.

Ulrich Telle · Answer 1 · Wed Nov 15 2023 04:33:01 GMT+0800 (China Standard Time)

It would be nice to have a conan package available for this project. I've created a conan2 recipe, let me know if you would like to review it.

Admittedly, I haven't used Conan up to now. However, it is certainly a good idea to support package management systems to make the component available for more users/developers.

So, yes, I'd like to review your Conan recipe.

Zoltan Szilvassy · Answer 2 · Thu Nov 16 2023 19:14:56 GMT+0800 (China Standard Time)

I will create a PR for Issue#109 regarding cmake install first. If the build system handles the copy of the files (headers, lib etc..), the conan recipe will be shorter and cleaner.
I have one question to this topic. This is the list of headers which are added to the cmake target:
src/cipher_common.h
src/cipher_config.h
src/fastpbkdf2.h
src/mystdint.h
src/rijndael.h
src/sha1.h
src/sha2.h
src/sqlite3.h
src/sqlite3ext.h
src/sqlite3mc.h
src/sqlite3mc_vfs.h
src/sqlite3userauth.h
src/test_windirent.h

Which ones shouldn't be used by users? I would only move the public ones to the include folder.

Ulrich Telle · Answer 3 · Thu Nov 16 2023 21:31:36 GMT+0800 (China Standard Time)

I will create a PR for Issue#109 regarding cmake install first.

Great. TIA.

If the build system handles the copy of the files (headers, lib etc..), the conan recipe will be shorter and cleaner.

The "problem" with SQLite is that there are so many configuration options.

I have one question to this topic. This is the list of headers which are added to the cmake target:

Which ones shouldn't be used by users? I would only move the public ones to the include folder.

Only the flagged header files should be made public, all not flagged header files are used internally only.

Zoltan Szilvassy · Answer 4 · Fri Nov 17 2023 02:18:27 GMT+0800 (China Standard Time)

I will add to the recipe that zlib and icu are conditionally required, based on the compiler flags. But I haven’t found information about the minimum required versions. Do you have a suggestion?

Ulrich Telle · Answer 5 · Fri Nov 17 2023 05:10:36 GMT+0800 (China Standard Time)

I will add to the recipe that zlib and icu are conditionally required, based on the compiler flags. But I haven’t found information about the minimum required versions. Do you have a suggestion?

Well, actually the SQLite extensions should work with any prior version. Actually, I haven't tested whether there are prior versions of zlib and icu, which would not work. We can set the following minimum version numbers for now, although it is very likely that prior versions could be used as well.

ICU: The first version used together with SQLite3 Multiple Ciphers was version 67.1.
zlib: let's use version 1.2.9 as the minimum version. That version was released in 2017.

Regarding zlib the user can choose to use the included miniz implementation avoiding the external dependency.

Zoltan Szilvassy · Answer 6 · Fri Nov 17 2023 23:08:04 GMT+0800 (China Standard Time)

Thank you for the information.
I have added the conan recipe in the PR. I wrote (the non-existing) version 1.7.5, since that will be the first version that contains the latest cmake changes required by this recipe. The SHA hash of the new package file will be needed in the conandata.yml.

I think you should be the one to upload the package to conan center as the project owner. You can find the guideline for the process here:
https://docs.conan.io/2/tutorial/conan_repositories/conan_center.html

But based on the guideline, a test_package is necessary. I'll have to add one to my recipe. I'll update my PR with it in a few days.

I believe the conan folder won't be necessary in the repo once the package is uploaded to the conan center.

Ulrich Telle · Answer 7 · Sat Nov 18 2023 00:27:37 GMT+0800 (China Standard Time)

I have added the conan recipe in the PR. I wrote (the non-existing) version 1.7.5, since that will be the first version that contains the latest cmake changes required by this recipe. The SHA hash of the new package file will be needed in the conandata.yml.

Thanks. We will see what the version number of the next release will be. Maybe 1.7.5, maybe 1.8.0 - depends on how my own activities will proceed.

I think you should be the one to upload the package to conan center as the project owner. You can find the guideline for the process here: https://docs.conan.io/2/tutorial/conan_repositories/conan_center.html

I will try to study the guidelines within the next days.

But based on the guideline, a test_package is necessary. I'll have to add one to my recipe.

What sort of test package? For example, my GitHub workflow executes some SQL scripts testing to access encrypted database files.

I'll update my PR with it in a few days.

Ok, I'll wait with merging, until I get the green light from you.

I believe the conan folder won't be necessary in the repo once the package is uploaded to the conan center.

Meaning what? IMHO it's not a bad idea to keep these files in a separate directory in the repo.

Zoltan Szilvassy · Answer 8 · Mon Nov 20 2023 06:04:39 GMT+0800 (China Standard Time)

What sort of test package? For example, my GitHub workflow executes some SQL scripts testing to access encrypted database files.

It's for testing the package itself rather than the underlying source code. It ensures that consumers can link to the package and use it.

Ok, I'll wait with merging, until I get the green light from you.

I have added a new commit, I think it's okay now. The remaining necessary modifications are to update the version numbers once the 1.7.5/1.8.0 is released, and to add the SHA to the conanfile.yml. Then the process of adding the package to the conan center can be started. When you check out the repo mentioned in the guide, there will be the recipes folder. There you can create the folder for the project, and everything from this repo's conan folder can go there.

Meaning what? IMHO it's not a bad idea to keep these files in a separate directory in the repo.

As far as I remember, projects usually don't contain the conan recipe in their repo, only on the conan center. But if you want to have it here as well, that's fine by me.

Zoltan Szilvassy · Answer 9 · Mon Nov 20 2023 15:11:32 GMT+0800 (China Standard Time)

I have found an issue during testing when I include sqlite3mc.h instead of sqlite3.h:
sqlite3mc.h(16,10): error C1083: Cannot open include file: 'sqlite3mc_version.h': No such file or directory
I'll have to check that.

A question: I have seen in many conan recipes, that they define the option fPIC true by default, and remove the option if the os is Windows. Do you think I should add this option to the recipe?

Ulrich Telle · Answer 10 · Mon Nov 20 2023 16:52:56 GMT+0800 (China Standard Time)

I have found an issue during testing when I include sqlite3mc.h instead of sqlite3.h: sqlite3mc.h(16,10): error C1083: Cannot open include file: 'sqlite3mc_version.h': No such file or directory I'll have to check that.

In the conan recipe no header files are mentioned. Maybe the header file sqlite3mc_version.h was not copied to wherever the header files are expected to be?

A question: I have seen in many conan recipes, that they define the option fPIC true by default, and remove the option if the os is Windows. Do you think I should add this option to the recipe?

Good question. AFAIK the autoconf/automake build files coming with sqlite3mc handle that automatically under the hood. But in CMake it may be necessary to specify the option explicitly.

Ulrich Telle · Answer 11 · Mon Nov 20 2023 17:02:55 GMT+0800 (China Standard Time)

What sort of test package? For example, my GitHub workflow executes some SQL scripts testing to access encrypted database files.

It's for testing the package itself rather than the underlying source code. It ensures that consumers can link to the package and use it.

Ok.

I have added a new commit, I think it's okay now. The remaining necessary modifications are to update the version numbers once the 1.7.5/1.8.0 is released, and to add the SHA to the conanfile.yml.

Ok.

Then the process of adding the package to the conan center can be started. When you check out the repo mentioned in the guide, there will be the recipes folder. There you can create the folder for the project, and everything from this repo's conan folder can go there.

I will check with the conan docs.

Meaning what? IMHO it's not a bad idea to keep these files in a separate directory in the repo.

As far as I remember, projects usually don't contain the conan recipe in their repo, only on the conan center. But if you want to have it here as well, that's fine by me.

On the one hand, it's logical to not have the recipe in the repo. At least not the final version due to the sha256 checksum.

On the other hand, it does no harm to have the recipe template in the repo. For now, I guess I will keep it in the repo.

Zoltan Szilvassy · Answer 12 · Mon Nov 20 2023 17:14:31 GMT+0800 (China Standard Time)

In the conan recipe no header files are mentioned. Maybe the header file sqlite3mc_version.h was not copied to wherever the header files are expected to be?

Yeah that was the issue. In this case, I have to modify the CMakeLists.txt to copy every header file during install.

Good question. AFAIK the autoconf/automake build files coming with sqlite3mc handle that automatically under the hood. But in CMake it may be necessary to specify the option explicitly.

I will add it to the recipe.

Ulrich Telle · Answer 13 · Mon Nov 20 2023 17:56:19 GMT+0800 (China Standard Time)

In the conan recipe no header files are mentioned. Maybe the header file sqlite3mc_version.h was not copied to wherever the header files are expected to be?

Yeah that was the issue. In this case, I have to modify the CMakeLists.txt to copy every header file during install.

Really EVERY header file? It would be good if that could be restricted to the header files required by applications using sqlite3mc.

Good question. AFAIK the autoconf/automake build files coming with sqlite3mc handle that automatically under the hood. But in CMake it may be necessary to specify the option explicitly.

I will add it to the recipe.

To the CMake recipe I guess.

I will still wait a bit before merging your PR.

Zoltan Szilvassy · Answer 14 · Mon Nov 20 2023 18:49:58 GMT+0800 (China Standard Time)

In the conan recipe no header files are mentioned. Maybe the header file sqlite3mc_version.h was not copied to wherever the header files are expected to be?

Yeah that was the issue. In this case, I have to modify the CMakeLists.txt to copy every header file during install.

Really EVERY header file? It would be good if that could be restricted to the header files required by applications using sqlite3mc.

You're right, I have checked and sqlite3mc_version.h is the only additional header that's needed, so I only add that to the list when installing.

Good question. AFAIK the autoconf/automake build files coming with sqlite3mc handle that automatically under the hood. But in CMake it may be necessary to specify the option explicitly.

I will add it to the recipe.

To the CMake recipe I guess.

I will still wait a bit before merging your PR.

Yes, I will update my PR this evening.

Zoltan Szilvassy · Answer 15 · Tue Nov 21 2023 00:10:12 GMT+0800 (China Standard Time)

I've pushed the latest commit, I think this one can be merged if no errors found.

Ulrich Telle · Answer 16 · Sat Nov 25 2023 05:24:36 GMT+0800 (China Standard Time)

@Myroendan: I created a PR in the Conan repository. Unfortunately, at least 2 checks failed.
Could you please take a look: conan-io/conan-center-index#21354? TIA.

Ulrich Telle · Answer 17 · Sat Nov 25 2023 17:03:17 GMT+0800 (China Standard Time)

In the meantime I applied a few minor adjustments (i.e. removing C++ compiler settings, because the library is a C library). Nevertheless, some build check steps still fail. The reason seems to be, that in the Conan check environment only an old CMake version 3.15.7 is available while sqlite3mc's CMake file asks for version 3.24.0 or higher.

Zoltan Szilvassy · Answer 18 · Sat Nov 25 2023 18:37:11 GMT+0800 (China Standard Time)

I see the yaml linter threw an error for the "---" beginning. I included that in the first place due to their guidelines: the yaml linter warned me locally that it was missing, well nevermind. I see you removed it from the conandata.yml, the config.yml also has it.
conan v1 pipeline: the recipe is not conan1 compatible. I don't know if its a requirement for a new recipe, but it would be weird, since they're trying to push everyone towards conan2.

The reason seems to be, that in the Conan check environment only an old CMake version 3.15.7 is available while sqlite3mc's CMake file asks for version 3.24.0 or higher

Yeah, this is also interesting. I think we could downgrade our CMake requirement to 3.15.7, I don't think we use features that require a higher version (I'll have to check it). But does this mean conan is not supporting projects using CMake with a version higher than 3.15.7? That's a 4 years old version.

Could you please clarify point 2 and 3 with them?

Ulrich Telle · Answer 19 · Sat Nov 25 2023 20:26:00 GMT+0800 (China Standard Time)

I see the yaml linter threw an error for the "---" beginning. I included that in the first place due to their guidelines: the yaml linter warned me locally that it was missing, well nevermind. I see you removed it from the conandata.yml, the config.yml also has it.

Yes, I saw that, after the checks failed again. I'll remove the line from that file, too, once I get information about the other issue.

conan v1 pipeline: the recipe is not conan1 compatible. I don't know if its a requirement for a new recipe, but it would be weird, since they're trying to push everyone towards conan2.

Personally, I don't see that failure as a real error, because all recipes should be using version 2. Interestingly, less than 20 recipes ofover 1600 recipes refer to version 2 at all in their required_conan_version, and if they do it is a combination like required_conan_version = ">=1.56.0 <2 || >=2.0.6".

Yeah, this is also interesting. I think we could downgrade our CMake requirement to 3.15.7, I don't think we use features that require a higher version (I'll have to check it). But does this mean conan is not supporting projects using CMake with a version higher than 3.15.7? That's a 4 years old version.

In the Conan documentation I see nowhere mentioned that CMake support is restricted to version below 3.15.7.

Could you please clarify point 2 and 3 with them?

I opened issue conan-io/conan-center-index#21358. We will see what answer we get.

Zoltan Szilvassy · Answer 20 · Sat Nov 25 2023 23:16:15 GMT+0800 (China Standard Time)

Personally, I don't see that failure as a real error, because all recipes should be using version 2. Interestingly, less than 20 recipes ofover 1600 recipes refer to version 2 at all in their required_conan_version, and if they do it is a combination like required_conan_version = ">=1.56.0 <2 || >=2.0.6".

I think it's because conan2 was released this February, and most of these projects were available since conan1. The recipes were adjusted to support conan2 as well, but they didn't want to break the already existing conan1 support.

SpaceIm · Answer 21 · Sun Nov 26 2023 02:29:33 GMT+0800 (China Standard Time)

Coming from conan-io/conan-center-index#21358

conancenter recipes adjust required_conan_version depending on features they rely on. It turns out that 95% of conancenter recipes use "conan v2" features implemented in conan<=1.53.0, and therefore there is absolutely no reason to ask for a more recent conan version to users (except when you don't like them).

This required_conan_version = ">=1.56.0 <2 || >=2.0.6" (which should be required_conan_version = ">=1.60.0 <2 || >=2.0.5" actually) you can see in few recipes comes from usage of a feature (foo/<host_version> in build requirements when foo is also in requirements) implemented in conan 2.0.5 and backported to conan v1.

Ulrich Telle · Answer 22 · Sun Nov 26 2023 07:42:47 GMT+0800 (China Standard Time)

conancenter recipes adjust required_conan_version depending on features they rely on. It turns out that 95% of conancenter recipes use "conan v2" features implemented in conan<=1.53.0, and therefore there is absolutely no reason to ask for a more recent conan version to users

Well, all over the Conan place one reads about the transition to version 2, and that recipes have to be compatible with version 2. Staying compatible with version 1 means that no "version 2 only" feature can be used.

(except when you don't like them).

Sorry, this remark was unnecessary and inadequate.

This required_conan_version = ">=1.56.0 <2 || >=2.0.6" (which should be required_conan_version = ">=1.60.0 <2 || >=2.0.5" actually) you can see in few recipes comes from usage of a feature (foo/<host_version> in build requirements when foo is also in requirements) implemented in conan 2.0.5 and backported to conan v1.

I changed the version requirement, but the Conan v1 Pipeline check still fails. However, the Conan v2 Pipeline check finally completed successfully a few minutes ago.

Ulrich Telle · Answer 23 · Sun Nov 26 2023 23:06:31 GMT+0800 (China Standard Time)

Finally, both Conan Pipelines succeeded. Now, we have to wait for the required 2 approving reviews.

Zoltan Szilvassy · Answer 24 · Mon Nov 27 2023 02:55:14 GMT+0800 (China Standard Time)

Finally, both Conan Pipelines succeeded. Now, we have to wait for the required 2 approving reviews.

Great! Thank you.

Zoltan Szilvassy · Answer 25 · Sat Jan 20 2024 18:10:10 GMT+0800 (China Standard Time)

HI,
did you receive any news from conan team since?

Ulrich Telle · Answer 26 · Sat Jan 20 2024 18:27:14 GMT+0800 (China Standard Time)

did you receive any news from conan team since?

No. The Conan pipelines all show green, but the PR is still waiting for a second review. On this page you can find a section "Time Spent in Review". The average is well above 30 days at the moment, with many entries way over 60 days. So, just keep your patience. I will close this issue, once the recipe will have been accepted.

Ulrich Telle · Answer 27 · Sun Mar 17 2024 21:22:46 GMT+0800 (China Standard Time)

Week after week is passing by ... nothing happens ... no comments to my/our remarks ... no new review ...

I have to admit that I'm gradually loosing interest in supporting a Conan recipe for my component.

Zoltan Szilvassy · Answer 28 · Sun Mar 17 2024 23:16:50 GMT+0800 (China Standard Time)

Yeah, I didn't expect it would take this much time. I also wonder what's the process of adding new versions, once you have an existing conan package. Hopefully you don't have to wait months for each release.

Ulrich Telle · Answer 29 · Fri Mar 22 2024 05:29:23 GMT+0800 (China Standard Time)

Finally, the Conan package is available: sqlite3mc/1.8.0.

I also wonder what's the process of adding new versions, once you have an existing conan package.

Me too. I guess a PR has to be done to update the Conan recipe for each new version.

Hopefully you don't have to wait months for each release.

I really hope that adding new versions will be a smoother process.