Issues regarding building this poject since merging in the NLP core
patrickgold opened this issue · comments
This issue serves as a collection for build issues (mainly regarding the native C++ code) using the new build config (state deaf329 or newer) which has been introduced by the NLP implementation.
Currently I am investigating and collecting all failures so I can try to resolve them. If you are affected by this, please make sure to post a comment describing the build issue you are facing, please make sure to also include some details on the host OS you try to compile FlorisBoard on, and if applicable please also include screenshots, CMake error output or logs (preferably as a details/summary block so this thread remains readable). Before you post please make sure you've installed all the build dependencies when building for Android though!
Again please note that this issue is exclusively for the current code base, any build issues regarding the build config from 0.4.0-alpha04 or earlier are not relevant here.
Thanks!
My findings:
- Major compilation/linking issues on MacOS, maybe linker flags are incorrect??
- Build instructions are unclear (reworked the README in florisboard/nlp#6)
- CMake script has Python path hard-coded (fixed in florisboard/nlp#6)
- CMake script needs to do stricter host OS checks (fixed in florisboard/nlp#6)
On my arch installation GNU make is used, but the command is make, not gmake.
To solve, i can create the link manually, ln -s /usr/bin/make ~/.local/bin/gmake. Then the build works.
ymmw if the path looks different or you have a different make
Interesting - for finding GNU make I use CMake's find_program() utility, so normally that path shouldn't even be hard-coded. But will check that anyways, thanks for bringing it up.
As discussed in the matrix channel, I had a few difficulties while trying to build the project but those are now hopefully behind :)
So I thought I'd share what I've done in a 'step by step' form
This is intended for Ubuntu 22.04. It has been tested on native Linux, and in docker on both Linux and Windows host (using WSL2)
- Prerequisites
apt update && apt install -y wget unzip
- Android SDK
apt install -y android-sdk
- Java 17
apt install -y openjdk-17-jdk openjdk-17-jre
- Android NDK r25
cd /root/ &&\
wget https://dl.google.com/android/repository/android-ndk-r25c-linux.zip && \
unzip android-ndk-r25c-linux.zip
- CMake 3.22+
apt install -y cmake
- Ninja 1.10+
wget -qO /usr/local/bin/ninja.gz https://github.com/ninja-build/ninja/releases/latest/download/ninja-linux.zip && \
gunzip /usr/local/bin/ninja.gz && \
chmod a+x /usr/local/bin/ninja
- GNU make 3.80+
apt install -y make
- Clang 14.x+ & libc++ (bundled with Android NDK)
apt install -y clang
- Python 3.10+
apt install -y python3
- Update the environment variable :
export ANDROID_HOME="/usr/lib/android-sdk" &&\
export ANDROID_NDK_HOME="/root/android-ndk-r25c"
- Accept the licences :
cd /root/ &&\
wget https://dl.google.com/android/repository/commandlinetools-linux-9477386_latest.zip && \
unzip commandlinetools-linux-9477386_latest.zip &&\
cd /root/cmdline-tools/bin && \
yes | ./sdkmanager --licenses --sdk_root=$ANDROID_HOME
- Update local.properties (may not be mandatory)
cd /path/to/florisboard/directory &&\
echo "sdk.dir=$ANDROID_HOME" >> local.properties
- After that, head to the Florisboard directory, and
./gradlew clean && ./gradlew assembleDebug
It should work :)
On my arch installation GNU make is used, but the command is make, not gmake.
@nettnikl In florisboard/nlp#6 this issue should now be fixed, I didn't hardcode the path but the exec name. Now either gmake or make is accepted. Can you try to build only the NLP core using the state from florisboard/nlp#6 and without your symlink to confirm this is fixed?
As discussed in the matrix channel, I had a few difficulties while trying to build the project but those are now hopefully behind :)
So I thought I'd share what I've done in a 'step by step' form
That's a really cool step-by-step guide. Do you mind if I might "steal" this guide for the planned BUILDING.md document (I will of course attribute you and link to this comment)?
That's a really cool step-by-step guide. Do you mind if I might "steal" this guide for the planned BUILDING.md document (I will of course attribute you and link to this comment)?
No problem, go on !
This was originally intended to create a docker building environment (so that people could just pull an image, start a container and be done with it) but of course the guide mainly stays valid for a native Debian environment
Hi all, I'm running into a new build issue wondering if anyone has encountered something similar. My purpose is just to be able to build it and install on my phone.
Steps I took:
- followed all the steps here #2218 (comment)
- cloned https://github.com/florisboard/florisboard.git and the recursively initialized all submodules
Error:
initially I thought this might be a python error since my system default comes with python3.8 but I aliased python3 as python 3.10.
> Task :app:configureCMakeDebug[armeabi-v7a] FAILED
C/C++: Traceback (most recent call last):
C/C++: File "/home/<user_name>/androidprojects/florisboard/app/src/main/cpp/nlp/utils/rewrite_cppm_to_header.py", line 53, in <module>
C/C++: def extract_hmp(extract_match: re.Match[str]) -> Tuple[str, str, str]:
C/C++: TypeError: 'type' object is not subscriptable
C/C++: CMake Error at nlp/CMakeModules/RewriteCppmToHeader.cmake:22 (message):
C/C++: Failed to rewrite source file
C/C++: /home/<user_name>/androidprojects/florisboard/app/src/main/cpp/nlp/icuext/src/icuext.cppm
C/C++: with error code '1' and message ''
C/C++: Call Stack (most recent call first):
C/C++: nlp/icuext/src/CMakeLists.txt:4 (target_module_sources)
my environment is
wsl2 with ubuntu 22
iirc nlp was merged sometime in early May (5/7), hard resetting head to some time before that (like April) results in a similar error
one more follow up question is, I'm not sure if I understand the purpose of the standalone nlp repository here
https://github.com/florisboard/nlp
am I supposed to clone that separately and build it, or would the recursive init submodules handle this?
Thanks!
Hello,
@HSQ8 I have been able to reproduce your issue in Ubuntu 22.04
In fact, the problem seems to arise with the command ./gradlew clean when using Python 3.8. My guess would be that your symlink could be broken (ie python3 --version would not return Python 3.10). If that's the case I can recommend either trying to fix your symlink or uninstalling 3.8 and installing 3.10 instead.
one more follow up question is, I'm not sure if I understand the purpose of the standalone nlp repository here
https://github.com/florisboard/nlp
am I supposed to clone that separately and build it, or would the recursive init submodules handle this?
The nlp repository is indeed supposed to be cloned as a florisboard submodules, so when setting up florisboard project : git clone --recurse https://github.com/florisboard/florisboard
In addition, it seems I have missed a few crucial steps in my previous message (those have now been added to the original message) :
- Install
unzipin the prerequisitesapt install unzip - Installation of android-sdk
apt install -y android-sdk - Grant permissions for ninja executable
chmod a+x /usr/local/bin/ninja - Add android sdk path to local.properties file
cd /path/to/florisboard/directory && echo "sdk.dir=$ANDROID_HOME" >> local.properties(this one may not actually be mandatory, but feels more secure to me than environment variables)
one more follow up question is, I'm not sure if I understand the purpose of the standalone nlp repository here
https://github.com/florisboard/nlp
am I supposed to clone that separately and build it, or would the recursive init submodules handle this?
The purpose of the separate NLP repository is because the NLP logic is really complex and the core serves a dual purpose of being the suggestion core on mobile devices and being the preprocess core on desktops (because the default dictionaries need to be custom built). Also debugging just the NLP core is easier on desktops than on mobile devices.
my environment is
wsl2 with ubuntu 22
This environment has been tested by me and as long as you use a custom clang-16 executable it compiles perfectly fine. However if you target ANDROID then the clang shipeed with the Android SDK tools should suffice.
C/C++: Traceback (most recent call last):
C/C++: File "/home/<user_name>/androidprojects/florisboard/app/src/main/cpp/nlp/utils/rewrite_cppm_to_header.py", line 53, in
C/C++: def extract_hmp(extract_match: re.Match[str]) -> Tuple[str, str, str]:
C/C++: TypeError: 'type' object is not subscriptable
This points to the fact that the Python rewrite utility fails, and looking at the error it isn't even a logic error but simply the fact that your Python typing module does not allow specifying a re.Match[str] for whatever reason. I have just investigated on my system and Python 3.10 should support this syntax. Looking at the documentation support for this was introduced in Python 3.9, so ensure again that you aren't using Python 3.8 or earlier.
Installation of android-sdk apt install -y android-sdk
That's something I need to add to the dependencies, I always assume that Android Studio is already installed which would do this in the background.
Install unzip in the prerequisites apt install unzip
Which command in the build process depends on unzip?
Which command in the build process depends on unzip?
I use it to extract the command line tools, only used to accept the android sdk licences. It is the only "automatic" way I have found to do so
Ah I see thanks for explaining
[at]everyone still affected with build issues: In #2209 I've updated the NLP submodule, which now both contains an improved error detection system in the CMake configs for dependencies as well as a reworked README. Using the new build instructions can you now build the project successfully?
For MacOS users: I have adjusted the compiler check and linker flags, however as I do not own a Macbook I currently do not have a simple way of testing it out on MacOS, so I would need additional feedback from you.
@patrickgold Regarding the macOS build, would it be of help if one implemented a CI pipeline to build the NLP submodule through Github Actions? Github apparently offers runners that run macOS 12 (see this post).
Regarding the macOS build, would it be of help if one implemented a CI pipeline to build the NLP submodule through Github Actions?
It would certainly be a help, yes, especially because it would allow me to test new changes in the build system in the future too.
I'll give that a try then :)
ref: https://github.com/leso-kn/florisboard/tree/feature/github-actions-macos
done @patrickgold. Okay, so I've extended the existing workflow by a matrix job definition, that will build for both ubuntu-latest and macos-latest.
At this point the build fails due to gmake not being found, but reading up on the conversation here I assume I just didn't pull your latest changes to my florisboard-fork.
The change is quite small really :) (see here) So the question now would be if I should create a PR for it, or do you prefer to cherry-pick the change to your feature branch first to do some testing on your end?
Side note: The matrix job does make it quite simple to include a windows-latest build, though that depends on your preference.
Building for multiple platforms in theory increases the maintenance effort for the workflow configuration (though in this case at least macos seems to be just happy with the defaults from the existing ubuntu-latest job).
Update: FYI, building against the latest version of florisboard/nlp (master) does result in a different error, see the build log (with nlp latest [master])
Hi all, I'm running into a new build issue wondering if anyone has encountered something similar. My purpose is just to be able to build it and install on my phone.
Steps I took:
- followed all the steps here Issues regarding building this poject since merging in the NLP core #2218 (comment)
- cloned https://github.com/florisboard/florisboard.git and the recursively initialized all submodules
Error:
initially I thought this might be a python error since my system default comes with python3.8 but I aliased python3 as python 3.10.
> Task :app:configureCMakeDebug[armeabi-v7a] FAILED C/C++: Traceback (most recent call last): C/C++: File "/home/<user_name>/androidprojects/florisboard/app/src/main/cpp/nlp/utils/rewrite_cppm_to_header.py", line 53, in <module> C/C++: def extract_hmp(extract_match: re.Match[str]) -> Tuple[str, str, str]: C/C++: TypeError: 'type' object is not subscriptable C/C++: CMake Error at nlp/CMakeModules/RewriteCppmToHeader.cmake:22 (message): C/C++: Failed to rewrite source file C/C++: /home/<user_name>/androidprojects/florisboard/app/src/main/cpp/nlp/icuext/src/icuext.cppm C/C++: with error code '1' and message '' C/C++: Call Stack (most recent call first): C/C++: nlp/icuext/src/CMakeLists.txt:4 (target_module_sources)my environment is wsl2 with ubuntu 22
iirc nlp was merged sometime in early May (5/7), hard resetting head to some time before that (like April) results in a similar error
one more follow up question is, I'm not sure if I understand the purpose of the standalone nlp repository here https://github.com/florisboard/nlp am I supposed to clone that separately and build it, or would the recursive init submodules handle this?
Thanks!
Hello. I also got the same error. Did you fix it yet? I am stuck at this error with Window WSL2. Thank you !
Getting this error -- The C compiler identification is Clang 17.0.2
-- The CXX compiler identification is Clang 17.0.2
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done