Consider making a future/test build with Cheerp
CetinSert opened this issue · comments
The developer of emscripten himself suggests using Cheerp for projects complaining about memory fragmentation (⚠ add reference).
I know it is too early to conclude anything yet but should we eventually happen to exhaust all of our options with #33, Cheerp might provide a solution. It is GPLv2 for open-source projects and PDFium is open-source with its BSD license.
This is just a suggestion.
https://leaningtech.com/cheerp/
https://medium.com/leaningtech/best-in-class-webassembly-with-cheerp-2-5-rc1-aa44951bc0f0
Getting started
mkdir cheerp
cd cheerp
wget https://launchpad.net/cheerp/trunk/cheerp-2.5/+download/cheerp-llvm-clang-2.5.tar.gz
tar xvf cheerp-llvm-clang-2.5.tar.gz
cd cheerp-llvm-clang-2.5
cat README.md
mkdir build
cd build
cmake -C ../CheerpCmakeConf.cmake -DCMAKE_BUILD_TYPE=Release ..
make #-j16
cd bin
lscetin@sert-works:~/dn/cheerp/cheerp-llvm-clang-2.5/build/bin$ ls
FileCheck clang-import-test llvm-c-test llvm-extract llvm-opt-report llvm-tblgen
arcmt-test clang-offload-bundler llvm-cat llvm-go llvm-pdbutil llvm-xray
bugpoint clang-rename llvm-config llvm-lib llvm-profdata not
c-arcmt-test clang-tblgen llvm-cov llvm-link llvm-ranlib obj2yaml
c-index-test count llvm-cvtres llvm-lit llvm-rc opt
clang diagtool llvm-cxxdump llvm-lto llvm-readelf sancov
clang++ llc llvm-cxxfilt llvm-lto2 llvm-readobj sanstats
clang-6.0 lli llvm-diff llvm-mc llvm-rtdyld scan-build
clang-check lli-child-target llvm-dis llvm-mcmarkup llvm-size scan-view
clang-cl llvm-PerfectShuffle llvm-dlltool llvm-modextract llvm-split verify-uselistorder
clang-cpp llvm-ar llvm-dsymutil llvm-mt llvm-stress yaml-bench
clang-diff llvm-as llvm-dwarfdump llvm-nm llvm-strings yaml2obj
clang-format llvm-bcanalyzer llvm-dwp llvm-objdump llvm-symbolizer
Hi,
Very interesting. I will try it.
I will make a new surgery, so i will be limited this days.
Thanks.
Hi,
I added a first PR here:
#55
The problem is my machine. Out of space to install all cheerp things:
If anyone can test to me and put other things in PR that left, will be nice!
@paulo-coutinho – I will test tomorrow (away from keyboard today) but I am slower than you are with builds.
@paulo-coutinho – DM-ed you on Twitter about disk space.
Hi,
I tried make a new docker image for cheerp, but it too a lot of space.
If anyone help me to build it will be nice.
I stop on problems with snap store install by Google script "install-build-deps.sh" (source code is here: https://chromium.googlesource.com/chromium/src/+/refs/heads/main/build/install-build-deps.sh). I don't know if it already need this package.
Thanks.
@paulo-coutinho – how much space did you have and was not enough?
My work machine has 256gb and can't be changed. My home macbook has 256gb too.
wow ... ok, I was going to set up a dedicated build VM just for cheerp so I am wondering how much space would be enough? 20GB, 50GB, more?
I have a lot of projects, docker images/containers and a lot of files that can be deleted. Im trying remove somethings to free space to still work and compile the projects.
I don't know what the full size because it stop before finish.
If anyone have a better machine, can make the docker image work instead of i need try build, delete all, build from scratch gain and repeat, repeat ...
It is stopping here:
=> => # Preparing to unpack .../000-snapcraft_3.0ubuntu1.1_all.deb ...
=> => # debconf: unable to initialize frontend: Dialog
=> => # debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.)
=> => # debconf: falling back to frontend: Readline
=> => # => Installing the Snapcraft snap
=> => # ==> Checking connectivity with the snap store
I have got a VM setup. I will also configure a laptop right after.
You can join in anytime! (We can exchange credentials on twitter)
- VM via SSH public key
- Laptop via RDP
...
starting with the files in https://github.com/paulo-coutinho/pdfium-lib/pull/55/files
=> => # ==> Checking connectivity with the snap store
I tried all things, now im trying debconf:
FROM ubuntu:20.04
# general
ARG DEBIAN_FRONTEND=noninteractive
ENV PROJ_TARGET="cheerp"
ENV JAVA_VERSION="8"
ENV JAVA_HOME="/usr/lib/jvm/java-8-openjdk-amd64/"
RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections
# packages
RUN apt-get -y update && \
apt-get install -y software-properties-common && \
add-apt-repository ppa:leaningtech-dev/cheerp-ppa && \
apt-get -y update && \
apt-get install -y build-essential sudo file git wget curl ninja-build zip unzip tar software-properties-common openjdk-${JAVA_VERSION}-jdk nano lsb-release libglib2.0-dev tzdata doxygen python3 python3-pip cheerp-core --no-install-recommends && \
apt-get upgrade && \
rm -rf /var/lib/apt/lists/* && \
apt-get clean
# define timezone
RUN echo "America/Sao_Paulo" > /etc/timezone
RUN dpkg-reconfigure -f noninteractive tzdata
RUN /bin/echo -e "LANG=\"en_US.UTF-8\"" > /etc/default/local
# python
RUN python3 --version
RUN pip3 --version
# cmake
RUN pip3 install cmake
RUN cmake --version
# java
ENV PATH=${PATH}:${JAVA_HOME}/bin
RUN echo ${JAVA_HOME}
RUN java -version
# google depot tools
RUN git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git /opt/depot-tools
ENV PATH=${PATH}:/opt/depot-tools
# pdfium - dependencies
RUN mkdir /build
WORKDIR /build
RUN gclient config --unmanaged https://pdfium.googlesource.com/pdfium.git
RUN gclient sync
WORKDIR /build/pdfium
RUN git checkout 31722577db9dc12a249ca6cdf2ff34e51a998360
RUN ln -sf /usr/bin/python3 /usr/bin/python
RUN ln -sf /usr/bin/pip3 /usr/bin/pip
# chromium dependencies
# https://chromium.googlesource.com/chromium/src/+/refs/heads/main/build/install-build-deps.sh
RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections
RUN apt-get install -o APT::Immediate-Configure=false -f apt \
&& apt-get -f install \
&& dpkg --configure -a \
&& apt-get -y dist-upgrade \
&& echo n | ./build/install-build-deps.sh \
&& rm -rf /build
# ninja
RUN ln -nsf /opt/depot-tools/ninja-linux64 /usr/bin/ninja
# dependencies
RUN pip3 install --upgrade pip
RUN pip3 install setuptools docopt python-slugify tqdm
# libjpeg
RUN mkdir /opt/libjpeg
WORKDIR /opt/libjpeg
RUN curl https://ijg.org/files/jpegsrc.v9c.tar.gz -o jpegsrc.v9c.tar.gz
RUN tar -xvf jpegsrc.v9c.tar.gz
WORKDIR /opt/libjpeg/jpeg-9c
RUN ./configure --prefix=/usr
RUN make && make install
# nodejs and npm
RUN curl -sL https://deb.nodesource.com/setup_14.x | sudo -E bash -
RUN apt-get install -y nodejs
RUN npm install -g npm@latest
# compiler version
RUN cc --version
# test
RUN bash -c 'echo "int main() { return 0; }" > /tmp/main.cc'
RUN bash -c 'cc -o /tmp/main.html /tmp/main.cc'
# working dir
WORKDIR /app
While trying to re-fork and run your actions, I have had time to look at the action logs.
I have then googled for people who have had similar problems and their solutions:
- stop and remove lxd
- use a debian (not ubuntu) docker image: https://bugs.launchpad.net/cloud-images/+bug/1869963
Finally i make it works.
The problem now is to generate the sample HTML, like emscripten. Do you know if it has this option?
When i run:
/opt/cheerp/bin/clang++ -o /tmp/main.html /tmp/main.cc
It generate on main.html file:
cat main.html
"use strict";/*Compiled using Cheerp (R) by Leaning Technologies Ltd*/var j=Math.imul;var k=Math.fround;var oSlot=0;var nullArray=[null];var nullObj={d:nullArray,o:0};function K(p){var b=null,f='function';if(typeof fetch===f)b=fetch(p).then(r=>r.arrayBuffer());else if(typeof require===f){p=require('path').join(__dirname, p);b=new Promise((y,n)=>{require('fs').readFile(p,(e,d)=>{if(e)n(e);else y(d);});});}else b=new Promise((y,n)=>{y(read(p,'binary'));});return b;}var __asm=null,__heap=null;function H(){throw new Error('this should be unreachable');};H.promise=K('main.wasm').then(L=>WebAssembly.instantiate(L,{i:{}})).then(L=>{__asm=L.instance.exports;__heap=__asm.I.buffer;G(__heap);__asm.h();});function G(L){}
Thanks.
Finally i make it works.
As github action or locally?
The problem now is to generate the sample HTML, like emscripten. Do you know if it has this option?
No idea, yet... still looking for references.
https://github.com/leaningtech/cheerp-meta/tree/master/pages
Ok, i will generate the JS file and maybe ignore the HTML for this target.
Ah, i make it work locally hacking the Google files inside the Dockerfile, and now im testing to upload to github actions after all steps to compile pdfium works (only left the cheerp compile it with success).
auhauhauhahuahu
Ok, i will generate the JS file and maybe ignore the HTML for this target.
Btw, as soon as you have the resulting .wasm file and its .js bindings, I will also be here to re-test for the memory leak issue!
I stopped on this problem:
https://github.com/leaningtech/cheerp-compiler/issues/39
For reference:
https://github.com/paulo-coutinho/pdfium-lib/runs/4185542707?check_suite_focus=true#step:13:30
Im searching for a solution.
(as a side not)
with a change (← not really sure whether this has had any effect actually), snap finally gave up after a looong time: https://github.com/pdf-ist-internal/pdfium-lib/runs/4185224289?check_suite_focus=true#step:8:3800 without canceling the action but the whole thing still failed here: https://github.com/pdf-ist-internal/pdfium-lib/runs/4185224289?check_suite_focus=true#step:8:5256
I solved this problem on mine. The problem now is with the clang version of cheerp.
I know, waiting for their response at https://github.com/leaningtech/cheerp-compiler/issues/39
They fixed it and the fix is on master, but they disabled the issues pages :(
I don't know if it is available on Ubuntu packages :(
The last version is from 2020:
https://launchpad.net/~leaningtech-dev/+archive/ubuntu/cheerp-ppa
On nightly cheer-core package have only for October: https://launchpad.net/~leaningtech-dev/+archive/ubuntu/cheerp-nightly-ppa
I talked with "Alessandro Pignotti" on Gitter and he sad that in two days +/- a nightly build will be released with the master version.
Hi,
All problems was fixed in new release.
You can check here:
https://github.com/paulocoutinhox/pdfium-lib/releases/tag/5407
The demo is here:
https://pdfviewer.github.io/
Im closing because problems was solved using EMSDK in latest release.
Feel free to donate to help project development.
Thanks.