Compile / Installation problem

Question

Compile / Installation problem

vinisalazar opened this issue 5 years ago · comments

Hi,

I saw your arXiv paper and was excited to try the lib. When I try to compile pugz, I get this:

(base)  vini@mussismilia  ~/code/pugz   master  make
  AR       libdeflate.a  
  CXX      programs/gunzip.o  
In file included from programs/../lib/deflate_decompress.hpp:58:0,  
                 from programs/../lib/gzip_decompress.hpp:40,  
                 from programs/gunzip.cpp:28:  
programs/../lib/memory.hpp: In function 'malloc_span<T> alloc_huge(size_t)':  
programs/../lib/memory.hpp:382:38: error: 'MADV_HUGEPAGE' was not declared in this scope  
     auto res = ::madvise(ptr, bytes, MADV_HUGEPAGE);
                                      ^~~~~~~~~~~~~
programs/../lib/memory.hpp:382:38: note: suggested alternative: 'MADV_MERGEABLE'
     auto res = ::madvise(ptr, bytes, MADV_HUGEPAGE);
                                      ^~~~~~~~~~~~~
                                      MADV_MERGEABLE
make: *** [Makefile:179: programs/gunzip.o] Error 1

I followed the CLI's suggestion and replaced those strings by running sed -i -- 's/MADV_HUGEPAGE/MADV_MERGEABLE/g' */* . This changed lib/memory.hpp and programs/prog_util.hpp Obviously changing source code unadvised is not recommended, but I was only experimenting.

I then would get this error:

(base)  vini@mussismilia  ~/code/pugz   master ●  make
  CXX      programs/gunzip.o  
  CXX      programs/prog_util.o  
  CXX      programs/tgetopt.o  
  CCLD     gunzip  
/home/vini/anaconda3/bin/../lib/gcc/x86_64-conda_cos6-linux-gnu/7.3.0/../../../../x86_64-conda_cos6-linux-gnu/bin/ld: /tmp/ccLWiWLa.ltrans0.ltrans.o: undefined reference to symbol 'pthread_create@@GLIBC_2.2.5'
/home/vini/anaconda3/bin/../lib/gcc/x86_64-conda_cos6-linux-gnu/7.3.0/../../../../x86_64-conda_cos6-linux-gnu/bin/ld: /home/vini/anaconda3/bin/../x86_64-conda_cos6-linux-gnu/sysroot/lib/libpthread.so.0: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
make: *** [Makefile:187: gunzip] Error 1

This was resolved by adding -pthread to the compiler flags in the Makefile.

So now the program compiled without errors. But it does not work:

(base)  vini@mussismilia  ~/code/pugz   master ●  ./gunzip predicted_bkp.tar.gz
terminate called after throwing an instance of 'gzip_error'
  what():  INVALID_LITERAL
[1]    41942 abort      ./gunzip predicted_bkp.tar.gz

Any ideas?

Thank you for any assistance you can provide.

PS: I read thorugh #8 but because I would get a different error, thought it would be good to create a different issue.

Rayan Chikhi · Answer 1 · Thu May 23 2019 03:06:53 GMT+0800 (China Standard Time)

Hi, what's your system and gcc version please? (commands uname -ar and gcc --version). Also tar.gz files cannot be decompressed using pugz yet as they're not text files, sorry.

Vini Salazar · Answer 2 · Thu May 23 2019 03:30:03 GMT+0800 (China Standard Time)

Output of uname -ar:
Linux mussismilia.cl.biologia.ufrj.br 3.7.10-1.40-desktop #1 SMP PREEMPT Thu Jul 10 11:22:12 UTC 2014 (9b06319) x86_64 x86_64 x86_64 GNU/Linux

Output of gcc --version:
gcc (SUSE Linux) 4.7.2 20130108 [gcc-4_7-branch revision 195012]

The .tar explanation makes sense. Thanks.

Vini Salazar · Answer 3 · Thu May 23 2019 03:38:47 GMT+0800 (China Standard Time)

Hi, I just tested it on a text file (not .tar) and it worked. I made these small changes like I mentioned, so it could compile.

Rayan Chikhi · Answer 4 · Thu May 23 2019 04:50:04 GMT+0800 (China Standard Time)

Thanks for this investigation. I think @Piezoid will follow-up.

Maël Kerbiriou · Answer 5 · Thu May 23 2019 07:56:07 GMT+0800 (China Standard Time)

Hi, thanks for the report.

The INVALID_LITERAL is triggered because non ASCII files are not supported.
In theory the algorithm could handle it, but with worse performance. So we don't have yet a mode that degrade to binary file handling.

It's seems that MADV_HUGEPAGE is not supported by your libc (from 2002 it seems, but your kernel does support it). It's only a hint to the virtual memory subsystem to spare some TLB entries. We should disable it when MADV_HUGEPAGE is not defined.

I'll substitute -lpthread for -pthread, it's seems necessary to support weird builds environment such as a half used anaconda... (If gcc is from your system, why it is trying to use anaconda's libs ?)

My freshly installed, but already crippled anaconda is stuck on "Solving environment: /" so I can't reproduce right now.

Could you try this and paste the resulting link, please ? It's for diagnostics.

git stash push Makefile # stash your modified makefile
{ echo LD_LIBRARY_PATH=$LD_LIBRARY_PATH; which "${CXX:-g++}"; make clean; make V=1 CFLAGS='-v -Wl,-v' } 2>&1 | curl -F 'sprunge=<-' http://sprunge.us 
git stash pop # get it back

Rayan Chikhi · Answer 6 · Thu May 23 2019 20:01:54 GMT+0800 (China Standard Time)

@Piezoid do you mean git push or git stash?

Maël Kerbiriou · Answer 7 · Thu May 23 2019 20:31:52 GMT+0800 (China Standard Time)

Oops, it's git stash push indeed.

Maël Kerbiriou · Answer 8 · Mon May 27 2019 19:33:56 GMT+0800 (China Standard Time)

This issue should be fixed in master. Can you give a try ?

Vini Salazar · Answer 9 · Wed May 29 2019 01:33:09 GMT+0800 (China Standard Time)

Hi @Piezoid and @rchikhi ,

I made a fresh clone from master and it compiled and worked flawlessly, so that seems to have fixed it.

Thank you very much for your prompt response and fix. I will be closing this issue.