HTS-lib dependency

Question

HTS-lib dependency

jaudoux opened this issue 5 years ago · comments

I would like to know if there is a way to compile a static binary that integrates the hts-lib avoiding the dynamic link to libhts.so ?

Best regards,
Jérôme.

Brent Pedersen · Answer 1 · Fri Jan 25 2019 22:26:55 GMT+0800 (China Standard Time)

hi @jaudoux, in theory, this is possible. see, for example: https://nim-lang.org/docs/nimc.html#dynliboverride

I think in practice, this would be a challenge due to the many dependencies of htslib. I will have a look again. Last time, I didn't get too far. If there's enough interest, I might also see how/if it's possible to sponsor this (for some nim/C expert) as it would be nice to have.

Brent Pedersen · Answer 2 · Fri Jan 25 2019 22:57:54 GMT+0800 (China Standard Time)

I just played around with this. In mosdepth I added a nim.cfg containing:

@if static:
  passl:"/home/brentp/src/htslib/libhts.a"
  passl:"/home/brentp/src/libdeflate/libdeflate.a"
  passl:"-lhts"
  passl:"-ldeflate"
  passl:"-lz"
  passl:"-llzma"
  passl:"-lcurl"
  passl:"-lbz2"
  passl:"-lcrypto"
  passl:"-lpthread"
  dynlibOverride:"hts"
@end

and then build with:

nim c -d:static -d:release mosdepth

this makes a binary that does not depend on libhts.so (just bz2.so, crypto, pthread, etc. :( ). But, it's a start.
Might be able to use a docker image to help make a static binary for all these libs since we can know the location of the static binaries, e.g. /usr/lib/x86_64-linux-gnu/libbz2.a.

This now seems more doable. I'll look into it more once I clear out my queue of things I need to finish.

jaudoux · Answer 3 · Sat Jan 26 2019 00:22:06 GMT+0800 (China Standard Time)

Hi @brentp,

Thanks a lots for looking into this. This is a nice solution, and yes a Docker image ready to compile nim-hts based software into static binary would help. I'll try on my side to do something and share it on this thread.

Also many thanks for developing hts-nim I recently had a small bioinformatic project with nim and it is really great, I can finally get rid of my perl scripts !!!

Best,
Jérôme.

Brent Pedersen · Answer 4 · Sat Jan 26 2019 00:57:29 GMT+0800 (China Standard Time)

sounds good. the somalier docker file: https://github.com/brentp/somalier/blob/master/docker/Dockerfile

could be a good place to start. it has htslib and nim installed. then you'd just track down the locations of all the static libs. I think we'd have to build lib-curl instead of relying on the apt version.

it's great to hear that hts-nim is getting use.

Brent Pedersen · Answer 5 · Sat Jan 26 2019 10:14:04 GMT+0800 (China Standard Time)

see also: https://github.com/kaushalmodi/hello_musl

jaudoux · Answer 6 · Sun Jan 27 2019 17:17:25 GMT+0800 (China Standard Time)

Okay thanks a lot for all those informations.

FYI : This is the small project that I was working one for to evaluate nim/hts-nim : https://github.com/SeqOne/kraf

Brent Pedersen · Answer 7 · Mon Jan 28 2019 23:58:42 GMT+0800 (China Standard Time)

cool. I added that here: https://github.com/brentp/hts-nim/wiki/Examples

I hope to have time to get an example going with a static build. I think the first step will be getting a static curl build (or just building htslib without libcurl) for the static lib.

Brent Pedersen · Answer 8 · Wed Feb 27 2019 02:24:11 GMT+0800 (China Standard Time)

I have succesfully built a static binary for mosdepth. The nim.cfg in the mosdepth repo contains this:

# in htslib: ./configure --with-libdeflate --disable-libcurl
# nim c -a -d:static -d:release mosdepth.nim
@if static:
  passC:"-static"
  passl:"-static"

  passl:"/home/brentp/src/htslib/libhts.a"
  passl:"/home/brentp/src/libdeflate/libdeflate.a"
  passl:"/usr/lib/x86_64-linux-gnu/liblzma.a"
  passl:"/usr/lib/x86_64-linux-gnu/libz.a"
  passl:"-lbz2"
  passl:"-llzma"
  passl:"-lpthread"
  dynlibOverride:"hts"
  dynlibOverride:"bz2"
  dynlibOverride:"pthread"
@end

I am working to make a docker image in the hts-nim repo that will facilitate making static binaries for anything that uses hts-nim. For now, it will not support libcurl (so no S3 or http/https support). We may be able to add that later.

Brent Pedersen · Answer 9 · Wed Feb 27 2019 12:29:53 GMT+0800 (China Standard Time)

I have this working in a branch: https://github.com/brentp/hts-nim/tree/static#static-builds

You can try this binary (which I built using this static machinery).
static_builder.gz

It will take a bit on the first run as it pulls hts-nim docker but it's pretty fast after that. I have nice static binaries for slivar and mosdepth working fine.

I'd be happy to have anyone who is interested read that section and then give this binary a try.

Brent Pedersen · Answer 10 · Fri Mar 01 2019 02:21:36 GMT+0800 (China Standard Time)

This is now in master. Use the binary here: https://github.com/brentp/hts-nim/releases/tag/v0.2.8

I am interested to hear any feedback.

Max Levine · Answer 11 · Sat Mar 02 2019 01:14:21 GMT+0800 (China Standard Time)

The static binary that I created for https://github.com/papaemmelab/split_bed_by_reads, appears to now have a memory leak.

Brent Pedersen · Answer 12 · Sat Mar 02 2019 01:17:59 GMT+0800 (China Standard Time)

this is (was) a bug in htslib. can you confirm that if you specify threads=0 that there is no leak?

this has been fixed in devel (samtools/htslib#822)
but I'm using the htslib 1.9 release so it will be present for now.

Max Levine · Answer 13 · Sat Mar 02 2019 03:20:24 GMT+0800 (China Standard Time)

Yep threads=0 fixes it!

Brent Pedersen · Answer 14 · Sat Mar 02 2019 03:38:44 GMT+0800 (China Standard Time)

ok. I'll update the docker image once the new version of htslib comes out so this will be resolved. it only manifests with large numbers of queries. thanks for giving it a try!

Brent Pedersen · Answer 15 · Thu Mar 14 2019 05:46:52 GMT+0800 (China Standard Time)

since I know at least 1 other project (besides mine) has static builds, I think this is solved. I hope to have libcurl support, but that will depend on updates for alpine linux itself.