nodegit / nodegit

Native Node bindings to Git.

Home Page:https://www.nodegit.org/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Cannot be installed under Docker Alpine

snics opened this issue · comments

I would like to use NodeGit under Docker with the Node alpine image but its doesn't work. Do you know one of the ways to solve the problem?

I install NodeGit with yarn i get the following error message:

Error: Error loading shared library /home/node/node_modules/nodegit/build/Release/nodegit.node: Exec format error
at Object.Module._extensions..node (module.js:598:18)
at Module.load (module.js:503:32)
at tryModuleLoad (module.js:466:12)
at Function.Module._load (module.js:458:3)
at Module.require (module.js:513:17)
at require (internal/module.js:11:18)
at Object.<anonymous> (/home/node/node_modules/nodegit/dist/nodegit.js:11:12)
at Module._compile (module.js:569:30)
at Object.Module._extensions..js (module.js:580:10)
at Module.load (module.js:503:32)
at tryModuleLoad (module.js:466:12)
at Function.Module._load (module.js:458:3)
at Module.require (module.js:513:17)
at require (internal/module.js:11:18)
at Object.<anonymous> (/home/node/src/utils/versioning.js:11:17)
at Module._compile (module.js:569:30)
at Object.Module._extensions..js (module.js:580:10)
at Module.load (module.js:503:32)
at tryModuleLoad (module.js:466:12)
at Function.Module._load (module.js:458:3)
at Module.require (module.js:513:17)
at require (internal/module.js:11:18)

I install NodeGit with npm i get the following error message:

> nodegit@0.20.1 install /home/node/node_modules/nodegit
> node lifecycleScripts/preinstall && node lifecycleScripts/install

[nodegit] Running pre-install script
[nodegit] Configuring libssh2.
{ Error: Command failed: /home/node/node_modules/nodegit/vendor/libssh2/configure --with-libssl-prefix=/home/node/node_modules/nodegit/vendor/openssl/openssl
/home/node/node_modules/nodegit/vendor/libssh2/missing: Unknown `--is-lightweight' option
Try `/home/node/node_modules/nodegit/vendor/libssh2/missing --help' for more information
configure: WARNING: 'missing' script is too old or missing
/home/node/node_modules/nodegit/vendor/libssh2/configure: ./configure.lineno: line 1: /usr/bin/file: not found
configure: error: No crypto library found!
Try --with-libssl-prefix=PATH
 or --with-libgcrypt-prefix=PATH
 or --with-wincng on Windows

    at ChildProcess.exithandler (child_process.js:252:12)
    at emitTwo (events.js:125:13)
    at ChildProcess.emit (events.js:213:7)
    at maybeClose (internal/child_process.js:887:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:208:5)
  killed: false,
  code: 1,
  signal: null,
  cmd: '/home/node/node_modules/nodegit/vendor/libssh2/configure --with-libssl-prefix=/home/node/node_modules/nodegit/vendor/openssl/openssl' }
/home/node/node_modules/nodegit/vendor/libssh2/missing: Unknown `--is-lightweight' option
Try `/home/node/node_modules/nodegit/vendor/libssh2/missing --help' for more information
configure: WARNING: 'missing' script is too old or missing
/home/node/node_modules/nodegit/vendor/libssh2/configure: ./configure.lineno: line 1: /usr/bin/file: not found
configure: error: No crypto library found!
Try --with-libssl-prefix=PATH
 or --with-libgcrypt-prefix=PATH
 or --with-wincng on Windows

[nodegit] ERROR - Could not finish preinstall
{ Error: Command failed: /home/node/node_modules/nodegit/vendor/libssh2/configure --with-libssl-prefix=/home/node/node_modules/nodegit/vendor/openssl/openssl
/home/node/node_modules/nodegit/vendor/libssh2/missing: Unknown `--is-lightweight' option
Try `/home/node/node_modules/nodegit/vendor/libssh2/missing --help' for more information
configure: WARNING: 'missing' script is too old or missing
/home/node/node_modules/nodegit/vendor/libssh2/configure: ./configure.lineno: line 1: /usr/bin/file: not found
configure: error: No crypto library found!
Try --with-libssl-prefix=PATH
 or --with-libgcrypt-prefix=PATH
 or --with-wincng on Windows

    at ChildProcess.exithandler (child_process.js:252:12)
    at emitTwo (events.js:125:13)
    at ChildProcess.emit (events.js:213:7)
    at maybeClose (internal/child_process.js:887:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:208:5)
  killed: false,
  code: 1,
  signal: null,
  cmd: '/home/node/node_modules/nodegit/vendor/libssh2/configure --with-libssl-prefix=/home/node/node_modules/nodegit/vendor/openssl/openssl' }
npm info lifecycle nodegit@0.20.1~install: Failed to exec install script
npm WARN vao-api@1.0.0 No repository field.

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! nodegit@0.20.1 install: `node lifecycleScripts/preinstall && node lifecycleScripts/install`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the nodegit@0.20.1 install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2017-08-30T10_48_59_977Z-debug.log

I did some testing myself but got stuck at libcurl-gnutls.so.4. I couldn't figure out what package would add that shared library to Alpine Linux.

> nodegit@0.20.1 postinstall /node_modules/nodegit
> node lifecycleScripts/postinstall

[nodegit] WARN - Could not finish postinstall
{ Error: Command failed: node "/node_modules/nodegit/dist/nodegit.js"
/node_modules/nodegit/dist/nodegit.js:15
    throw ex;
    ^

Error: Error loading shared library libcurl-gnutls.so.4: No such file or directory (needed by /node_modules/nodegit/build/Release/nodegit.node)
    at Object.Module._extensions..node (module.js:602:18)
    at Module.load (module.js:507:32)
    at tryModuleLoad (module.js:470:12)
    at Function.Module._load (module.js:462:3)
    at Module.require (module.js:517:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/node_modules/nodegit/dist/nodegit.js:11:12)
    at Module._compile (module.js:573:30)
    at Object.Module._extensions..js (module.js:584:10)
    at Module.load (module.js:507:32)

    at ChildProcess.exithandler (child_process.js:270:12)
    at emitTwo (events.js:125:13)
    at ChildProcess.emit (events.js:213:7)
    at maybeClose (internal/child_process.js:927:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:211:5)
  killed: false,
  code: 1,
  signal: null,
  cmd: 'node "/node_modules/nodegit/dist/nodegit.js"' }

@rcjsuen I have installed the following package libgit2-dev and I think this has all dependencies for NodeGit. For example I add my dockerfile below

FROM node:8.1.0-alpine

COPY ./ /home/node

WORKDIR /home/node

# Git install
RUN mkdir /nodegit && \
    cd /nodegit && \
    apk update && \
    apk upgrade && \
    apk add git libgit2-dev && \
    apk add python tzdata pkgconfig build-base && \
    yarn add -E nodegit@0.20 && \
    apk del python tzdata pkgconfig build-base && \
    rm yarn.lock package.json && \
    rm -rf /tmp/* /var/cache/apk/* && \
    yarn cache clean && \

# Install Node modules
RUN rm -rf package-lock.json node_modules \
    && apk add --update --no-cache --virtual .build-deps \
    curl \
    g++ \
    gcc \
    gnupg \
    libgcc \
    make \
    alpine-sdk \
    python \
    && npm config set unsafe-perm true \
    && npm install \
    && npm prune --production \
    && apk del .build-deps \
    && rm -rf /usr/share/man /tmp/* /var/cache/apk/* \
    && ls -a /nodegit

EXPOSE 8000

USER node
CMD ["node", "index"]

I think this is a duplicate of #1246.

There are two ways to solve the problem.

  1. Create the missing symlink that the precompiled binary is depending on.

     sudo ln -s /usr/lib64/libcurl.so.4 /usr/lib64/libcurl-gnutls.so.4
    
  2. Force nodegit to be recompiled at install time.

     BUILD_ONLY=true npm install nodegit
    

The second option slows down installation considerably.

I couldn't figure out what package would add that shared library to Alpine Linux.

The problem is, this is not the normal name of this library. It's normally called libcurl.so.4 and installed by libcurl.

I'm trying to run nodegit in a Docker container using the base image node:8-alpine but having no luck with the error:

RUN npm install nodegit
 ---> Running in b99980a70b1b

> nodegit@0.20.3 install /node_modules/nodegit
> node lifecycleScripts/preinstall && node lifecycleScripts/install

[nodegit] Running pre-install script
[nodegit] Configuring libssh2.
{ Error: Command failed: /node_modules/nodegit/vendor/libssh2/configure --with-libssl-prefix=/node_modules/nodegit/vendor/openssl/openssl
/node_modules/nodegit/vendor/libssh2/missing: Unknown `--is-lightweight' option
Try `/node_modules/nodegit/vendor/libssh2/missing --help' for more information
configure: WARNING: 'missing' script is too old or missing
configure: error: in `/node_modules/nodegit/vendor/libssh2':
configure: error: no acceptable C compiler found in $PATH
See `config.log' for more details

    at ChildProcess.exithandler (child_process.js:275:12)
    at emitTwo (events.js:126:13)
    at ChildProcess.emit (events.js:214:7)
    at maybeClose (internal/child_process.js:925:16)
    at Socket.stream.socket.on (internal/child_process.js:346:11)
    at emitOne (events.js:116:13)
    at Socket.emit (events.js:211:7)
    at Pipe._handle.close [as _onclose] (net.js:554:12)
  killed: false,
  code: 1,
  signal: null,
  cmd: '/node_modules/nodegit/vendor/libssh2/configure --with-libssl-prefix=/node_modules/nodegit/vendor/openssl/openssl' }
/node_modules/nodegit/vendor/libssh2/missing: Unknown `--is-lightweight' option
Try `/node_modules/nodegit/vendor/libssh2/missing --help' for more information
configure: WARNING: 'missing' script is too old or missing
configure: error: in `/node_modules/nodegit/vendor/libssh2':
configure: error: no acceptable C compiler found in $PATH
See `config.log' for more details

[nodegit] ERROR - Could not finish preinstall
{ Error: Command failed: /node_modules/nodegit/vendor/libssh2/configure --with-libssl-prefix=/node_modules/nodegit/vendor/openssl/openssl
/node_modules/nodegit/vendor/libssh2/missing: Unknown `--is-lightweight' option
Try `/node_modules/nodegit/vendor/libssh2/missing --help' for more information
configure: WARNING: 'missing' script is too old or missing
configure: error: in `/node_modules/nodegit/vendor/libssh2':
configure: error: no acceptable C compiler found in $PATH
See `config.log' for more details

    at ChildProcess.exithandler (child_process.js:275:12)
    at emitTwo (events.js:126:13)
    at ChildProcess.emit (events.js:214:7)
    at maybeClose (internal/child_process.js:925:16)
    at Socket.stream.socket.on (internal/child_process.js:346:11)
    at emitOne (events.js:116:13)
    at Socket.emit (events.js:211:7)
    at Pipe._handle.close [as _onclose] (net.js:554:12)
  killed: false,
  code: 1,
  signal: null,
  cmd: '/node_modules/nodegit/vendor/libssh2/configure --with-libssl-prefix=/node_modules/nodegit/vendor/openssl/openssl' }
npm WARN microclimate-portal@1.0.0 No description
npm WARN microclimate-portal@1.0.0 No repository field.

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! nodegit@0.20.3 install: `node lifecycleScripts/preinstall && node lifecycleScripts/install`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the nodegit@0.20.3 install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2018-02-20T13_59_09_275Z-debug.log
The command '/bin/sh -c npm install nodegit --production' returned a non-zero code: 1

I've tried both the suggested options above and installing via a git clone but it still doesn't work.

Any suggestions of things I can try? apk packages to add?

Thanks in advance

In order to install on Alpine, you need the following packages:

apk --no-cache -q add build-base libgit2-dev

You also need this symlink:

ln -s /usr/lib/libcurl.so.4 /usr/lib/libcurl-gnutls.so.4

Then you can install and use nodegit on Alpine.

@mojavelinux Thanks for getting back to me.

The above allows me to run nodegit but when I Git.clone a valid repository it crashes my application with a segmentation error.
I've not seen anything to do with segmentation errors in alpine in an issue so is this new or is there already a known fix?

Segmentation fault
npm ERR! code ELIFECYCLE
npm ERR! errno 139
npm ERR! github-api@1.0.0 start: `node server.js`
npm ERR! Exit status 139
npm ERR! 
npm ERR! Failed at the github-api@1.0.0 start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2018-02-21T09_59_06_487Z-debug.log

I'm using nodegit successfully in my application on Alpine, so I know it works in general (including clone). However, what I've learned is that if you forget to await an operation and close the repository prematurally, then nodegit will hard crash. So if not called correctly, nodegit is known to segfault in my experience.

What I recommend is to create a minimal script that reproduces the problem. That will either allow you to pinpoint the problem or identify a problem in nodegit.

@mojavelinux this is all I am doing to cause it to crash.
In my case I don't use the repository object. All I want to do is clone a repository and then send a 200 status back in the response from my API.
You've mentioned using await but I'm not sure where/why I'd add that in

const Git = require("nodegit");
const express = require('express');
const bodyParser = require('body-parser');
const app = express();
const serverPort = 9030;
const server = app.listen(serverPort);
const path = require('path');

app.use(bodyParser.urlencoded({ extended: false }));

app.post('/api/v1/import', function(req, res) {
  if (!req.body.repo) res.sendStatus(400).end();
  try {
    let repo = req.body.repo;
    // get the name of the git repository from the repo given
    let dir = path.basename(repo.replace('.git', ''));
    if (req.body.dir) dir = req.body.dir;
    // make dir the full path
    dir = `${__dirname}${dir}`;
    console.log(repo);
    console.log(dir);
    Git.Clone(repo, dir).then(function() {
      res.sendStatus(200).end();
    }).catch(function(err) {
      if (err.message.indexOf('authentication required') > -1) {
        res.header('authentication-required', 'true');
      }
      res.sendStatus(500).end();
      console.log(err);
    });
  } catch (err) {
    res.sendStatus(500).end();
    console.log(err);
  }
});

Here's how I use it. Notice I can't clone an https repo without providing fetch options.

https://gitlab.com/antora/antora/blob/master/packages/content-aggregator/lib/aggregate-content.js#L129

Notice I'm using Git.Clone.clone, not Git.Clone.

I have over 100 tests cloning repositories of different types, so I'm pretty confident that call to clone is a good one.

@mojavelinux I've added the Git.Clone.clone and added your fetchOpts and its function as a test case, still failing with a segmentation error.

I've simplified my function to be:

function checkWorks() {
 // Uses the gitFetchOptions from your gitlab
  Git.Clone.clone('https://github.com/nodegit/nodegit.git', '/tmp/nodegit', { bare: 1, fetchOpts: getFetchOptions() }).then(function(repo) {
    console.log('done');
  }).catch(function(err) {
    console.log(err);
  });
}

But I'm still getting the same segmentation error. At this point i'm clueless on what I can do to fix it. Should I be adding an await somewhere or a repo.close()?

Thanks again

This is my Dockerfile in case I've created the issue in it

FROM node:8-alpine

RUN apk --update --no-cache -q add docker zip git grep bash sudo curl build-base libgit2-dev
RUN ln -s /usr/lib/libcurl.so.4 /usr/lib/libcurl-gnutls.so.4

ADD package.json server.js ./

RUN npm install

CMD ["npm", "start"]

You are indeed correct. Cloning a remote repository on Alpine fails in a segfault.

I distilled the script even further:

const git = require('nodegit')

;(async () => {
  const repo = await git.Clone.clone('http://github.com/sindresorhus/awesome.git', 'clone', {
    fetchOpts: {
      callbacks: {
        certificateCheck: () => 1,
      },
    },
  })
})()

You may be interested in researching isomorphic git (isogit for short). It's a git client written completely in JavaScript. It's quite as fast as nodegit, but it is a heck of a lot more stable and easy to install. I'm evaluating it as a substitute git client in my application.

I now have a deeper understanding of this issue and have discovered a workaround.

To install the pre-built nodegit package on Alpine, the following packages are required:

apk --no-cache add gcc libc-dev libressl-dev

That allows you to install nodegit and use it to read and write local repositories. However, if you attempt to clone a remote repository, the process will segfault. The workaround for this problem is to rebuild nodegit at the time it is installed.

To rebuild nodegit at installation time, you first need to following additional packages:

apk --no-cache add make python curl-dev g++

Next, install nodegit with the BUILD_ONLY=true environment variable:

BUILD_ONLY=true npm install nodegit

or, with yarn:

BUILD_ONLY=true yarn add nodegit

The BUILD_ONLY=true environment variable can be used when installing any package which itself depends on nodegit. It just needs to be set at the time nodegit is being installed.

Compilation takes a long time, but I can confirm that it does solve the segfault problem. I tested this process using the node:8-alpine Docker image.

I've distilled these steps in the following comment: https://gitlab.com/antora/docker-antora/issues/2#note_61275108

Thanks for sharing your workaround folks! Here's my solution for a microservice on node 10:

FROM node:10.15.1-alpine

RUN mkdir -p /app/
WORKDIR /app/
COPY package.json yarn.lock /app/

## https://github.com/nodegit/nodegit/issues/1361
RUN apk --no-cache add curl-dev g++ make python \
    && BUILD_ONLY=true YARN_CACHE_FOLDER=/dev/shm/yarn_cache yarn --production \
    && apk del g++ make python

COPY src /app/src/

CMD ["yarn", "start:prod"]

Note that I install and delete packages in one RUN. This makes the final image significantly smaller since only curl-dev makes it into it.

Hi @mojavelinux Hi have today experienced issues with nodegit inside docker container node:8-alpine , pretty much the same as James

So I just added the commands you suggested :

In order to install on Alpine, you need the following packages:

apk --no-cache -q add build-base libgit2-dev

You also need this symlink:

ln -s /usr/lib/libcurl.so.4 /usr/lib/libcurl-gnutls.so.4

Then you can install and use nodegit on Alpine.

I still get errors, which you can reproduce with a tag I made for you :

https://github.com/Jean-Baptiste-Lasselle/provision-mjml-server/releases/tag/MOJAVELINUX_REFERENCE

To run the test, just execute :

docker-compose down --rmi all && git pull && docker-compose down --rmi all && docker-compose up -d --force-recreate

Thank you all team for leaving so many comments, working on issues, feels great to feel support (and I'm dying to get my new MJML web editor ... :) )

@mojavelinux Thanks, this worked for us, too.

However, the libgit2 compilation indeed takes some time (~3 minutes for us), so we decided to switch to the "slim" variant of the official debian node image (node:10-stretch-slim). There, you can just apt install libcurl3-gnutls to get libcurl-gnutls.so.4.

As for me, it worked fine on alpine 3.8 but not with node:10-alpine3.9 it faults Error loading shared library /usr/src/Hilary/node_modules/nodegit/build/Release/nodegit.node: Exec format error

Any ideas?

@rcjsuen I have installed the following package libgit2-dev and I think this has all dependencies for NodeGit. For example I add my dockerfile below

FROM node:8.1.0-alpine

COPY ./ /home/node

WORKDIR /home/node

# Git install
RUN mkdir /nodegit && \
    cd /nodegit && \
    apk update && \
    apk upgrade && \
    apk add git libgit2-dev && \
    apk add python tzdata pkgconfig build-base && \
    yarn add -E nodegit@0.20 && \
    apk del python tzdata pkgconfig build-base && \
    rm yarn.lock package.json && \
    rm -rf /tmp/* /var/cache/apk/* && \
    yarn cache clean && \

# Install Node modules
RUN rm -rf package-lock.json node_modules \
    && apk add --update --no-cache --virtual .build-deps \
    curl \
    g++ \
    gcc \
    gnupg \
    libgcc \
    make \
    alpine-sdk \
    python \
    && npm config set unsafe-perm true \
    && npm install \
    && npm prune --production \
    && apk del .build-deps \
    && rm -rf /usr/share/man /tmp/* /var/cache/apk/* \
    && ls -a /nodegit

EXPOSE 8000

USER node
CMD ["node", "index"]

is this works ?

I have some error like this:

Error loading shared library libgssapi_krb5.so.2: No such file or directory

using multi stage dockerfile. Should I install krb5-libs or other libs in the last stage ?

@prakasa-tkpd +1 same problem

RUN apk --no-cache add krb5 pcre

it's work for me @prakasa-tkpd

commented

Has anyone gotten this to work on node:12-alpine? The rebuild solution worked for me with alpine 3.8, but not 3.9.

FROM node:12-alpine

RUN apk --update --no-cache -q add libgit2-dev

ADD package.json index.js ./

RUN apk --no-cache --virtual .node-gyp-compilation-dependencies add \
		g++ \
		make \
		python \
		curl-dev \
	&& apk --no-cache add \
		bash \
		ca-certificates \
		krb5-dev \
		libgit2-dev

RUN BUILD_ONLY=true npm install

RUN apk del .node-gyp-compilation-dependencies

CMD ["node", "index.js"]
const nodegit = require('nodegit');
nodegit.Clone.clone('https://github.com/nodegit/nodegit/', 'test-test', {})
	.then(() => console.log('all good!'))
	.catch(e => console.log('error!', e));
{
  "name": "test",
  "version": "1.0.0",
  "main": "index.js",
  "dependencies": {
    "nodegit": "0.27.0"
  }
}
> docker build -t tester .
> docker run tester
Segmentation fault

@pheel Im having issues dockerizing nodegit 0.27.0 as well. Were you able to get this working?

commented

@Perronef5 low-hanging solution for us was to use node:12-buster-slim as a base instead. Works out of the box with npm's compiled nodegit. Just make sure to RUN apt-get update && apt-get install -y libgssapi-krb5-2 in the build.

@pheel That works! Thanks 🔥

@pheel Could you share the whole Dockerfile with the node:12-buster-slim approach? I can't seem to make it work by just adding that line so probably I am missing something.

For others that face this issue, I was able to address it using #1361 (comment).

FROM node:14-buster-slim
RUN apt-get update && apt-get install -y libgssapi-krb5-2
...