/etc/mesos-slave/modules

Question

/etc/mesos-slave/modules

akamalov opened this issue 9 years ago · comments

** Environment **

OS: RHEL 7.2
Apache Mesos: 0.26.0
Rexray: 0.3.1
DVDI: 0.4.1-dev

Built DVDI Isolator module for Mesos 0.26.0 as per documentation. Trying to load the load the module on Mesos-slave upon start:

File: _/etc/mesos-slave/modules_

file:///usr/lib/dvdi-mod.json

File _/etc/mesos-slave/isolation_

com_emccode_mesos_DockerVolumeDriverIsolator

Trying to start mesos-slave:

systemctl daemon-reload && systemctl restart mesos-slave

Unfortunately it is failing on startup:

systemctl -l status mesos-slave
● mesos-slave.service - Mesos Slave
   Loaded: loaded (/usr/lib/systemd/system/mesos-slave.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/mesos-slave.service.d
           └─mesos-slave-containerizer.conf
   Active: activating (auto-restart) (Result: exit-code) since Fri 2016-01-29 15:08:47 EST; 5s ago
  Process: 2202 ExecStart=/usr/bin/mesos-init-wrapper slave (code=exited, status=1/FAILURE)
 Main PID: 2202 (code=exited, status=1/FAILURE)

Jan 29 15:08:47 node1.local.net systemd[1]: Unit mesos-slave.service entered failed state.
Jan 29 15:08:47 node1.local.net systemd[1]: mesos-slave.service failed.
[root@node1 mesos-slave]#

_error log entries_

Jan 29 15:00:00 node1.local.net mesos-slave[1741]: Error loading modules: Error opening library: '/usr/lib/dvdi_isolator-0.26.0.so': Could not load library '/usr/lib/dvdi_isolator-0.26.0.so': /usr/lib/dvdi_isolator-0.26.0.so: cannot dynamically load executable
Jan 29 15:00:00 node1.local.net systemd: mesos-slave.service: main process exited, code=exited, status=1/FAILURE
Jan 29 15:00:00 node1.local.net systemd: Unit mesos-slave.service entered failed state.
Jan 29 15:00:00 node1.local.net systemd: mesos-slave.service failed.

Any clues where I should be focusing on ?

Thanks,

Alex

Alex Kamalov · Answer 1 · Sat Jan 30 2016 04:13:33 GMT+0800 (China Standard Time)

Reopening with updated problem description...

Alex Kamalov · Answer 2 · Sat Jan 30 2016 04:29:04 GMT+0800 (China Standard Time)

Also, there is a bug with the only available Mesos Docker module 0.23.0:

[root@node1 proj]# docker run -ti -v `pwd`:/isolator emccode/mesos-module-dvdi-dev:0.23.0
/bin/sh: 1: cd: can't cd to /isolator/isolator
[root@node1 proj]#

Steve Wong · Answer 3 · Sat Jan 30 2016 05:20:49 GMT+0800 (China Standard Time)

Regarding previous comment, There are Mesos prebuilt Docker modules for 0.24.1, 0.25.0 and 0.26.0 here in my personal docker hub repo:
https://hub.docker.com/r/cantbewong/mesos-build-module-dev/tags/

But the docker file in this github repo will let you build your own, and unless you fully trust the source, this is thought by many to be prudent because essentially you are taking a binary built by somebody else and running it as root on your host.

So for example, assuming I have done a git clone of the isolator to /home/steve/git/cantbewong/mesos-module-dvdi

I do this to build from my docker hub image:
sudo docker run -ti -v /home/steve/git/cantbewong/mesos-module-dvdi/isolator/:/isolator cantbewong/mesos-build-module-dev:0.26.0 /bin/bash

To produce your own docker hub image, do this instead (while in /home/steve/git/cantbewong/mesos-module-dvdi/):
sudo docker build -t my-dockerhub-accountname/mesos-build-module-dev:0.26.0 - < Dockerfile-mesos-modules-dev

sudo docker run -ti -v /home/steve/git/cantbewong/mesos-module-dvdi/isolator/:/isolator my-dockerhub-accountname/mesos-build-module-dev:0.26.0 /bin/bash

Alex Kamalov · Answer 4 · Sat Jan 30 2016 05:21:58 GMT+0800 (China Standard Time)

Already did. Getting error as far as loading a library.

Alex Kamalov · Answer 5 · Sat Jan 30 2016 05:22:44 GMT+0800 (China Standard Time)

Error loading modules: Error opening library: '/usr/lib/dvdi_isolator-0.26.0.so': Could not load library '/usr/lib/dvdi_isolator-0.26.0.so': /usr/lib/dvdi_isolator-0.26.0.so: cannot dynamically load executable

Steve Wong · Answer 6 · Sat Jan 30 2016 05:43:53 GMT+0800 (China Standard Time)

could you check that the .so file has execute permission enabled:

ubuntu@ec2-52-33-34-141:~$ ls -la /usr/lib/libmesos_dvdi_isolator-0.26.0.so
-rwxr-xr-x 1 root root 3478979 Dec 15 05:37 /usr/lib/libmesos_dvdi_isolator-0.26.0.so

Steve Wong · Answer 7 · Sat Jan 30 2016 05:54:46 GMT+0800 (China Standard Time)

Can you dump and post the content of the json file, in particular it contains a version suffix that I sometimes have forgotten to update:

ubuntu@ec2-52-33-34-141:~$ cat /usr/lib/dvdi-mod.json
{
"libraries": [
{
"file": "/usr/lib/libmesos_dvdi_isolator-0.26.0.so",
"modules": [
{
"name": "com_emccode_mesos_DockerVolumeDriverIsolator"
}
]
}
]
}

Alex Kamalov · Answer 8 · Sat Jan 30 2016 06:32:38 GMT+0800 (China Standard Time)

Execute bit is on:

ls -ld dvdi_isolator-0.26.0.so
-rwxr-xr-x 1 root root 4685688 Jan 29 12:58 dvdi_isolator-0.26.0.so

Content of dvdi-mod.json:

{
"libraries": [
{
"file": "/usr/lib/dvdi_isolator-0.26.0.so",
"modules": [
{
"name": "com_emccode_mesos_DockerVolumeDriverIsolator"
}
]
}
]
}

Alex Kamalov · Answer 9 · Sat Jan 30 2016 06:36:51 GMT+0800 (China Standard Time)

By the way, when compiled, it gets compiled as "dvdi_isolator" and not "libmesos_dvdi_isolator-0.26.so".

Example:

[root@node1 mesos-module-dvdi]# ls -al dvdi
total 4588
drwxr-xr-x 2 root root      65 Jan 29 17:34 .
drwxr-xr-x 5 root root    4096 Jan 29 10:44 ..
-rwxr-xr-x 1 root root 4685688 Jan 29 09:47 dvdi_isolator
-rw-r--r-- 1 root root     219 Jan 29 13:01 dvdi-mod.json
-rw-r--r-- 1 root root     363 Jan 29 09:47 modules.json
[root@node1 mesos-module-dvdi]#

Steve Wong · Answer 10 · Sat Jan 30 2016 06:46:11 GMT+0800 (China Standard Time)

So when I build at a command line within the Docker container, I have been doing this:

cd /isolator
export ISOLATOR_VERSION=0.26.0
export LD_LIBRARY_PATH=LD_LIBRARY_PATH:/usr/local/lib
./bootstrap
rm -Rf build
mkdir build
cd build
../configure --with-mesos-root=/mesos --with-mesos-build-dir=/mesos
make all

The ISOLATOR_VERSION environment variable is what appends the version.
If left empty, this might omit version.
Don't know why yours wouldn't end in .so though.

I have always used ubuntu and not redhat though.
Have you tried running the .so published under releases on github. I know its ubuntu and not redhat so there might be issues, but if it even loads this might tell us something

Steve Wong · Answer 11 · Sat Jan 30 2016 06:54:53 GMT+0800 (China Standard Time)

after doing the above steps, I get this content in the .libs directory under the build directory:

root@87c526820d72:/isolator/build# ls -la .libs
total 3444
drwxr-xr-x 2 root root    4096 Jan 29 22:52 .
drwxr-xr-x 4 root root    4096 Jan 29 22:52 ..
-rwxr-xr-x 1 root root 3514103 Jan 29 22:52 libmesos_dvdi_isolator-0.26.0.so
lrwxrwxrwx 1 root root      28 Jan 29 22:52 libmesos_dvdi_isolator.la -> ../libmesos_dvdi_isolator.la
-rw-r--r-- 1 root root    1039 Jan 29 22:52 libmesos_dvdi_isolator.lai
lrwxrwxrwx 1 root root      32 Jan 29 22:52 libmesos_dvdi_isolator.so -> libmesos_dvdi_isolator-0.26.0.so

Steve Wong · Answer 12 · Sat Jan 30 2016 08:32:32 GMT+0800 (China Standard Time)

Do you have SELinux engaged in enforcing mode?

David vonThenen · Answer 13 · Sat Jan 30 2016 09:08:21 GMT+0800 (China Standard Time)

I stood the following standalone Mesos node in Amazon EC2:
OS: RHEL 7.2
Apache Mesos: 0.26.0
Rexray: 0.3.1
DVDCLI: 0.1.0
Mesos DVDI Isolator: 0.4.0 (https://github.com/emccode/mesos-module-dvdi/releases/tag/v0.4.0)

I had no problems dispatching a task that created 2 new volumes. Have you tried using the binaries posted in the 0.4.0 release? Or are locked into building your own?

Also did you install docker? and enable the docker containerizer in Mesos?
echo "docker,mesos" | tee /etc/mesos-slave/containerizers

[root@ip-172-31-29-143 ec2-user]# getenforce
Enforcing

[root@ip-172-31-29-143 ec2-user]# ls -lZ /usr/lib/libmesos_dvdi_isolator-0.26.0.so
-rwxr-xr-x. ec2-user ec2-user unconfined_u:object_r:user_tmp_t:s0 /usr/lib/libmesos_dvdi_isolator-0.26.0.so

Alex Kamalov · Answer 14 · Sat Jan 30 2016 09:25:05 GMT+0800 (China Standard Time)

Hmm..this is interesting. Here is my version:

[root@node1 mesos-module-dvdi]# cat VERSION
0.4.1-dev
[root@node1 mesos-module-dvdi]#

Alex Kamalov · Answer 15 · Sat Jan 30 2016 09:25:19 GMT+0800 (China Standard Time)

Let me try with 0.4.0 release...

Alex Kamalov · Answer 16 · Sat Jan 30 2016 09:44:41 GMT+0800 (China Standard Time)

Can anyone post me their /etc/mesos-slave/modules file ?

Alex Kamalov · Answer 17 · Sat Jan 30 2016 09:46:25 GMT+0800 (China Standard Time)

Yep, containerizer is enabled for docker. I will purge try to rebuild it

Alex Kamalov · Answer 18 · Sat Jan 30 2016 10:13:23 GMT+0800 (China Standard Time)

SELInux is disabled. Tried to pull the pre-built binary off the site. Tried to run with it. Getting the following error:

Jan 29 21:09:35 node1 mesos-slave[8253]: Error loading modules: Error loading module 'com_emccode_mesos_DockerVolumeDriverIsolator': Error looking up symbol 'com_emccode_mesos_DockerVolumeDriverIsolator' in '/usr/lib/libmesos-0.26.0.so' : /lib/libmesos-0.26.0.so: undefined symbol: com_emccode_mesos_DockerVolumeDriverIsolator

Steve Wong · Answer 19 · Sat Jan 30 2016 11:04:12 GMT+0800 (China Standard Time)

ubuntu@ec2-52-33-34-141:~$ cat /etc/mesos-slave/modules
file:///usr/lib/dvdi-mod.json

Alex Kamalov · Answer 20 · Sat Jan 30 2016 11:14:07 GMT+0800 (China Standard Time)

Ok, it worked. I went ahead and instead of building it on my own downloaded off this Git. Don't know why would the version I built didn't work (however it compiled fine....)

Thank you very much indeed everyone for helping!!!

Alex

Steve Wong · Answer 21 · Sat Jan 30 2016 11:17:13 GMT+0800 (China Standard Time)

Here are expected results:

get .so
ubuntu@ec2-52-33-34-141:/tmp$ wget https://github.com/emccode/mesos-module-dvdi/releases/download/v0.4.0/libmesos_dvdi_isolator-0.26.0.so
--2016-01-30 03:14:44-- https://github.com/emccode/mesos-module-dvdi/releases/download/v0.4.0/libmesos_dvdi_isolator-0.26.0.so
Resolving github.com (github.com)... 192.30.252.131
Connecting to github.com (github.com)|192.30.252.131|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://github-cloud.s3.amazonaws.com/releases/43106864/a9abd238-b940-11e5-8f47-9b045bf88792.so?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAISTNZFOVBIJMK3TQ%2F20160130%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20160130T031306Z&X-Amz-Expires=300&X-Amz-Signature=b49109195d1ffa09286ecb1cbdddc510988bf45670ddc2467b396a622b85f75d&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dlibmesos_dvdi_isolator-0.26.0.so&response-content-type=application%2Foctet-stream [following]
--2016-01-30 03:14:44-- https://github-cloud.s3.amazonaws.com/releases/43106864/a9abd238-b940-11e5-8f47-9b045bf88792.so?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAISTNZFOVBIJMK3TQ%2F20160130%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20160130T031306Z&X-Amz-Expires=300&X-Amz-Signature=b49109195d1ffa09286ecb1cbdddc510988bf45670ddc2467b396a622b85f75d&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dlibmesos_dvdi_isolator-0.26.0.so&response-content-type=application%2Foctet-stream
Resolving github-cloud.s3.amazonaws.com (github-cloud.s3.amazonaws.com)... 54.231.81.200
Connecting to github-cloud.s3.amazonaws.com (github-cloud.s3.amazonaws.com)|54.231.81.200|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3514103 (3.4M) [application/octet-stream]
Saving to: ‘libmesos_dvdi_isolator-0.26.0.so’
checksum
ubuntu@ec2-52-33-34-141:/tmp$ sha1sum libmesos_dvdi_isolator-0.26.0.so
c06ddb21f742cb0fb1a0957588cdc48f7154099b libmesos_dvdi_isolator-0.26.0.so
check for presence of symbol your system claims is unresolved
ubuntu@ec2-52-33-34-141:/tmp$ nm -gC libmesos_dvdi_isolator-0.26.0.so | grep com_emccode
0000000000237ba0 B com_emccode_mesos_DockerVolumeDriverIsolator

Steve Wong · Answer 22 · Sat Jan 30 2016 11:18:55 GMT+0800 (China Standard Time)

put in my last comment before I saw yours

Is it possible build was done with this environment variable undefined?
export ISOLATOR_VERSION=0.26.0

build would complete but probably not work if this happened

Alex Kamalov · Answer 23 · Sat Jan 30 2016 12:02:33 GMT+0800 (China Standard Time)

It is a possibility! Thanks so much Steve!

Alex

On Fri, Jan 29, 2016 at 10:18 PM, Steve Wong notifications@github.com
wrote:

put in my last comment before I saw yours

Is it possible build was done with this environment variable undefined?
export ISOLATOR_VERSION=0.26.0

—
Reply to this email directly or view it on GitHub
#68 (comment)
.

Pradeep Chhetri · Answer 24 · Sun Jan 31 2016 00:14:29 GMT+0800 (China Standard Time)

Can you please upload the working build for mesos-0.26.0 on github too ?

Alex Kamalov · Answer 25 · Sun Jan 31 2016 01:22:00 GMT+0800 (China Standard Time)

Are you talking about the module ? If yes, then the module is available here:

https://github.com/emccode/mesos-module-dvdi/releases/tag/v0.4.0

Alex

Steve Wong · Answer 26 · Sun Jan 31 2016 06:21:01 GMT+0800 (China Standard Time)

pradeepchhetri, I expect that going forward we will publish future releases with centos binaries in addition to ubuntu binaries. In the meantime, we have a report that the ubuntu build does run on centos/redhat

Pradeep Chhetri · Answer 27 · Sun Jan 31 2016 06:34:54 GMT+0800 (China Standard Time)

Thank you Steve. I am having a weird error after upgrading my mesos version to 0.26.0 from 0.25.0 although i downloaded the libmesos_dvdi_isolator-0.26.0.so file as well.

I am getting the following error while starting jenkins framework over mesos:

SEVERE: Error loading modules: Error opening library: '/usr/lib/libmesos_dvdi_isolator-0.26.0.so': Could not load library '/usr/lib/libmesos_dvdi_isolator-0.26.0.so': /usr/lib/libmesos_dvdi_isolator-0.26.0.so: undefined symbol: _ZTIN6google8protobuf7MessageE

I tried rebuilding the module myself for mesos-0.26.0 but still getting the same error. Not sure how to debug.

Alex Kamalov · Answer 28 · Sun Jan 31 2016 07:06:30 GMT+0800 (China Standard Time)

Try to re-download it. Also, don't know if matters: try to set execute bit
on.

Alex

On Sat, Jan 30, 2016 at 5:34 PM, Pradeep Chhetri notifications@github.com
wrote:

Thank you Steve. I am having a weird error after upgrading my mesos
version to 0.26.0 from 0.25.0 although i downloaded the
libmesos_dvdi_isolator-0.26.0.so file as well.

I am getting the following error while starting jenkins framework over
mesos:

SEVERE: Error loading modules: Error opening library: '/usr/lib/libmesos_dvdi_isolator-0.26.0.so': Could not load library '/usr/lib/libmesos_dvdi_isolator-0.26.0.so': /usr/lib/libmesos_dvdi_isolator-0.26.0.so: undefined symbol: _ZTIN6google8protobuf7MessageE

I tried rebuilding the module myself for mesos-0.26.0 but still getting
the same error. Not sure how to debug.

—
Reply to this email directly or view it on GitHub
#68 (comment)
.