how to debug

Question

how to debug

osallou opened this issue 9 years ago · comments

Hi,
how can I debug the fact that mesos slave does not call the module ?
Using dvdcli is fine.

I launched my slave with:

/usr/sbin/mesos-slave --master=zk://a.b.c.d:2181/mesos --log_dir=/var/log/mesos --containerizers=docker,mesos --modules=file:///usr/lib/dvdi-mod.json

I can see at startup:

.... --log_dir="/var/log/mesos" --logbufsecs="0" --logging_level="INFO" --master="zk://a.b.c.d:2181/mesos" --modules="libraries {
  file: "/usr/lib/libmesos_dvdi_isolator-0.23.0.so"
  modules {
    name: "com_emccode_mesos_DockerVolumeDriverIsolator"
  }
}

When I call my docker plugin with dvdcli, I can see my plugin is activated etc... However, when I execute a job, nothing happens.
I have set env variables in mesos offer prototype, in commandinfo, though I cannot check they are correctly set. The job is executed "as usual", without calling the docker plugin.

I use mesos 0.23.0

Thanks

Olivier Sallou · Answer 1 · Tue Jan 12 2016 23:06:29 GMT+0800 (China Standard Time)

Additional info:

I can see at startup the module loading

I0112 15:01:01.919692  9762 containerizer.cpp:111] Using isolation: com_emccode_mesos_DockerVolumeDriverIsolator
I0112 15:01:01.919965  9762 docker_volume_driver_isolator.cpp:710] Loading Docker Volume Driver Isolator module
I0112 15:01:01.920300  9762 docker_volume_driver_isolator.cpp:89] create() called
I0112 15:01:01.920522  9762 docker_volume_driver_isolator.cpp:121] using /var/lib/mesos/dvdi/dvdimounts.json

Clint Kitson · Answer 2 · Tue Jan 12 2016 23:30:35 GMT+0800 (China Standard Time)

I am assuming that when a task is launched that you do see it start on this
slave? Can you share the Json for the task or application?

On Tuesday, January 12, 2016, Olivier Sallou notifications@github.com
wrote:

Additional info:

I can see at startup the module loading

I0112 15:01:01.919692 9762 containerizer.cpp:111] Using isolation: com_emccode_mesos_DockerVolumeDriverIsolator
I0112 15:01:01.919965 9762 docker_volume_driver_isolator.cpp:710] Loading Docker Volume Driver Isolator module
I0112 15:01:01.920300 9762 docker_volume_driver_isolator.cpp:89] create() called
I0112 15:01:01.920522 9762 docker_volume_driver_isolator.cpp:121] using /var/lib/mesos/dvdi/dvdimounts.json

—
Reply to this email directly or view it on GitHub
#63 (comment)
.

David vonThenen · Answer 3 · Tue Jan 12 2016 23:32:33 GMT+0800 (China Standard Time)

Hi,

Seems like you answered your own question. We have all our logging inline with the normal agent log. You can find our debug info by looking for the CPP file with its corresponding line number as seen below.

I0112 15:01:01.919965 9762 docker_volume_driver_isolator.cpp:710] Loading Docker Volume Driver Isolator module

If you have a specific error, please attach a mesos agent log file, tell us what your backing storage is, include a copy of your REX-Ray config file (please delete any creds your might have in the file), and also run the "rexray env" command (again delete any creds that might show up).

Olivier Sallou · Answer 4 · Tue Jan 12 2016 23:33:10 GMT+0800 (China Standard Time)

I have only 1 slave, and can see it though mesos webui.
How can I get the json of the task on mesos side ? I use my own mesos framework (in python) and submit the task via an Offer.
Or do you mean the docker inspect result for the task?

Olivier Sallou · Answer 5 · Tue Jan 12 2016 23:38:56 GMT+0800 (China Standard Time)

@dvonthenen I have my own Docker volume plugin (in devpt), and as i said it works with dvdcli. Issue is only when ran via my mesos framework. I mount a local storage, but whatever it is, my Docker volume plugin is never called/triggered (no error, just not called).

Clint Kitson · Answer 6 · Tue Jan 12 2016 23:48:41 GMT+0800 (China Standard Time)

Quick suggestion here would be to disable to Docker containerizer. If your
task ceases to launch all together then we know you're issuing the task to
the wrong containerizer.

On Tuesday, January 12, 2016, Olivier Sallou notifications@github.com
wrote:

@dvonthenen https://github.com/dvonthenen I have my own Docker volume
plugin (in devpt), and as i said it works with dvdcli. Issue is only when
ran via my mesos framework. I mount a local storage, but whatever it is, my
Docker volume plugin is never called/triggered (no error, just not called).

—
Reply to this email directly or view it on GitHub
#63 (comment)
.

Olivier Sallou · Answer 7 · Tue Jan 12 2016 23:57:05 GMT+0800 (China Standard Time)

If I disable the docker containerizer on slave, task fail with

I0112 15:55:07.581176 10106 slave.cpp:1573] Queuing task '15574' for executor 15574 of framework '20160112-155427-2561659146-5050-1486-0000
E0112 15:55:07.581416 10106 slave.cpp:3269] Container 'a739a5c9-afbd-42e6-8025-b50623e2758c' for executor '15574' of framework '20160112-155427-2561659146-5050-1486-0000' failed to start: None of the enabled containerizers (mesos) could create a container for the provided TaskInfo/ExecutorInfo message.

Clint Kitson · Answer 8 · Wed Jan 13 2016 00:02:28 GMT+0800 (China Standard Time)

So there's the problem I believe. Your task is launching using Docker
containerizer so it never makes its way to our isolator.

For this case the volume driver would be used natively through Docker. See
the following docs for the Marathon framework example of using native
parameters as key values.

http://rexray.readthedocs.org/en/stable/user-guide/third-party/mesos/

On Tuesday, January 12, 2016, Olivier Sallou notifications@github.com
wrote:

If I disable the docker containerizer on slave, task fail with

I0112 15:55:07.581176 10106 slave.cpp:1573] Queuing task '15574' for executor 15574 of framework '20160112-155427-2561659146-5050-1486-0000
E0112 15:55:07.581416 10106 slave.cpp:3269] Container 'a739a5c9-afbd-42e6-8025-b50623e2758c' for executor '15574' of framework '20160112-155427-2561659146-5050-1486-0000' failed to start: None of the enabled containerizers (mesos) could create a container for the provided TaskInfo/ExecutorInfo message.

—
Reply to this email directly or view it on GitHub
#63 (comment)
.

Olivier Sallou · Answer 9 · Wed Jan 13 2016 00:07:41 GMT+0800 (China Standard Time)

ok, I understand. There is an issue however.
Suggested use of parameters:

"parameters": [{
            "key": "volume-driver",
            "value": "rexray"
        }, {
            "key": "volume",
            "value": "nginx-data:/data/www"
        }]

parameters are strongly discouraged in mesos interface (for future compatiblity). In addition, the volume-driver command option of Docker is deprecated. Though it still works, it will disappear in future release, requesting the use of docker volume commands before and after the docker run commands.

This means that solution will work for the moment only.

Clint Kitson · Answer 10 · Wed Jan 13 2016 00:13:52 GMT+0800 (China Standard Time)

Yes the long term approach is to use the mesos modules and isolators. You
cans also modify which containerizer you are using to be able to leverage
mesos-module-dvdi today to align to future strategies.

On Tuesday, January 12, 2016, Olivier Sallou notifications@github.com
wrote:

ok, I understand. There is an issue however.
Suggested use of parameters:

"parameters": [{
"key": "volume-driver",
"value": "rexray"
}, {
"key": "volume",
"value": "nginx-data:/data/www"
}]

parameters are strongly discouraged in mesos interface (for future
compatiblity). In addition, the volume-driver command option of Docker is
deprecated. Though it still works, it will disappear in future release,
requesting the use of docker volume commands before and after the docker
run commands.

This means that solution will work for the moment only.

—
Reply to this email directly or view it on GitHub
#63 (comment)
.

Clint Kitson · Answer 11 · Wed Jan 13 2016 03:27:20 GMT+0800 (China Standard Time)

If you are satisfied can you close the issue @osallou?