Issues running release in AWS Lambda

Question

Issues running release in AWS Lambda

jschoch opened this issue 8 years ago · comments

I'm playing with exrm on AWS Lambda using a phoenix server to serve requests.

First a few odd things about lambda. First the files alll show up as owned by user slicer, however the invoking users can be random. This makes file permissions tricky. chmod of the files before zipping and sending seems to take and i've made some things like tmp, log, running_config 777 perms.

next I'm running MIX_ENV=prod mix release to build my release, and then unpacking it into a different directory for mutation pre deploy to lambda. This includes injecting an index.js and some other .js libs so Lambda can be invoked and make a local request to the phoenix server packaged into my release.

I am using node's child_process.exec. I tried spawn, but it does not behave. It seems exrm needs a shell. If this is not the case please correct me.

Now on to the odd problem. Lambda will run arbitrary javascript on loading of my index.js. I have this starting the phoenix server packaged in the release. start does not work, the process does not stay alive, no idea why. foreground will start with out complaint, however it can't seem to serve any requests. Only when I run a separate child_process.exec (assuming this is a completely new process/thread) telling the release to restart does it start serving request. I've literally tried everything I can think of to try to figure this out, however I can't log directly into the environment which makes troubleshoothing this tough.

I have a simple shell exec that allows me to interact with the container.

sh update.sh
... lots of msgs but success ...

Looking at the logs

iex(11)> L.cmd "cat burn/log/*"
OUT:
=====
===== LOGGING STARTED Wed May 11 04:43:48 UTC 2016
=====
Using /home/elixir1.2/test/ddbpholam/burn/burn/releases/0.0.1/ddbpholam.sh
Exec: /home/elixir1.2/test/ddbpholam/burn/burn/erts-7.0/bin/erlexec -boot /home/elixir1.2/test/ddbpholam/burn/burn/releases/0.0.1/ddbpholam -mode embedded -config /home/elixir1.2/test/ddbpholam/burn/burn/running-config/sys.config -boot_var ERTS_LIB_DIR /home/elixir1.2/test/ddbpholam/burn/burn/erts-7.0/../lib -env ERL_LIBS /home/elixir1.2/test/ddbpholam/burn/burn/lib -pa /home/elixir1.2/test/ddbpholam/burn/burn/lib/ddbpholam-0.0.1/consolidated -args_file /home/elixir1.2/test/ddbpholam/burn/burn/running-config/vm.args -user Elixir.IEx.CLI -extra --no-halt +iex -- console
Root: /home/elixir1.2/test/ddbpholam/burn/burn
/home/elixir1.2/test/ddbpholam/burn/burn
Erlang/OTP 18 [erts-7.0] [source] [64-bit] [async-threads:10] [hipe] [kernel-poll:false]

04:43:49.511 [info] Running Ddbpholam.Endpoint with Cowboy using http on port 9080
Interactive Elixir (1.2.1) - press Ctrl+C to exit (type h() ENTER for help)
iex(ddbpholam@stink)1> run_erl [13828] Wed May 11 04:43:48 2016
Args before exec of shell:
run_erl [13828] Wed May 11 04:43:48 2016
argv[0] = sh
run_erl [13828] Wed May 11 04:43:48 2016
argv[1] = -c
run_erl [13828] Wed May 11 04:43:48 2016
argv[2] = exec "/home/elixir1.2/test/ddbpholam/burn/burn/bin/ddbpholam" "console"

ERR:
:ok

Looking for the process

iex(12)> L.cmd "ps -ef |grep pho"
OUT: 492          7     1  0 04:44 ?        00:00:00 /bin/sh /var/task/burn/releases/0.0.1/ddbpholam.sh foreground
492         44     1  0 04:44 ?        00:00:00 /bin/sh -c ps -ef |grep pho
492         46    44  0 04:44 ?        00:00:00 grep pho

ERR:
:ok

Testing the URL

iex(13)> L.cmd "curl localhost:9080/api/ping"
FAIL: Command failed: /bin/sh -c curl localhost:9080/api/ping
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 9080: Connection refused

:ok

issuing restart

iex(15)> L.cmd "burn/bin/ddbpholam restart"
FAIL: Command failed: /bin/sh -c burn/bin/ddbpholam restart

:ok

testing the url again

iex(16)> L.cmd "curl localhost:9080/api/ping"
OUT: {"ping":"pong"}
ERR:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    15  100    15    0     0    512      0 --:--:-- --:--:-- --:--:--   517

:ok

really no clue why this is doing this, looking for any ideas on debugging this.

Jesse Schoch · Answer 1 · Wed May 11 2016 13:24:01 GMT+0800 (China Standard Time)

Ok, looking at the other related issues they seemed to be related to pipes, and this also seems related. Since I can't control my username and it may be random, any ideas on getting around this?

chown -R works as expected but...

iex(6)> L.cmd "ls -al burn/tmp"
OUT: total 12
drwxrwxrwx 3 slicer 497 4096 May 10 16:53 .
drwxrwxr-x 9 slicer 497 4096 May 11 05:05 ..
drwxrwxrwx 3 slicer 497 4096 May 10 16:53 erl_pipes

ERR:
:ok
iex(7)> L.cmd "ls -al burn/tmp/erl_pipes"
OUT: total 12
drwxrwxrwx 3 slicer 497 4096 May 10 16:53 .
drwxrwxrwx 3 slicer 497 4096 May 10 16:53 ..
drwxrwxrwx 2 slicer 497 4096 May 11 05:06 ddbpholam

ERR:
:ok

and the kicker

iex(8)> L.cmd "ls -al burn/tmp/erl_pipes/ddbpholam"
OUT: total 8
drwxrwxrwx 2 slicer       497 4096 May 11 05:06 .
drwxrwxrwx 3 slicer       497 4096 May 10 16:53 ..
prw-rw-r-- 1 sbx_user1058 488    0 May 11 05:06 erlang.pipe.1.r
prw-rw-r-- 1 sbx_user1058 488    0 May 11 05:06 erlang.pipe.1.w

ERR:
:ok
iex(9)> L.cmd "whoami"
OUT: sbx_user1053

ERR:
:ok

Jesse Schoch · Answer 2 · Wed May 11 2016 13:24:44 GMT+0800 (China Standard Time)

oh it gets stranger, where'd the pipes go?

iex(11)> L.cmd "whoami"
OUT: sbx_user1059

ERR:
:ok
iex(12)> L.cmd "ls -al burn/tmp/erl_pipes/ddbpholam"
OUT: total 8
drwxrwxrwx 2 slicer 497 4096 May 11 05:14 .
drwxrwxrwx 3 slicer 497 4096 May 10 16:53 ..

ERR:
:ok
iex(13)> L.cmd "ps -ef |grep pho"
OUT: 488          7     1  0 05:22 ?        00:00:00 /var/task/burn/erts-7.0/bin/beam.smp -Bd -- -root /var/task/burn -progname var/task/burn/releases/0.0.1/ddbpholam.sh -- -home /var/task/burn -- -noshell -noshell -noinput -boot /var/task/burn/releases/0.0.1/ddbpholam -mode embedded -config /var/task/burn/running-config/sys.config -boot_var ERTS_LIB_DIR /var/task/burn/erts-7.0/../lib -pa /var/task/burn/lib/ddbpholam-0.0.1/consolidated -sname ddbpholam -setcookie ddbpholam -- foreground
488         67     1  0 05:22 ?        00:00:00 /bin/sh -c ps -ef |grep pho
488         69    67  0 05:22 ?        00:00:00 grep pho

ERR:
:ok
iex(14)> L.cmd "burn/bin/ddbpholam restart"
FAIL: Command failed: /bin/sh -c burn/bin/ddbpholam restart

:ok
iex(15)> L.cmd "ls -al burn/tmp/erl_pipes/ddbpholam"
OUT: total 8
drwxrwxrwx 2 slicer 497 4096 May 11 05:14 .
drwxrwxrwx 3 slicer 497 4096 May 10 16:53 ..

ERR:
:ok
iex(16)> L.cmd "curl localhost:9080/api/ping"
OUT: {"ping":"pong"}
ERR:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    15  100    15    0     0    410      0 --:--:-- --:--:-- --:--:--   416

:ok
iex(17)> L.cmd "ls -al burn/tmp/erl_pipes/ddbpholam"
OUT: total 8
drwxrwxrwx 2 slicer 497 4096 May 11 05:14 .
drwxrwxrwx 3 slicer 497 4096 May 10 16:53 ..

ERR:
:ok

Paul Schoenfelder · Answer 3 · Wed May 25 2016 00:14:19 GMT+0800 (China Standard Time)

Hmmm, that's really strange. But a few things:

When running the release in Lambda (and this applies to Docker too actually), use the foreground task - that way you can capture any logging output - since presumably the daemonization provided by start isn't necessary in those environments. Using console indeed requires a tty.

Could you paste the output of running the foreground task?