Issue installing dockprom on Amazon AMI
mobeigi opened this issue · comments
So I was installing this on an Amzon AMI and run into some issues.
The docker compose would fail but if I kept running it over and over it would eventually setup all containers. I ran into some issues where I was getting a 502 Bad Gateway error if Caddy was installed before Grafana and stuff like that.
Right now it seems like all containers are set up and working. I'm still exploring everything so I'm not 100% sure everything is set up correctly given installation errors.
Docker Version
Docker version 19.03.6-ce, build 369ce74
Docker Info
[ec2-user@ip-172-31-44-87 dockprom]$ docker info
Client:
Debug Mode: false
Server:
Containers: 7
Running: 7
Paused: 0
Stopped: 0
Images: 7
Server Version: 19.03.6-ce
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: ff48f57fc83a8c44cf4ad5d672424a98ba37ded6
runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.14.181-142.260.amzn2.x86_64
Operating System: Amazon Linux 2
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 461.1MiB
Name: ip-172-31-44-87.ap-southeast-2.compute.internal
ID: AC7A:MV65:6Q3U:2YGD:HTRA:U4JD:CWZC:QUVB:AKBU:ZYDH:STAF:MNCG
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Docker Composer Version
[ec2-user@ip-172-31-44-87 dockprom]$ docker-compose version
docker-compose version 1.26.2, build eefe0d31
docker-py version: 4.2.2
CPython version: 3.7.7
OpenSSL version: OpenSSL 1.1.0l 10 Sep 2019
Install logs
[ec2-user@ip-172-31-44-87 dockprom]$ ADMIN_USER=admin ADMIN_PASSWORD=admin sudo /usr/local/bin/docker-compose up -d
Pulling prometheus (prom/prometheus:v2.19.2)...
v2.19.2: Pulling from prom/prometheus
76df9210b28c: Pull complete
559be8e06c14: Pull complete
9a0df092b3bc: Pull complete
70a369cc7d1c: Pull complete
3a438dac417e: Pull complete
f77384112445: Pull complete
d1b93b8f2529: Pull complete
a65c5f474fc6: Pull complete
698d3745d05b: Pull complete
c36b8b23ce2c: Pull complete
866c4df90526: Pull complete
5098ca813218: Pull complete
Digest: sha256:cd134bd4fca0f60ff8b4c679cebe5c5c5cf5e2da5f4886b2ae933da821915f92
Status: Downloaded newer image for prom/prometheus:v2.19.2
Pulling alertmanager (prom/alertmanager:v0.21.0)...
v0.21.0: Pulling from prom/alertmanager
0f8c40e1270f: Pull complete
626a2a3fee8c: Pull complete
74ad1ee664e6: Pull complete
addd0e4e1dc5: Pull complete
012b22e92c79: Pull complete
518f5393dbbc: Pull complete
Digest: sha256:24a5204b418e8fa0214cfb628486749003b039c279c56b5bddb5b10cd100d926
Status: Downloaded newer image for prom/alertmanager:v0.21.0
Pulling nodeexporter (prom/node-exporter:v1.0.1)...
v1.0.1: Pulling from prom/node-exporter
86fa074c6765: Pull complete
ed1cd1c6cd7a: Pull complete
ff1bb132ce7b: Pull complete
Digest: sha256:cf66a6bbd573fd819ea09c72e21b528e9252d58d01ae13564a29749de1e48e0f
Status: Downloaded newer image for prom/node-exporter:v1.0.1
Pulling cadvisor (gcr.io/cadvisor/cadvisor:v0.37.0)...
v0.37.0: Pulling from cadvisor/cadvisor
df20fa9351a1: Pull complete
69166b12f0e3: Pull complete
843794b5c008: Pull complete
99bbb51fff9a: Pull complete
Digest: sha256:aa9694377d3be56d53b81b0aeecd5880c0832100923e3ed50a8229ef3e1704b1
Status: Downloaded newer image for gcr.io/cadvisor/cadvisor:v0.37.0
Pulling grafana (grafana/grafana:7.0.6)...
7.0.6: Pulling from grafana/grafana
cbdbe7a5bc2a: Pull complete
25d9c02c8756: Pull complete
216d1d99b347: Pull complete
f5d25fb4941e: Pull complete
32e62631bc5c: Pull complete
1bb17e6904b4: Pull complete
Digest: sha256:41e7f0c787118a84f068d6e84c89f3aea769374dc4678b72d6237dd47c325768
Status: Downloaded newer image for grafana/grafana:7.0.6
Pulling pushgateway (prom/pushgateway:v1.2.0)...
v1.2.0: Pulling from prom/pushgateway
0f8c40e1270f: Already exists
626a2a3fee8c: Already exists
07965f5a7bc1: Pull complete
f0801f1351d9: Pull complete
Digest: sha256:0a9031142481adfb964fe7f65e698d01a6260de982f0322bc44496c3dcc866f1
Status: Downloaded newer image for prom/pushgateway:v1.2.0
Pulling caddy (stefanprodan/caddy:)...
latest: Pulling from stefanprodan/caddy
88286f41530e: Pull complete
1d69571c0ad9: Pull complete
86c5f6279072: Pull complete
4c1f60f6eae1: Pull complete
8ecb576689d7: Pull complete
5669d55a34cf: Pull complete
Digest: sha256:44541cfacb66f4799f81f17fcfb3cb757ccc8f327592745549f5930c42d115c9
Status: Downloaded newer image for stefanprodan/caddy:latest
Creating grafana ...
Creating alertmanager ... error
Creating cadvisor ...
Creating caddy ...
Creating prometheus ...
Creating pushgateway ...
Creating nodeexporter ...
ERROR: for alertmanager Cannot start service alertmanager: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:432: running prestart hook 0 caused \\\"error running hook: exit status 2, stdout: , stderr: fatal error: runtime: out of memory\\\\n\\\\nruntime stack:\\\\nruntime.throw(0x5574527cf359, 0x16)\\\\n\\\\t/usr/lib/golang/src/runtime/panic.go:774 +0x74 fp=0x7ffff0c05128 sp=0x7ffff0c050f8 pc=0x557450e7b714\\\\nruntime.sysMap(0xc000000000, 0x4000000, 0x557454c1b1f8)\\\\n\\\\t/usr/lib/golang/src/runtime/mem_linux.go:169 +0xc7 fp=0x7ffff0c05168 sp=0x7ffff0c05128 pc=0x557450e64457\\\\nruntime.(*mheap).sysAlloc(0x557454bfea40, 0x2000, 0x0, 0x0)\\\\n\\\\t/usr/lib/golang/src/runtime/malloc.go:701 +0x1cf fp=0x7ffff0c05210 sp=0x7ffff0c05168 pc=0x557450e573ff\\\\nruntime.(*mheap).grow(0x557454bfea40, 0x1, 0xffffffff)\\\\n\\\\t/usr/lib/golang/src/runtime/mheap.go:1255 +0xa5 fp=0x7ffff0c05260 sp=0x7ffff0c05210 pc=0x557450e72605\\\\nruntime.(*mheap).allocSpanLocked(0x557454bfea40, 0x1, 0x557454c1b208, 0x1)\\\\n\\\\t/usr/lib/golang/src/runtime/mheap.go:1170 +0x268 fp=0x7ffff0c052e0 sp=0x7ffff0c05260 pc=0x557450e72498\\\\nruntime.(*mheap).alloc_m(0x557454bfea40, 0x1, 0x557450ea002a, 0x0)\\\\n\\\\t/usr/lib/golang/src/runtime/mheap.go:1022 +0xc6 fp=0x7ffff0c05330 sp=0x7ffff0c052e0 pc=0x557450e71d36\\\\nruntime.(*mheap).alloc.func1()\\\\n\\\\t/usr/lib/golang/src/runtime/mheap.go:1093 +0x4e fp=0x7ffff0c05368 sp=0x7ffff0c05330 pc=0x557450ea79de\\\\nruntime.(*mheap).alloc(0x557454bfea40, 0x1, 0x1002a, 0x40000)\\\\n\\\\t/usr/lib/golang/src/runtime/mheap.go:1092 +0x8c fp=0x7ffff0c053b8 sp=0x7ffff0c05368 pc=0x557450e7201c\\\\nruntime.(*mcentral).grow(0x557454bff878, 0x0)\\\\n\\\\t/usr/lib/golang/src/runtime/mcentral.go:255 +0x7d fp=0x7ffff0c053f8 sp=0x7ffff0c053b8 pc=0x557450e63e7d\\\\nruntime.(*mcentral).cacheSpan(0x557454bff878, 0x7f56b7ce5000)\\\\n\\\\t/usr/lib/golang/src/runtime/mcentral.go:106 +0x300 fp=0x7ffff0c05458 sp=0x7ffff0c053f8 pc=0x557450e639a0\\\\nruntime.(*mcache).refill(0x7f56b7ce5008, 0x2a)\\\\n\\\\t/usr/lib/golang/src/runtime/mcache.go:138 +0x87 fp=0x7ffff0c05478 sp=0x7ffff0c05458 pc=0x557450e63437\\\\nruntime.(*mcache).nextFree(0x7f56b7ce5008, 0x557454c1b22a, 0x7f56b7ce5008, 0x8, 0xfffffffffffffff8)\\\\n\\\\t/usr/lib/golang/src/runtime/malloc.go:854 +0x89 fp=0x7ffff0c054b0 sp=0x7ffff0c05478 pc=0x557450e57c29\\\\nruntime.mallocgc(0x180, 0x55745397dea0, 0x1, Creating cadvisor ... error
+0x3a fp=0x7ffff0c05580 sp=0x7ffff0c05550 pc=0x557450e5897a\\\\nruntime.malg(0x2d900008000, 0x557454c010e8)\\\\n\\\\t/usr/lib/golang/src/runtime/proc.go:3232 +0x33 fp=0x7ffff0c055c0 sp=0x7ffff0c05580 pc=0x557450e84be3\\\\nruntime.mpreinit(...)\\\\n\\\\t/usr/lib/golang/src/runtime/os_linux.go:324\\\\nruntime.mcommoninit(0x557454bf9500)\\\\n\\\\t/usr/lib/golang/src/runtime/proc.go:626 +0xc6 fp=0x7ffff0c055f8 sp=0x7ffff0c055c0 pc=0x557450e7e516\\\\nruntime.schedinit()\\\\n\\\\t/usr/lib/golang/src/runtime/proc.go:543 +0x76 fp=0x7ffff0c05650 sp=0x7ffff0c055f8 pc=0x557450e7e176\\\\nruntime.rt0_go(0x7ffff0c05758, 0x4, 0x7ffff0c05758, 0x0, 0x7f56b6ca202a, 0x4, 0x7ffff0c05758, 0x4f0c05780, 0x557450ea9da0, 0x0, ...)\\\\n\\\\t/usr/lib/golang/src/runtime/asm_amd64.s:214 +0x129 fp=0x7ffff0c05658 sp=0x7ffff0c05650 pc=0x557450ea9ed9\\\\n\\\"\"": unknown
ERROR: for cadvisor Cannot start service cadvisor: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:432: running prestart hook 0 caused \\\"error running hook: exit status 2, stdout: , stderr: fatal error: runtime: out of memory\\\\n\\\\nruntime stack:\\\\nruntime.throw(0x55cad6775359, 0x16)\\\\n\\\\t/usr/lib/golang/src/runtime/panic.go:774 +0x74 fp=0x7ffd948353c8 sp=0x7ffd94835398 pc=0x55cad4e21714\\\\nruntime.sysMap(0xc000000000, 0x4000000, 0x55cad8bc11f8)\\\\n\\\\t/usr/lib/golang/src/runtime/mem_linux.go:169 +0xc7 fp=0x7ffd94835408 sp=0x7ffd948353c8 pc=0x55cad4e0a457\\\\nruntime.(*mheap).sysAlloc(0x55cad8ba4a40, 0x2000, 0x0, 0x0)\\\\n\\\\t/usr/lib/golang/src/runtime/malloc.go:701 +0x1cf fp=0x7ffd948354b0 sp=0x7ffd94835408 pc=0x55cad4dfd3ff\\\\nruntime.(*mheap).grow(0x55cad8ba4a40, 0x1, 0xffffffff)\\\\n\\\\t/usr/lib/golang/src/runtime/mheap.go:1255 +0xa5 fp=0x7ffd94835500 sp=0x7ffd948354b0 pc=0x55cad4e18605\\\\nruntime.(*mheap).allocSpanLocked(0x55cad8ba4a40, 0x1, 0x55cad8bc1208, 0x1)\\\\n\\\\t/usr/lib/golang/src/runtime/mheap.go:1170 +0x268 fp=0x7ffd94835580 sp=0x7ffd94835500 pc=0x55cad4e18498\\\\nruntime.(*mheap).alloc_m(0x55cad8ba4a40, 0x1, 0x55cad4e4002a, 0x0)\\\\n\\\\t/usr/lib/golang/src/runtime/mheap.go:1022 +0xc6 fp=0x7ffd948355d0 sp=0x7ffd94835580 pc=0x55cad4e17d36\\\\nruntime.(*mheap).alloc.func1()\\\\n\\\\t/usr/lib/golang/src/runtime/mheap.go:1093 +0x4e fp=0x7ffd94835608 sp=0x7ffd948355d0 pc=0x55cad4e4d9de\\\\nruntime.(*mheap).alloc(0x55cad8ba4a40, 0x1, 0x1002a, 0x40000)\\\\n\\\\t/usr/lib/golang/src/runtime/mheap.go:1092 +0x8c fp=0x7ffd94835658 sp=0x7ffd94835608 pc=0x55cad4e1801c\\\\nruntime.(*mcentral).grow(0x55cad8ba5878, 0x0)\\\\n\\\\t/usr/lib/golang/src/runtime/mcentral.go:255 +0x7d fp=0x7ffd94835698 sp=0x7ffd94835658 pc=0x55cad4e09e7d\\\\nruntime.(*mcentral).cacheSpan(0x55cad8ba5878, 0x7f3f08073000)\\\\n\\\\t/usr/lib/golang/src/runtime/mcentral.go:106 +0x300 fp=0x7ffd948356f8 sp=0x7ffd9483569Creating grafana ... error
, 0x55cad8bc122a, 0x7f3f08073008, 0x8, 0xfffffffffffffff8)\\\\n\\\\t/usr/lib/golang/src/runtime/malloc.go:854 +0x89 fp=0x7ffd94835750 sp=0x7ffd94835718 pc=0x55cad4dfdc29\\\\nruntime.mallocgc(0x180, 0x55cad7923ea0, 0x1, 0x55cad8bc1260)\\\\n\\\\t/usr/lib/golang/src/runtime/malloc.go:1022 +0x7a5 fp=0x7ffd948357f0 sp=0x7ffd94835750 pc=0x55cad4dfe575\\\\nruntime.newobject(0x55cad7923ea0, 0x4000)\\\\n\\\\t/usr/lib/golang/src/runtime/malloc.go:1151 +0x3a fp=0x7ffd94835820 sp=0x7ffd948357f0 pc=0x55cad4dfe97a\\\\nruntime.malg(0x2d900008000, 0x55cad8ba70e8)\\\\n\\\\t/usr/lib/golang/src/runtime/proc.go:3232 +0x33 fp=0x7ffd94835860 sp=0x7ffd94835820 pc=0x55cad4e2abe3\\\\nruntime.mpreinit(...)\\\\n\\\\t/usr/lib/golang/src/runtime/os_linux.go:324\\\\nruntime.mcommoninit(0x55cad8b9f500)\\\\n\\\\t/usr/lib/golang/src/runtime/proc.go:626 +0xc6 fp=0x7ffd94835898 sp=0x7ffd94835860 pc=0x55cad4e24516\\\\nruntime.schedinit()\\\\n\\\\t/usr/lib/golang/src/runtime/proc.go:543 +0x76 fp=0x7ffd948358f0 sp=0x7ffd94835898 pc=0x55cad4e24176\\\\nruntime.rt0_go(0x7ffd948359f8, 0x4, 0x7ffd948359f8, 0x0, 0x7f3f0703002a, 0x4, 0x7ffd948359f8, 0x494835a20, 0x55cad4e4fda0, 0x0, ...)\\\\n\\\\t/usr/lib/golang/src/runtime/asm_amd64.s:214 +0x129 fp=0x7ffd948358f8 sp=0x7ffd948358f0 pc=0x55cad4e4fed9\\\\n\\\"\"": unknown
ERROR: for grafana Cannot start service grafana: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:432: running prestart hook 0 caused \\\"error running hook: exit status 2, stdout: , stderr: fatal error: runtime: out of memory\\\\n\\\\nruntime stack:\\\\nruntime.throw(0x55d9cb642359, 0x16)\\\\n\\\\t/usr/lib/golang/src/runtime/panic.go:774 +0x74 fp=0x7ffea2ec7248 sp=0x7ffea2ec7218 pc=0x55d9c9cee714\\\\nruntime.sysMap(0xc000000000, 0x4000000, 0x55d9cda8e1f8)\\\\n\\\\t/usr/lib/golang/src/runtime/mem_linux.go:169 +0xc7 fp=0x7ffea2ec7288 sp=0x7ffea2ec7248 pc=0x55d9c9cd7457\\\\nruntime.(*mheap).sysAlloc(0x55d9cda71a40, 0x2000, 0x0, 0x0)\\\\n\\\\t/usr/lib/golang/src/runtime/malloc.go:701 +0x1cf fp=0x7ffea2ec7330 sp=0x7ffea2ec7288 pc=0x55d9c9cca3ff\\\\nruntime.(*mheap).grow(0x55d9cda71a40, 0x1, 0xffffffff)\\\\n\\\\t/usr/lib/golang/src/runtime/mheap.go:1255 +0xa5 fp=0x7ffea2ec7380 sp=0x7ffea2ec7330 pc=0x55d9c9ce5605\\\\nruntime.(*mheap).allocSpanLocked(0x55d9cda71a40, 0x1, 0x55d9cda8e208, 0x1)\\\\n\\\\t/usr/lib/golang/src/runtime/mheap.go:1170 +0x268 fp=0x7ffea2ec7400 sp=0x7ffea2ec7380 pc=0x55d9c9ce5498\\\\nruntime.(*mheap).alloc_m(0x55d9cda71a40, 0x1, 0x55d9c9d1002a, 0x0)\\\\n\\\\t/usr/lib/golang/src/runtime/mheap.go:1022 +0xc6 fp=0x7ffea2ec7450 sp=0x7ffea2ec7400 pc=0x55d9c9ce4d36\\\\nruntime.(*mheap).alloc.func1()\\\\n\\\\t/usr/lib/golang/src/runtime/mheap.go:1093 +0x4e fp=0x7ffea2ec7488 sp=0x7ffea2ec7450 pc=0x55d9c9d1a9de\\\\nruntime.(*mheap).alloc(0x55d9cda71a40, 0x1, 0x1002a, 0x40000)\\\\n\\\\t/usr/lib/golang/src/runtime/mheap.go:1092 +0x8c fp=0x7ffea2ec74d8 sp=0x7ffea2ec7488 pc=0x55d9c9ce501c\\\\nruntime.(*mcentral).grow(0x55d9cda72878, 0x0)\\\\n\\\\t/usr/lib/golang/src/runtime/mcentral.go:255 +0x7d fp=0x7ffea2ec7518 sp=0x7ffea2ec74d8 pc=0x55d9c9cd6e7d\\\\nruntime.(*mcentral).cacheSpan(0x55d9cda72878, 0x7f52c3bd6000)\\\\n\\\\t/usr/lib/golang/src/runtime/mcentral.go:106 +0x300 fp=0x7ffea2ec7578 sp=0x7ffea2ec7518 pc=0x55d9c9cd69a0\\\\nruntime.(*mcache).refill(0x7f52c3bd6008, 0x2a)\\\\n\\\\t/usr/lib/golang/src/runtime/mcache.go:138 +0x87 fp=0x7ffea2ec7598 sp=0x7ffea2ec7578 pc=0x55d9c9cd6437\\\\nruntime.(*mcache).nextFree(0x7f52c3bd6008, 0x55d9cda8e22a, 0x7f52c3bd6008, 0x8, 0xfffffffffffffff8)\\\\n\\\\t/usr/lib/golang/src/runtime/malloc.go:854 +0x89 fp=0x7ffea2ec75d0 sp=0x7ffea2ec7598 pc=0x55d9c9ccac29\\\\nruntime.mallocgc(0x180, 0x55d9cc7f0ea0, 0x1, 0x55d9cda8e260)\\\\n\\\\t/usr/lib/golang/src/runtime/malloc.go:1022 +0x7a5 fp=0x7ffea2ec7670 sp=0x7ffea2ec75d0 pc=0x55d9c9ccb575\\\\nruntime.newobject(0x55d9cc7f0ea0, 0x4000)\\\\n\\\\t/usr/lib/golang/src/runtime/malloc.go:1151 +0x3a fp=0Creating caddy ... error
t(...)\\\\n\\\\t/usr/lib/golang/src/runtime/os_linux.go:324\\\\nruntime.mcommoninit(0x55d9cda6c500)\\\\n\\\\t/usr/lib/golang/src/runtime/proc.go:626 +0xc6 fp=0x7ffea2ec7718 sp=0x7ffea2ec76e0 pc=0x55d9c9cf1516\\\\nruntime.schedinit()\\\\n\\\\t/usr/lib/golang/src/runtime/proc.go:543 +0x76 fp=0x7ffea2ec7770 sp=0x7ffea2ec7718 pc=0x55d9c9cf1176\\\\nruntime.rt0_go(0x7ffea2ec7878, 0x4, 0x7ffea2ec7878, 0x0, 0x7f52c2b9302a, 0x4, 0x7ffea2ec7878, 0x4a2ec78a0, 0Creating prometheus ... done
Creating pushgateway ... done
Creating nodeexporter ... done
ERROR: for alertmanager Cannot start service alertmanager: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:432: running prestart hook 0 caused \\\"error running hook: exit status 2, stdout: , stderr: fatal error: runtime: out of memory\\\\n\\\\nruntime stack:\\\\nruntime.throw(0x5574527cf359, 0x16)\\\\n\\\\t/usr/lib/golang/src/runtime/panic.go:774 +0x74 fp=0x7ffff0c05128 sp=0x7ffff0c050f8 pc=0x557450e7b714\\\\nruntime.sysMap(0xc000000000, 0x4000000, 0x557454c1b1f8)\\\\n\\\\t/usr/lib/golang/src/runtime/mem_linux.go:169 +0xc7 fp=0x7ffff0c05168 sp=0x7ffff0c05128 pc=0x557450e64457\\\\nruntime.(*mheap).sysAlloc(0x557454bfea40, 0x2000, 0x0, 0x0)\\\\n\\\\t/usr/lib/golang/src/runtime/malloc.go:701 +0x1cf fp=0x7ffff0c05210 sp=0x7ffff0c05168 pc=0x557450e573ff\\\\nruntime.(*mheap).grow(0x557454bfea40, 0x1, 0xffffffff)\\\\n\\\\t/usr/lib/golang/src/runtime/mheap.go:1255 +0xa5 fp=0x7ffff0c05260 sp=0x7ffff0c05210 pc=0x557450e72605\\\\nruntime.(*mheap).allocSpanLocked(0x557454bfea40, 0x1, 0x557454c1b208, 0x1)\\\\n\\\\t/usr/lib/golang/src/runtime/mheap.go:1170 +0x268 fp=0x7ffff0c052e0 sp=0x7ffff0c05260 pc=0x557450e72498\\\\nruntime.(*mheap).alloc_m(0x557454bfea40, 0x1, 0x557450ea002a, 0x0)\\\\n\\\\t/usr/lib/golang/src/runtime/mheap.go:1022 +0xc6 fp=0x7ffff0c05330 sp=0x7ffff0c052e0 pc=0x557450e71d36\\\\nruntime.(*mheap).alloc.func1()\\\\n\\\\t/usr/lib/golang/src/runtime/mheap.go:1093 +0x4e fp=0x7ffff0c05368 sp=0x7ffff0c05330 pc=0x557450ea79de\\\\nruntime.(*mheap).alloc(0x557454bfea40, 0x1, 0x1002a, 0x40000)\\\\n\\\\t/usr/lib/golang/src/runtime/mheap.go:1092 +0x8c fp=0x7ffff0c053b8 sp=0x7ffff0c05368 pc=0x557450e7201c\\\\nruntime.(*mcentral).grow(0x557454bff878, 0x0)\\\\n\\\\t/usr/lib/golang/src/runtime/mcentral.go:255 +0x7d fp=0x7ffff0c053f8 sp=0x7ffff0c053b8 pc=0x557450e63e7d\\\\nruntime.(*mcentral).cacheSpan(0x557454bff878, 0x7f56b7ce5000)\\\\n\\\\t/usr/lib/golang/src/runtime/mcentral.go:106 +0x300 fp=0x7ffff0c05458 sp=0x7ffff0c053f8 pc=0x557450e639a0\\\\nruntime.(*mcache).refill(0x7f56b7ce5008, 0x2a)\\\\n\\\\t/usr/lib/golang/src/runtime/mcache.go:138 +0x87 fp=0x7ffff0c05478 sp=0x7ffff0c05458 pc=0x557450e63437\\\\nruntime.(*mcache).nextFree(0x7f56b7ce5008, 0x557454c1b22a, 0x7f56b7ce5008, 0x8, 0xfffffffffffffff8)\\\\n\\\\t/usr/lib/golang/src/runtime/malloc.go:854 +0x89 fp=0x7ffff0c054b0 sp=0x7ffff0c05478 pc=0x557450e57c29\\\\nruntime.mallocgc(0x180, 0x55745397dea0, 0x1, 0x557454c1b260)\\\\n\\\\t/usr/lib/golang/src/runtime/malloc.go:1022 +0x7a5 fp=0x7ffff0c05550 sp=0x7ffff0c054b0 pc=0x557450e58575\\\\nruntime.newobject(0x55745397dea0, 0x4000)\\\\n\\\\t/usr/lib/golang/src/runtime/malloc.go:1151 +0x3a fp=0x7ffff0c05580 sp=0x7ffff0c05550 pc=0x557450e5897a\\\\nruntime.malg(0x2d900008000, 0x557454c010e8)\\\\n\\\\t/usr/lib/golang/src/runtime/proc.go:3232 +0x33 fp=0x7ffff0c055c0 sp=0x7ffff0c05580 pc=0x557450e84be3\\\\nruntime.mpreinit(...)\\\\n\\\\t/usr/lib/golang/src/runtime/os_linux.go:324\\\\nruntime.mcommoninit(0x557454bf9500)\\\\n\\\\t/usr/lib/golang/src/runtime/proc.go:626 +0xc6 fp=0x7ffff0c055f8 sp=0x7ffff0c055c0 pc=0x557450e7e516\\\\nruntime.schedinit()\\\\n\\\\t/usr/lib/golang/src/runtime/proc.go:543 +0x76 fp=0x7ffff0c05650 sp=0x7ffff0c055f8 pc=0x557450e7e176\\\\nruntime.rt0_go(0x7ffff0c05758, 0x4, 0x7ffff0c05758, 0x0, 0x7f56b6ca202a, 0x4, 0x7ffff0c05758, 0x4f0c05780, 0x557450ea9da0, 0x0, ...)\\\\n\\\\t/usr/lib/golang/src/runtime/asm_amd64.s:214 +0x129 fp=0x7ffff0c05658 sp=0x7ffff0c05650 pc=0x557450ea9ed9\\\\n\\\"\"": unknown
ERROR: for cadvisor Cannot start service cadvisor: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:432: running prestart hook 0 caused \\\"error running hook: exit status 2, stdout: , stderr: fatal error: runtime: out of memory\\\\n\\\\nruntime stack:\\\\nruntime.throw(0x55cad6775359, 0x16)\\\\n\\\\t/usr/lib/golang/src/runtime/panic.go:774 +0x74 fp=0x7ffd948353c8 sp=0x7ffd94835398 pc=0x55cad4e21714\\\\nruntime.sysMap(0xc000000000, 0x4000000, 0x55cad8bc11f8)\\\\n\\\\t/usr/lib/golang/src/runtime/mem_linux.go:169 +0xc7 fp=0x7ffd94835408 sp=0x7ffd948353c8 pc=0x55cad4e0a457\\\\nruntime.(*mheap).sysAlloc(0x55cad8ba4a40, 0x2000, 0x0, 0x0)\\\\n\\\\t/usr/lib/golang/src/runtime/malloc.go:701 +0x1cf fp=0x7ffd948354b0 sp=0x7ffd94835408 pc=0x55cad4dfd3ff\\\\nruntime.(*mheap).grow(0x55cad8ba4a40, 0x1, 0xffffffff)\\\\n\\\\t/usr/lib/golang/src/runtime/mheap.go:1255 +0xa5 fp=0x7ffd94835500 sp=0x7ffd948354b0 pc=0x55cad4e18605\\\\nruntime.(*mheap).allocSpanLocked(0x55cad8ba4a40, 0x1, 0x55cad8bc1208, 0x1)\\\\n\\\\t/usr/lib/golang/src/runtime/mheap.go:1170 +0x268 fp=0x7ffd94835580 sp=0x7ffd94835500 pc=0x55cad4e18498\\\\nruntime.(*mheap).alloc_m(0x55cad8ba4a40, 0x1, 0x55cad4e4002a, 0x0)\\\\n\\\\t/usr/lib/golang/src/runtime/mheap.go:1022 +0xc6 fp=0x7ffd948355d0 sp=0x7ffd94835580 pc=0x55cad4e17d36\\\\nruntime.(*mheap).alloc.func1()\\\\n\\\\t/usr/lib/golang/src/runtime/mheap.go:1093 +0x4e fp=0x7ffd94835608 sp=0x7ffd948355d0 pc=0x55cad4e4d9de\\\\nruntime.(*mheap).alloc(0x55cad8ba4a40, 0x1, 0x1002a, 0x40000)\\\\n\\\\t/usr/lib/golang/src/runtime/mheap.go:1092 +0x8c fp=0x7ffd94835658 sp=0x7ffd94835608 pc=0x55cad4e1801c\\\\nruntime.(*mcentral).grow(0x55cad8ba5878, 0x0)\\\\n\\\\t/usr/lib/golang/src/runtime/mcentral.go:255 +0x7d fp=0x7ffd94835698 sp=0x7ffd94835658 pc=0x55cad4e09e7d\\\\nruntime.(*mcentral).cacheSpan(0x55cad8ba5878, 0x7f3f08073000)\\\\n\\\\t/usr/lib/golang/src/runtime/mcentral.go:106 +0x300 fp=0x7ffd948356f8 sp=0x7ffd94835698 pc=0x55cad4e099a0\\\\nruntime.(*mcache).refill(0x7f3f08073008, 0x2a)\\\\n\\\\t/usr/lib/golang/src/runtime/mcache.go:138 +0x87 fp=0x7ffd94835718 sp=0x7ffd948356f8 pc=0x55cad4e09437\\\\nruntime.(*mcache).nextFree(0x7f3f08073008, 0x55cad8bc122a, 0x7f3f08073008, 0x8, 0xfffffffffffffff8)\\\\n\\\\t/usr/lib/golang/src/runtime/malloc.go:854 +0x89 fp=0x7ffd94835750 sp=0x7ffd94835718 pc=0x55cad4dfdc29\\\\nruntime.mallocgc(0x180, 0x55cad7923ea0, 0x1, 0x55cad8bc1260)\\\\n\\\\t/usr/lib/golang/src/runtime/malloc.go:1022 +0x7a5 fp=0x7ffd948357f0 sp=0x7ffd94835750 pc=0x55cad4dfe575\\\\nruntime.newobject(0x55cad7923ea0, 0x4000)\\\\n\\\\t/usr/lib/golang/src/runtime/malloc.go:1151 +0x3a fp=0x7ffd94835820 sp=0x7ffd948357f0 pc=0x55cad4dfe97a\\\\nruntime.malg(0x2d900008000, 0x55cad8ba70e8)\\\\n\\\\t/usr/lib/golang/src/runtime/proc.go:3232 +0x33 fp=0x7ffd94835860 sp=0x7ffd94835820 pc=0x55cad4e2abe3\\\\nruntime.mpreinit(...)\\\\n\\\\t/usr/lib/golang/src/runtime/os_linux.go:324\\\\nruntime.mcommoninit(0x55cad8b9f500)\\\\n\\\\t/usr/lib/golang/src/runtime/proc.go:626 +0xc6 fp=0x7ffd94835898 sp=0x7ffd94835860 pc=0x55cad4e24516\\\\nruntime.schedinit()\\\\n\\\\t/usr/lib/golang/src/runtime/proc.go:543 +0x76 fp=0x7ffd948358f0 sp=0x7ffd94835898 pc=0x55cad4e24176\\\\nruntime.rt0_go(0x7ffd948359f8, 0x4, 0x7ffd948359f8, 0x0, 0x7f3f0703002a, 0x4, 0x7ffd948359f8, 0x494835a20, 0x55cad4e4fda0, 0x0, ...)\\\\n\\\\t/usr/lib/golang/src/runtime/asm_amd64.s:214 +0x129 fp=0x7ffd948358f8 sp=0x7ffd948358f0 pc=0x55cad4e4fed9\\\\n\\\"\"": unknown
ERROR: for grafana Cannot start service grafana: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:432: running prestart hook 0 caused \\\"error running hook: exit status 2, stdout: , stderr: fatal error: runtime: out of memory\\\\n\\\\nruntime stack:\\\\nruntime.throw(0x55d9cb642359, 0x16)\\\\n\\\\t/usr/lib/golang/src/runtime/panic.go:774 +0x74 fp=0x7ffea2ec7248 sp=0x7ffea2ec7218 pc=0x55d9c9cee714\\\\nruntime.sysMap(0xc000000000, 0x4000000, 0x55d9cda8e1f8)\\\\n\\\\t/usr/lib/golang/src/runtime/mem_linux.go:169 +0xc7 fp=0x7ffea2ec7288 sp=0x7ffea2ec7248 pc=0x55d9c9cd7457\\\\nruntime.(*mheap).sysAlloc(0x55d9cda71a40, 0x2000, 0x0, 0x0)\\\\n\\\\t/usr/lib/golang/src/runtime/malloc.go:701 +0x1cf fp=0x7ffea2ec7330 sp=0x7ffea2ec7288 pc=0x55d9c9cca3ff\\\\nruntime.(*mheap).grow(0x55d9cda71a40, 0x1, 0xffffffff)\\\\n\\\\t/usr/lib/golang/src/runtime/mheap.go:1255 +0xa5 fp=0x7ffea2ec7380 sp=0x7ffea2ec7330 pc=0x55d9c9ce5605\\\\nruntime.(*mheap).allocSpanLocked(0x55d9cda71a40, 0x1, 0x55d9cda8e208, 0x1)\\\\n\\\\t/usr/lib/golang/src/runtime/mheap.go:1170 +0x268 fp=0x7ffea2ec7400 sp=0x7ffea2ec7380 pc=0x55d9c9ce5498\\\\nruntime.(*mheap).alloc_m(0x55d9cda71a40, 0x1, 0x55d9c9d1002a, 0x0)\\\\n\\\\t/usr/lib/golang/src/runtime/mheap.go:1022 +0xc6 fp=0x7ffea2ec7450 sp=0x7ffea2ec7400 pc=0x55d9c9ce4d36\\\\nruntime.(*mheap).alloc.func1()\\\\n\\\\t/usr/lib/golang/src/runtime/mheap.go:1093 +0x4e fp=0x7ffea2ec7488 sp=0x7ffea2ec7450 pc=0x55d9c9d1a9de\\\\nruntime.(*mheap).alloc(0x55d9cda71a40, 0x1, 0x1002a, 0x40000)\\\\n\\\\t/usr/lib/golang/src/runtime/mheap.go:1092 +0x8c fp=0x7ffea2ec74d8 sp=0x7ffea2ec7488 pc=0x55d9c9ce501c\\\\nruntime.(*mcentral).grow(0x55d9cda72878, 0x0)\\\\n\\\\t/usr/lib/golang/src/runtime/mcentral.go:255 +0x7d fp=0x7ffea2ec7518 sp=0x7ffea2ec74d8 pc=0x55d9c9cd6e7d\\\\nruntime.(*mcentral).cacheSpan(0x55d9cda72878, 0x7f52c3bd6000)\\\\n\\\\t/usr/lib/golang/src/runtime/mcentral.go:106 +0x300 fp=0x7ffea2ec7578 sp=0x7ffea2ec7518 pc=0x55d9c9cd69a0\\\\nruntime.(*mcache).refill(0x7f52c3bd6008, 0x2a)\\\\n\\\\t/usr/lib/golang/src/runtime/mcache.go:138 +0x87 fp=0x7ffea2ec7598 sp=0x7ffea2ec7578 pc=0x55d9c9cd6437\\\\nruntime.(*mcache).nextFree(0x7f52c3bd6008, 0x55d9cda8e22a, 0x7f52c3bd6008, 0x8, 0xfffffffffffffff8)\\\\n\\\\t/usr/lib/golang/src/runtime/malloc.go:854 +0x89 fp=0x7ffea2ec75d0 sp=0x7ffea2ec7598 pc=0x55d9c9ccac29\\\\nruntime.mallocgc(0x180, 0x55d9cc7f0ea0, 0x1, 0x55d9cda8e260)\\\\n\\\\t/usr/lib/golang/src/runtime/malloc.go:1022 +0x7a5 fp=0x7ffea2ec7670 sp=0x7ffea2ec75d0 pc=0x55d9c9ccb575\\\\nruntime.newobject(0x55d9cc7f0ea0, 0x4000)\\\\n\\\\t/usr/lib/golang/src/runtime/malloc.go:1151 +0x3a fp=0x7ffea2ec76a0 sp=0x7ffea2ec7670 pc=0x55d9c9ccb97a\\\\nruntime.malg(0x2d900008000, 0x55d9cda740e8)\\\\n\\\\t/usr/lib/golang/src/runtime/proc.go:3232 +0x33 fp=0x7ffea2ec76e0 sp=0x7ffea2ec76a0 pc=0x55d9c9cf7be3\\\\nruntime.mpreinit(...)\\\\n\\\\t/usr/lib/golang/src/runtime/os_linux.go:324\\\\nruntime.mcommoninit(0x55d9cda6c500)\\\\n\\\\t/usr/lib/golang/src/runtime/proc.go:626 +0xc6 fp=0x7ffea2ec7718 sp=0x7ffea2ec76e0 pc=0x55d9c9cf1516\\\\nruntime.schedinit()\\\\n\\\\t/usr/lib/golang/src/runtime/proc.go:543 +0x76 fp=0x7ffea2ec7770 sp=0x7ffea2ec7718 pc=0x55d9c9cf1176\\\\nruntime.rt0_go(0x7ffea2ec7878, 0x4, 0x7ffea2ec7878, 0x0, 0x7f52c2b9302a, 0x4, 0x7ffea2ec7878, 0x4a2ec78a0, 0x55d9c9d1cda0, 0x0, ...)\\\\n\\\\t/usr/lib/golang/src/runtime/asm_amd64.s:214 +0x129 fp=0x7ffea2ec7778 sp=0x7ffea2ec7770 pc=0x55d9c9d1ced9\\\\n\\\"\"": unknown
ERROR: for caddy Cannot start service caddy: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"read init-p: connection reset by peer\"": unknown
ERROR: Encountered errors while bringing up the project.
[ec2-user@ip-172-31-44-87 dockprom]$
[ec2-user@ip-172-31-44-87 dockprom]$
[ec2-user@ip-172-31-44-87 dockprom]$ docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
55e0e745c9da prom/node-exporter:v1.0.1 "/bin/node_exporter …" 57 seconds ago Up 54 seconds 9100/tcp nodeexporter
4e3d82c83b9a prom/prometheus:v2.19.2 "/bin/prometheus --c…" 57 seconds ago Up 53 seconds 9090/tcp prometheus
9d6b6fd0ab00 prom/pushgateway:v1.2.0 "/bin/pushgateway" 57 seconds ago Up 53 seconds 9091/tcp pushgateway
[ec2-user@ip-172-31-44-87 dockprom]$ ADMIN_USER=admin ADMIN_PASSWORD=admin sudo /usr/local/bin/docker-compose up -d
nodeexporter is up-to-date
Starting caddy ...
prometheus is up-to-date
Starting cadvisor ...
pushgateway is up-to-date
Starting alertmanager ... error
Starting grafana ...
ERROR: for alertmanager Cannot start service alertmanager: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:432: running prestStarting caddy ... error
ERROR: for grafana Cannot start service grafana: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:432: running prestart hook 0 caused \\\"error running hook: signal: killed, stdout: , stderr: \\\"\"": unknown
Starting cadvisor ... done
ERROR: for caddy Cannot start service caddy: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:432: running prestart hook 0 caused \\\"error running hook: signal: killed, stdout: , stderr: \\\"\"": unknown
ERROR: for alertmanager Cannot start service alertmanager: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:432: running prestart hook 0 caused \\\"error running hook: signal: killed, stdout: , stderr: \\\"\"": unknown
ERROR: for grafana Cannot start service grafana: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:432: running prestart hook 0 caused \\\"error running hook: signal: killed, stdout: , stderr: \\\"\"": unknown
ERROR: for caddy Cannot start service caddy: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:432: running prestart hook 0 caused \\\"error running hook: signal: killed, stdout: , stderr: \\\"\"": unknown
ERROR: Encountered errors while bringing up the project.
[ec2-user@ip-172-31-44-87 dockprom]$
[ec2-user@ip-172-31-44-87 dockprom]$
[ec2-user@ip-172-31-44-87 dockprom]$
[ec2-user@ip-172-31-44-87 dockprom]$ docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
55e0e745c9da prom/node-exporter:v1.0.1 "/bin/node_exporter …" About a minute ago Up About a minute 9100/tcp nodeexporter
4e3d82c83b9a prom/prometheus:v2.19.2 "/bin/prometheus --c…" About a minute ago Up About a minute 9090/tcp prometheus
9d6b6fd0ab00 prom/pushgateway:v1.2.0 "/bin/pushgateway" About a minute ago Up About a minute 9091/tcp pushgateway
814ed12a0698 gcr.io/cadvisor/cadvisor:v0.37.0 "/usr/bin/cadvisor -…" About a minute ago Up 8 seconds (health: starting) 8080/tcp cadvisor
[ec2-user@ip-172-31-44-87 dockprom]$ ADMIN_USER=admin ADMIN_PASSWORD=admin sudo /usr/local/bin/docker-compose up -d
cadvisor is up-to-date
prometheus is up-to-date
Starting alertmanager ...
Starting grafana ...
nodeexporter is up-to-date
Starting caddy ...
Starting caddy ... error
ERROR: for caddy Cannot start service caddy: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:432: running prestart hook 0 caused \\\"error running hook: exit status 2, stdout: , stderr: fatal error: runtime: out of memory\\\\n\\\\nruntime stack:\\\\nruntime.throw(0x559ab258d359, 0x16)\\\\n\\\\t/usr/lib/golang/src/runtime/panic.go:774 +0x74 fp=0x7fff01e8ecf8 sp=0x7fff01e8ecc8 pc=0x559ab0c39714\\\\nruntime.sysMap(0xc000000000, 0x4000000, 0x559ab49d91f8)\\\\n\\\\t/usr/lib/golang/src/runtime/mem_linux.go:169 +0xc7 fp=0x7fff01e8ed38 sp=0x7fff01e8ecf8 pc=0x559ab0c22457\\\\nruntime.(*mheap).sysAlloc(0x559ab49bca40, 0x2000, 0x0, 0x0)\\\\n\\\\t/usr/lib/golang/src/runtime/malloc.go:701 +0x1cf fp=0x7fff01e8ede0 sp=0x7fff01e8ed38 pc=0x559ab0c153ff\\\\nruntime.(*mheap).grow(0x559ab49bca40, 0x1, 0xffffffff)\\\\n\\\\t/usr/lib/golang/src/runtime/mheap.go:1255 +0xa5 fp=0x7fff01e8ee30 sp=0x7fff01e8ede0 pc=0x559ab0c30605\\\\nruntime.(*mheap).allocSpanLocked(0x559ab49bca40, 0x1, 0x559ab49d9208, 0x1)\\\\n\\\\t/usr/lib/golang/src/runtime/mheap.go:1170 +0x268 fp=0x7fff01e8eeb0 sp=0x7fff01e8ee30 pc=0x559ab0c30498\\\\nruntime.(*mheap).alloc_m(0x559ab49bca40, 0x1, 0x559ab0c6002a, 0x0)\\\\n\\\\t/usr/lib/golang/src/runtime/mheap.go:1022 +0xc6 fp=0x7fff01e8ef00 sp=0x7fff01e8eeb0 pc=0x559ab0c2fd36\\\\nruntime.(*mheap).alloc.func1()\\\\n\\\\t/usr/lib/golang/src/runtime/mheap.go:1093 +0x4e fp=0x7fff01e8ef38 sp=0x7fff01e8ef00 pc=0x559ab0c659de\\\\nruntime.(*mheap).alloc(0x559ab49bca40, 0x1, 0x1002a, 0x40000)\\\\n\\\\t/usr/lib/golang/src/runtime/mheap.go:1092 +0x8c fp=0x7fff01e8ef88 sp=0x7fff01e8ef38 pc=0x559ab0c3001c\\\\nruntime.(*mcentral).grow(0x559ab49bd878, 0x0)\\\\n\\\\t/usr/lib/golang/src/runtime/mcentral.go:255 +0x7d fp=0x7fff01e8efc8 sp=0x7fff01e8ef88 pc=0x559ab0c21e7d\\\\nruntime.(*mcentral).cacheSpan(0x559ab49bd878, 0x7fb77cb98000)\\\\n\\\\t/usr/lib/golang/src/runtime/mcentral.go:106 +0x300 fp=0x7fff01e8f028 sp=0x7fff01e8efc8 pc=0x559ab0c219a0\\\\nruntime.(*mcache).refill(0x7fb77cb98008, 0x2a)\\\\n\\\\t/usr/lib/golang/src/runtime/mcache.go:138 +0x87 fp=0x7fff01e8f048 sp=0x7fff01e8f028 pc=0x559ab0c21437\\\\nruntime.(*mcache).nextFree(0x7fb77cb98008, 0x559ab49d922a, 0x7fb77cb98008, 0x8, 0xfffffffffffffff8)\\\\n\\\\t/usr/lib/golang/src/runtime/malloc.go:854 +0x89 fp=0x7fff01e8f080 sp=0x7fff01e8f048 pc=0x559ab0c15c29\\\\nruntime.mallocgc(0x180, 0x559ab373bea0, 0x1, 0x559ab49d9260)\\\\n\\\\t/usr/lib/golang/src/runtime/malloc.go:1022 +0x7a5 fp=0x7fff01e8f120 sp=0x7fff01e8f080 pc=0x559ab0c16575\\\\nruntime.newobject(0x559ab373bea0, 0x4000)\\\\n\\\\t/usr/lib/golang/src/runtime/malloc.go:1151 +0x3a fp=0x7fff01e8f150 sp=0x7fff01e8f120 pc=0x559ab0c1697a\\\\nruntime.malg(0x31000008000, 0x559ab49bf0e8)\\\\n\\\\t/usr/lib/golang/src/runtime/proc.go:3232 +0x33 fp=0x7fff01e8f190 sp=0x7fff01e8f150 pc=0x559ab0c42be3\\\\nruntime.mpreinit(..Starting alertmanager ... done
Starting grafana ... done
ab0c67da0, 0x0, ...)\\\\n\\\\t/usr/lib/golang/src/runtime/asm_amd64.s:214 +0x129 fp=0x7fff01e8f228 sp=0x7fff01e8f220 pc=0x559ab0c67ed9\\\\n\\\"\"": unknown
ERROR: for caddy Cannot start service caddy: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:432: running prestart hook 0 caused \\\"error running hook: exit status 2, stdout: , stderr: fatal error: runtime: out of memory\\\\n\\\\nruntime stack:\\\\nruntime.throw(0x559ab258d359, 0x16)\\\\n\\\\t/usr/lib/golang/src/runtime/panic.go:774 +0x74 fp=0x7fff01e8ecf8 sp=0x7fff01e8ecc8 pc=0x559ab0c39714\\\\nruntime.sysMap(0xc000000000, 0x4000000, 0x559ab49d91f8)\\\\n\\\\t/usr/lib/golang/src/runtime/mem_linux.go:169 +0xc7 fp=0x7fff01e8ed38 sp=0x7fff01e8ecf8 pc=0x559ab0c22457\\\\nruntime.(*mheap).sysAlloc(0x559ab49bca40, 0x2000, 0x0, 0x0)\\\\n\\\\t/usr/lib/golang/src/runtime/malloc.go:701 +0x1cf fp=0x7fff01e8ede0 sp=0x7fff01e8ed38 pc=0x559ab0c153ff\\\\nruntime.(*mheap).grow(0x559ab49bca40, 0x1, 0xffffffff)\\\\n\\\\t/usr/lib/golang/src/runtime/mheap.go:1255 +0xa5 fp=0x7fff01e8ee30 sp=0x7fff01e8ede0 pc=0x559ab0c30605\\\\nruntime.(*mheap).allocSpanLocked(0x559ab49bca40, 0x1, 0x559ab49d9208, 0x1)\\\\n\\\\t/usr/lib/golang/src/runtime/mheap.go:1170 +0x268 fp=0x7fff01e8eeb0 sp=0x7fff01e8ee30 pc=0x559ab0c30498\\\\nruntime.(*mheap).alloc_m(0x559ab49bca40, 0x1, 0x559ab0c6002a, 0x0)\\\\n\\\\t/usr/lib/golang/src/runtime/mheap.go:1022 +0xc6 fp=0x7fff01e8ef00 sp=0x7fff01e8eeb0 pc=0x559ab0c2fd36\\\\nruntime.(*mheap).alloc.func1()\\\\n\\\\t/usr/lib/golang/src/runtime/mheap.go:1093 +0x4e fp=0x7fff01e8ef38 sp=0x7fff01e8ef00 pc=0x559ab0c659de\\\\nruntime.(*mheap).alloc(0x559ab49bca40, 0x1, 0x1002a, 0x40000)\\\\n\\\\t/usr/lib/golang/src/runtime/mheap.go:1092 +0x8c fp=0x7fff01e8ef88 sp=0x7fff01e8ef38 pc=0x559ab0c3001c\\\\nruntime.(*mcentral).grow(0x559ab49bd878, 0x0)\\\\n\\\\t/usr/lib/golang/src/runtime/mcentral.go:255 +0x7d fp=0x7fff01e8efc8 sp=0x7fff01e8ef88 pc=0x559ab0c21e7d\\\\nruntime.(*mcentral).cacheSpan(0x559ab49bd878, 0x7fb77cb98000)\\\\n\\\\t/usr/lib/golang/src/runtime/mcentral.go:106 +0x300 fp=0x7fff01e8f028 sp=0x7fff01e8efc8 pc=0x559ab0c219a0\\\\nruntime.(*mcache).refill(0x7fb77cb98008, 0x2a)\\\\n\\\\t/usr/lib/golang/src/runtime/mcache.go:138 +0x87 fp=0x7fff01e8f048 sp=0x7fff01e8f028 pc=0x559ab0c21437\\\\nruntime.(*mcache).nextFree(0x7fb77cb98008, 0x559ab49d922a, 0x7fb77cb98008, 0x8, 0xfffffffffffffff8)\\\\n\\\\t/usr/lib/golang/src/runtime/malloc.go:854 +0x89 fp=0x7fff01e8f080 sp=0x7fff01e8f048 pc=0x559ab0c15c29\\\\nruntime.mallocgc(0x180, 0x559ab373bea0, 0x1, 0x559ab49d9260)\\\\n\\\\t/usr/lib/golang/src/runtime/malloc.go:1022 +0x7a5 fp=0x7fff01e8f120 sp=0x7fff01e8f080 pc=0x559ab0c16575\\\\nruntime.newobject(0x559ab373bea0, 0x4000)\\\\n\\\\t/usr/lib/golang/src/runtime/malloc.go:1151 +0x3a fp=0x7fff01e8f150 sp=0x7fff01e8f120 pc=0x559ab0c1697a\\\\nruntime.malg(0x31000008000, 0x559ab49bf0e8)\\\\n\\\\t/usr/lib/golang/src/runtime/proc.go:3232 +0x33 fp=0x7fff01e8f190 sp=0x7fff01e8f150 pc=0x559ab0c42be3\\\\nruntime.mpreinit(...)\\\\n\\\\t/usr/lib/golang/src/runtime/os_linux.go:324\\\\nruntime.mcommoninit(0x559ab49b7500)\\\\n\\\\t/usr/lib/golang/src/runtime/proc.go:626 +0xc6 fp=0x7fff01e8f1c8 sp=0x7fff01e8f190 pc=0x559ab0c3c516\\\\nruntime.schedinit()\\\\n\\\\t/usr/lib/golang/src/runtime/proc.go:543 +0x76 fp=0x7fff01e8f220 sp=0x7fff01e8f1c8 pc=0x559ab0c3c176\\\\nruntime.rt0_go(0x7fff01e8f328, 0x4, 0x7fff01e8f328, 0x0, 0x7fb77bb5502a, 0x4, 0x7fff01e8f328, 0x401e8f350, 0x559ab0c67da0, 0x0, ...)\\\\n\\\\t/usr/lib/golang/src/runtime/asm_amd64.s:214 +0x129 fp=0x7fff01e8f228 sp=0x7fff01e8f220 pc=0x559ab0c67ed9\\\\n\\\"\"": unknown
ERROR: Encountered errors while bringing up the project.
[ec2-user@ip-172-31-44-87 dockprom]$
[ec2-user@ip-172-31-44-87 dockprom]$
[ec2-user@ip-172-31-44-87 dockprom]$ docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
55e0e745c9da prom/node-exporter:v1.0.1 "/bin/node_exporter …" About a minute ago Up About a minute 9100/tcp nodeexporter
4e3d82c83b9a prom/prometheus:v2.19.2 "/bin/prometheus --c…" About a minute ago Up About a minute 9090/tcp prometheus
9d6b6fd0ab00 prom/pushgateway:v1.2.0 "/bin/pushgateway" About a minute ago Up About a minute 9091/tcp pushgateway
814ed12a0698 gcr.io/cadvisor/cadvisor:v0.37.0 "/usr/bin/cadvisor -…" About a minute ago Up 22 seconds (health: starting) 8080/tcp cadvisor
800d775c3994 prom/alertmanager:v0.21.0 "/bin/alertmanager -…" About a minute ago Up 3 seconds 9093/tcp alertmanager
8abda3eb8499 grafana/grafana:7.0.6 "/run.sh" About a minute ago Up 3 seconds 3000/tcp grafana
[ec2-user@ip-172-31-44-87 dockprom]$
[ec2-user@ip-172-31-44-87 dockprom]$
[ec2-user@ip-172-31-44-87 dockprom]$
[ec2-user@ip-172-31-44-87 dockprom]$ ADMIN_USER=admin ADMIN_PASSWORD=admin sudo /usr/local/bin/docker-compose up -d
alertmanager is up-to-date
pushgateway is up-to-date
nodeexporter is up-to-date
cadvisor is up-to-date
grafana is up-to-date
prometheus is up-to-date
Starting caddy ... done
[ec2-user@ip-172-31-44-87 dockprom]$
[ec2-user@ip-172-31-44-87 dockprom]$
[ec2-user@ip-172-31-44-87 dockprom]$ docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
55e0e745c9da prom/node-exporter:v1.0.1 "/bin/node_exporter …" 2 minutes ago Up 2 minutes 9100/tcp nodeexporter
4e3d82c83b9a prom/prometheus:v2.19.2 "/bin/prometheus --c…" 2 minutes ago Up 2 minutes 9090/tcp prometheus
9d6b6fd0ab00 prom/pushgateway:v1.2.0 "/bin/pushgateway" 2 minutes ago Up 2 minutes 9091/tcp pushgateway
814ed12a0698 gcr.io/cadvisor/cadvisor:v0.37.0 "/usr/bin/cadvisor -…" 2 minutes ago Up 41 seconds (healthy) 8080/tcp cadvisor
800d775c3994 prom/alertmanager:v0.21.0 "/bin/alertmanager -…" 2 minutes ago Up 21 seconds 9093/tcp alertmanager
566bd13be427 stefanprodan/caddy "/sbin/tini -- caddy…" 2 minutes ago Up 3 seconds 0.0.0.0:3000->3000/tcp, 0.0.0.0:9090-9091->9090-9091/tcp, 0.0.0.0:9093->9093/tcp caddy
8abda3eb8499 grafana/grafana:7.0.6 "/run.sh" 2 minutes ago Up 21 seconds 3000/tcp grafana
[ec2-user@ip-172-31-44-87 dockprom]$
Okay so it seems the issue was just the limited amount of ram (was running this on 512mb box). Bumping it to 1GB seems to have made things better.