cisagov / Malcolm

Malcolm is a powerful, easily deployable network traffic analysis tool suite for full packet capture artifacts (PCAP files), Zeek logs and Suricata alerts.

Home Page:https://cisagov.github.io/Malcolm/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

install.py script not installing docker compose and docker-compose pull is not working

Pascar85 opened this issue Β· comments

πŸ› Summary

Install.py script is not installing docker-compose and after manual install of docker-compose, when docker-compose pull command is issued nothing is happening form the Malcolm directory.

What's wrong? Please be specific.

Hi there.. i was trying to install Malcolm on my proxmox test lab today and i noticed that the install script is not installing docker-compose. i did a reboot after the first part of the install script, then after the ./scripts/auth_setup i tried do a docker-compose pull and it give an error that docker-compose is not installed. then i installed it manually. but when i do a docker-compose pull from the malcolm directory, nothing happens. could it be that i missed a step somewere parhaps. i have shared the links i used bellow
im trying to install on ubuntu 22.04 LTS server fully updated.

for malcolm installation in used:
https://malcolm.fyi/docs/ubuntu-install-example.html#InstallationExample
for docker installation is used:
https://support.netfoundry.io/hc/en-us/articles/360057865692-Installing-Docker-and-docker-compose-for-Ubuntu-20-04

To reproduce

Steps to reproduce the behavior:

  1. Do this
  2. Then this

Expected behavior

What did you expect to happen that didn't?

Any helpful log output or screenshots

Paste the results here:

Add any screenshots of the problem here.

I'll check into the automatic installation of docker compose and report back here.

As of the latest Malcolm, we have got some "profiles" specified in the docker-compose.yml. Try docker-compose --profile malcolm pull and see if that's better for you. I'll look over the documentation and see if I need to call that out specifically in the documentation. Maybe I can find a way to make the "malcolm" profile default.

okay sir.. il will be waiting.. mean while il try out the profile option.

Imaged to pull the images with the --profile macolm option, but then now i cant start malcolm with the ./start script. i have shared the errors im getting bellow. i think its a perssion error.

controller@soc-malcolm-srv01:/Malcolm/scripts$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
ghcr.io/idaholab/malcolm/nginx-proxy 23.10.0 787f638e77f5 6 days ago 168MB
ghcr.io/idaholab/malcolm/api 23.10.0 ba8c629b39bf 6 days ago 186MB
ghcr.io/idaholab/malcolm/logstash-oss 23.10.0 a872ad58acc1 6 days ago 1.33GB
ghcr.io/idaholab/malcolm/arkime 23.10.0 64f654f8f708 6 days ago 834MB
ghcr.io/idaholab/malcolm/dashboards 23.10.0 20c87b20e2ea 6 days ago 1.6GB
ghcr.io/idaholab/malcolm/file-monitor 23.10.0 9ff69d8be044 7 days ago 685MB
ghcr.io/idaholab/malcolm/netbox 23.10.0 3d2df04aa076 7 days ago 1.27GB
ghcr.io/idaholab/malcolm/file-upload 23.10.0 df7081c70298 7 days ago 250MB
ghcr.io/idaholab/malcolm/suricata 23.10.0 345d5a60512a 7 days ago 318MB
ghcr.io/idaholab/malcolm/htadmin 23.10.0 d705d8b2c04f 7 days ago 245MB
ghcr.io/idaholab/malcolm/filebeat-oss 23.10.0 45372bce9eef 7 days ago 387MB
ghcr.io/idaholab/malcolm/opensearch 23.10.0 0d13aeb8fb37 7 days ago 1.61GB
ghcr.io/idaholab/malcolm/pcap-monitor 23.10.0 aa9258525c7b 7 days ago 176MB
ghcr.io/idaholab/malcolm/pcap-capture 23.10.0 6062596f108e 7 days ago 137MB
ghcr.io/idaholab/malcolm/freq 23.10.0 8966c13a5d75 7 days ago 153MB
ghcr.io/idaholab/malcolm/dashboards-helper 23.10.0 a27e63284942 7 days ago 185MB
ghcr.io/idaholab/malcolm/redis 23.10.0 b8627e86b40e 7 days ago 47.3MB
ghcr.io/idaholab/malcolm/postgresql 23.10.0 f67044ae20d3 7 days ago 294MB
ghcr.io/idaholab/malcolm/zeek 23.10.0 77ac798542a1 7 days ago 1.22GB
controller@soc-malcolm-srv01:
/Malcolm/scripts$ ./start
PermissionError: [Errno 13] Permission denied: '/home/controller/Malcolm/opensearch/nodes'
controller@soc-malcolm-srv01:/Malcolm/scripts$ sudo ./start
Exception: start should not be run as root
controller@soc-malcolm-srv01:
/Malcolm/scripts$

As indicated by the errors you see, Malcolm's not meant to be run as root. You should run it as the same user as the UID/GID you set up during configuration (in the ./config/process.env configuration file). In this case that appears to be the user controller?

So, do something like:

sudo chown -R controller:controller /home/controller/Malcolm

Then try the start.

Okay, thank you sir.. pardon me I'm not that of an expert in linux. but it seems to be starting now.

I'll update the documentation as per your suggestion. Thanks.

is this out put normal? there seems to be some exceptions or errors

pcap-monitor_1 | socket.gaierror: [Errno -2] Name or service not known
pcap-monitor_1 |
pcap-monitor_1 | During handling of the above exception, another exception occurred:
pcap-monitor_1 |
pcap-monitor_1 | urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fab5a242d50>: Failed to establish a new connection: [Errno -2] Name or service not known
pcap-monitor_1 | 2023-10-31 15:23:54 ERROR: pcap_watcher.py: 1 connection error: ConnectionError(<urllib3.connection.HTTPConnection object at 0x7fab5a242d50>: Failed to establish a new connection: [Errno -2] Name or service not known) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7fab5a242d50>: Failed to establish a new connection: [Errno -2] Name or service not known)
pcap-monitor_1 | 2023-10-31 15:23:55 WARNING: GET http://opensearch:9200/_cluster/health [status:N/A request:0.018s]
pcap-monitor_1 | socket.gaierror: [Errno -2] Name or service not known
pcap-monitor_1 |
pcap-monitor_1 | During handling of the above exception, another exception occurred:
pcap-monitor_1 |
pcap-monitor_1 | urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fab5a1d39d0>: Failed to establish a new connection: [Errno -2] Name or service not known
pcap-monitor_1 | 2023-10-31 15:23:55 WARNING: GET http://opensearch:9200/_cluster/health [status:N/A request:0.035s]
pcap-monitor_1 | socket.gaierror: [Errno -2] Name or service not known
pcap-monitor_1 |
pcap-monitor_1 | During handling of the above exception, another exception occurred:
pcap-monitor_1 |
pcap-monitor_1 | urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fab5a1d1410>: Failed to establish a new connection: [Errno -2] Name or service not known
pcap-monitor_1 | 2023-10-31 15:23:55 WARNING: GET http://opensearch:9200/_cluster/health [status:N/A request:0.022s]
pcap-monitor_1 | socket.gaierror: [Errno -2] Name or service not known
pcap-monitor_1 |
pcap-monitor_1 | During handling of the above exception, another exception occurred:
pcap-monitor_1 |
pcap-monitor_1 | urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fab5a236a90>: Failed to establish a new connection: [Errno -2] Name or service not known
pcap-monitor_1 | 2023-10-31 15:23:55 WARNING: GET http://opensearch:9200/_cluster/health [status:N/A request:0.052s]
pcap-monitor_1 | socket.gaierror: [Errno -2] Name or service not known
pcap-monitor_1 |
pcap-monitor_1 | During handling of the above exception, another exception occurred:
pcap-monitor_1 |
pcap-monitor_1 | urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fab5a213590>: Failed to establish a new connection: [Errno -2] Name or service not known
pcap-monitor_1 | 2023-10-31 15:23:55 ERROR: pcap_watcher.py: 1 connection error: ConnectionError(<urllib3.connection.HTTPConnection object at 0x7fab5a213590>: Failed to establish a new connection: [Errno -2] Name or service not known) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7fab5a213590>: Failed to establish a new connection: [Errno -2] Name or service not known)

sorry didnt know you closed the issue, but its fine if you are updating the documentation.

It's ok we can still follow it up here. Openseaerch isn't staying running, which means you've probably got it not assigned enough resources.

  1. What is the amount of RAM on the system?
  2. How much do you have assigned to opensearch?
    • grep Xms ./config/*.env
  3. ./scripts/logs -s opensearch
    • share the output of those logs

the system has 32 Gigs of RAM.

controller@soc-malcolm-srv01:/Malcolm$ grep Xms ./config/*.env
./config/logstash.env:LS_JAVA_OPTS=-server -Xms2500m -Xmx2500m -Xss1536k -XX:-HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/./urandom -Dlog4j.formatMsgNoLookups=true
./config/opensearch.env:OPENSEARCH_JAVA_OPTS=-server -Xms16g -Xmx16g -Xss256k -XX:-HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/./urandom -Dlog4j.formatMsgNoLookups=true
controller@soc-malcolm-srv01:
/Malcolm$

controller@soc-malcolm-srv01:~/Malcolm$ ./scripts/logs -s opensearch
Name Command State Ports

malcolm_opensearch_1 /usr/bin/tini -- /usr/loca ... Exit 1

Attaching to malcolm_opensearch_1
opensearch_1 | root
opensearch_1 | uid=0(root) gid=0(root) groups=0(root)
opensearch_1 |
opensearch_1 | OpenSearch cannot run as root. Please start your container as another user.
malcolm_opensearch_1 exited with code 1

controller@soc-malcolm-srv01:~/Malcolm$

Okay, you've got bad values in your ./config/process.env file.

PUID=1000
PGID=1000

So during configuration you told Malcolm to run as root, I guess? I should prevent that from happening. Those values should be whatever the ID/GID of your controller user is. So either run through ./scripts/configure again or just edit that file and set PUID and PGID to your user's UID and GID (e.g., id -u and id -g, respectively). You may also need to run the chmod command from above again.

Okay.. Il try again in a while in a bit and update you im a bit far from the box now

On which file should chmod again?

sudo chown -R controller:controller /home/controller/Malcolm

Okay.. I will do that sir

What is the ideal RAM size and hard disk size for production Malcolm.. We want to deply it in our environment.

Minimum is 16GB with 4 CPU cores (for like a VM or laptop installation). On any long-term deployment I'd recommend 32GB or up and 8+ cores. Hard disk size depends on the amount of retention you want to have, I'd recommend at least a couple hundred gigabytes as a base and then more depending on how long you want index retention

Okay.. I think we have enough resources for that

Hallo again sir.. i implemented the solutions you shared earlier and tried to start macolm again..im still getting some erros.

pcap-monitor_1 |
pcap-monitor_1 | urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f57a4c6b050>: Failed to establish a new connection: [Errno 111] Connection refused
pcap-monitor_1 | 2023-11-01 00:43:18 ERROR: pcap_watcher.py: 1 connection error: ConnectionError(<urllib3.connection.HTTPConnection object at 0x7f57a4c6b050>: Failed to establish a new connection: [Errno 111] Connection refused) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7f57a4c6b050>: Failed to establish a new connection: [Errno 111] Connection refused)
pcap-monitor_1 | 2023-11-01 00:43:19 WARNING: GET http://opensearch:9200/_cluster/health [status:N/A request:0.065s]
pcap-monitor_1 | ConnectionRefusedError: [Errno 111] Connection refused
pcap-monitor_1 |
pcap-monitor_1 | During handling of the above exception, another exception occurred:
pcap-monitor_1 |
pcap-monitor_1 | urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f57a4dc9b10>: Failed to establish a new connection: [Errno 111] Connection refused
pcap-monitor_1 | 2023-11-01 00:43:19 WARNING: GET http://opensearch:9200/_cluster/health [status:N/A request:0.130s]
pcap-monitor_1 | ConnectionRefusedError: [Errno 111] Connection refused
pcap-monitor_1 |
pcap-monitor_1 | During handling of the above exception, another exception occurred:
pcap-monitor_1 |
pcap-monitor_1 | urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f57a4dca4d0>: Failed to establish a new connection: [Errno 111] Connection refused
pcap-monitor_1 | 2023-11-01 00:43:19 WARNING: GET http://opensearch:9200/_cluster/health [status:N/A request:0.101s]
pcap-monitor_1 | ConnectionRefusedError: [Errno 111] Connection refused
pcap-monitor_1 |
pcap-monitor_1 | During handling of the above exception, another exception occurred:
pcap-monitor_1 |
pcap-monitor_1 | urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f57a4c4db50>: Failed to establish a new connection: [Errno 111] Connection refused
pcap-monitor_1 | 2023-11-01 00:43:20 WARNING: GET http://opensearch:9200/_cluster/health [status:N/A request:0.148s]
pcap-monitor_1 | ConnectionRefusedError: [Errno 111] Connection refused
pcap-monitor_1 |
pcap-monitor_1 | During handling of the above exception, another exception occurred:
pcap-monitor_1 |
pcap-monitor_1 | urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f57a4c4e0d0>: Failed to establish a new connection: [Errno 111] Connection refused
pcap-monitor_1 | 2023-11-01 00:43:20 ERROR: pcap_watcher.py: 1 connection error: ConnectionError(<urllib3.connection.HTTPConnection object at 0x7f57a4c4e0d0>: Failed to establish a new connection: [Errno 111] Connection refused) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7f57a4c4e0d0>: Failed to establish a new connection: [Errno 111] Connection refused)
pcap-monitor_1 | 2023-11-01 00:43:21 WARNING: GET http://opensearch:9200/_cluster/health [status:N/A request:0.144s]
pcap-monitor_1 | ConnectionRefusedError: [Errno 111] Connection refused
pcap-monitor_1 |
pcap-monitor_1 | During handling of the above exception, another exception occurred:
pcap-monitor_1 |
pcap-monitor_1 | urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f57a4e7a2d0>: Failed to establish a new connection: [Errno 111] Connection refused

when i issue the ./scripts/logs -s opensearch im getting the out bellow.

controller@soc-malcolm-srv01:~/Malcolm$ ./scripts/logs -s opensearch
Name Command State Ports

malcolm_opensearch_1 /usr/bin/tini -- /usr/loca ... Up (unhealthy) 9200/tcp, 9300/tcp, 9600/tcp, 9650/tcp

Attaching to malcolm_opensearch_1
opensearch_1 | usermod: no changes
opensearch_1 | opensearch
opensearch_1 | uid=1000(opensearch) gid=1000(opensearch) groups=1000(opensearch)
opensearch_1 |
opensearch_1 | OpenSearch Security Plugin does not exist, disable by default
opensearch_1 | OpenSearch Performance Analyzer Plugin does not exist, disable by default
opensearch_1 | WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/li b/opensearch-2.8.0.jar)
opensearch_1 | WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
opensearch_1 | WARNING: System::setSecurityManager will be removed in a future release
opensearch_1 | WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/ opensearch-2.8.0.jar)
opensearch_1 | WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
opensearch_1 | WARNING: System::setSecurityManager will be removed in a future release

any progress on the updating of the documentation?

The documentation will be updated in the next release of Malcolm, sometime in November.

That's the entire output of the opensearch log? That doesn't tell me much. The status says unhealthy but it doesn't seem to indicate what happened to it. Is the opensearch container getting high cpu usage? Running docker stats will show container resource usage.

i issued the command and nothing really seams to pop up, just random once.. but mostly it showing this

CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
8517af397b62 malcolm_nginx-proxy_1 -- -- / -- -- -- -- --
1524399d2a03 malcolm_dashboards_1 -- -- / -- -- -- -- --
c20c3d1cbee3 malcolm_netbox_1 -- -- / -- -- -- -- --
6000518361e3 malcolm_arkime_1 26.18% 166.8MiB / 31.34GiB 0.52% 3.83MB / 2.13MB 115MB / 7.3MB 35
b60b72d66b75 malcolm_dashboards-helper_1 -- -- / -- -- -- -- --
28f116e0f5b2 malcolm_zeek_1 -- -- / -- -- -- -- --
f8da263ba1c6 malcolm_logstash_1 -- -- / -- -- -- -- --
6ad032958c4d malcolm_pcap-monitor_1 -- -- / -- -- -- -- --
9feaa381f4a2 malcolm_api_1 -- -- / -- -- -- -- --
64a9ee674bef malcolm_pcap-capture_1 -- -- / -- -- -- -- --
39426d65a5e9 malcolm_netbox-redis-cache_1 -- -- / -- -- -- -- --
13c5a16f7708 malcolm_zeek-live_1 -- -- / -- -- -- -- --
050ec6b3b952 malcolm_suricata_1 -- -- / -- -- -- -- --
9d6fd7ed1074 malcolm_htadmin_1 -- -- / -- -- -- -- --
04aa0ab9f8e4 malcolm_upload_1 -- -- / -- -- -- -- --
4b9bc263193e malcolm_netbox-redis_1 -- -- / -- -- -- -- --
4a1f816d29f8 malcolm_netbox-postgres_1 -- -- / -- -- -- -- --
ff3e3a81a3e4 malcolm_freq_1 -- -- / -- -- -- -- --
f22e20d9921b malcolm_suricata-live_1 -- -- / -- -- -- -- --
dcefc99e43ab malcolm_filebeat_1 -- -- / -- -- -- -- --
c04308c7eb00 malcolm_file-monitor_1 -- -- / -- -- -- -- --
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
8517af397b62 malcolm_nginx-proxy_1 -- -- / -- -- -- -- --
1524399d2a03 malcolm_dashboards_1 -- -- / -- -- -- -- --
c20c3d1cbee3 malcolm_netbox_1 -- -- / -- -- -- -- --
6000518361e3 malcolm_arkime_1 26.18% 166.8MiB / 31.34GiB 0.52% 3.83MB / 2.13MB 115MB / 7.3MB 35
b60b72d66b75 malcolm_dashboards-helper_1 -- -- / -- -- -- -- --
28f116e0f5b2 malcolm_zeek_1 -- -- / -- -- -- -- --
f8da263ba1c6 malcolm_logstash_1 -- -- / -- -- -- -- --
6ad032958c4d malcolm_pcap-monitor_1 -- -- / -- -- -- -- --
9feaa381f4a2 malcolm_api_1 -- -- / -- -- -- -- --
64a9ee674bef malcolm_pcap-capture_1 -- -- / -- -- -- -- --
39426d65a5e9 malcolm_netbox-redis-cache_1 -- -- / -- -- -- -- --
13c5a16f7708 malcolm_zeek-live_1 -- -- / -- -- -- -- --
050ec6b3b952 malcolm_suricata_1 -- -- / -- -- -- -- --
9d6fd7ed1074 malcolm_htadmin_1 -- -- / -- -- -- -- --
04aa0ab9f8e4 malcolm_upload_1 -- -- / -- -- -- -- --
4b9bc263193e malcolm_netbox-redis_1 -- -- / -- -- -- -- --
4a1f816d29f8 malcolm_netbox-postgres_1 -- -- / -- -- -- -- --
ff3e3a81a3e4 malcolm_freq_1 -- -- / -- -- -- -- --
f22e20d9921b malcolm_suricata-live_1 -- -- / -- -- -- -- --
dcefc99e43ab malcolm_filebeat_1 -- -- / -- -- -- -- --
c04308c7eb00 malcolm_file-monitor_1 -- -- / -- -- -- -- --

should i perhaps redo the whole installation from scratch? then maybe you just advice me on the config options i should look out for

I think you should do that, yeah. Just blow away the whole thing and start over. Read over this document as you answer the configuration question: https://cisagov.github.io/Malcolm/docs/malcolm-hedgehog-e2e-iso-install.html#MalcolmConfig

okay, thank i will do that, i will reach out again if things dont go right