Index / Visualization Mismatch
koelslaw opened this issue · comments
Indexes are present but the dashboards error out.
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .kibana_7.12.1_001 ordWlk6IRyK7laqYYdmhhQ 1 0 930 996 7.5mb 7.5mb
green open .monitoring-kibana-7-2021.05.16 3W3wuq_LQJGcf_OllHxGdg 1 0 348 0 189.2kb 189.2kb
green open ecs-suricata-network-2021.05.16 gUPiWrOlRoOzADHtwsqh6Q 1 0 489 0 2.4mb 2.4mb
green open ecs-zeek-observations-2021.05.16 bNo206X4TD2ycj3Dfyrnzw 1 0 2 0 18.3kb 18.3kb
green open .apm-agent-configuration 8eMb3Pl4QoaWOJ_beMO6bA 1 0 0 0 208b 208b
green open .kibana_task_manager_7.12.1_001 9LIMNTACSdqCLStf82nM_Q 1 0 9 199 280.7kb 280.7kb
green open ecs-zeek-miscellaneous-2021.05.16 gZeD5cGRRSy-voYkG08vpA 1 0 23 0 56.6kb 56.6kb
green open ecs-zeek-network-2021.05.16 r_TV1DqkT1-CeounuVTymQ 1 0 16 0 71.3kb 71.3kb
green open .kibana-event-log-7.12.1-000001 K0ny7VC5TkeMfSKEXukYOg 1 0 2 0 11kb 11kb
green open .tasks roxuTzyEQzedwrMMjKxiwA 1 0 2 0 13.7kb 13.7kb
green open .monitoring-logstash-7-2021.05.16 xEgfFpBOQcGxpOLJShZxmw 1 0 74406 0 29.5mb 29.5mb
green open ecs-zeek-diagnostic-2021.05.16 M9kC17OKR3Oe-oFFZCnacQ 1 0 1762 0 319kb 319kb
green open .apm-custom-link A0O5HyMPTXmvuGoL3_iz3Q 1 0 0 0 208b 208b
green open .async-search GYZJhfZNQkW_YFS6khIDmg 1 0 15 0 12.7kb 12.7kb
green open .monitoring-es-7-2021.05.16 bN0BQnOBTQSNwPvq0pxAXg 1 0 4385 488 5.2mb 5.2mb
Conditions:
4 Cores
2 NICs
16 GB RAM
Automated Install
Single Node
rocknsm-20210514-14_14_30.iso
This could be an issue with Elastic 7.11 it looks like. I will keep this open but want to try another elastic version to see if we get the same errors with the dashboards before changing them.
As discussed I believe a good place to look for this issue is in the lighttpd proxy config. I confirmed this in 7.11 but didn't test 7.12. Here is a snippet of the lighttpd error log:
2021-03-11 16:10:38: (http-header-glue.c.1032) proxy backend sent invalid response header (Transfer-Encoding) to HTTP/1.0 request
2021-03-11 16:10:38: (gw_backend.c.2125) response not received, request sent: 927 on socket: tcp:127.0.0.1:5601 for /internal?, closing connection
Ended up removing the lighttpd which was buggy and installed nginx.
You can use this simple configuration as it works, but if your rock web access is in a closed network only.
If you feel like switching to a SSL certificate and auth you will have a little bit more work to do.
location / {
proxy_pass http://127.0.0.1:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-for $remote_addr;
port_in_redirect off;
proxy_connect_timeout 3600;
}
Also in kibana.yml add this so it doesn't buzz all the time with the publicbaseurl notice:
server.publicBaseUrl: "http://127.0.0.1:5601"
server.host: 127.0.0.1
server.port: 5601
Hope it helps.