Elasticsearch: NoNodeAvailableException

Question

Elasticsearch: NoNodeAvailableException

sidharth15 opened this issue 2 years ago · comments

Hi,

Good day to you.

I'm trying to conduct a benchmarking test of the Elasticsearch cluster I have deployed on Azure using Elastic cloud (Managed Elasticsearch).

I have the cluster up and running at https://nosql-logging-cluster.es.uksouth.azure.elastic-cloud.com on port 9243. I verified this by hitting the API using Postman. I also have Azure functions that query and ingest data into the Elasticsearch cluster, and they are all working fine.

However, when I try to run the ycsb tool on this cluster using the command given below, I get the following error:

Command used:
./bin/ycsb.sh run elasticsearch -s -P workloads/workloada -p es.remote=true -p es.hosts.list=nosql-logging-cluster.es.uksouth.azure.elastic-cloud.com:9243 -p path.home=~/benchmarking/ycsb-0.17.0/some-path -p cluster.name=nosql-logging-cluster

Output:

/usr/lib/jvm/java-8-openjdk-amd64/bin/java  -classpath /home/sid/benchmarking/ycsb-0.17.0/conf:/home/sid/benchmarking/ycsb-0.17.0/lib/HdrHistogram-2.1.4.jar:/home/sid/benchmarking/ycsb-0.17.0/lib/core-0.17.0.jar:/home/sid/benchmarking/ycsb-0.17.0/lib/htrace-core4-4.1.0-incubating.jar:/home/sid/benchmarking/ycsb-0.17.0/lib/jackson-core-asl-1.9.4.jar:/home/sid/benchmarking/ycsb-0.17.0/lib/jackson-mapper-asl-1.9.4.jar:/home/sid/benchmarking/ycsb-0.17.0/elasticsearch-binding/lib/HdrHistogram-2.1.4.jar:/home/sid/benchmarking/ycsb-0.17.0/elasticsearch-binding/lib/commons-cli-1.3.1.jar:/home/sid/benchmarking/ycsb-0.17.0/elasticsearch-binding/lib/compress-lzf-1.0.2.jar:/home/sid/benchmarking/ycsb-0.17.0/elasticsearch-binding/lib/elasticsearch-2.4.0.jar:/home/sid/benchmarking/ycsb-0.17.0/elasticsearch-binding/lib/elasticsearch-binding-0.17.0.jar:/home/sid/benchmarking/ycsb-0.17.0/elasticsearch-binding/lib/guava-18.0.jar:/home/sid/benchmarking/ycsb-0.17.0/elasticsearch-binding/lib/hppc-0.7.1.jar:/home/sid/benchmarking/ycsb-0.17.0/elasticsearch-binding/lib/jackson-core-2.8.1.jar:/home/sid/benchmarking/ycsb-0.17.0/elasticsearch-binding/lib/jackson-dataformat-cbor-2.8.1.jar:/home/sid/benchmarking/ycsb-0.17.0/elasticsearch-binding/lib/jackson-dataformat-smile-2.8.1.jar:/home/sid/benchmarking/ycsb-0.17.0/elasticsearch-binding/lib/jackson-dataformat-yaml-2.8.1.jar:/home/sid/benchmarking/ycsb-0.17.0/elasticsearch-binding/lib/jna-4.1.0.jar:/home/sid/benchmarking/ycsb-0.17.0/elasticsearch-binding/lib/joda-convert-1.2.jar:/home/sid/benchmarking/ycsb-0.17.0/elasticsearch-binding/lib/joda-time-2.9.4.jar:/home/sid/benchmarking/ycsb-0.17.0/elasticsearch-binding/lib/jsr166e-1.1.0.jar:/home/sid/benchmarking/ycsb-0.17.0/elasticsearch-binding/lib/lucene-analyzers-common-5.5.2.jar:/home/sid/benchmarking/ycsb-0.17.0/elasticsearch-binding/lib/lucene-backward-codecs-5.5.2.jar:/home/sid/benchmarking/ycsb-0.17.0/elasticsearch-binding/lib/lucene-core-5.5.2.jar:/home/sid/benchmarking/ycsb-0.17.0/elasticsearch-binding/lib/lucene-grouping-5.5.2.jar:/home/sid/benchmarking/ycsb-0.17.0/elasticsearch-binding/lib/lucene-highlighter-5.5.2.jar:/home/sid/benchmarking/ycsb-0.17.0/elasticsearch-binding/lib/lucene-join-5.5.2.jar:/home/sid/benchmarking/ycsb-0.17.0/elasticsearch-binding/lib/lucene-memory-5.5.2.jar:/home/sid/benchmarking/ycsb-0.17.0/elasticsearch-binding/lib/lucene-misc-5.5.2.jar:/home/sid/benchmarking/ycsb-0.17.0/elasticsearch-binding/lib/lucene-queries-5.5.2.jar:/home/sid/benchmarking/ycsb-0.17.0/elasticsearch-binding/lib/lucene-queryparser-5.5.2.jar:/home/sid/benchmarking/ycsb-0.17.0/elasticsearch-binding/lib/lucene-sandbox-5.5.2.jar:/home/sid/benchmarking/ycsb-0.17.0/elasticsearch-binding/lib/lucene-spatial-5.5.2.jar:/home/sid/benchmarking/ycsb-0.17.0/elasticsearch-binding/lib/lucene-spatial3d-5.5.2.jar:/home/sid/benchmarking/ycsb-0.17.0/elasticsearch-binding/lib/lucene-suggest-5.5.2.jar:/home/sid/benchmarking/ycsb-0.17.0/elasticsearch-binding/lib/netty-3.10.6.Final.jar:/home/sid/benchmarking/ycsb-0.17.0/elasticsearch-binding/lib/securesm-1.0.jar:/home/sid/benchmarking/ycsb-0.17.0/elasticsearch-binding/lib/snakeyaml-1.15.jar:/home/sid/benchmarking/ycsb-0.17.0/elasticsearch-binding/lib/spatial4j-0.5.jar:/home/sid/benchmarking/ycsb-0.17.0/elasticsearch-binding/lib/t-digest-3.0.jar site.ycsb.Client -t -db site.ycsb.db.ElasticsearchClient -s -P workloads/workloada -p es.remote=true -p es.hosts.list=nosql-logging-cluster.es.uksouth.azure.elastic-cloud.com:9243 -p path.home=\~/benchmarking/ycsb-0.17.0/some-path -p cluster.name=nosql-logging-cluster
Command line: -t -db site.ycsb.db.ElasticsearchClient -s -P workloads/workloada -p es.remote=true -p es.hosts.list=nosql-logging-cluster.es.uksouth.azure.elastic-cloud.com:9243 -p path.home=~/benchmarking/ycsb-0.17.0/some-path -p cluster.name=nosql-logging-cluster
YCSB Client 0.17.0

Loading workload...
Starting test.
2022-08-10 09:03:39:547 0 sec: 0 operations; est completion in 0 second
Elasticsearch starting node = nosql-logging-cluster
Elasticsearch node path.home = ~/benchmarking/ycsb-0.17.0/some-path
Elasticsearch Remote Mode = true
Elasticsearch Remote Hosts = nosql-logging-cluster.es.uksouth.azure.elastic-cloud.com:9243
Aug 10, 2022 9:03:39 AM org.elasticsearch.plugins.PluginsService <init>
INFO: [Charles Xavier] modules [], plugins [], sites []
Aug 10, 2022 9:03:40 AM org.elasticsearch.client.transport.TransportClientNodesService$SniffNodesSampler$1$1 handleException
INFO: [Charles Xavier] failed to get local cluster state for {#transport#-1}{52.151.125.244}{52.151.125.244:9243}, disconnecting...
NodeDisconnectedException[[][52.151.125.244:9243][cluster:monitor/state] disconnected]

Exception in thread "Thread-3" NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{52.151.125.244}{52.151.125.244:9243}]]
        at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:290)
        at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:207)
        at org.elasticsearch.client.transport.support.TransportProxyClient.execute(TransportProxyClient.java:55)
        at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:288)
        at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359)
        at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:348)
        at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1221)
        at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.exists(AbstractClient.java:1241)
        at site.ycsb.db.ElasticsearchClient.init(ElasticsearchClient.java:149)
        at site.ycsb.DBWrapper.init(DBWrapper.java:86)
        at site.ycsb.ClientThread.run(ClientThread.java:91)
        at java.lang.Thread.run(Thread.java:750)
2022-08-10 09:03:40:445 0 sec: 0 operations; est completion in 106751991167300 days 15 hours
[OVERALL], RunTime(ms), 934
[OVERALL], Throughput(ops/sec), 0.0
[TOTAL_GCS_PS_Scavenge], Count, 4
[TOTAL_GC_TIME_PS_Scavenge], Time(ms), 23
[TOTAL_GC_TIME_%_PS_Scavenge], Time(%), 2.462526766595289
[TOTAL_GCS_PS_MarkSweep], Count, 0
[TOTAL_GC_TIME_PS_MarkSweep], Time(ms), 0
[TOTAL_GC_TIME_%_PS_MarkSweep], Time(%), 0.0
[TOTAL_GCs], Count, 4
[TOTAL_GC_TIME], Time(ms), 23
[TOTAL_GC_TIME_%], Time(%), 2.462526766595289

I tried with other ports as well (9200, 9300)- just to make sure - but it still gives the same exception. Could you please look into this?

Thank you

Sid

Daniel Seybold · Answer 1 · Wed Aug 10 2022 20:15:32 GMT+0800 (China Standard Time)

Hi Sid,

while the error message indicates that no node is reachable this sounds like the your Azure deployment is not reachable from the instance you are running YCSB.

But since you verified this already via postman, there is also the possibility of using an unsupported ElasticSearch version.

Which version of ElasticSearch are you running?

Currently, the YCSB only support 6.x but there is an open pull request that add support for 7.x, see #1575

Sidharth Ramesh · Answer 2 · Wed Aug 10 2022 20:18:24 GMT+0800 (China Standard Time)

Hi Daniel,

I'm using Elasticsearch version 8.3.

Regards,
Sid

Daniel Seybold · Answer 3 · Wed Aug 10 2022 20:21:07 GMT+0800 (China Standard Time)

Hi Sid,

unfortunately this version is not yet supported with 0.17.0 and the current master branch.

Sidharth Ramesh · Answer 4 · Wed Aug 10 2022 20:23:14 GMT+0800 (China Standard Time)

Hi Daniel,

Okay. Can a feature request be created to add support for the latest version (8.x +) ?

Sean Busbey · Answer 5 · Tue Aug 23 2022 22:43:30 GMT+0800 (China Standard Time)

we're a volunteer project. if you'd like to see support done, then please a) check for an outstanding PR that provides it and test the change out or b) if there isn't one, make a PR that adds it.