redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool

Question

redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool

justinTM opened this issue 2 years ago · comments

I can't connect to an AWS MemoryDB cluster using SSL; I get an exception: redis.clients.jedis.exceptions.JedisConnectionException.

Using the TablePlus app on macOS, I was able to connect with the same params as in DataBricks (using this library):

user=admin
auth=*****
host=******
port=****
ssl=true

But when executing on DataBricks cluster:

redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
---------------------------------------------------------------------------
Py4JJavaError                             Traceback (most recent call last)
<command-744056925752582> in <module>
      7     .option("spark.redis.auth", dev_redis_auth)
      8 #     .option("spark.redis.db", 1)
----> 9     .option("spark.redis.ssl", "true")
     10     .save()
     11 )

/databricks/spark/python/pyspark/sql/readwriter.py in save(self, path, format, mode, partitionBy, **options)
    735             self.format(format)
    736         if path is None:
--> 737             self._jwrite.save()
    738         else:
    739             self._jwrite.save(path)

/databricks/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py in __call__(self, *args)
   1255         answer = self.gateway_client.send_command(command)
   1256         return_value = get_return_value(
-> 1257             answer, self.gateway_client, self.target_id, self.name)
   1258 
   1259         for temp_arg in temp_args:

/databricks/spark/python/pyspark/sql/utils.py in deco(*a, **kw)
     61     def deco(*a, **kw):
     62         try:
---> 63             return f(*a, **kw)
     64         except py4j.protocol.Py4JJavaError as e:
     65             s = e.java_exception.toString()

/databricks/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
    326                 raise Py4JJavaError(
    327                     "An error occurred while calling {0}{1}{2}.\n".
--> 328                     format(target_id, ".", name), value)
    329             else:
    330                 raise Py4JError(

Py4JJavaError: An error occurred while calling o1111.save.
: redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
	at redis.clients.jedis.util.Pool.getResource(Pool.java:84)
	at redis.clients.jedis.JedisPool.getResource(JedisPool.java:377)
	at com.redislabs.provider.redis.ConnectionPool$.connect(ConnectionPool.scala:35)
	at com.redislabs.provider.redis.RedisEndpoint.connect(RedisConfig.scala:75)
	at com.redislabs.provider.redis.RedisConfig.clusterEnabled(RedisConfig.scala:199)
	at com.redislabs.provider.redis.RedisConfig.getNodes(RedisConfig.scala:339)
	at com.redislabs.provider.redis.RedisConfig.getHosts(RedisConfig.scala:239)
	at com.redislabs.provider.redis.RedisConfig.<init>(RedisConfig.scala:138)
	at org.apache.spark.sql.redis.RedisSourceRelation.<init>(RedisSourceRelation.scala:34)
	at org.apache.spark.sql.redis.DefaultSource.createRelation(DefaultSource.scala:21)
	at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
	at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
	at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
	at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:86)
	at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:152)
	at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:140)
	at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$5.apply(SparkPlan.scala:193)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
	at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:189)
	at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:140)
	at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:117)
	at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:115)
	at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:711)
	at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:711)
	at org.apache.spark.sql.execution.SQLExecution$$anonfun$withCustomExecutionEnv$1.apply(SQLExecution.scala:113)
	at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:243)
	at org.apache.spark.sql.execution.SQLExecution$.withCustomExecutionEnv(SQLExecution.scala:99)
	at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:173)
	at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:711)
	at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:307)
	at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:293)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:380)
	at py4j.Gateway.invoke(Gateway.java:295)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:251)
	at java.lang.Thread.run(Thread.java:748)
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: Failed to create socket.
	at redis.clients.jedis.DefaultJedisSocketFactory.createSocket(DefaultJedisSocketFactory.java:110)
	at redis.clients.jedis.Connection.connect(Connection.java:226)
	at redis.clients.jedis.BinaryClient.connect(BinaryClient.java:144)
	at redis.clients.jedis.BinaryJedis.connect(BinaryJedis.java:314)
	at redis.clients.jedis.BinaryJedis.initializeFromClientConfig(BinaryJedis.java:92)
	at redis.clients.jedis.BinaryJedis.<init>(BinaryJedis.java:297)
	at redis.clients.jedis.Jedis.<init>(Jedis.java:169)
	at redis.clients.jedis.JedisFactory.makeObject(JedisFactory.java:177)
	at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:571)
	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:298)
	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:223)
	at redis.clients.jedis.util.Pool.getResource(Pool.java:75)
	... 41 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:607)
	at redis.clients.jedis.DefaultJedisSocketFactory.createSocket(DefaultJedisSocketFactory.java:80)
	... 52 more

Any ideas?

Oleksii Diagiliev · Answer 1 · Wed Jun 15 2022 05:36:27 GMT+0800 (China Standard Time)

Hi @justinTM ,
could you try it with spark-redis 3.1.0? Also, could you try to specify connection parameters when creating Spark session rather than as dataframe save options?

Tse Kit Yam · Answer 2 · Fri Jul 08 2022 16:36:50 GMT+0800 (China Standard Time)

I am also experiencing the same issue.

When I do

import redis

r = redis.StrictRedis(host=redis_host, port=int(redis_port), ssl=True, decode_responses=True)

print(r.ping())

I can see True being printed.

However, I got Caused by: java.net.ConnectException: Connection refused (Connection refused) when I do

loadedDf = (
  spark
  .read
  .format("org.apache.spark.sql.redis")
  .option("spark.redis.host", redis_host)
  .option("spark.redis.port", redis_port)
  .option("spark.redis.ssl", "true")
  .load()
)
loadedDf.show()

I also try

from pyspark.sql import SparkSession

another_spark = (
  SparkSession
  .builder
  .config("spark.redis.host", redis_host)
  .config("spark.redis.port", redis_port)
  .config("spark.redis.ssl", "true")
  .getOrCreate()
)

loadedDf = (
  another_spark
  .read
  .format("org.apache.spark.sql.redis")
  .load()
)
loadedDf.show()

However, the error are still the same

P.S. I am using 3.1.0

Tse Kit Yam · Answer 3 · Fri Jul 08 2022 16:57:05 GMT+0800 (China Standard Time)

I can connect to the server with the following code

loadedDf = (
  spark
  .read
  .format("org.apache.spark.sql.redis")
  .option("host", redis_host)
  .option("port", redis_port)
  .option("ssl", "true")
  .option("infer.schema", True)
  .load()
)
loadedDf.show()

JDRanpariya · Answer 4 · Thu Sep 15 2022 15:25:25 GMT+0800 (China Standard Time)

Same Exception using master branch of spark-redis and spark 3.3.0

sql( "create table if not exists leads( id int, first_name string, last_name string, email string, gender string, city string, country string) using org.apache.spark.sql.redis options (table 'leads')" )