Dataframe TTL doesn't include schema keys
folfix opened this issue · comments
folfix commented
Hi!
I've noticed that following configuration:
df.write()
.format("org.apache.spark.sql.redis")
.option("table", "example")
.option("ttl", 60)
.save();
allows me to keep data in Redis for 60 seconds. It works as expected. But I've observed that schema key (_spark:example:schema
) is being kept. This causes a problem when reading: I can't tell if data is in cache or not. Because, following:
Dataset<Row> loadedDataset = spark.getSession().read()
.format("org.apache.spark.sql.redis")
.option("table", "example")
.load();
responds with valid Dataset (none exception thrown).
Is there any way to propagate TTL to schema keys as well? Thanks!
Oleksii Diagiliev commented
Hi @folfix,
the TTL option affects rows only and not table schema. Can you check that returned dataframe is empty?
M Sazzadul Hoque commented
Closed due to inactivity.