spark.ui.enabled configuration ignored

[amin-nejad]

Disabling the spark UI with a configuration option using spark.ui.enabled (as documented here) doesn't appear to be respected. e.g.

from pyspark import SparkConf, SparkContext

conf = SparkConf()
conf.set("spark.executor.memory", "2g")
conf.set("spark.ui.enabled", "false") # or conf.set("spark.ui.enabled", False)
SparkContext()

import databricks.koalas as ks

I have tried both a string value and a boolean value. I still get this warning when I run koalas with or without that option:

21/06/29 00:05:58 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.

(or indeed if I try setting it to "true"/True)

[Hyukjin Kwon]

In which env did you run the codes above? It's likely that SparkContext is already initialized before you running these codes.

[amin-nejad]

I think I forgot the conf=conf in SparkContext... my bad