TaskSchedulerImpl: Initial job has not accepted any resources

[Shanmugathas Vigneswaran]

Hi all,

I create cluster based on k8s yaml file.
Spark ui master, worker are good.

screencapture-spark-dev-spark-dev-10-133-245-111-xip-io-2019-10-04-10_58_39.pdf

I aslo created an ubuntu pod for testing Spark cluster inside k8s cluster.
Precision : spark master + ds => namespace = spark-dev
Ubuntu pod => namespace = default

Using Ipython + Pyspark:

`
from pyspark.sql import SparkSession
cluster = '10.42.11.124:7077'
spark = SparkSession
.builder
.master('spark://'+cluster)
.appName('k8s-test-mini-confg')
.config('spark.cores.max', '1')
.config('spark.executor.memory', '1g')
.config('spark.executor.instances', '1')
.getOrCreate()

lines = spark._sc.textFile('out.txt')
pairs = lines.map(lambda s: (s, 1))
counts = pairs.reduceByKey(lambda a, b: a + b)
counts.collect()
`

I get :
19/10/04 09:00:40 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
19/10/04 09:00:55 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
19/10/04 09:01:10 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

Job is not executed, still freeze.

Did someone faced on same issue ?

[Giannis Mouchakis]

Hi @shanisma.
Is the "Ubuntu pod" reachable from the workers?
Can you please provide the pod definition you used for the Pod so we can try to reproduce this?

[Shanmugathas Vigneswaran]

Hi @gmouchakis

I added in metadata.label => app: spark-client

metadata:
namespace: spark-dev
name: spark-submit-rest-api
labels:
app: spark-client

It's working fine :)