"Exception: Java gateway process exited before sending its port number" when using pyspark and sagemaker feature store

[Amit Arora]

Following instructions provided here https://docs.aws.amazon.com/sagemaker/latest/dg/batch-ingestion-spark-connector-setup.html, getting this exception:
"Exception: Java gateway process exited before sending its port number"
when running the following code

from pyspark.sql import SparkSession
from feature_store_pyspark.FeatureStoreManager import FeatureStoreManager
import feature_store_pyspark

extra_jars = ",".join(feature_store_pyspark.classpath_jars())
spark = SparkSession.builder \
.config("spark.jars", extra_jars) \
.getOrCreate()

Steps followed:

1. Create a new SageMaker Notebook (not SageMaker Studio, just regular SageMaker notebook).
2. Create a new notebook using the conda_python3 kernel.
3. Install the packages as provided in the instructions here https://docs.aws.amazon.com/sagemaker/latest/dg/batch-ingestion-spark-connector-setup.html. Specifically, copy paste the following in a notebook cell and run.

import os
    
original_spark_version = "2.4.0"
os.environ['SPARK_HOME'] = '/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/pyspark'
    
# Install a newer versiion of Spark which is compatible with spark library
!pip3 install pyspark==3.1.1
!pip3 install sagemaker-feature-store-pyspark --no-binary :all:

See screenshots


.

I created a custom conda environment (not on this notebook, on a completely different sagemaker notebook) with Python3.7. Then install pyspark, sagemaker and friends, did not get this error but then started getting other errors related to missing Java packages. Not putting that here to avoid any confusion, just mentioning it as a data point that part of this problem could be that the built in Python3 conda environment is Python3.6, maybe it needs to be Python3.7. There is a conda_mxnet_latest_python3.7 environment also but then that runs into other different problems.

[cansun]

The code is broken because SageMaker notebook upgraded their python version from 3.6 to 3.8. Thus the spark home changed. Running spark with an incorrect spark home resulted in the exception above.

For notebook users, it is not necessary to specify the SPARK_HOME env and the path may change depending on notebook's python version upgrades.

I resolved this issue by simply commenting out

os.environ['SPARK_HOME'] = '/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/pyspark'

We should update the documentation besides callout the specific kernel customer should use which is compatible with connector.

[parsash2]

Hi there!

Thank you for taking the time to let us know about the issue! I've used your feedback to update the documentation.

Have a great day!