Giters

big-data-europe / docker-spark

Apache Spark docker image

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Inconsistencies between running containers on macOS and Linux

dusandjovanovic opened this issue · comments

commented

Hello all,

I don't have a specific issue to report since I managed to bring up the needed containers in the end. However, I did notice there are inconsistencies between running these on Linux and macOS.

On Linux, everything went as expected. My infrastructure containers were stable and submitting a pyspark application was a breeze. On macOs, it's a whole different story since resourcemanager deamons would occasionally go down. More often, I would not manage to execute my application. Every time I tried submitting the application, eventually the process would end being killed - even in the case of simplest transformations like .select(). Did anyone else encounter issues like this? If so how did you overcome them?

BDE images: tried many, both for Hadoop 2.x and 3.x
macOS version: 10.15.7
Docker engine version: 20.10.6

image

commented

HI @dusandjovanovic ,

thanks a lot for reporting this. Unfortunately, I can't really reproduce it as I'm on Linux machine and thus can't reproduce it. Have you thought of using WSL as a solution to run/use Linux images within WIndows via Docker Desktop instead of using just Docker Desktop on Windows? I may not be able to test it out to run them fully on Windows as I didn't really test them out.

Feel free to let us know if you did resolve this issue.

Best regards,

Update:: Apologies, you were referring macOS and I was thinking of Windows :( -- docker desktop got confused there. That is indeed strange why it isn't working on macOS. Maybe others who have mac notebooks can try it out and confirm this.

commented

@GezimSejdiu I forgot to write a follow-up for this.

In the end, it turned out that my Docker resource allocation on macOS was faulted. The containers were lacking resources (cores and memory) and were crashing unexpectedly. Everything went smoothly after updating the Docker engine and sorting out my local issues.

Thereby, this issue can be closed and marked as resolved since there was never a container-related issue to begin with.