PyCOMPSs + Jupyter Tutorial Notebooks

Introduction

This repository contains all PyCOMPSs related tutorial notebooks.

It is divided into three main folders:

syntax: Contains the main tutorial notebooks. They cover the syntax and main functionalities of PyCOMPSs.
hands-on: Contains example applications and hands-on exercises.
demos: Contains demonstration notebooks.

PyCOMPSs + Jupyter Tutorial Notebooks

Prerequisites

This notebooks can be used with the pycompss-cli (recommended for Linux and Mac-OS), Virtual Machine provided by BSC (recommended for Windows). Alternatively, they can also been used locally.

Using pycompss-cli:
- python3
- pycompss-cli
Using Docker:
- Docker
- Git
Using Virtual Machine:
- VirtualBox
For local execution:
- Git
- Python 3
- COMPSs installation
- Jupyter (with the desired ipykernel)
- ipywidgets (only for some hands-on notebooks)
- numpy (only for some notebooks)
- dislib (only for some notebooks)

Instructions

Using pycompss-cli:
- Install docker:
  - Linux: apt-get install docker (depends on your distribution)
  - Mac-os: direct download from docker.com. You can find instructions here: https://docs.docker.com/docker-for-mac/install/
- Get COMPSs docker image:
```
docker pull compss/compss-tutorial:3.1
```
- Install pycompss-cli:
  - Linux:
```
sudo python3 -m pip install pycompss-cli
# or
python3 -m pip install pycompss-cli --user
```
  - Mac-os:
```
pip install pycompss-cli
```
- Get tutorials:
```
git clone https://github.com/bsc-wdc/tutorial_apps.git
git clone https://github.com/bsc-wdc/notebooks.git
```
  Note: If the docker pull command fails be sure you have internet connection, the Docker service is running (sudo service docker start) and your user is in the docker group (sudo usermod -aG docker $USER)
- Use pycompss-cli:
  - Initialize a COMPSs cluster:
```
pycompss init -n compss-tutorial docker –i compss/compss-tutorial:3.1
```
  - Start Jupyter-notebook:
```
pycompss jupyter
```
  - Open a web browser with the address: http://localhost:8888 or http://172.17.0.2:8888
  - Check the pycompss-cli documentation for more information.
- Stop pycompss-cli after playing with PyCOMPSs:
```
pycompss env remove compss-tutorial
```
Using Virtual Machine (recommended for Windows):
- Download the OVA from: http://compss.bsc.es/releases/vms/COMPSs-latest-VM.ova
- Import the OVA from VirtualBox
- Start the Virtual Machine
  - User: compss
  - Password: compss2021
- Get COMPSs docker image:
```
docker pull compss/compss-tutorial:3.1
```
  Note: If the docker pull command fails be sure you have internet connection, the Docker service is running (sudo service docker start) and your user is in the docker group (sudo usermod -aG docker $USER)
- Get tutorials:
```
git clone https://github.com/bsc-wdc/tutorial_apps.git
git clone https://github.com/bsc-wdc/notebooks.git
```
- Use pycompss-cli:
  - Initialize a COMPSs cluster:
```
pycompss init -n compss-tutorial docker –i compss/compss-tutorial:3.1
```
  - Start Jupyter-notebook:
```
pycompss jupyter
```
  - Open a web browser with the address: http://localhost:8888 or http://172.17.0.2:8888
  - Check the pycompss-cli documentation for more information.
- Stop pycompss-cli after playing with PyCOMPSs:
```
pycompss env remove compss-tutorial
```

Using Docker:

Run in your machine:

git clone https://github.com/bsc-wdc/notebooks.git
docker pull compss/compss-tutorial:3.1
# Update the path to the notebooks path in the next command before running it
docker run --name mycompss -p 8888:8888 -p 8080:8080 -v /PATH/TO/notebooks:/home/notebooks -itd compss/compss-tutorial:3.1
docker exec -it mycompss /bin/bash

Now that docker is running and you are connected:

cd /home/notebooks
/etc/init.d/compss-monitor start
jupyter-notebook --no-browser --allow-root --ip=172.17.0.2 --NotebookApp.token=

From local web browser:
- Open COMPSs monitor: http://localhost:8080/compss-monitor/index.zul
- Open Jupyter notebook interface: http://localhost:8888/

Local Execution instructions

Install all requirements described in Prerequisites setion.

Usage:

git clone https://github.com/bsc-wdc/notebooks.git
cd notebooks
/etc/init.d/compss-monitor start
jupyter-notebook

Then:
- Open COMPSs monitor: http://localhost:8080/compss-monitor/index.zul
- Open Jupyter notebook interface: http://localhost:8888/
  - Look for the application.ipynb of interest.

Important Notes

Using COMPSs versions < 2.8: It is necessary to RESTART the python kernel from Jupyter after the execution of any notebook.

Troubleshooting

ISSUE 1: Cannot connect using docker pull.

REASON: The docker service is not running:

# Error messsage:
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
# SOLUTION: Restart the docker service:
sudo service docker start

ISSUE 2: The notebooks folder is empty or contains other data using docker.

REASON: The notebooks path in the docker run command is wrong.

# Remove the docker instance and reinstantiate with the appropriate notebooks path
exit
docker stop mycompss
docker rm mycompss
# Pay attention and UPDATE: /PATH/TO in the next command
docker run --name mycompss -p 8888:8888 -p 8080:8080 -v /PATH/TO/notebooks:/home/notebooks -itd compss/compss-tutorial:2.8
# Continue as normal

ISSUE 3: COMPSs does not start in Jupyter.

REASON: There are COMPSs' python/java processes running

# SOLUTION: Restart the python kernel from Jupyter and check that there are no COMPSs' python/java processes running.

ISSUE 4: Numba is not working with the VM or Docker.

REASON: Numba is not installed in the VM or docker

# SOLUTION: Install Numba in the VM/Docker
#           Open a console in the VM/Docker and follow the next steps.
# For Python 2:
sudo python2 -m pip install numba
# or
python2 -m pip install numba --user
# For Python 3:
sudo python3 -m pip install numba
# or
python3 -m pip install numba --user

ISSUE 5: Matplotlib is not working with the VM or Docker.

REASON: Matplotlib is not installed in the VM or docker

# SOLUTION: Install Matplotlib in the VM/Docker
#           Open a console in the VM/Docker and follow the next steps.
# For Python 2:
sudo python2 -m pip install matplotlib
# or
python2 -m pip install matplotlib --user
# For Python 3:
sudo python3 -m pip install matplotlib
# or
python3 -m pip install matplotlib --user

ISSUE 6: Can not install dislib. Raises an exception with dependency package.

REASON: enum34 is present in the docker instance

# SOLUTION: Uninstall enum34 from the docker instance
#           Open a console in the Docker and follow the next steps.
# For Python 3:
sudo python3 -m pip uninstall -i enum34

Contact

support-compss@bsc.es

bsc-wdc / notebooks