The assignment can be completed either locally on your own machine (natively on macOS / Ubuntu or inside Docker container) or in the cloud leveraging GitHub Codespaces. Accordingly, either follow the instructions in Section 1, Section 2, or Section 3. Important: A GPU-enabled environment/workstation is not required for this assignment.
The code should run natively on Ubuntu (preferred), macOS, and Windows. A native installation is the most resource efficient option and allows you to leverage the full power of your machine. However, it requires you to install all dependencies on your machine. Therefore, the setup is a bit more involved than with (some of) the other options.
Our codebase relies heavily on JAX, which natively supports Linux x86_64 (with NVIDIA GPU support), macOS hosts (both x86_64 / Intel & arm64 / Apple Silicon, no GPU support), and Windows (no GPU support on host, experimental support for NVIDIA GPUs on WSL2).
Furthermore, you can either use our provided scripts to install the required dependencies in a Conda environment (Option 1), or install all packages manually (Option 2). We strongly recommend to use Conda, as this will allow you to easily select the desired Python version and prevent any version conflicts.
JAX is now supported on Windows. Therefore, this codebase should also run on Windows. However, all the development and testing of this practical assignment was done on Ubuntu. Accordingly, we cannot guarantee that everything will work as expected on Windows and can provide only limited support for Windows users. We recommend to setup a dual boot installation with Windows & Ubuntu. Alternatively, also Linux Subsystem for Windows can be used.
Please download the code template from the GitHub repository by clicking on Code -> Download ZIP or by cloning the repository using git:
git clone https://github.com/tud-phi/ics-pa-svWe primarily support the installation of the required dependencies using Conda. Please first install the latest version of Conda or Miniconda using the instructions on the Conda website.
Afterwards, please open a terminal and navigate to the top level directory of the repository using the cd command. Then, please follow the instructions below. Important note for Windows users: You might want to use the Anaconda Prompt instead of the Windows Command Prompt to make sure that the conda command is available.
Then, run our bash script to create the new Conda environment ics with all required dependencies. On Ubuntu / macOS, you can do so by running:
./00-conda-setup.shIf you encounter permission issues, please run chmod +x /00-conda-setup.sh to give executable permissions to the
bash script.
On Windows, please run (in the Anaconda Prompt):
.\00-conda-setup.batSubsequently, we need to activate the Conda environment ics and add the assignment folder to the PYTHONPATH environment variable such that we can import Python modules from the assignment folder.
On Ubuntu / macOS, you can do so by running:
conda activate ics && ./02-add-to-pythonpath.shOn Windows, please run (in the Anaconda Prompt):
conda activate ics && .\02-add-to-pythonpath.batIf you are running on Ubuntu and want to leverage a NVIDIA GPU for increasing the LNN training speed in Problem 2c, you need to install JAX with GPU support. Please note that we will not offer any support for installing JAX with GPU support and it is totally up to your discretion to follow this installation step as the assignment can also be completed solely using the CPU.
Please run in the ics Conda environment:
pip install -U "jax[cuda12_pip]" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.htmlIn case you are encountering issues with the installation or if JAX does not find your GPU, please refer to the JAX README.
We only support a manual installation of the required dependencies on Ubuntu. Please follow the instructions below to install all required dependencies manually. This framework requires Python 3.10/3.11. For example, Ubuntu 22.04 ships with Python 3.10 and is therefore a suitable choice. Please note that some required dependencies (for example PyTorch) might not be updated yet to work with Python 3.12 or newer.
FFmpeg is required to create Matplotlib animations and save them as .mp4 video files. Please follow installation instructions online such as this one. On Ubuntu, the package can be easily installed via:
sudo apt update && apt install -y ffmpegNode.js is required to install JupyterLab extensions for interactive plots. On Ubuntu, the package can be easily installed via:
sudo apt update && apt install -y nodejsYou can install the jax_double_pendulum package and all other necessary pip dependencies such as JAX, PyTorch, Matplotlib etc., by running the following command in the top level directory of the repository:
pip3 install .If you encounter any issues with the installation of JAX, it is recommended to follow the installation instructions in the JAX repository.
As we import some Python modules from folders outside a package, we need to add the assignment folder to the PYTHONPATH environment variable.
./02-add-to-pythonpath.shPlease don't forget, if applicable, to activate the Conda environment before running any scripts or notebooks.
As mentioned in the assignment description, most of the code needs to be completed within the interactive Jupyter notebooks. You can use the following command to start a Jupyter notebook server in the top level directory of the repository:
jupyter notebookA codespace is a development environment that's hosted in the cloud. Different instance types ranging from 2 CPU cores, 4 GB of RAM, and 32 GB of storage and up to 16 CPU cores, 32 GB of RAM and 128 GB of storage are available.
Important: 180 core-hours of GitHub Codespaces usage per month are included for free in GitHub Pro, which is offered as part of the GitHub Student Developer Pack. If you haven't already, please register for the pack using your TU Delft email address to get access to this free usage quota. This will be sufficient for 90 hours per month of continuous usage with the smallest 2-core instance type. Make sure to stop your Codespaces instance when you are not using it to avoid unnecessary usage of your quota. Also, if you have used up the free quota, you will need to pay for any additional usage (not covered by this course).
We strongly advise you to carefully study Section 2.5 to prevent any loss of code.
As you are studying this README, you probably know that the code template is available on GitHub in the
tud-phi/ics-pa-sv repository.
Please click on Use this template and then Create new repository to create a new repository for the
assignment solution in your own personal GitHub account. Please make sure to make the repository private.
Then, please open the new repository in a GitHub Codespaces instance by clicking on Code -> Open with Codespaces.
No worries, all dependencies are automatically installed in the GitHub Codespaces environment. You should be able to start working right away.
The python executable is symlinked to /usr/local/bin/python.
If you are encountering issues with missing Python dependencies or you observe a jupyter: command not found error when executing the ./10-start-notebook-as-student.sh bash script, please run the following command in the GitHub Codespaces terminal:
./01-pip-install.shAfterwards, the issues should be resolved.
You can open the Jupyter notebooks in the editor and then use the integrated Jupyter notebook extension to execute them. If you are prompted to select a kernel, please choose the kernel Python 3.11.x /usr/local/bin/python.
Alternatively, you can also start a Jupyter notebook server in the VS Code terminal, for which port-forwarding should be configured automatically:
./10-start-notebook-as-student.shComplimentary to saving your code on the Codespaces instance, you will also want to push the code changes to your GitHub repository, so that your code is not lost when the instance is deleted. We refer to the internet for comprehensive guides on git. In the following, we will only point out the basic usage of pushing code from Codespaces to the main branch of the GitHub repository:
- Click on Source Control in the left sidebar.
- Here, you can see all files you have modified / added / deleted. You can add / stage changes by clicking on the
+symbol right of the filename. - You can commit the Staged Changes by writing a concise message descriping the changes into the text box and then clicking on Commit.
- Click on Sync Changes or Push to mirror the local commits to the remote GitHub repository.
A standard 2-core instance of GitHub Codespaces has a memory limit of 4 GB.
This memory limit can be exhausted quite quickly, in particular when running neural network trainings. You can monitor the current memory usage of your instance by running htop in the terminal. When the instance runs out of memory, this usually results in a Jupyter notebook kernel crash. You would for example see the following or a very similar error message:
The Kernel crashed while executing code in the the current cell or a previous cell. Please review the code in the cell(s) to identify a possible cause of the failure.
When you are running low on memory, please stop and close any Jupyter notebooks you might not be using at the moment. This will free some memory.
If you keep encountering kernel crashes, we recommend upgrading to a larger GitHub Codespaces instance. One option is the 4-core instance type with 8 GB of memory. The various instance options are documented here. Please note that in the GitHub Student Developer Pack, you have 180 core-hours of Codespaces per month included for free. For a 4-core instance, this evaluates to 45 hours of continuous usage of the instance per month. To change your instance to an 4-core instance, please click on Code -> Codespaces -> On current branch -> ... -> Change machine type. Then, select the 4-core instance type.
There are two ways to work with Matplotlib plots within standard Python scripts (i.e. not Jupyter notebooks):
- Add the following line to the top of your Python script, which allows you to run the script interactively in VS Code.
# %%
# Now you can add the rest of your code here
import matplotlib.pyplot as plt
plt.figure()
plt.show()Then you can run the script interactively by clicking on the Run Cell button in the top left corner of the code cell.
- Instead of showing the Matplotlib code, you can also just save the plot to a file and open it in a new tab in VS Code.
import matplotlib.pyplot as plt
plt.figure()
plt.savefig("my_plot.png")Visual Studio Code Dev Containers allow you to open the repository in a containerized development environment. It relies on Docker to virtualize a pre-configured Ubuntu system with all necessary dependencies installed. This allows you to execute code also on a natively unsupported host operating system (e.g. Windows) or to avoid installing all dependencies on your host system. Please note that we are currently not providing support for the Dev Container setup.
In the following, we will describe the basic setup and usage of the Dev Container. You find a more comprehensive guide here. Please also consult the Docker documentation for more information on Docker.
Please install Docker Desktop on Windows / Mac / Ubuntu. If you are one Windows, please make sure to install and enable the WSL 2 backend in the Docker Desktop settings.
Docker runs in a Virtual Machine (VM) while limiting the available resources such as CPU cores, memory, and storage, which might result in insufficient resources for executing the code in this repository. Accordingly, you might need to allocate more resources to the Docker VM than provided by default.
We recommend at least 4 CPUs, 6 GB of memory, and 32 GB of disk space. However, allocating too many resources to the VM might slow down / freeze your host system.
If you have followed the provided indications and configured Docker to use WSL2, you will find out that you have no access to the advanced settings in Docker that allow you to limit the RAM and number of CPUs of your system (WSL does not provide a means to set the disk space). To change the resources of your containers when Docker is using WSL as backend, you need to make use of a .wslconfig file. To do so, you need to take the following steps (you require Visual Studio Code installed, see section 3.3 of this manual).
- In Visual Studio Code select in the top toolbar File -> New File... and name it
.wslconfig. VS Code will prompt you to indicate the location to which you have to save the file. The file should be located under your user folder:
C:\Users\<your_user_name>\.wslconfig
-
Once the file is open in VS Code, you need to make sure that the endline character of the file is LF. You can do that in the blue Status Bar at the bottom of the VS Code window. Towards the right there should be a piece of text indicating CRLF or LF. If the text displayed is LF then you don't need to perform any further action. If the text displayed is CRLF, then click on it and select LF in the command palette at the top of the window.
-
Copy and paste the following text into your
.wslconfigfile. Notice that if the computational resources of your computer are limited you may need to reduce some of these values (e.g. if you only have 4GB of RAM, you would need to reduce the memory value, although that can result in issues while running the code). If you have a more powerful computer, feel free to increase the values without forgetting to leave some resources for your host system.
[wsl2]
memory=6GB
processors=4
-
Save the changes (Ctrl + S) and close the window. If you have already installed Docker you will need to restart your computer for the changes to be effective.
-
You can verify that the changes are taking effect by running the command below. The value displayed directly below total should match the value you have assigned to
memoryin your.wslconfigfile.
free --gigaPlease open Docker Desktop and go to Settings -> Resources -> Advanced and modify the resources as described as in the Docker Desktop macOS and Linux manuals.
Please install Visual Studio Code and subsequently also the Dev Container extension. With the Dev Containers extension installed, you will see a new green status bar item at the far bottom left.
As you are studying this README, you probably know that the code template is available on GitHub in the
tud-phi/ics-pa-sv repository.
Please click on Use this template and then Create new repository to create a new repository for the
assignment solution in your own personal GitHub account. Please make sure to make the repository private.
Please clone the repository you have just created on your own GitHub account to your local machine. For example, you can do so by opening a new window in VS Code, then clicking on Source Control in the left sidebar, and then on Clone Repository.
Now, please open the folder with the cloned repository in Visual Studio Code. You will see a green status bar item at the far bottom left. Click on it and select Reopen in Container.
Alternatively, you might also be automatically prompted to reopen the folder in a container. This will use the settings in the .devcontainer folder to build a new Dev container and then open a new VS Code window with the repository opened in the containerized environment.
After VS Code has been openend in the container, you will see a terminal window at the bottom of the screen. The process of installing Python dependencies should be started automatically. If not, please run the following command in the terminal:
./01-pip-install.shYou can open the Jupyter notebooks in the editor and then use the integrated Jupyter notebook extension to execute them. If you are prompted to select a kernel, please choose the kernel Python 3.11.x /usr/local/bin/python.
Alternatively, you can also start a Jupyter notebook server in the VS Code terminal, for which port-forwarding should be configured automatically:
./10-start-notebook-as-student.shWith respect to backing up your code, the same principle of GitHub Codespaces also applies to Dev Containers: Complimentary to saving your code on the Codespaces instance, you will also want to push the code changes to your GitHub repository, so that your code is not lost when the Dev container is (accidentally) deleted. We refer to the internet for comprehensive guides on git. In the following, we will only point out the basic usage of pushing code from Codespaces to the main branch of the GitHub repository:
- Click on Source Control in the left sidebar.
- Here, you can see all files you have modified / added / deleted. You can add / stage changes by clicking on the
+symbol right of the filename. - You can commit the Staged Changes by writing a concise message descriping the changes into the text box and then clicking on Commit.
- Click on Sync Changes or Push to mirror the local commits to the remote GitHub repository.
When changing the content of functions implemented in a Jupyter notebook and used in other notebooks, it might (sometimes) be necessary to save all notebooks and then restarting the notebook kernel(s). This procedure will allow the function in all notebooks relying on it to be re-loaded.
Zou are able to validate the syntax of your code, the removal of all NotImplementedError exceptions, and the
passing of all public tests.
On Ubuntu / macOS, you can do so by running:
./20-validate-assignment.shOn Windows, please run (in the Anaconda Prompt):
.\20-validate-assignment.batFirst, make sure to double check the following:
- Your contact information is filled in the corresponding cells at the top of the notebooks. Please do so as well for partly completed and incomplete notebooks.
- Make sure that you have NOT INSERTED NOR DELETED any cells in any of the notebooks, to avoid potential issues with the autograding.
- Make sure that your notebooks run top-to-bottom (or as far as you have implemented) after restarting the kernel. The provided script
20-validate-assignment.sh/20-validate-assignment.batwill do this for you (see Section 4.2).
Please create a zip archive of the entire repository folder. The name of the submitted ZIP file should be in the format studentNumber_FirstName_LastName.zip to facilitate the identification of the owner of the code. Please, stick to that convention by:
- Omitting middle names
- Omitting accents (e.g. Rubén --> Ruben)
- Merging compound first/last names using CamelCase style (e.g 1234567_Ruben_MartinRodriguez.zip or 1112233_Name_vanDenSurname.zip)
In order to make sure that we receive all necessary metadata and that the structure of the archive is as expected, please use the provided script for this:
python ./25-create-submission-archive.pyAfterwards, you will find the zip archive on the top level of the repository folder. Please upload this zip archive to Brightspace.