SIL NLP

SIL NLP provides a set of pipelines for performing experiments on various NLP tasks with a focus on resource-poor and minority languages.

Supported Pipelines

Neural Machine Translation
Statistical Machine Translation
Word Alignment

SILNLP Prerequisites

These are the main requirements for the SILNLP code to run on a local machine. Since there are many Python packages that need to be used with complex versioning requirements, we use a Python package called Poetry to mangage all of those. So here is a rough heirarchy of SILNLP with the major dependencies.

Requirement	Reason
GIT	to get the repo from github
Python	to run the silnlp code
Poetry	to manage all the Python packages and versions
NVIDIA GPU	Required to run on a local machine
Nvidia drivers	Required for the GPU
CUDA Toolkit	Required for the Machine learning with the GPU
Environment variables	To tell SILNLP where to find the data, etc.

Environment Setup

If using a local GPU, install the corresponding NVIDIA driver.

On Ubuntu, the driver can alternatively be installed through the GUI by opening Software & Updates, navigating to Additional Drivers in the top menu, and selecting the newest NVIDIA driver with the labels proprietary and tested.

After installing the driver, reboot your system.
Download and install Docker Desktop.
- If using Linux (not WSL), add your user to the docker group by using a terminal to run: sudo usermod -aG docker $USER
- Reboot after installing, confirm that all installation steps are complete before the next step.
Pull Docker image

In a terminal, run:
```
docker pull ghcr.io/sillsdev/silnlp:latest
```
- For Windows, use CMD Prompt
- If there is an error like "request returned Internal Server Error for API route and version , check if the server supports the requested API version" Check that the Docker Desktop installation steps are complete. Reopen CMD prompt and try again.
Create Docker container based on the image

If you're using a local GPU, then in a terminal, run:
```
docker create -it --gpus all --name silnlp ghcr.io/sillsdev/silnlp:latest
```
Otherwise, run:
```
docker create -it --name silnlp ghcr.io/sillsdev/silnlp:latest
```
A docker container should be created. You should be able to see a container named 'silnlp' on the Containers page of Docker Desktop.
Create file for environment variables

Create a text file with the following content and edit as necessary:
```
CLEARML_API_HOST="https://api.sil.hosted.allegro.ai"
CLEARML_API_ACCESS_KEY=xxxxx
CLEARML_API_SECRET_KEY=xxxxx
AWS_REGION="us-east-1"
AWS_ACCESS_KEY_ID=xxxxx
AWS_SECRET_ACCESS_KEY=xxxxx
SIL_NLP_DATA_PATH="/aqua-ml-data"
```
- If you do not intend to use SILNLP with ClearML and/or AWS, you can leave out the respective variables. If you need to generate ClearML credentials, see ClearML setup.
- Note that this does not give you direct access to an AWS S3 bucket from within the Docker container, it only allows you to run scripts referencing files in the bucket.
Start container

In a terminal, run:
```
   docker start silnlp
   docker exec -it --env-file path/to/env_vars_file silnlp bash
```
- After this step, the terminal should change to say root@xxxxx:~/silnlp#, where xxxxx is a string of letters and numbers, instead of your current working directory. This is the command line for the docker container, and you're able to run SILNLP scripts from here.
- To leave the container, run exit, and to stop it, run docker stop silnlp. It can be started again by repeating step 6. Stopping the container will not erase any changes made in the container environment, but removing it will.

Development Environment Setup

Follow the instructions below to set up a Dev Container in VS Code. This is the recommended way to develop in SILNLP. For manual setup, see Manual Setup.

If using a local GPU, install the corresponding NVIDIA driver.
- On Ubuntu, the driver can alternatively be installed through the GUI by opening Software & Updates, navigating to Additional Drivers in the top menu, and selecting the newest NVIDIA driver with the labels proprietary and tested.
- After installing the driver, reboot your system.
Download and install Docker Desktop.
- Reboot after installing and completing the relevant steps below, confirm that all installation steps are complete before the next step.
- Linux users should have no additional steps given that they follow Docker's distribution-specific setup.
Windows (non-WSL) and macOS:
- Open Settings in Docker Desktop and under the Resources tab, update File Sharing with any locations your source code is kept.
WSL:
- Enable WSL 2 backend:
  - Open Settings in Docker Desktop and check "Use WSL 2 based engine" under the General tab. It may already be checked.
  - To verify, check under the Resources tab in Settings for a message saying that you are using the WSL 2 backend. Linux:
- Add your user to the docker group by using a terminal to run: sudo usermod -aG docker $USER
- Sign out and back in again so your changes take effect
Set up ClearML.
Define environment variables.

Set the following environment variables with your respective credentials: CLEARML_API_ACCESS_KEY, CLEARML_API_SECRET_KEY, AWS_ACCESS_KEY_ID, and AWS_SECRET_ACCESS_KEY. Additionally, set AWS_REGION. The typical value is "us-east-1".
- Linux: To set environment variables permanently, add each variable as a new line to the .bashrc file in your home directory with the format
```
export VAR="VAL"
```
  Close and reopen any open terminals for the changes to take effect.
- Windows:
  1. Open Settings and go to the System tab.
  2. Under the "Device Specifications" section, in the "Related links", click "Advanced system settings".
  3. Click "Environment Variables".
  4. In the "System Variables" section, click "New".
  5. Enter the name and value of the variable and click "Ok". Repeat for as many variables as you need.
  6. Click "Ok" on the Environment Variables page to save your changes.
  7. Close and reopen any open command prompt terminals for the changes to take effect.
Install Visual Studio Code.
Clone the silnlp repo.
Open up silnlp folder in VS Code.
Install the Dev Containers extension for VS Code.
Build the dev container and open the silnlp folder in the container.
- Click on the Remote Indicator in the bottom left corner.
- Select "Reopen in Container" and choose the silnlp dev container if necessary. This will take a while the first time because the container has to build.
- If it was successful, the window will refresh and it will say "Dev Container: SILNLP" in the bottom left corner.
- Note: If you don't have a local GPU, you may need to comment out the gpus --all part of the runArgs field of the .devcontainer/devcontainer.json file.
Install and activate Poetry environment.
- In the VS Code terminal, run poetry install to install the necessary Python libraries, and then run poetry shell to enter the environment in the terminal.
(Optional) Locally mount the S3 bucket. This will allow you to interact directly with the S3 bucket from your local terminal (outside of the dev container). See instructions here.

To get back into the dev container and poetry environment each subsequent time, open the silnlp folder in VS Code, select the "Reopen in Container" option from the Remote Connection menu (bottom left corner), and use the poetry shell command in the terminal.

Setting Up and Running Experiments

See the wiki for information on setting up and running experiments. The most important pages for getting started are the ones on file structure, model configuration, and running experiments. A lot of the instructions are specific to NMT, but are still helpful starting points for doing other things like alignment.

See this page for information on using the VS code debugger.

If you need to use a tool that is supported by SILNLP but is not installable as a Python library (which is probably the case if you get an error like "RuntimeError: eflomal is not installed."), follow the appropriate instructions here.

.NET Machine alignment models

If you need to run the .NET versions of the Machine alignment models, you will need to install .NET Core SDK 8.0. After installing, run dotnet tool restore.

Windows: .NET Core SDK
Linux: Installation instructions can be found here.

sillsdev / silnlp