Yarn Cleaner
This module can be used to monitor and clean the temporary directories that are populated during Yarn jobs.
Features
- Checks disk usage of Yarn and kills the application that is using up more than a given threshold
- Connects to a remote node via SSH and runs commands
Prerequisites
- python3
- paramiko
- argparse
Installation
Clone this repository:
git clone https://github.com/stvoutsin/yarncleaner.git
cd into the repository:
cd yarncleaner
Install dependencies:
pip install -r requirements.txt
Usage
Using the Command Line This module can be run from the command line with the following command:
sh
python yarncleaner.py [--workers [WORKERS [WORKERS ...]]] [--ssh-username SSH_USERNAME] [--ssh-key-file SSH_KEY_FILE] [--usercache-dir USERCACHE_DIR] [--threshold-percent THRESHOLD_PERCENT]
Arguments
| Argument | Description |
|---|---|
| --workers | A list of workers |
| --ssh-username | The username to use for SSH |
| --ssh-key-file | The path to the private key file to use for SSH |
| --usercache-dir | The directory where the usercache is located |
| --threshold-percent | The percentage of disk usage to trigger a clean |
Using the API
This module can also be used as an API:
from yarncleaner import YarnCleaner
workers = ["worker01", "worker02", "worker03"]
ssh_username = "username"
ssh_key_file = "/path/to/key/file.pem"
cleaner = YarnCleaner(workers=workers, ssh_username=ssh_username, ssh_key_file=ssh_key_file)
cleaner.clean()
License
This module is licensed under the GNU GENERAL PUBLIC LICENSE.