This project allows for the automated backup of MySQL databases to an S3 bucket and restoration of those backups. It is configured to run as a service to ensure regular backups without manual intervention.
- Python 3.x
- MariaDB or MySQL Server
- A Wasabi or AWS S3 Bucket
- An environment to run the service (e.g., a server where the database is hosted)
-
Clone the Repository
git clone https://github.com/yashodhank/mysql_backup_s3 cd mysql_backup_s3
-
Create a Virtual Environment and Activate It
python3 -m venv venv-mysql_backup_s3 source venv-mysql_backup_s3/bin/activate
-
Install Required Packages
pip3 install -r requirements.txt
-
Secure Sensitive Information
To secure sensitive information like database passwords and API keys, it is recommended to use environment variables. This is a good practice to avoid exposing sensitive information. Set the following environment variables in your shell, or better, in your
.bashrc
or.bash_profile
file. After setting them, restart your terminal session so that the changes take effect. If environment variables are not set, whenbackup.py configure
is invoked, these details will be prompted and stored inbackup_config.json
in the same directory, which is less secure.export DB_USER='mariadb_user' export DB_PASSWORD='mariadb_password' export AWS_ACCESS_KEY_ID='AKIAIOSFODNN7EXAMPLE' export AWS_SECRET_ACCESS_KEY='wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY'
-
Configure the Backup Script
Run the configuration wizard to set up the backup script:
python3 backup.py configure
Follow the prompts to configure the S3 bucket name, region name, and enable verbose or debug mode if needed. If the environment variables for sensitive information are not set, they will be prompted during this step. If the required database user and password are not found in standard locations, they will also be prompted.
Enter AWS Access Key ID [AKIAIOSFODNN7EXAMPLE]: Enter AWS Secret Access Key [wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY]: Enter S3 Bucket Name: mydatabase-backups Enter S3 Region Name [us-east-1]: Enter Database User [mariadb_user]: Enter Database Password [mariadb_password]: Enter Telegram Bot Token (optional, press enter to skip): 123456:ABC-DEF1234ghIkl-zyx57W2v1u123ew11 Enter Telegram Chat ID (optional, press enter to skip): 123456789 Enter Pushover Application Token (optional, press enter to skip): azGDORePK8gMaC0QOYAMyEEuzJnyUi Enter Pushover User Key (optional, press enter to skip): uQiRzpo4DXghDmr9QzzfQu27cmVRsG Enable verbose mode? (yes/no): yes Enable debug mode? (yes/no): no
If you skip the optional fields, you won't receive notifications via Telegram or Pushover. If you enable verbose or debug mode, more detailed information will be printed to the console and log file during backup operations.
-
Create a Systemd Service File
Create a systemd service file to run the backup script as a service:
sudo nano /etc/systemd/system/mysql_backup_s3.service
Add the following content to the service file, replacing
/path/to/mysql_backup_s3
with the actual path to yourmysql_backup_s3
directory:[Unit] Description=MySQL Backup to S3 After=network.target [Service] User=root WorkingDirectory=/path/to/mysql_backup_s3 Environment="PATH=/path/to/mysql_backup_s3/venv-mysql_backup_s3/bin" ExecStart=/path/to/mysql_backup_s3/venv-mysql_backup_s3/bin/python3 /path/to/mysql_backup_s3/backup.py Restart=on-failure RestartSec=30s [Install] WantedBy=default.target
-
Start and Enable the Service
sudo systemctl start mysql_backup_s3.service sudo systemctl enable mysql_backup_s3.service
This will start the service and ensure it starts on boot.
Once the service is started, it will automatically back up your databases to the configured S3 bucket at regular intervals without any manual intervention.
To restore a database, use the restore.py
script:
-
Activate the Virtual Environment
source venv-mysql_backup_s3/bin/activate
-
Run the Restore Script
python3 restore.py
Follow the prompts to restore a specific backup from the S3 bucket to the local MySQL/MariaDB server.
If the script can locate MySQL/MariaDB credentials from standard or control panel-specific locations, it will use them. If not, it will prompt you to enter the database user and password. Enter the name of the database to restore and the path of the backup file. If the database already exists, it will ask for confirmation twice before overwriting it.
-
Telegram Notifications:
- The script can send Telegram notifications for successful backups/restores and critical errors.
- Configure the
TELEGRAM_TOKEN
andTELEGRAM_CHAT_ID
during the configuration step or set them as environment variables.
-
Pushover Notifications:
- The script can send Pushover notifications for successful backups/restores and critical errors.
- Configure the
PUSHOVER_TOKEN
andPUSHOVER_USER
during the configuration step or set them as environment variables.
- The script logs all events to
backup.log
andrestore.log
in the script's directory. - You can monitor these log files for any issues or to verify successful backups/restores.
- Secure the log files and avoid logging sensitive information.
For example, if you have a database named mydatabase
and you have configured the S3 bucket as mybucket
, the backup script will automatically back up the mydatabase
database to the mybucket
S3 bucket.
For instance, to restore the mydatabase
database from a backup file named mydatabase_backup_20230924.sql.bz2
located in the mybucket
S3 bucket, run the restore script and enter the database name as mydatabase
and the backup file
path as mydatabase_backup_20230924.sql.bz2
.
- Secure the configuration file and environment variables containing sensitive information, such as passwords and API keys.
- Regularly monitor the log files and set up log rotation to avoid filling up the disk space.
- Set up an alerting system to notify you if the script encounters any errors.
Feel free to fork the project, create a feature branch, and open a Pull Request. For bugs, please open an issue.
This project is open-source and available under the MIT License.
- The above README assumes that the user has basic knowledge of using the command line, Python, and MariaDB/MySQL.
- The README provides a basic outline. You might want to customize and enhance it according to your project’s specific needs, adding more details, sections, or clarifications as necessary.
- Make sure to test all the instructions provided in the README to ensure they are accurate and clear before publishing it.