kurtcms / web-js-monitor

Web Monitoring: Monitor a JavaScript Rendered Web Page for Updates

Web Monitoring: Monitor a JavaScript Rendered Web Page for Updates

This Python app is containerised with Docker Compose for a modular and cloud native deployment that fits in any microservice architecture.

It does the following:

1. Use the Python libraries, Requests and Beautiful Soup, to download a copy of the web page of interest and any JavaScript files that are referred to in the page;
2. Use the hashlib module to compute a SHA 256-bit checksum with the downloaded contents; and
3. Compare the checksum with a previous build, if one exists, and on mismatch save the checksum and a copy of the downloaded contents, before using the smtplib module to send a notification email to the predefined recipient if it is required. Otherwise if no previous checksum exists, save the checksum the rest of the downloaded contents for future matching and references.

A detailed walk-through is available here.

Table of Content

• Getting Started
  • Git Clone
  • Environment Variable
  • Crontab
  • Docker Container
    • Docker Compose
    • Build and Run
  • Standalone Python Script
    • Dependencies
    • Usage and Option
    • Cron
• Checksum and the Downloaded Contents
• Reference

Getting Started

Get started in three simple steps:

1. Download a copy of the app;
2. Create the environment variables for email notification if needed and modify the crontab; and
3. Docker Compose or build and run the image manually to start the app, or alternatively run the Python script as a standalone service.

Git Clone

Download a copy of the script with git clone. Be sure to pass the --recurse-submodules argument to initialise and update each submodule in the repository.

$ git clone --recurse-submodules https://github.com/kurtcms/web-js-monitor /app/web-js-monitor/

Environment Variables

If email notification is required, the app expects the SMTPS port number, SMTP server address, the notification receiver email address, the notification sender email address and password as environment variables in a .env file in the same directory.

Create the .env file.

$ nano /app/web-js-monitor/.env

And define the variables accordingly.

EMAIL_SSL_PORT = 465
EMAIL_SMTP_SERVER = 'smtp.kurtcms.org'
EMAIL_SENDER = '[e-mail]'
EMAIL_RECEIVER = '[e-mail]'
EMAIL_SENDER_PASSWORD = '(redacted)'

Crontab

By default the app is scheduled with cron to pull a copy of the web page and check for changes every 15 minutes, with stdout and stderr redirected to the main process for Docker logs.

Modify the crontab to feed in the URL of interest, and signify to the Python script whether email notification is needed. Change to a different schedule if required.

$ nano /app/web-js-monitor/crontab

And define the variables accordingly.

*/15 * * * * /usr/bin/python3 /app/web-js-monitor/web-js-monitor.py -u https://lookingglass.pccwglobal.com/ > /proc/1/fd/1 2>/proc/1/fd/2
#
# Usage: web-js-monitor.py [-e] -u <url>
#
# Option:
#   -h
#     Display usage
#   -e, --email
#     Send email notification on changes
#   -u, --url
#     The URL of interest

Docker Container

Packaged as a container, the app is a standalone, executable package that may be run on Docker Engine. Be sure to have Docker installed.

Docker Compose

With Docker Compose, the app may be provisioned with a single command.

Install Docker and Docker Compose with the Bash script that comes with app.

$ chmod +x /app/web-js-monitor/docker-compose/docker-compose.sh \
    && /app/web-js-monitor/docker-compose/docker-compose.sh

Start the containers with Docker Compose.

$ docker-compose -f /app/web-js-monitor/docker-compose.yml up -d

Stopping the container is as simple as a single command.

$ docker-compose -f /app/web-js-monitor/docker-compose.yml down

Build and Run

Otherwise the Docker image can also be built manually.

$ docker build -t web-js-monitor /app/web-js-monitor/

Run the image with Docker once it is ready.

$ docker run -it --rm --name web-js-monitor web-js-monitor

Standalone Python Script

Alternatively the web-js-monitor.py script may be deployed as a standalone service.

Dependencies

In which case be sure to install the following required libraries:

1. Requests
2. Python-dotenv
3. Beautiful Soup

Install them with pip3:

$ pip3 install requests python-dotenv bs4

Usage and Option

The script reads the URL of interest from the -u argument and sends email notification on changes if the -e option is given.

Usage: web-js-monitor.py [-e] -u <url>

Option:
  -h
    Display usage
  -e, --email
    Send email notification on changes
  -u, --url
    The URL of interest

Cron

It may then be executed with a task scheduler such as cron that runs it once every 15 minutes for example.

$ (crontab -l; echo "*/15 * * * * /usr/bin/python3 /app/web-js-monitor/web-js-monitor.py -u https://lookingglass.pccwglobal.com/") | crontab -

Checksum and the Downloaded Contents

The SHA 256-bit checksum and the rest of the downloaded contents will be saved as separate files on a Docker volume that is mounted in the same directory of the docker-compose.yml file on the Docker host. If the Python script is run as a standalone service, the files will be in the same directory of the script instead.

In any case, the files are stored under a directory by the URL of the web page of interest, with the checksum stored in a file directly under it and the rest of the downloaded contents in a nested directory named by the full date and time of the download to ease access.

.
└── url/
    ├── url-sha256hash
    └── YYYY-MM-DD-HH-MM-SS/
        ├── index.html
        ├── javaScript1.js
        ├── javaScript2.js
        └── javaScript3.js

Reference

• Requests Documentation
• Beautiful Soup Documentation