Visualizing an XML sitemap using Python
Source code repository for the "How to visualize an XML sitemap using Python" Ayima blog post.
Extraction, Categorization and Visualization
The code has been split into three scripts that can be downloaded here. Running these requires Python and external library dependencies as detailed in the section below.
Reproducing our Results
Before plugging in a new sitemap, you may wish to test the script by reproducing a result from our blog post. This can be done by opening a terminal session, navigating to the folder containing the three .py scripts and running the following commands:
python extract_urls.py
python categorize_urls.py
python visualize_urls.py --depth 1
Plugging in an XML Sitemap URL
A custom sitemap URL can be ingested by passing arguments to the extraction script. The commands to execute would look something like this (where we are categorizing and visualizing with a granularity depth of 3):
python extract_urls.py --url "site.com/sitemap-index.xml"
python categorize_urls.py --depth 3
python visualize_urls.py --depth 3 --title "My Sitemap" --size "20"
If your XML sitemap file contains the page URLs (instead of linking to other sitemaps) then make sure to pass the --not_index argument:
python extract_urls.py --url "site.com/sitemap.xml" --not_index
There is also built in support for compressed XML files:
python extract_urls.py --url "site.com/sitemap.xml" --not_index --gzip
The visualize_urls.py script also has a --limit argument that can be passed. This can be used to limit the number of edges spawning from a node, and is useful for creating deep sitemap visualizations that don't grow out of control. For example:
python categorize_urls.py --depth 6
python visualize_urls.py --depth 6 --limit 2 --size "30"
The graph save format can be specified with --output-format e.g. python visualize_urls.py --output-format png.
Additionally, select nodes can be skipped (restrict children from rendering) using the --skip argument. For example:
python extract_urls.py
python categorize_urls.py
python visualize_urls.py --depth 2 --skip 'product,brands,categories,find,campaigns,clearance,stores'
More detailed usage instructions are included in the header of each file.
Categorizing and Visualizing a list of URLs
If you already have a list of URLs, they can be compiled into a file named sitemap_urls.dat that contains one URL per line, and then processed by running the latter two scripts:
python categorize_urls.py
python visualize_urls.py
Please ensure you have installed the dependencies before attempting to run the scripts.
Dependencies
The code can run in Python 2 or 3 and the external library dependencies are as follows:
- Requests and BeautifulSoup4 for
extract_urls.py - Pandas for
categorize_urls.py - Graphviz for
visualize_urls.py
Once you have Python, these libraries can most likely be installed on any operating system with the following terminal commands:
pip install requests
pip install beautifulsoup4
pip install pandas
The Graphviz library is more difficult to install. On Mac it can be done with the help of homebrew:
brew install graphviz
pip install graphviz
For other operating systems or alternate methods, check out the installation instructions in the Graphviz documentation.
Contact
We are here to help! If you run into any problems you can reach out to us on twitter @ayima.