A Python library for automating interaction with websites. MechanicalSoup automatically stores and sends cookies, follows redirects, and can follow links and submit forms.
From PyPI
pip install MechanicalSoup
WWW-Mechanize and Mechanize are similar Perl and Ruby libraries. Unfortunately, their Python sister project has become inactive, and is now incompatible with recent Python versions (Python 3.x). This library provides a similar API, built on Python giants Requests (for http sessions) and BeautifulSoup (for document navigation).
From example.py, code to log into the GitHub website:
import mechanicalsoup
browser = mechanicalsoup.Browser()
# request github login page
login_page = browser.get("https://github.com/login")
# find login form
login_form = login_page.soup.select("#login")[0].select("form")[0]
# specify username and password
login_form.select("#login_field")[0]['value'] = "username"
login_form.select("#password")[0]['value'] = "password"
# submit form
page2 = browser.submit(login_form, login_page.response.url)
# verify we are now logged in
assert page2.soup.select(".logout-form")
# verify we remain logged in (thanks to cookies) as we browse the rest of the site
page3 = browser.get("https://github.com/colonelpanic/MechanicalSoup")
assert page3.soup.select(".logout-form")
For an example with a more complicated form (with checkboxes, radio buttons and textareas), read tests/test_browser.py.
py.test