Robotex does not support comments

[Martha Thompson]

Rules with trailing comments are not being applied correctly.

Example:
https://ask.fmcsa.dot.gov/robots.txt

User-agent: *                    # ADDED BY HMS
Disallow: /                      # ADDED BY HMS

The above rule should disallow everything on the site. Instead:

> robotex = Robotex.new('testing')
> robotex.allowed?('https://ask.fmcsa.dot.gov/anything')
true

PR: #3

Reference: https://developers.google.com/search/reference/robots_txt#file-format