A tree query language for HTML - think of a combination of regular expressions and CSS selectors, applied to tree structures.
import bs4
import tql
html = bs4.BeautifulSoup("<div id='find-me' data-attr='world'>hello</div>")
qry = tql.compile("div#find-me[txt, .data-attr]")
(text, data_attr), = qry.match(html)
print(text, data_attr) # Prints "hello world"
tql.compile(expr: str) -> tql.Node
- Returns the compiled query patterntql.Node.match(html: bs4.BeautifulSoup, **filterfuncs: Callable[[bs4.Tag], bool]) -> Iterable[tuple]
- Performs a search over the provided HTML document and returns an iterator over the results. The results are tuples, and the contents of the tuple depends on the extractors used and the structure of the pattern. Extractors usually return strings, so most results are a tuple of strings. However, using repetition operators (like*
and+
) will result in a nested tuple structure.filterfuncs
are the functions used in filters using the$f
notation, callingqry.match(..., my_func=lambda node: ...)
will result in$my_func
calling that lambda.
Terms used:
- "tag" - A HTML element, like a
div
ortable
; "parent", "child", "neighbour", and "attribute" have the same meanings as in HTML
@
- Matches any tagdiv
- Matches adiv
tag.centered
- Matches a tag with thecentered
CSS class#root
- Matches a tag with theroot
iddiv.centered#root
- Matched a div tag with thecentered
CSS class and theroot
id
div[txt]
- Extract the text from a tagdiv[node]
- Extract the bs4 node for that tagdiv[.data-attr]
- Extract the value of thedata-attr
attribute
div > a
- Matches adiv
with ana
tag as a direct childdiv >> a
- Matches adiv
with ana
tag as a direct or indirect childdiv > table >> li > a
- Matches ana
that s a child of ali
which is the descendent of atable
that is a child of adiv
TQL can match "sideways" as well as downwards, similar to the CSS :first-child
selectors. However, this syntax is much more powerful.
{}
- Switch mode: All queries start in depth mode, and can be switched to breadth mode using curly braces; using curly braces in breadth mode, switches back to depth mode{ div : a }
- Match adiv
with ana
as a direct neighbour{ div :: a }
- Match adiv
with ana
as a direct or indirect neighbour (i.e.div
thena
ORdiv
then some other tags thena
){ div : { li > a } }
- Match ana
that is the direct child of ali
that is the direct neighbour of adiv
div { div : div : div }
- Match adiv
tag with threediv
tags as children
$ > @[txt]
- Extract the text from the root tagdiv[txt] > $
- Extract the text from adiv
tag with no childrenhead { $ : @[txt] }
- Extract the first child of ahead
taghead { @[txt] : $ }
- Extract the last child of ahead
tag
{ $ : (a :}* : $ }
- Match a tag which has zero or morea
tags as its only children{ $ : (a : b :)* : $ }
- Match a tag which has zero or morea
followed byb
tags as its only childrendiv > (span >)* > a
- Match ana
tag which has adiv
ancestor and all (zero or more) parents that are descendants of thatdiv
tag arespan
sdiv > (span >)+ > a
- Match ana
tag which has adiv
ancestor and all parents that are descendants of thatdiv
tag arespan
s and there must be at least one span parent
div > span? > a
- Match ana
tag which either has aspan
parent which has adiv
parent, or adiv
parent
.a.b
- Match a tag that has both thea
andb
CSS class(.a | .b)
- Match a tag with thea
or theb
(or both) CSS class(a | b)
- Match ana
orb
tagdiv > (span | table > tbody) > a
- Match ana
tag that either has aspan
parent withdiv
parent, or atbody
parent with atable
parent with adiv
parentdiv!.a
- Match adiv
tag that does not has thea
CSS class
div~(.data-x)
- Match adiv
tag with adata-x
attribute that is non-emptydiv~(.data-x == 'hello')
- Match adiv
tag with adata-x
attribute that equals"hello"
div~(.data-x != 'hello')
- Match adiv
tag with adata-x
attribute that does not equal"hello"
div~(.data-x ~~ 'hello')
- Match adiv
tag with adata-x
attribute that matches"hello"
(this is done using a regex search)div~(.data-x !~ 'hello')
- Match adiv
tag with adata-x
attribute that does not match"hello"
(this is done using a regex search)div~(.data-x && .data-y)
- Match adiv
tag with adata-x
anddata-y
attribute, both of which are non-emptydiv~(.data-x || .data-y)
- Match adiv
tag with either adata-x
ordata-y
(or both) attribute that is non-emptydiv~(.data-x && $my_func)
- Match adiv
tag with adata-x
attribute that is non-empty and for which the filter functionmy_func
returns a truthy value
Phew, that's a lot of stuff! Check out the tests for verified examples. Also, try experimenting. Just like regular expressions, seeing the results is much more intuitive than reading descriptions of the operators.