This Zillabyte Component will crawl an entire web domain and return the results as a new dataset.
You can execute this component from the command line:
$ zillabyte execute "domain_crawl" "docs.zillabyte.com"Or, you can stitch this component into your own Data App.
app = Zillabyte.app("your_app")
stream = app.source(...)
stream = stream.call_component("domain_crawl")
stream.sink(...)