This has been generated by the StormCrawler Maven Archetype as a starting point for building your own crawler. Have a look at the code and resources and modify them to your heart's content.
mvn clean compile exec:java -Dexec.mainClass=net.pic.crawler.CrawlTopology -Dexec.args="-conf crawler-conf.yaml -local"to run the demo CrawlTopology in local mode, without Storm installed.
With Storm installed, you can generate an uberjar:
mvn clean packageand then submit the topology using the storm command:
storm jar target/stormcrawler-1.0-SNAPSHOT.jar net.pic.crawler.CrawlTopology -conf crawler-conf.yaml -localto run in local mode. Simply remove the '-local' to run the topology in distributed mode.
You can also use Flux to do the same:
storm jar target/stormcrawler-1.0-SNAPSHOT.jar org.apache.storm.flux.Flux --local crawler.flux