Prerequisites: nodejs w/ npm
- Clone the repo locally
- From within repo directory, run
npm install
to install all dependencies - To execute script, run
node [script].js
- To manually stop script, press
ctrl + c
from active CLI.
./lawyer-data/
contains individual lawyer data in json files./practice-areas/
contains lists of all lawyers' URLs in json files./all-lawyers.json
contains all lawyer data compiled viacompile-lawyers.js
from data within./lawyer-data/
./all-lawyers-hrefs.json
contains all lawyers's URLs compiled viacompile-lawyer-hrefs.js
from data within./practice-areas/
./compile-lawyer-hrefs.js
reads data within./practice-areas/
and compiles to./all-lawyers-hrefs.json
./compile-lawyers.js
reads data within./lawyer-data/
and compiles to./all-lawyers.json
./scrape-lawyers.js
grabs the data for individual lawyers by digesting./all-lawyers-hrefs.json
./scrape-practice-areas.js
traverses Avvo's directory structure to grab URLs for lawyers./scrape-results-all-practice-areas.sjon
(you will not interact with this file) this file is consumed by./scrape-practice-areas.js
- Run
node scrape-practice-areas.js
to scrape the URLs for all of the individual Lawyer pages. - Run
node compile-lawyer-hrefs.js
to compile all of the Lawyer's hrefs into one file for consumption. - Run
node scrape-lawyers.js
to scrape each lawyer's data. - Run
node compile-lawyers.js
to compile all lawyer data into one file.
scrape-practice-areas.js
is responsible for grabbing all of the URLs for each Lawyer in every practice area in every state. Once the entire practice area has been scraped, it is removed from the all-practice-areas array so that the script can be successfully resumed in case of error, timeout, or resuming scrape.
compile-lawyer-hrefs.js
will read all of the data saved in practice-areas
and compile it into one file to be consumed by scrape-lawyers.js
scrape-lawyers.js
will visit each URL listed in all-lawyers-hrefs.json
and write the data scraped for each lawyer to a file in lawyer-data
named after the lawyer and prefaced with the state in which they are located.
compile-lawyers.js
will combine all data for each lawyer into a single file for easy consumption.
Dealing with Timeouts:
It's possible for a timeout to occur under some circumstances. If this happens, simple hit ctrl + c
and restart the script. All previously scraped data will be preserved.