General Scraper

An ambitious lieutenant scraper. Parameterised with env variables - for now.

Installation

This project uses pipenv for handling package dependencies. To set up your development environment, follow the steps below:

Ensure you have pipenv installed. If you do not have pipenv installed, you can install it using pip:

pip install pipenv

For more information on using pipenv, you can refer to the official documentation

Clone the repository:

git clone https://github.com/DeeprajPandey/lieutenant-scraper.git
cd lieutenant-scraper

Install dependencies:

Use pipenv to install the project dependencies from the Pipfile:
```
pipenv install
```
This will create a virtual environment and install the necessary packages.
Activate the virtual environment:
```
pipenv shell
```
This will spawn a new shell subprocess, which can be deactivated by simply closing it, or by running exit.
Run the application:

With the virtual environment activated and config, you can now run the application:
```
python main.py
```

To install a new package for your project:
```
pipenv install package-name
```
To install dev-packages (packages that are only needed for development, not in production):
```
pipenv install package-name --dev
```
To generate a Pipfile.lock, which should be committed:
```
pipenv lock
```

Add .env.example and config.yaml.example to root and add edit instructions after activating virtual environemnt in installation instructions.