alxxtexxr / webgen

WebGen model pipeline

🌐🤖 WebGen

Data Generation

The data generation process utilizes the Alpaca Self-Instruct pipeline, but with the OpenAI chat model (e.g., gpt-3.5-turbo). Please note tthat his pipeline does not employ a batch system since the chat model does not support prompt batching.

python -m generate_instruction generate_instruction_following_data \
  --output_dir="./" \
  --num_instructions_to_generate=100 \
  --num_instructions_to_generate_per_request=3 \
  --model_name="gpt-3.5-turbo-16k" \
  --similarity_threshold=0.6

Fine-tuning

The fine-tuning process utilizes the WizardLM-13B-V1.2 model.

python finetune.py \
  --hf_token hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx \
  --base_model_id alimtegar/WizardLM-13B-V1.2-sharded \
  --dataset_id alimtegar/webgen-dataset-2 \
  --output_dir "./webgen-wizardlm-13b-lora" \
  --output_model_id alimtegar/webgen-wizardlm-13b-lora \
  --commit_message "Finetune for 3 epochs" \
  --vastai_api_key xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx \
  --vastai_instance_id 1234567
  --stop_vastai_instance 0