jlertle / FinBERT

BERT for Finance : UC Berkeley MIDS w266 Final Project

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

FinBERT: Pre-Trained on SEC Filings for Financial NLP Tasks

Vinicio DeSola, Kevin Hanna, Pri Nonis

MODEL WEIGHTS

PUBLICATION

MOTIVATIONS

Goal 1 FinBERT-Prime_128MSL-500K+512MSL-10K vs BERT

  • Compare mask LM prediction accurracy on technical financial sentences
  • Compare analogy on financial relationships

Goal 2 FinBERT-Prime_128MSL-500K vs FinBERT-Pre2K_128MSL-500K

  • Compare mask LM prediction accuracy on financial news from 2019
  • Compare analogy on financial relationship, measure shift in understanding : risk vs climate in 1999 vs 2019

Goal 3 FinBERT-Prime_128MSL-500K vs FinBERT-Prime_128MSK-500K+512MSL-10K

  • Compare mask LM prediction accuracy on long financial sentences

Goal 4 FinBERT-Combo_128MSL-250K vs FinBERT-Prime_128MSL-500K+512MSL-10K

  • Compare mask LM prediction accuracy on financial sentences : can we get same accuracy with less training by building on original BERT weights.

TERMINOLOGY

  • Prime Pre-trained from scratch on 2017, 2018, 2019 SEC 10K dataset

  • Pre2K Pre-traind from scratch on 1998, 1999 SEC 10K dataset

  • Combo Pre-trained continued from original BERT on 2017, 2018, 2019 SEC 10K dataset

ANALYSIS

About

BERT for Finance : UC Berkeley MIDS w266 Final Project


Languages

Language:C++ 47.1%Language:Python 36.8%Language:Jupyter Notebook 6.8%Language:Shell 6.0%Language:Starlark 2.3%Language:Dockerfile 0.5%Language:Perl 0.3%Language:Smarty 0.1%Language:Makefile 0.1%