scottpitcher / Decoder_LLM

Dive deeper into LLMs via creating a micro-LLM decoder from scratch to best understand the architecture.

Personal Project: Building a Decoder LLM From Scratch

Tools Used

OpenWebText Download: https://skylion007.github.io/OpenWebTextCorpus/

Amazon Web Services (AWS): https://aws.amazon.com

Storage (S3), CloudComputing (EC2)

Visual Studio Code: https://code.visualstudio.com

Set-Up

Ensure you are using Python 3.10+

In Terminal: which python3

Installing Requirements pip install -r requirements.txt

Social Links

Linkedin: https://www.linkedin.com/in/scottpitcher1

Github: https://github.com/scottpitcher

Website: https://www.scottpitcher.com/

Sources For This Project

[Initial Guide] Create a LLM from Scratch with Python: https://youtu.be/UU1WVnMk4E8?si=_AdR3Qk2RrF2OFYP

[Supplemental Talk] Andrej Karpathy [1hr Talk] Intro to LLMs: https://youtu.be/zjkBMFhNj_g?si=K6ooQAK8XZbAjeTs

[Research Paper] Attention is All You Need: https://arxiv.org/pdf/1706.03762.pdf

[Research Paper] A Survey of LLMs: https://arxiv.org/pdf/2303.18223.pdf