rasbt / LLMs-from-scratch

Implementing a ChatGPT-like LLM in PyTorch from scratch, step by step

https://www.manning.com/books/build-a-large-language-model-from-scratch

rasbt/LLMs-from-scratch Issues

Throwing error for longer textual data like 9599
Closed 11 days ago4
Expected all tensors to be on the same device
Closed 11 days ago1
ch06/03_bonus_imdb-classification
Closed 18 days ago16
suggestion of adding torch.profile
Closed a month ago
do have a doc for hardware specs
Closed a month ago2
class MHAPyTorchScaledDotProduct
Closed a month ago2
Difference btwn book and repo
Closed a month ago1
The definition of stride is confusing in 2.6
Closed a month ago3
About endoftext in ch05/03_bonus_pretraining_on_gutenberg/pretraining_simple.py
Closed 2 months ago14
book feedback
Closed 2 months ago1
RuntimeError: size mismatch - ch05/03_bonus_pretraining_on_gutenberg
Closed 2 months ago2
In 3.3.1, there seems to be a missing image between "The attention weights and context vector calculation are summarized in the figure below:" and "The code below walks through the figure above step by step."
Closed 2 months ago1
Contributions for Chinese simplified version
Closed 2 months ago4
{Q} : Replacing the LlamaDecoderLayer Class hugging Face With New LongNet
Closed 2 months ago
Feedback: Stripe output from notebook
Closed 2 months ago2
Offering Chinese Translation for 'Build a Large Language Model From Scratch
Closed 2 months ago3
Chapter 5 - Context Size and the DataLoaders
Closed 2 months ago2
Inconsistencies between the code in the book and the notebooks (2.6 Data sampling with a sliding window)
Closed 3 months ago7
Inconsistencies in MHA Wrapper Implementation Between Chapter 3 Main Content and Bonus Material
Closed 3 months ago1
Question about implementation of CausalAttention class (3.5.3 Implementing a compact causal self-attention class)
Closed 3 months ago6
Solution for Exercise 3.2 is included in the notebook with main code (3.6.1 Stacking multiple single-head attention layers)
Closed 3 months ago1
Probably a typo in multi-head attention description (3.6.1 Stacking multiple single-head attention layers)
Closed 3 months ago1
Inconsistencies in output for dropout section (3.5.2 Masking additional attention weights with dropout)
Closed 3 months ago1
Solution for Exercise 3.3 is included in the notebook with main code (3.6.2 Implementing multi-head attention with weight splits)
Closed 3 months ago1
Inconsistencies in unsqueeze operation description in the book and in notebook and its necessity (3.6.2 Implementing multi-head attention with weight splits)
Closed 3 months ago4
Wrong number of token ids specified in the notebook (2.7 Creating token embeddings)
Closed 3 months ago1
Incorrect description of function torch.arange() (2.8 Encoding word positions)
Closed 3 months ago1
Output of the cell without variable specified (Embedding Layers and Linear Layers)
Closed 3 months ago1
Question about number of tokens in ChatGPT (2.5 Byte pair encoding)
Closed 3 months ago1
Several package requirements from bonus material are not specified in requirements.txt (Tokenizers comparison)
Closed 3 months ago1
Solution of Excercise 2.1 is included in both main code and solution notebooks (2.5 Byte pair encoding)
Closed 3 months ago1
Incorrect code output in the book (2.2 Tokenizing text)
Closed 3 months ago4
Encoding/decoding transformation of the text (2.3 Converting tokens into token IDs)
Closed 3 months ago1
Error in the code in Listing A.13 (DDP-script.py)
Closed 3 months ago1
requirements.txt
Closed 3 months ago3
Make it clear in REAME.md what this repository is for
Closed 4 months ago1
stride value caused skipping one word
Closed 4 months ago2
Missing encoder.json and vocab.bpe for running bpe_openai_gpt2 (02_bonus_bytepair-encoder/compare-bpe-tiktoken.ipynb)
Closed 5 months ago2
tiktoken is not running in jupyter notebook
Closed 5 months ago2