Generative AI (including language, image, time etc) is massive new area of data science, with its own curriculum. Below is a core curriculum for enter this field
Subject areas
- data preparation and hardware
- neural architectures
- training and loss
- prompting techniques
- efficient serving
- evaluation
- software architectures
- toolkits: Torch, Tensorflow, Jax, MXNet
- linear algebra for deep learning: tensor multiplication
- tokenization
- byte-pair encoding
- data augmentation
- GPU computation
- transformers
- encoder and decoder
- large language models (LLMs)
- CNN
- RNN
- variational auto-encoders
- masked language modeling
- dropout
- activation functions in hidden layers
- activation functions in output layer
- quantization
- loss functions
- optimizers
- training schedules
- teacher forcing
- loss spike and stabilization
- vanishing gradients
- parameter-efficient fine-tuning, e.g. LoRA
- supervised fine-tuning
- RLHF
- instruction following
- in-context learning
- zero and few-shot learning
- prompting techniques, e.g. chain-of-thought, medprompt
- LLMP output evaluation, e.g. critics, ROUGE
- hallucinations and countermeasures
- toxicity
- vectorstore and cosine similarity
- retrieval-augmented generation (RAG)
- efficient inference
- multi-agent systems
- distillation
- PPO, DPO
- CLIP
- ResNet
- twin output models: Siamese, TarNet
- ViT visual transformer