creyesp / ml-resources

ml-resources

• Agile in DS - https://eugeneyan.com/writing/data-science-and-agile-what-works-and-what-doesnt/
• Data Storyteling - https://shopifyengineering.myshopify.com/blogs/engineering/data-storytelling-shopify
• Data scientist storytelling technical presentation - https://www.susanshu.com/data-scientist-storytelling-technical-presentation
• ML interview book - https://github.com/chiphuyen/ml-interviews-book
• Problem Framing - https://developers.google.com/machine-learning/problem-framing
• Rules of ML - https://developers.google.com/machine-learning/guides/rules-of-ml
• ML Education at Uber: Frameworks Inspired by Engineering Principles - https://www.uber.com/en-PL/blog/ml-education-at-uber/
• 5 Essential Management Strategies For A Data Science Project - https://medium.com/analytics-vidhya/5-essential-management-strategies-for-a-data-science-project-d38e9c850aeb
• Machine Learning that Matters 2012
• Understanding UMAP

Python

• https://towardsdatascience.com/best-practices-for-setting-up-a-python-environment-d4af439846a
• https://towardsdatascience.com/data-scientists-guide-to-efficient-coding-in-python-670c78a7bf79
• https://www.mihaileric.com/posts/setting-up-a-machine-learning-project/
• https://towardsdatascience.com/on-writing-clean-jupyter-notebooks-abdf6c708c75
• https://venthur.de/2021-06-26-python-packaging.html
• https://medium.com/@jessicachenfan/taming-your-python-dictionaries-with-dataclasses-marshmallow-and-desert-388dbffedaec

ML Pattern design

• Design patterns (a software point of view)- https://eugeneyan.com/writing/design-patterns/
• ML design patterns - https://github.com/msaroufim/ml-design-patterns
• ML design patterns (exercide of book) https://github.com/GoogleCloudPlatform/ml-design-patterns
• Rules of Machine Learnng (Google) - https://developers.google.com/machine-learning/guides/rules-of-ml

MLOPS

Open source resources to learn about MLOps:

• Made with ML by Goku Mohandas: https://madewithml.com/
• Machine Learning Systems Design by Chip Huyen: https://stanford-cs329s.github.io/ | book
• Part of NYU course by Jacopo Tagliabue: https://github.com/jacopotagliabue/FREE_7773
• The Machine Learning Engineering book by Andriy Burkov http://www.mlebook.com/
• Full Stack Deep Learning course https://fullstackdeeplearning.com/
• Metaflow tutorials
• Building a Machine Learning Platform [Definitive Guide]

Blogs

• Reproducible deep learning course - https://www.sscardapane.it/teaching/reproducibledl/
• https://services.google.com/fh/files/misc/practitioners_guide_to_mlops_whitepaper.pdf
• https://cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning
• https://medium.com/eliiza-ai/getting-started-with-mlops-d10301cef521
  • Improvement 1: Reproducibility
  • Improvement 2: Modularity
  • Improvement #3: Centralised Caching
  • Improvement #4: Scalability
• https://medium.com/geekculture/enhancing-kubeflow-with-mlflow-8983373d0cac
  • mlflow + kubeflow
• https://winder.ai/how-to-build-a-robust-ml-workflow-with-pachyderm-and-seldon/
• https://gradientflow.com/machine-learning-model-monitoring/
  • Model Monitoring Enables Robust Machine Learning Applications
• https://www.ambiata.com/blog/2020-12-07-mlops-tools/
• https://databaseline.tech/ml-cards/
• https://towardsdatascience.com/mlflow-part-2-deploying-a-tracking-server-to-minikube-a2d6671e6455
• https://medium.com/ibm-data-ai/automate-your-machine-learning-workflow-tasks-using-elyra-and-apache-airflow-adf297adc455
• https://medium.com/everything-full-stack/machine-learning-model-serving-overview-c01a6aa3e823
• https://towardsdatascience.com/how-to-measure-data-quality-815076010b37
• https://huyenchip.com/machine-learning-systems-design/toc.html
• Machine Learning Production Pipeline - https://docs.google.com/presentation/d/1mvmJ1PnCe7lWGmSoL80CjLe7N2QpEwkU8x7l62BawME/edit#slide=id.g7eb0adee5f_0_854
• MLOps without much ops | Coveo series
• Breaking up with Flask & FastAPI: Why ML model serving requires a specialized framework
• Stack for Machine Learning
  • The Rapid Evolution of the Canonical Stack for Machine Learning - https://opendatascience.com/the-rapid-evolution-of-the-canonical-stack-for-machine-learning/
  • Navigating the MLOps tooling landscape (Part 1: The Lifecycle) - https://ljvmiranda921.github.io/notebook/2021/05/10/navigating-the-mlops-landscape/
  • Introducing TWIML’s New ML and AI Solutions Guide - https://twimlai.com/solutions/introducing-twiml-ml-ai-solutions-guide/
• Papers:
• Machine Learning Operations (MLOps): Overview, Definition, and Architecture
• Frameworks
  • https://github.com/bodywork-ml/bodywork-core
• ML platform in the industry
  • Instacart
• An overview of gradient descent optimization algorithms
• A Review of Location Encoding for GeoAI: Methods and Applications
• A step-by-step guide to using MLFlow Recipes to refactor messy notebooks
• Streamlining Machine Learning Operations (MLOps) with Kubernetes and Terraform

Feature store

• Featureform - https://www.featureform.com/embeddinghub
• Do You Really Need a Feature Store?

Serving

• https://medium.com/@apoor/serving-ml-models-with-grpc-2116cf8374dd

Drift

• https://medium.com/@elena.samuylova/my-data-drifted-whats-next-how-to-handle-ml-model-drift-in-production-78719ef007b1

Monitoring and Alerting

• A comprensive guide od ML monitoring in production
• Monitoring with Boxkite and grafana

GCP

• Vertex

  • Model Serving at Scale with Vertex AI : custom container deployment with pre and post processing - https://medium.com/@piyushpandey282/model-serving-at-scale-with-vertex-ai-custom-container-deployment-with-pre-and-post-processing-12ac62f4ce76

• ML Checklist — Best Practices for a Successful Model Deployment - https://medium.com/analytics-vidhya/ml-checklist-best-practices-for-a-successful-model-deployment-2cff5495efed

• Google MLOps template - https://github.com/GoogleCloudPlatform/mlops-with-vertex-ai

Algorithms / Technique

NLP

• Complete collection of NLP Resources - https://github.com/ivan-bilan/The-NLP-Pandect
• A Complete Guide to Natural Language Processing
• Bulk labeling - https://github.com/RasaHQ/rasalit

Embeddings

• What Are Word Embeddings for Text? - https://machinelearningmastery.com/what-are-word-embeddings/
• An implementation guide to Word2Vec using NumPy and Google Sheets - https://towardsdatascience.com/an-implementation-guide-to-word2vec-using-numpy-and-google-sheets-13445eebd281
• Word2vec from Scratch - https://jaketae.github.io/study/word2vec/
• Word2Vec Tutorial - The Skip-Gram Model (2016) - http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/
• The Illustrated Word2vec - https://jalammar.github.io/illustrated-word2vec/
• Embeddings with Word2Vec in non-NLP Contexts (Details with papers) - https://towardsdatascience.com/embeddings-with-word2vec-in-non-nlp-contexts-details-e879b093d34d
• InferSent

Word endedding

• Papers:
  • A Neural Probabilistic Language Model (2003) - https://proceedings.neurips.cc/paper/2000/file/728f206c2a01bf572b5940d7d9a8fa4c-Paper.pdf
  • Efficient Estimation of Word Representations in Vector Space (2013 word2vec) - https://arxiv.org/abs/1301.3781
  • Swivel: Improving Embeddings by Noticing What's Missing (2016 Google) - https://arxiv.org/pdf/1602.02215.pdf

Sentence Embedding

• Universal Sentence Encoder for English (Google 2018)
• Supervised Learning of Universal Sentence Representations from Natural Language Inference Data - InferSent (Facebook 2018)
• SentEval: An Evaluation Toolkit for Universal Sentence Representations (2018 Facebook)
• Multilingual Universal Sentence Encoder for Semantic Retrieval (Google 2019)
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (2019)
• Learning Thematic Similarity Metric Using Triplet Networks / wikipedia sentences similarity

Tokenizer

• SentencePiece Tokenizer Demystified (2021)- https://towardsdatascience.com/sentencepiece-tokenizer-demystified-d0a3aac19b15

Attention

• A Guide to the Encoder-Decoder Model and the Attention Mechanism - https://betterprogramming.pub/a-guide-on-the-encoder-decoder-model-and-the-attention-mechanism-401c836e2cdb
• Visualizing A Neural Machine Translation Model (Mechanics of Seq2seq Models With Attention) - https://jalammar.github.io/visualizing-neural-machine-translation-mechanics-of-seq2seq-models-with-attention/
• Attn: Illustrated Attention - https://towardsdatascience.com/attn-illustrated-attention-5ec4ad276ee3
• Attention? Attention!
• Papers:
  • Neural machine translation by jointly learning to align and translate - https://arxiv.org/pdf/1508.04025.pdf
  • Effective Approaches to Attention-based Neural Machine Translation - https://arxiv.org/pdf/1409.0473.pdf

Tansformer

BERT user self--supervice loss call next sentence prediction (NSP) ALBERT Snetence Order prediciction (SOP) wich clain that model is force to learn mode fine-grain datils ELECTRA (GAN) DistilBert (2019) TinyBert (2020) MobileBert Logformer (hybrid local en global attention)

• The Transformer Family
• Curated list of transformer (Dair)- https://github.com/dair-ai/Transformers-Recipe
• Illustrated transformer- https://jalammar.github.io/illustrated-transformer/
• Transformers Explained Visually
  • (Part 1): Overview of Functionality - https://towardsdatascience.com/transformers-explained-visually-part-1-overview-of-functionality-95a6dd460452
  • (Part 3): Multi-head Attention, deep dive - https://towardsdatascience.com/transformers-explained-visually-part-3-multi-head-attention-deep-dive-1c1ff1024853
  • (Part 2): How it works, step-by-step - https://towardsdatascience.com/transformers-explained-visually-part-2-how-it-works-step-by-step-b49fa4a64f34
• Illustrated: Self-Attention - https://towardsdatascience.com/illustrated-self-attention-2d627e33b20a
• https://towardsdatascience.com/galerkin-transformer-a-one-shot-experiment-at-neurips-2021-96efcbaefd3e
• Dive into Deep Learning: Coding Session#5 Attention Mechanism II - https://www.youtube.com/watch?v=rRQcS1O58xk
• The Illustrated Retrieval Transformer - https://jalammar.github.io/illustrated-retrieval-transformer/
• Transformers from Scratch (Brandon Rohrer 2021) - https://e2eml.school/transformers
• Transformer Recipe - https://github.com/dair-ai/Transformers-Recipe
• Code to train Language model (hugging face)- https://github.com/huggingface/transformers/tree/master/examples/pytorch/language-modeling
• BERT-ology at 100 kmph - https://thenlp.space/blog/bert-ology-at-100-kmph
• Customize transformer models to your domain - https://thenlp.space/blog/customize-transformer-models-to-your-domain
• Papers:
  • Attention Is All You Need- https://arxiv.org/pdf/1706.03762.pdf
  • Improving Language Models by Retrieving from Trillions of Tokens (DeepMind’s RETRO (Retrieval-Enhanced TRansfOrmer) Dec 2021) - https://deepmind.com/research/publications/2021/improving-language-models-by-retrieving-from-trillions-of-tokens
  • Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks (2020 ALLEN) - https://arxiv.org/pdf/2004.10964.pdf
  • Natural Language Processing (NLP) for Semantic Search Online Book (pinecone.io) - https://www.pinecone.io/learn/dense-vector-embeddings-nlp/

BERT

• Explaining BERT Simply Using Sketches - https://mlwhiz.medium.com/explaining-bert-simply-using-sketches-ba30f6f0c8cb
• How to Train a BERT Model From Scratch - https://towardsdatascience.com/how-to-train-a-bert-model-from-scratch-72cfce554fc6
• LawBERT: Towards a Legal Domain-Specific BERT? - https://towardsdatascience.com/lawbert-towards-a-legal-domain-specific-bert-716886522b49
• Distillation of BERT-Like Models: The Theory - https://towardsdatascience.com/distillation-of-bert-like-models-the-theory-32e19a02641f

Distillation

• Distillation of BERT-Like Models: The Theory
• Distillation of BERT-like models: the code

BigBird

• BigBird Research Ep. 1 - Sparse Attention Basics - https://www.youtube.com/watch?v=YvA9nqPmGWg

Courses

• http://web.stanford.edu/class/cs224n/
• https://www.coursera.org/specializations/natural-language-processing
• https://github.com/dair-ai/ML-YouTube-Courses/blob/main/README.md

REcSYs

• See more here

Reinforment learning

Next best action

• NBA - https://blog.griddynamics.com/building-a-next-best-action-model-using-reinforcement-learning/
  • https://github.com/ikatsov/tensor-house
• Next-Best-Action Recommendation https://ambiata.com/blog/2020-09-21-next-best-action-concepts/
• Bandits - https://eugeneyan.com/writing/bandits/
• Contextual bandits for ads recommendations - https://bytes.swiggy.com/contextual-bandits-for-ads-recommendations-ec210775fcf
• HuggingFace Deep Reinforcement Learning course - https://github.com/huggingface/deep-rl-class

Frameworks

• ReAgent (Facebook) - https://github.com/facebookresearch/ReAgent
• Open Multi-bandit pipeline - https://github.com/st-tech/zr-obp

Graph

• Knowledge Graphs in Natural Language Processing @ ACL 2021 - https://towardsdatascience.com/knowledge-graphs-in-natural-language-processing-acl-2021-6cac04f39761
• Graph ML in 2022: Where Are We Now? - https://towardsdatascience.com/graph-ml-in-2022-where-are-we-now-f7f8242599e0

Time Series

• https://towardsdatascience.com/temporal-convolutional-networks-the-next-revolution-for-time-series-8990af826567
• https://towardsdatascience.com/introducing-pytorch-forecasting-64de99b9ef46
• IJCAI 2021 Tutorial: Modern Aspects of Big Time Series Forecasting
• M4 Forecasting Competition: Introducing a New Hybrid ES-RNN Model (Uber) - https://eng.uber.com/m4-forecasting-competition/
• Interpretable Deep Learning for Time Series Forecasting (Google) - https://ai.googleblog.com/2021/12/interpretable-deep-learning-for-time.html
• Anomali detection on TS

Papers:

• N-BEATS: Neural basis expansion analysis for interpretable time series forecasting - https://openreview.net/pdf?id=r1ecqn4YwB
• Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting - https://arxiv.org/pdf/1907.00235.pdf
• Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting - https://arxiv.org/pdf/1912.09363.pdf

Learn to rank

• https://medium.com/swlh/ranknet-factorised-ranknet-lambdarank-explained-implementation-via-tensorflow-2-0-part-i-1e71d8923132
• https://bytes.swiggy.com/learning-to-rank-restaurants-c6a69ba4b330?gi=b000dfdf0130
• https://bendersky.github.io/res/TF-Ranking-ICTIR-2019.pdf

Search

• https://medium.com/@shahrukhx01/keyword-search-using-dense-vectors-filter-recommendation-using-deep-learning-and-haystack-5b242de176d1

ONESHOT

• https://medium.com/@crimy/one-shot-learning-siamese-networks-and-triplet-loss-with-keras-2885ed022352
• https://medium.datadriveninvestor.com/nlp-in-healthcare-entity-linking-48845a762ed7
• https://bytes.swiggy.com/find-my-food-semantic-embeddings-for-food-search-using-siamese-networks-abb55be0b639 (Michel)
• https://towardsdatascience.com/interpreting-semantic-text-similarity-from-transformer-models-ba1b08e6566c

Constrastive learning (supervised / self-supervised)

Contrastive learning is a self-supervised, task-independent deep learning technique that allows a model to learn about data, even without labels.

• Understanding Contrastive Learning - https://towardsdatascience.com/understanding-contrastive-learning-d5b19fd96607
• Contrastive Representation Learning - https://lilianweng.github.io/lil-log/2021/05/31/contrastive-representation-learning.html
• Introduction to Dense Text Representations - https://www.youtube.com/watch?v=t4Gf4LruVZ4&list=PL7kaex1gKh6BDLHEwEeO45wZRDm5QlRil
  • Global and local structute of vector space
  • Losses: Multiple Negative Ranking Loss (Training with in-batch negative InfoNCE or NTXentloss) / Batch Hard Triplet Loss / Triplet Loss / Contrative loss / CosineSimilarity loss

• The InfoNCE loss in self-supervised learning (deeplearning) - https://crossminds.ai/video/the-infonce-loss-in-self-supervised-learning-606fef0bf43a7f2f827c1583/

• Papers:
  • 2019
    • Self-Supervised Learning of Pretext-Invariant Representations FAIR
  • 2020
    • Momentum Contrast for Unsupervised Visual Representation Learning `FAIR``
    • SimCLR - A Simple Framework for Contrastive Learning of Visual Representations Google Brain
    • SimCRLv2 - Big Self-Supervised Models are Strong Semi-Supervised Learners
  • 2022
    • Data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language Facebook
• https://github.com/voidism/DiffCSE

Others

• Rethinking Pre-training and Self-training
• Confident Learning: Estimating Uncertainty in Dataset Labels

Applied ml in the industry (papers)

• https://github.com/eugeneyan/applied-ml

Producto categorization

• Deep Learning: Product Categorization and Shelving - https://medium.com/walmartglobaltech/deep-learning-product-categorization-and-shelving-630571e81e96
• Semantic Vector Search: Tales from the Trenches - https://medium.com/grensesnittet/semantic-vector-search-tales-from-the-trenches-fa8b61ea3680

Attribute extractyion in a e-commerce

• Learning to Extract Attribute Value from Product via Question Answering: A Multi-task Approach (2020 Google)

Product matching

Papers:

• Product Matching in eCommerce using deep learning (medium)
• Neural Network based Extreme Classification and Similarity Models for Product Matching (Ebay)
• BERT-based similarity learning for product matching
• Deep Entity Matching with Pre-Trained Language Models (Megagon Labs)
• Salient Phrase Aware Dense Retrieval: Can a Dense Retriever Imitate a Sparse One?

Entity matching

• DeepMatch https://github.com/anhaidgroup/deepmatcher
• DeepER http://www.vldb.org/pvldb/vol11/p1454-ebraheem.pdf
• EMTA https://github.com/brunnurs/entity-matching-transformer
• Auto-EM https://www.microsoft.com/en-us/research/uploads/prod/2019/04/Auto-EM.pdf
• Ditto https://arxiv.org/pdf/2004.00584.pdf
• GROOV (facebook) - https://arxiv.org/pdf/2209.06148.pdf

Foodbert

• http://pic2recipe.csail.mit.edu/
• https://github.com/ChantalMP/Exploiting-Food-Embeddings-for-Ingredient-Substitution
• https://github.com/chambliss/foodbert
• https://deepnote.notion.site/NLP-in-Notebooks-Competition-6616e415f0a44e5c95982e7bc1cb89dd
• Paper:
  • Exploiting Food Embeddings for Ingredient Substitution - https://www.scitepress.org/Papers/2021/102020/102020.pdf

item2vec

• Moving Beyond Meta for Better Product Embeddings (MET) - https://medium.com/1mgofficial/moving-beyond-meta-better-product-embeddings-for-better-recommendations-fa6dd1578777
• Item2Vec: Neural Item Embeddings to enhance recommendations - https://tech.olx.com/item2vec-neural-item-embeddings-to-enhance-recommendations-1fd948a6f293
• Papers:
  • Product recommendation at scale (prod2vec yahoo) - https://dl.acm.org/doi/pdf/10.1145/2783258.2788627
  • item2vec (2016) - https://arxiv.org/pdf/1603.04259.pdf
  • Meta-Prod2Vec - Product Embeddings Using Side-Information for Recommendation (2016)- https://arxiv.org/pdf/1607.07326.pdf
  • Billion-scale Commodity Embedding for E-commerce Recommendation in Alibaba (2018): https://arxiv.org/pdf/1803.02349.pdf
  • Deep neural network marketplace recommenders in online experiments by Avito - https://arxiv.org/pdf/1809.02130.pdf
  • BERTSCORE: EVALUATING TEXT GENERATION WITH BERT (2019) - https://arxiv.org/pdf/1904.09675.pdf

XIA

• Explainability and Auditability in ML: Definitions, Techniques, and Tools - https://neptune.ai/blog/explainability-auditability-ml-definitions-techniques-tools
• The right way to compute your Shapley Values - https://towardsdatascience.com/the-right-way-to-compute-your-shapley-values-cfea30509254
• A Brief Overview of Methods to Explain AI (XAI) - https://towardsdatascience.com/a-brief-overview-of-methods-to-explain-ai-xai-fe0d2a7b05d6

Ab testing

Bayesian A/B Testing for Business Decisions Statistical Challenges in Online Controlled Experiments: A Review of A/B Testing Methodology

Frameworks

Tensorflow

• https://www.tensorflow.org/guide/keras/preprocessing_layers
• https://www.tensorflow.org/api_docs/python/tf/keras/layers/StringLookup#adapt
• https://towardsdatascience.com/tensorflow-template-for-deep-learning-beginners-3b976d0ee084

Pytorch

• Declarative Deep Learning - https://medium.com/pytorch/ludwig-on-pytorch-1241776417fc

Education

MLE certification

• https://sathishvj.medium.com/notes-from-my-google-cloud-professional-machine-learning-engineer-certification-exam-2110998db0f5
• https://towardsdatascience.com/how-i-passed-the-gcp-professional-ml-engineer-certification-47104f40bec5
• https://cloud.google.com/training/machinelearning-ai?skip_cache=true
• https://www.coursera.org/specializations/machine-learning-engineering-for-production-mlops
• https://www.tensorflow.org/certificate

Course

• University of Amsterdam Master
• Dive into Deep Learning
• Full stack deep learning
• Machine Learning University Amazon github
• Aman Chadha resources
• Standform Tranformers
• DEEP LEARNING - SPRING 2020 - NYU CENTER FOR DATA SCIENCE
• Stanford CS330: Deep Multi-Task & Meta Learning I Autumn 2021I
• Deep learnign fundamental Pytorch ligthning
• Applied Deep Learning Course
• Stat Rethinking

Other topics

• State of AI Report 2021 - https://www.stateof.ai/
• https://medium.datadriveninvestor.com/why-machine-learning-engineers-are-replacing-data-scientists-769d81735553
• https://datatalks.club/books.html