pre-training

There are 5 repositories under pre-training topic.

RUCAIBox / LLMSurvey
The official GitHub page for the survey paper "A Survey of Large Language Models".
chain-of-thought chatgpt in-context-learning instruction-tuning large-language-models llm llms natural-language-processing pre-trained-language-models pre-training rlhf
Language:Python 9007
dbiir / UER-py
Open Source Pre-training Model Framework in PyTorch & Pre-trained Model Zoo
bert pre-training fine-tuning gpt chinese natural-language-processing pytorch elmo classification ner t5 unilm roberta albert clue gpt-2 model-zoo bart pegasus xlm-roberta
Language:Python 2916
modelscope / data-juicer
A one-stop data processing system to make data higher-quality, juicier, and more digestible for LLMs! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷为大语言模型提供更高质量、更丰富、更易”消化“的数据！
chinese data-analysis data-science data-visualization dataset gpt gpt-4 instruction-tuning large-language-models llama llava llm llms multi-modal nlp opendata pre-training pytorch sora streamlit
Language:Python 1547
ChandlerBang / awesome-self-supervised-gnn
Papers about pretraining and self-supervised learning on Graph Neural Networks (GNN).
graph-neural-networks pretraining self-supervised-learning deep-learning machine-learning graph-mining pre-training graph-self-supervised-learning
Language:Python 1482
modelscope / swift
ms-swift: Use PEFT or Full-parameter to finetune 200+ LLMs or 15+ MLLMs
agent awq deploy dpo finetune galore llama llama3 llava llm lora modelscope multimodal peft pissa pre-training qwen sft unsloth
Language:Python 1471
EgoAlpha / prompt-in-context-learning
Awesome resources for in-context learning and prompt engineering: Mastery of the LLMs such as ChatGPT, GPT-3, and FlanT5, with up-to-date and cutting-edge updates.
chain-of-thought chatbot chatgpt chatgpt-api cot in-context-learning language-modeling language-understanding large-language-model llm pre-training prompt prompt-based-learning prompt-design prompt-engineering prompt-learning prompt-toolkit prompt-tuning
Language:Jupyter Notebook 1371
LirongWu / awesome-graph-self-supervised-learning
Code for TKDE paper "Self-supervised learning on graphs: Contrastive, generative, or predictive"
self-supervised-learning machine-learning unsupervised-learning graph-neural-networks pre-training data-augmentation pretext-task representation-learning transfer-learning deep-learning
1292
yzhuoning / Awesome-CLIP
Awesome list for research on CLIP (Contrastive Language-Image Pre-Training).
clip contrastive-learning pre-training
1034
zjunlp / KnowLM
An Open-sourced Knowledgable Large Language Model Framework.
llama large-language-models pre-trained-language-models language-model instruction-following deep-learning chinese english instructions models reasoning gpt-3 deepspeed instruction-tuning lora pre-training bilingual pre-trained-model knowlm instructie
Language:Python 1034
Oscar
microsoft / Oscar
Oscar and VinVL
vision-and-language pre-training image-captioning vqa image-text-search oscar vinvl
Language:Python 1032
Tencent / TencentPretrain
Tencent Pre-training framework in PyTorch & Pre-trained Model Zoo
albert bart bert chinese classification clue elmo fine-tuning gpt gpt-2 model-zoo natural-language-processing ner pegasus pre-training pytorch roberta t5 unilm xlm-roberta
Language:Python 986
brightmart / bert_language_understanding
Pre-training of Deep Bidirectional Transformers for Language Understanding: pre-train TextCNN
bert-model language-model attention-is-all-you-need transformer-encoder pre-training language-understanding text-classification document-classification self-attention transfer-learning nlp question-answering textcnn fasttext
Language:Python 959
qingsongedu / Awesome-TimeSeries-SpatioTemporal-LM-LLM
A professional list on Large (Language) Models and Foundation Models (LLM, LM, FM) for Time Series, Spatiotemporal, and Event Data.
anomalydetection autoscaling deeplearning forecasting foundation-models large-language-models large-models machinelearning pre-training rca timeseries
766
ChenRocks / UNITER
Research code for ECCV 2020 paper "UNITER: UNiversal Image-TExt Representation Learning"
transformers pre-training vision-and-language pytorch
Language:Python 765
jackroos / VL-BERT
Code for ICLR 2020 paper "VL-BERT: Pre-training of Generic Visual-Linguistic Representations".
pre-training representation-learning self-supervised-learning vision-and-language bert pytorch iclr2020 vl-bert
Language:Jupyter Notebook 734
Shen-Lab / GraphCL
[NeurIPS 2020] "Graph Contrastive Learning with Augmentations" by Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, Yang Shen
graph-neural-network contrastive-learning self-supervised-learning pre-training
Language:Python 529
acbull / GPT-GNN
Code for KDD'20 "Generative Pre-Training of Graph Neural Networks"
graph-neural-networks graph-representation-learning pre-training self-supervised-learning
Language:Python 474
SalesforceAIResearch / uni2ts
Unified Training of Universal Time Series Forecasting Transformers
deep-learning forecasting machine-learning pre-trained-models pre-training representation-learning time-series time-series-forecasting transformers universal-forecasting
Language:Jupyter Notebook 470
princeton-nlp / LLM-Shearing
[ICLR 2024] Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning
efficiency llama llama2 llm nlp pre-training pruning
Language:Python 467
microsoft / XPretrain
Multi-modality pre-training
multimodal-learning pre-training multimedia computer-vision nlp
Language:Python 442
nancheng58 / Awesome-LLM4RS-Papers
Large Language Model-enhanced Recommender System Papers
large-language-models llm papers pre-training prompt-tuning recommender-system survey awesome-list gpt
412
GAIR-NLP / MathPile
Generative AI for Math: MathPile
corpus language-model large-language-models math pre-training
Language:JavaScript 347
google-research-datasets / conceptual-12m
Conceptual 12M is a dataset containing (image-URL, caption) pairs collected for vision-and-language pre-training.
vision-and-language pre-training multimodal-dataset
326
GCC
THUDM / GCC
GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training @ KDD 2020
contrastive-learning graph-neural-networks pre-training
Language:Python 319
zezhishao / STEP
Code for our SIGKDD'22 paper Pre-training-Enhanced Spatial-Temporal Graph Neural Network For Multivariate Time Series Forecasting.
multivariate-time-series graph-neural-networks pre-training traffic-forecasting
Language:Python 312
sayakpaul / probing-vits
Probing the representations of Vision Transformers.
attention image-recognition keras pre-training self-supervision tensorflow transformers vits explaining-vits
Language:Jupyter Notebook 302
linwhitehat / ET-BERT
The repository of ET-BERT, a network traffic classification model on encrypted traffic. The work has been accepted as The Web Conference (WWW) 2022 accepted paper.
pre-training transformer-architecture burst-analysis mask-burst-modeling same-origin-burst-prediction pytorch encrypted-traffic-analysis
Language:Python 284
showlab / all-in-one
[CVPR2023] All in One: Exploring Unified Video-Language Pre-training
codebase pre-training video-language pytorch
Language:Python 273
mczhuge / Kaleido-BERT
💐Kaleido-BERT: Vision-Language Pre-training on Fashion Domain
bert e-commerce fashion pre-training multimodal vision-language
Language:Python 264
wangxiao5791509 / MultiModal_BigModels_Survey
[MIR-2023-Survey] A continuously updated paper list for multi-modal pre-trained big models
anhui-university audio big-models depth event-camera multi-modal natural-language pengchenglab point-cloud pre-training radar review rgb-text-audio self-attention survey thermal-infrared transformers
252
akanyaani / gpt-2-tensorflow2.0
OpenAI GPT2 pre-training and sequence prediction implementation in Tensorflow 2.0
gpt-2 implementation gpt2 pretraining nlp transformer tensorflow2 tensorflow gpt openai pre-training text-generation
Language:Python 251
DeepGraphLearning / GearNet
GearNet and Geometric Pretraining Methods for Protein Structure Representation Learning, ICLR'2023 (https://arxiv.org/abs/2203.06125)
graph-neural-networks pre-training protein-representation-learning
Language:Python 247
Lupin1998 / Awesome-MIM
[Survey] Masked Modeling for Self-supervised Representation Learning on Vision and Beyond (https://arxiv.org/abs/2401.00897)
awesome-list awesome-mim masked-image-modeling self-supervised-learning vision-transformer masked-autoencoder computer-vision deep-learning pre-training bert gpt generative-models mae masked-modeling representation-learning
Language:Python 244
westlake-repl / Recommendation-Systems-without-Explicit-ID-Features-A-Literature-Review
Paper List of Pre-trained Foundation Recommender Models
chatgpt foundation-model llm llm4rec multimodal pre-training recommender-system transfer-learning chatgpt4rec cross-domainrecommendation gpt4rec chatgpt3 language-model multimodal-deep-learning recommendation-system transferable multimodalrecommendation cross-domain-recommendation large-language-model llm-recommendation
239
ViTAE-Transformer / SAMRS
The official repo for [NeurIPS'23] "SAMRS: Scaling-up Remote Sensing Segmentation Dataset with Segment Anything Model"
dataset deep-learning sam pre-training remote-sensing semantic-segmentation transfer-learning segment-anything-model
Language:Python 235
lucidrains / electra-pytorch
A simple and working implementation of Electra, the fastest way to pretrain language models from scratch, in Pytorch
artificial-intelligence deep-learning pre-training transformer
Language:Python 218