pre-training

There are 6 repositories under pre-training topic.

RUCAIBox / LLMSurvey
The official GitHub page for the survey paper "A Survey of Large Language Models".
chain-of-thought chatgpt in-context-learning instruction-tuning large-language-models llm llms natural-language-processing pre-trained-language-models pre-training rlhf
Language:Python 11811
modelscope / data-juicer
Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷
data data-analysis data-pipeline data-processing data-science data-visualization foundation-models instruction-tuning large-language-models llm llms multi-modal pre-training synthetic-data
Language:Python 5192
dbiir / UER-py
Open Source Pre-training Model Framework in PyTorch & Pre-trained Model Zoo
albert bart bert chinese classification clue elmo fine-tuning gpt gpt-2 model-zoo natural-language-processing ner pegasus pre-training pytorch roberta t5 unilm xlm-roberta
Language:Python 3080
ChandlerBang / awesome-self-supervised-gnn
Papers about pretraining and self-supervised learning on Graph Neural Networks (GNN).
graph-neural-networks pretraining self-supervised-learning deep-learning machine-learning graph-mining pre-training graph-self-supervised-learning
Language:Python 1685
EgoAlpha / prompt-in-context-learning
Awesome resources for in-context learning and prompt engineering: Mastery of the LLMs such as ChatGPT, GPT-3, and FlanT5, with up-to-date and cutting-edge updates. - Professor Yu Liu
chain-of-thought chatbot chatgpt chatgpt-api cot in-context-learning language-modeling large-language-model llm pre-training prompt prompt-based-learning prompt-engineering prompt-learning ai-agent
Language:Jupyter Notebook 1643
LirongWu / awesome-graph-self-supervised-learning
Code for TKDE paper "Self-supervised learning on graphs: Contrastive, generative, or predictive"
self-supervised-learning machine-learning unsupervised-learning graph-neural-networks pre-training data-augmentation pretext-task representation-learning transfer-learning deep-learning
1409
zjunlp / KnowLM
An Open-sourced Knowledgable Large Language Model Framework.
bilingual chinese deep-learning deepspeed english gpt-3 instructie instruction-following instruction-tuning instructions knowlm language-model large-language-models llama lora models pre-trained-language-models pre-trained-model pre-training reasoning
Language:Python 1341
SalesforceAIResearch / uni2ts
Unified Training of Universal Time Series Forecasting Transformers
deep-learning forecasting machine-learning pre-trained-models pre-training representation-learning time-series time-series-forecasting transformers universal-forecasting
Language:Jupyter Notebook 1251
yzhuoning / Awesome-CLIP
Awesome list for research on CLIP (Contrastive Language-Image Pre-Training).
clip contrastive-learning pre-training
1215
qingsongedu / Awesome-TimeSeries-SpatioTemporal-LM-LLM
A professional list on Large (Language) Models and Foundation Models (LLM, LM, FM) for Time Series, Spatiotemporal, and Event Data.
anomalydetection autoscaling deeplearning forecasting foundation-models large-language-models large-models machinelearning pre-training rca timeseries
1135
Tencent / TencentPretrain
Tencent Pre-training framework in PyTorch & Pre-trained Model Zoo
albert bart bert chinese classification clue elmo fine-tuning gpt gpt-2 model-zoo natural-language-processing ner pegasus pre-training pytorch roberta t5 unilm xlm-roberta
Language:Python 1078
Oscar
microsoft / Oscar
Oscar and VinVL
image-captioning image-text-search oscar pre-training vinvl vision-and-language vqa
Language:Python 1049
brightmart / bert_language_understanding
Pre-training of Deep Bidirectional Transformers for Language Understanding: pre-train TextCNN
bert-model language-model attention-is-all-you-need transformer-encoder pre-training language-understanding text-classification document-classification self-attention transfer-learning nlp question-answering textcnn fasttext
Language:Python 965
ChenRocks / UNITER
Research code for ECCV 2020 paper "UNITER: UNiversal Image-TExt Representation Learning"
transformers pre-training vision-and-language pytorch
Language:Python 792
jackroos / VL-BERT
Code for ICLR 2020 paper "VL-BERT: Pre-training of Generic Visual-Linguistic Representations".
pre-training representation-learning self-supervised-learning vision-and-language bert pytorch iclr2020 vl-bert
Language:Jupyter Notebook 744
nancheng58 / Awesome-LLM4RS-Papers
Large Language Model-enhanced Recommender System Papers
awesome-list gpt large-language-models llm papers pre-training prompt-tuning recommender-system survey
708
princeton-nlp / LLM-Shearing
[ICLR 2024] Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning
efficiency llama llama2 llm nlp pre-training pruning
Language:Python 630
cxcscmu / Craw4LLM
Official repository for "Craw4LLM: Efficient Web Crawling for LLM Pretraining"
crawler crawling large-language-models llm pre-training pretraining web-crawler web-crawling
Language:Python 607
Shen-Lab / GraphCL
[NeurIPS 2020] "Graph Contrastive Learning with Augmentations" by Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, Yang Shen
graph-neural-network contrastive-learning self-supervised-learning pre-training
Language:Python 603
linwhitehat / ET-BERT
The repository of ET-BERT, a network traffic classification model on encrypted traffic. The work has been accepted as The Web Conference (WWW) 2022 accepted paper.
burst-analysis encrypted-traffic-analysis mask-burst-modeling pre-training pytorch same-origin-burst-prediction transformer-architecture
Language:Python 527
microsoft / XPretrain
Multi-modality pre-training
computer-vision multimedia multimodal-learning nlp pre-training
Language:Python 506
acbull / GPT-GNN
Code for KDD'20 "Generative Pre-Training of Graph Neural Networks"
graph-neural-networks graph-representation-learning pre-training self-supervised-learning
Language:Python 492
GAIR-NLP / MathPile
[NeurlPS D&B 2024] Generative AI for Math: MathPile
corpus language-model large-language-models math pre-training
Language:Python 417
GestaltCogTeam / STEP
Code for our SIGKDD'22 paper Pre-training-Enhanced Spatial-Temporal Graph Neural Network For Multivariate Time Series Forecasting.
multivariate-time-series graph-neural-networks pre-training traffic-forecasting
Language:Python 409
google-research-datasets / conceptual-12m
Conceptual 12M is a dataset containing (image-URL, caption) pairs collected for vision-and-language pre-training.
vision-and-language pre-training multimodal-dataset
386
westlake-repl / Recommendation-Systems-without-Explicit-ID-Features-A-Literature-Review
Paper List of Pre-trained Foundation Recommender Models
chatgpt foundation-model llm llm4rec multimodal pre-training recommender-system transfer-learning chatgpt4rec cross-domainrecommendation gpt4rec chatgpt3 language-model multimodal-deep-learning recommendation-system transferable multimodalrecommendation cross-domain-recommendation large-language-model llm-recommendation
363
ViTAE-Transformer / SAMRS
The official repo for [NeurIPS'23] "SAMRS: Scaling-up Remote Sensing Segmentation Dataset with Segment Anything Model"
dataset deep-learning sam pre-training remote-sensing semantic-segmentation transfer-learning segment-anything-model
Language:Python 357
Lupin1998 / Awesome-MIM
[Survey] Masked Modeling for Self-supervised Representation Learning on Vision and Beyond (https://arxiv.org/abs/2401.00897)
awesome-list awesome-mim masked-image-modeling self-supervised-learning vision-transformer masked-autoencoder computer-vision deep-learning pre-training bert gpt generative-models mae masked-modeling representation-learning
Language:Python 343
ZigeW / data_management_LLM
Collection of training data management explorations for large language models
instruction-tuning large-language-models natural-language-processing pre-training pretraining
334
sayakpaul / probing-vits
Probing the representations of Vision Transformers.
attention image-recognition keras pre-training self-supervision tensorflow transformers vits explaining-vits
Language:Jupyter Notebook 330
GCC
THUDM / GCC
GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training @ KDD 2020
contrastive-learning graph-neural-networks pre-training
Language:Python 326
OpenDriveLab / ViDAR
[CVPR 2024 Highlight] Visual Point Cloud Forecasting
autonomous-driving point-cloud-forecasting pre-training world-model
Language:Python 305
wangxiao5791509 / MultiModal_BigModels_Survey
[MIR-2023-Survey] A continuously updated paper list for multi-modal pre-trained big models
audio depth event-camera multi-modal natural-language pengchenglab point-cloud pre-training radar review self-attention survey thermal-infrared transformers anhui-university big-models rgb-text-audio
286
DeepGraphLearning / GearNet
GearNet and Geometric Pretraining Methods for Protein Structure Representation Learning, ICLR'2023 (https://arxiv.org/abs/2203.06125)
graph-neural-networks pre-training protein-representation-learning
Language:Python 284
showlab / all-in-one
[CVPR2023] All in One: Exploring Unified Video-Language Pre-training
codebase pre-training pytorch video-language
Language:Python 282
metauto-ai / Kaleido-BERT
💐Kaleido-BERT: Vision-Language Pre-training on Fashion Domain
bert pre-training multimodal vision-language
Language:Python 269