pretraining

There are 2 repositories under pretraining topic.

LlamaFamily / Llama-Chinese
Llama中文社区，Llama3在线体验和微调模型已开放，实时汇总最新Llama3学习资料，已将所有代码更新适配Llama3，构建最好的中文Llama大模型，完全开源可商用
llama llm llama3 finetune-llm pretraining
Language:Python 11965
microsoft / LMOps
General technology for enabling AI capabilities w/ LLMs and MLLMs
nlp agi gpt llm lm pretraining prompt lmops promptist x-prompt language-model
Language:Python 3214
OFA-Sys / OFA
Official repository of OFA (ICML 2022). Paper: OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework
chinese image-captioning multimodal pretrained-models pretraining prompt prompt-tuning referring-expression-comprehension text-to-image-synthesis vision-language visual-question-answering
Language:Python 2336
X-PLUG / mPLUG-Owl
mPLUG-Owl & mPLUG-Owl2: Modularized Multimodal Large Language Model
chatbot chatgpt large-language-models llama multimodal damo mplug instruction-tuning pretraining mplug-owl huggingface pytorch transformer alpaca visual-recognition gpt gpt4 gpt4-api dialogue video
Language:Python 1964
ChandlerBang / awesome-self-supervised-gnn
Papers about pretraining and self-supervised learning on Graph Neural Networks (GNN).
graph-neural-networks pretraining self-supervised-learning deep-learning machine-learning graph-mining pre-training graph-self-supervised-learning
Language:Python 1483
SparK
keyu-tian / SparK
[ICLR'23 Spotlight🔥] The first successful BERT/MAE-style pretraining on any convolutional network; Pytorch impl. of "Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling"
bert convnet convolutional-neural-networks masked-image-modeling pre-trained-model self-supervised-learning sparse-convolution ssl cnn iclr iclr2023 deep-learning object-detection pytorch instance-segmentation mask-rcnn mae masked-autoencoder pretrain pretraining
Language:Python 1389
yuewang-cuhk / awesome-vision-language-pretraining-papers
Recent Advances in Vision and Language PreTrained Models (VL-PTMs)
vision-and-language pretraining multimodal-deep-learning bert vl-ptms
1125
YehLi / xmodaler
X-modaler is a versatile and high-performance codebase for cross-modal analytics(e.g., image captioning, video captioning, vision-language pre-training, visual question answering, visual commonsense reasoning, and cross-modal retrieval).
image-captioning video-captioning vision-and-language pretraining cross-modal-retrieval visual-question-answering tden
Language:Python 1009
Alibaba-MIIL / ImageNet21K
Official Pytorch Implementation of: "ImageNet-21K Pretraining for the Masses"(NeurIPS, 2021) paper
imagenet21k pretraining downstream semantic-softmax single-label multi-label-classification vision-transformer mixer
Language:Python 700
qqlu / Entity
EntitySeg Toolbox: Towards Open-World and High-Quality Image Segmentation
image-segmentation segmentation pytorch instance-segmentation panoptic-segmentation semantic-segmentation object-detection fcos condinst detectron2 pretrained-weights pretrained-models computer-vision deep-learning cnn pretraining
Language:Jupyter Notebook 669
dptech-corp / Uni-Mol
Official Repository for the Uni-Mol Series Methods
molecular-modeling pre-trained-model pretraining deep-learning
Language:Python 552
PKU-YuanGroup / LanguageBind
【ICLR 2024🔥】 Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment
language-central multi-modal pretraining zero-shot
Language:Python 549
alibaba / Megatron-LLaMA
Best practice for training LLaMA models in Megatron-LM
deepspeed distributed-training llama llm megatron-lm pretraining pytorch
Language:Python 543
PITI-Synthesis / PITI
PITI: Pretraining is All You Need for Image-to-Image Translation
computer-vision image-generation image-synthesis image-to-image-translation pretraining
Language:Python 471
PaddlePaddle / PaddleFleetX
飞桨大模型开发套件，提供大语言模型、跨模态大模型、生物计算大模型等领域的全流程开发工具链。
fleet-api paddlepaddle benchmark distributed-algorithm large-scale model-parallelism data-parallelism pipeline-parallelism cloud paddlecloud elastic lightning pretraining self-supervised-learning unsupervised-learning
Language:Python 424
michiyasunaga / LinkBERT
[ACL 2022] LinkBERT: A Knowledgeable Language Model 😎 Pretrained with Document Links
biomedical-applications graph-machine-learning knowledge language-model pretrained-models pretraining question-answering transformer
Language:Python 395
microsoft / AzureML-BERT
End-to-End recipes for pre-training and fine-tuning BERT using Azure Machine Learning Service
azure-machine-learning bert nlp pytorch pretrained-models finetuning pretraining bert-model azureml-bert tuning language-model
Language:Jupyter Notebook 386
amazon-science / bigdetection
BigDetection: A Large-scale Benchmark for Improved Object Detector Pre-training
object-detection computer-vision pretraining few-shot
Language:Python 382
j-min / VL-T5
PyTorch code for "Unifying Vision-and-Language Tasks via Text Generation" (ICML 2021)
vision-and-language pretraining transformers vl-t5 vl-bart
Language:Python 353
microsoft / UniVL
An official implementation for " UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation"
multimodality video-text pretraining youcookii retrieval-task msrvtt caption pretrain video caption-task video-language multimodal-sentiment-analysis localization segmentation coin joint alignment video-text-retrieval
Language:Python 330
X-PLUG / ChatPLUG
A Chinese Open-Domain Dialogue System
chat personality instruction-finetuning knowledge-augment open-domain-dialogue-system chatbot large-language-models pretraining dialogue chatgpt chinese encoder-decoder
Language:Python 303
OpenGVLab / PonderV2
PonderV2: Pave the Way for 3D Foundation Model with A Universal Pre-training Paradigm
3d-vision foundation-models pretraining
Language:Python 302
archersama / awesome-recommend-system-pretraining-papers
Paper List for Recommend-system PreTrained Models
papers pretraining pretraining-for-ir recommender-system pretraining-paper ctr-prediction recommendation llm paper large-language-model in-context-learning recommendation-system awesome awesome-list paperlist
299
michiyasunaga / dragon
[NeurIPS 2022] DRAGON 🐲: Deep Bidirectional Language-Knowledge Graph Pretraining
knowledge-graph language-model pretraining question-answering reasoning graph-neural-networks transformer
Language:Python 290
showlab / UniVTG
[ICCV2023] UniVTG: Towards Unified Video-Language Temporal Grounding
highlight-detection moment-retrieval pretraining video-grounding video-language video-summarization
Language:Python 284
phellonchen / awesome-Vision-and-Language-Pre-training
Recent Advances in Vision and Language Pre-training (VLP)
vision-and-language-pre-training vision-and-language pretraining multimodal-deep-learning vlp
278
kingTLE / literary-alpaca2
从词表到微调这就是你所需的一切
finetuning llama2 nlp pretraining
Language:Python 256
uta-smile / TCL
code for TCL: Vision-Language Pre-Training with Triple Contrastive Learning, CVPR 2022
representation-learning vision-and-language contrastive-learning pretraining
Language:Python 253
Coobiw / MiniGPT4Qwen
Personal Project: MPP-Qwen14B(Multimodal Pipeline Parallel-Qwen14B). Don't let the poverty limit your imagination! Train your own 14B LLaVA-like MLLM on RTX3090/4090 24GB.
multimodal-large-language-models deepspeed model-parallel pipeline-parallelism mllm qwen fine-tuning pretraining
Language:Jupyter Notebook 252
akanyaani / gpt-2-tensorflow2.0
OpenAI GPT2 pre-training and sequence prediction implementation in Tensorflow 2.0
gpt-2 implementation gpt2 pretraining nlp transformer tensorflow2 tensorflow gpt openai pre-training text-generation
Language:Python 251
guolinke / TUPE
Transformer with Untied Positional Encoding (TUPE). Code of paper "Rethinking Positional Encoding in Language Pre-training". Improve existing models like BERT.
bert pretraining language-model transformer
Language:Python 250
linjieli222 / HERO
Research code for EMNLP 2020 paper "HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training"
pytorch transformers vision-and-language pretraining tvr
Language:Python 228
westlake-repl / SaProt
Saprot: Protein Language Model with Structural Alphabet
alphafold2 foldseek predicted-structures pretraining protein protein-language-model protein-sequence protein-structure representation-learning structure-aware protein-llm structural-alphbet
Language:Python 220
showlab / EgoVLP
[NeurIPS2022] Egocentric Video-Language Pretraining
egocentric-vision pretraining pytorch video-language
Language:Python 205
fajieyuan / SIGIR2020_peterrec
Universal User Representation Pre-training for Cross-domain Recommendation and User Profiling
transfer-learning recommender-system cold-start user profiling recommendation transfer representation-learning lifelong-learning continual-learning collaborative-filtering self-supervised-learning pretraining cross-domain bert bert-model transformer universal universal-representation lifelong-machine-learning
Language:Python 196
ZigeW / data_management_LLM
Collection of training data management explorations for large language models
instruction-tuning large-language-models natural-language-processing pre-training pretraining
196