multi-modal

There are 16 repositories under multi-modal topic.

modelscope
modelscope / modelscope
ModelScope: bring the notion of Model-as-a-Service to life.
cv deep-learning machine-learning multi-modal nlp python science speech
Language:Python 6048
lucidrains / DALLE-pytorch
Implementation / replication of DALL-E, OpenAI's Text to Image Transformer, in Pytorch
artificial-intelligence attention-mechanism deep-learning multi-modal text-to-image transformers
Language:Python 5492
THUDM / CogVLM
a state-of-the-art-level open visual language model | 多模态预训练模型
cross-modality language-model multi-modal pretrained-models visual-language-models
Language:Python 4992
valhalla / valhalla
Open Source Routing Engine for OpenStreetMap
astar dijkstra directions isochrones multi-modal openstreetmap routing routing-engine tiled traveling-salesman
Language:C++ 4179
marqo
marqo-ai / marqo
Unified embedding generation and search engine. Also available on cloud - cloud.marqo.ai
deep-learning information-retrieval machinelearning vector-search tensor-search clip multi-modal search-engine transformers vision-language machine-learning semantic-search visual-search natural-language-processing hnsw knn hacktoberfest chatgpt gpt large-language-models
Language:Python 4116
THUDM / VisualGLM-6B
Chinese and English multimodal conversational language model | 多模态中英双语对话语言模型
chatglm-6b gpt multi-modal
Language:Python 3930
OFA-Sys / Chinese-CLIP
Chinese version of CLIP which achieves Chinese cross-modal retrieval and representation generation.
chinese clip computer-vision contrastive-loss coreml-models deep-learning image-text-retrieval multi-modal multi-modal-learning nlp pretrained-models pytorch transformers vision-and-language-pre-training vision-language
Language:Python 3611
zjunlp / DeepKE
[EMNLP 2022] An Open Toolkit for Knowledge Graph Extraction and Construction
knowledge-graph relation-extraction chinese named-entity-recognition attribute-extraction low-resource document-level information-extraction pytorch deepke ner nlp few-shot prompt deep-learning kg multi-modal lightner knowprompt instructie
Language:Python 2929
docarray
docarray / docarray
Represent, send, store and search multimodal data
docarray data-structures multimodal cross-modal neural-search deep-learning nested-data qdrant weaviate nearest-neighbor-search protobuf elasticsearch dataclass multi-modal semantic-search machine-learning pytorch fastapi pydantic
Language:Python 2740
PKU-YuanGroup / Video-LLaVA
Video-LLaVA: Learning United Visual Representation by Alignment Before Projection
instruction-tuning large-vision-language-model multi-modal
Language:Python 2378
SciSharp / LLamaSharp
A C#/.NET library to run LLM models (🦙LLaMA/LLaVA) on your local device efficiently.
chatbot gpt llama llamacpp llm semantic-kernel llava multi-modal llama2 llama3 llama-cpp
Language:C# 1878
Kav-K / GPTDiscord
A robust, all-in-one GPT interface for Discord. ChatGPT-style conversations, image generation, AI-moderation, custom indexes/knowledgebase, youtube summarizer, and more!
artificial-intelligence asyncio chatbot code-interpreter collaborate dalle2 digitalocean discord embeddings extractive-question-answering github gpt3 hacktoberfest help-wanted moderator-bot multi-modal openai openai-api pinecone python
Language:Python 1780
PKU-YuanGroup / MoE-LLaVA
Mixture-of-Experts for Large Vision-Language Models
large-vision-language-model mixture-of-experts moe multi-modal
Language:Python 1664
modelscope / data-juicer
A one-stop data processing system to make data higher-quality, juicier, and more digestible for LLMs! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷为大语言模型提供更高质量、更丰富、更易”消化“的数据！
chinese data-analysis data-science data-visualization dataset gpt gpt-4 instruction-tuning large-language-models llama llava llm llms multi-modal nlp opendata pre-training pytorch sora streamlit
Language:Python 1477
dvlab-research / LISA
Project Page for "LISA: Reasoning Segmentation via Large Language Model"
large-language-model llm multi-modal segmentation
Language:Python 1449
hcaptcha-challenger
QIN2DIM / hcaptcha-challenger
🥂 Gracefully face hCaptcha challenge with MoE(ONNX) embedded solution.
yolov5 hcaptcha opencv-python onnx-models hcaptcha-solver solver onnx yolo onnxruntime playwright clip multi-modal zero-shot-classification multi-modal-learning computer-vision image-segmentation object-detection
Language:Python 1415
DirtyHarryLYL / Transformer-in-Vision
Recent Transformer-based CV and related works.
computer-vision deep-learning multi-modal paper self-attention transformer vision-transformers visual-language
1290
OpenMotionLab / MotionGPT
[NeurIPS 2023] MotionGPT: Human Motion as a Foreign Language, a unified motion-language generation model using LLMs
3d-generation chatgpt gpt language-model motion motion-generation text-driven text-to-motion motiongpt multi-modal
Language:Python 1223
tangxyw / RecSysPapers
推荐/广告/搜索领域工业界经典以及最前沿论文集合。A collection of industry classics and cutting-edge papers in the field of recommendation/advertising/search.
calibration causal-inference cold-start contrastive-learning debias distillation diverse fairness feedback-delay look-alike match multi-modal multi-scenario multi-task papers pre-rank rank re-rank recommendation-system reinforcement-learning
Language:Python 981
MedMNIST
MedMNIST / MedMNIST
[pip install medmnist] 18x Standardized Datasets for 2D and 3D Biomedical Image Classification
dataset benchmark automl mnist medical medical-image-analysis medmnist multi-modal decathlon medical-imaging medical-image-computing deep-learning machine-learning 3d 2d classification few-shot-learning pytorch federated-learning
Language:Python 972
fastRAG
IntelLabs / fastRAG
Efficient Retrieval Augmentation and Generation Framework
nlp benchmark colbert information-retrieval semantic-search sentence-transformers summarization transformers diffusion knowledge-graph multi-modal question-answering generative-ai llm
Language:Python 897
vercel / modelfusion
The TypeScript library for building AI applications.
ai artificial-intelligence chatbot claude dall-e embedding gpt-3 huggingface javascript js llamacpp llm mistral multi-modal ollama openai stable-diffusion ts typescript whisper
Language:TypeScript 891
OpenGVLab / InternVL
InternVL Family: A Pioneering Open-Source Alternative to GPT-4V. [CVPR 2024 Oral]
image-classification image-text-retrieval llm mme multi-modal semantic-segmentation video-classification vision-language-model vit-22b vit-6b
Language:Python 832
bytedance / SALMONN
SALMONN: Speech Audio Language Music Open Neural Network
audio audio-processing large-language-models multi-modal speech speech-recognition bytedance tsinghua-university music iclr2024
Language:Python 796
jokieleung / awesome-visual-question-answering
A curated list of Visual Question Answering(VQA)(Image/Video Question Answering),Visual Question Generation ,Visual Dialog ,Visual Commonsense Reasoning and related area.
attention-networks awesome-list multi-modal multi-modal-learning vqa
642
farmvibes-ai
microsoft / farmvibes-ai
FarmVibes.AI: Multi-Modal GeoSpatial ML Models for Agriculture and Sustainability
agriculture ai geospatial geospatial-analytics stac sustainability multi-modal remote-sensing weather
Language:Jupyter Notebook 627
salesforce / UniControl
Unified Controllable Visual Generation Model
aigc generation multi-modal
Language:Python 577
PKU-YuanGroup / LanguageBind
【ICLR 2024🔥】 Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment
language-central multi-modal pretraining zero-shot
Language:Python 525
modelscope / agentscope
Start building LLM-empowered multi-agent applications in an easier way.
agent chatbot gpt-4 large-language-models llm llm-agent multi-agent distributed-agents multi-modal llama3
Language:Python 492
open-compass / VLMEvalKit
Open-source evaluation toolkit of large vision-language models (LVLMs), support GPT-4v, Gemini, QwenVLPlus, 30+ HF models, 15+ benchmarks
gpt-4v large-language-models llava multi-modal openai vqa llm openai-api mplug-owl qwen gpt minigpt4 computer-vision pytorch gpt4 chatgpt clip vit benchmark evaluation
Language:Python 364
v-iashin / SpecVQGAN
Source code for "Taming Visually Guided Sound Generation" (Oral at the BMVC 2021)
transformer vqvae gan pytorch audio-generation video-features melgan multi-modal video-understanding vggsound vas bmvc evaluation-metrics audio video
Language:Jupyter Notebook 317
boschresearch / OASIS
Official implementation of the paper "You Only Need Adversarial Supervision for Semantic Image Synthesis" (ICLR 2021)
semantic-image-synthesis gan image-to-image-translation computer-vision multi-modal generative-adversarial-networks deep-learning pytorch image-generation label-to-image-translation oasis iclr2021 machine-learning bcai
Language:Python 316
wangsuzhen / Audio2Head
code for paper "Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motion" in the conference of IJCAI 2021
talking-head multi-modal talking-face paper ijcai2021 codes
Language:Python 291
EndlessSora / TSIT
[ECCV 2020 Spotlight] A Simple and Versatile Framework for Image-to-Image Translation
generative-adversarial-network gan image-to-image-translation image-generation image-manipulation two-stream-networks versatile feature-transformation multi-scale style-transfer multi-modal semantic-image-synthesis eccv2020
Language:Python 271
kyegomez / RT-2
Democratization of RT-2 "RT-2: New model translates vision and language into action"
artificial-intelligence attention-mechanism embodied-agent gpt4 multi-modal robotics transformer
Language:Python 266
patrickjohncyh / fashion-clip
FashionCLIP is a CLIP-like model fine-tuned for the fashion domain.
nlp clip ecommerce nlp-machine-learning fashion multi-modal transformer
Language:Python 252