pineking

chameleon

Repository for Meta Chameleon, a mixed-modal early-fusion foundation model from FAIR.

mmaction2

OpenMMLab's Next Generation Video Understanding Toolbox and Benchmark

RecordRTC

RecordRTC is WebRTC JavaScript library for audio/video as well as screen activity recording. It supports Chrome, Firefox, Opera, Android, and Microsoft Edge. Platforms: Linux, Mac and Windows.

Recorder

html5 js 录音 mp3 wav ogg webm amr g711a g711u 格式，支持pc和Android、iOS部分浏览器、Hybrid App（提供Android iOS App源码）、微信，提供ASR语音识别转文字 H5版语音通话聊天示例 DTMF编码解码

pyannote-audio

Neural building blocks for speaker diarization: speech activity detection, speaker change detection, overlapped speech detection, speaker embedding

pipecat

Open Source framework for voice and multimodal conversational AI

VoiceStreamAI

Near-Realtime audio transcription using self-hosted Whisper and WebSocket in Python/JS

selfservicekiosk-audio-streaming

A best practice for streaming audio from a browser microphone to Dialogflow or Google Cloud STT by using websockets.

Awesome-Speaker-Diarization

Some comprehensive papers about speaker diarization

insanely-fast-whisper

Languagecodec

Language-Codec: Reducing the Gaps Between Discrete Codec Representation and Speech Language Models

SpeechTokenizer

This is the code for the SpeechTokenizer presented in the SpeechTokenizer: Unified Speech Tokenizer for Speech Language Models. Samples are presented on

AcademiCodec

AcademiCodec: An Open Source Audio Codec Model for Academic Research

vector-quantize-pytorch

Vector (and Scalar) Quantization, in Pytorch

audiolm-pytorch

Implementation of AudioLM, a SOTA Language Modeling Approach to Audio Generation out of Google Research, in Pytorch

descript-audio-codec

State-of-the-art audio codec with 90x compression factor. Supports 44.1kHz, 24kHz, and 16kHz mono/stereo audio.

DALLE-pytorch

Implementation / replication of DALL-E, OpenAI's Text to Image Transformer, in Pytorch

DALL-E

PyTorch package for the discrete VAE used for DALL·E.

ChatTTS

A generative speech model for daily dialogue.

speech-trident

Awesome speech/audio LLMs, representation learning, and codec models

USLM

Unified Speech Language Model for paper "SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models"(ICLR 2024)

FunCodec

FunCodec is a research-oriented toolkit for audio quantization and downstream applications, such as text-to-speech synthesis, music generation et.al.

SpeechGPT

SpeechGPT Series: Speech Large Language Models

MuLan

MuLan: Adapting Multilingual Diffusion Models for 110+ Languages (无需额外训练为任意扩散模型支持多语言能力)

MiniCPM-V

MiniCPM-Llama3-V 2.5: A GPT-4V Level Multimodal LLM on Your Phone

OneChart

[ACM'MM 2024 Oral] Official code for "OneChart: Purify the Chart Structural Extraction via One Auxiliary Token"

AdvancedLiterateMachinery

A collection of original, innovative ideas and algorithms towards Advanced Literate Machinery. This project is maintained by the OCR Team in the Language Technology Lab, Tongyi Lab, Alibaba Group.

AniPortrait

AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animation

Awesome-Chart-Understanding

A curated list of recent and past chart understanding work based on our survey paper: From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Models.

YOLO-World

[CVPR 2024] Real-Time Open-Vocabulary Object Detection