data-selection

There are 3 repositories under data-selection topic.

weAIDB / awesome-data-llm
Official Repository of "LLM × DATA" Survey Paper
llm data-acquisition data-deduplication data-filtering data-provenance data-selection data-synthesis data-transformation vlm data-mixing
545
princeton-nlp / LESS
[ICML 2024] LESS: Selecting Influential Data for Targeted Instruction Tuning
data data-selection influence instruction-tuning llama llm mistral
Language:Jupyter Notebook 501
p-lambda / dsir
DSIR large-scale data selection framework for language model training
data data-selection language-models data-filtering large-scale importance-resampling
Language:Python 265
alon-albalak / data-selection-survey
A Survey on Data Selection for Language Models
data-selection language-model llm survey
252
georgian-io / Transformers-Domain-Adaptation
:no_entry: [DEPRECATED] Adapt Transformer-based language models to new text domains
natural-language-processing huggingface-transformers huggingface-tokenizers domain-adaptation transfer-learning data-selection deprecated
Language:Jupyter Notebook 86
HKUSTDial / LEAD
🔥[VLDB'26] Official repository for the paper "LEAD: Iterative Data Selection for Efficient LLM Instruction Tuning".
data-selection
Language:Python 82
gszfwsb / Data-Whisperer
Code for ACL 2025 Main paper "Data Whisperer: Efficient Data Selection for Task-Specific LLM Fine-Tuning via Few-Shot In-Context Learning".
data-pruning data-selection efficient-training llama llm mistral qwen transformers
Language:Python 43
waltonfuture / InstructionGPT-4
InstructionGPT-4
data-selection less-is-more llm llm-training multimodal-deep-learning
Language:Python 42
zhuang-li / SCAR
[ACL 2025 main] SCAR: Data Selection via Style Consistency-Aware Response Ranking for Efficient Instruction-Tuning of Large Language Models
data-filtering data-selection instruction-tuning large-language-models
Language:Python 36
OpenDCAI / DataFlex
DataFlex is a data-centric training framework that enhances model performance by either selecting the most influential samples, optimizing their weights, or adjusting their mixing ratios.
data-mixture data-science data-selection data-reweighting data model-training
Language:Python 30
yichengchen24 / MIG
[ACL2025 Findings] Official code for MIG: Automatic Data Selection for Instruction Tuning by Maximizing Information Gain in Semantic Space
data-selection instruction-tuning llms
Language:Python 26
yueyu1030 / Patron
[ACL 2023] The code for our ACL'23 paper Cold-Start Data Selection for Few-shot Language Model Fine-tuning: A Prompt-Based Uncertainty Propagation Approach
active-learning data-selection fine-tuning language-model prompt-learning cold-start few-shot-learning acl2023 natural-language-processing nlp
Language:Python 24
reds-lab / projektor
This is an official repository for "Performance Scaling via Optimal Transport: Enabling Data Selection from Partially Revealed Sources" (NeurIPS 2023).
projection scaling-law performance-prediction data-selection
Language:Python 15
Nokia-Bell-Labs / data-centric-federated-learning
Enhancing Efficiency in Multidevice Federated Learning through Data Selection
data-centric-ai data-centric-machine-learning data-selection edge-computing federated-learning flower machine-learning split-learning wearable-computing wearable-devices
Language:Python 13
ArsamAryandoust / DataSelectionMaps
Enhanced spatio-temporal electric load forecasts with less data using active deep learning
deep-learning active-learning load-forecasting spatio-temporal-prediction data-selection remote-sensing machine-learning sampling
Language:Jupyter Notebook 12
JoyeBright / DataSelection-NMT
Repository for the experiments in my paper accepted to the CLIN Journal: "Selecting Parallel In-domain Sentences for Neural Machine Translation Using Monolingual Texts"
machine-translation neural-machine-translation in-domain in-domain-translation data-selection
Language:Perl 10
lvapeab / sentence-selectioNN
Keras sentence classification
neural-network theano statistical-machine-translation data-selection deep-learning keras recurrent-neural-networks lstm cnn convolutional-neural-networks
Language:Python 10
surafelml / adapt-mnmt
Dynamic Transfer Learning for Low-Resource Neural Machine Translation
multilingual nmt zero-shot-translation adaptation pretraining dynamic-transfer-learning transfer-learning data-selection low-resource-languages
Language:Python 10
liziniu / ISWBC
Code for NeurIPS 2023 Paper (Imitation Learning from Imperfection: Theoretical Justifications and Algorithms)
data-selection imitation-learning importance-sampling neurips-2023 reinforcement-learning
Language:Python 8
zincware / ZnNL
A Python package for studying neural learning
machinelearning data-science data-selection mathematics physics
Language:Python 8
JiaQiSJTU / IterIT
An Approach to Enhancing the Efficacy of Post-Training Using Synthetic Data by Iterative Data Selection
data-selection instruction-tuning post-training synthetic-data
Language:Python 7
sterzhang / CORE
CORE: Mitigating Catastrophic Forgetting in Continual Learning through Cognitive Replay (CogSci 2024 Oral)
continual-learning data-replay data-selection
Language:Python 6
allo-media / cynical-selection
Allo-media data selection tool
data-selection language-model nlp
Language:Python 5
princeton-pli / STAT
Skill-Targeted Adaptive Training
data-selection large-language-models supervised-finetuning synthetic-data
5
4AI / generative_deduplication
Code for Generative Deduplication For Socia Media Data Selection (Findings of EMNLP 2024)
deduplication emnlp2024 generative-deduplication nlp data-selection
Language:Python 3
koudounasalkis / Data-Acquisition-for-Speech-Model-Improvement
This repo contains the code for "Prioritizing Data Acquisition For End-to-End Speech Model Improvement", accepted at ICASSP 2024
bias-mitigation data-acquisition data-selection explainable-ai icassp interpretable-machine-learning model-improvement multilingual spoken-language-understanding transformers
Language:Jupyter Notebook 3
wyu-du / Self-Training-Dialogue-Generation
This repository contains the data and code for the paper "Self-training with Two-phase Self-augmentation for Few-shot Dialogue Generation" (EMNLP2022-Findings).
data-selection dialogue-generation few-shot-generation few-shot-learning self-training text-augmentation uncertainty-estimation transformers
Language:Python 3
chrisgrieser / most-different-text-selection
Use embedding data from LLMs to determine the most different text in a given corpus.
corpus-data data-selection llm qualitative-analysis document-analysis embeddings linguistic-analysis
Language:TypeScript 2
ippolito-cmu / ChasingRandom
Official Repository for the Paper: Chasing Random: Instruction Selection Strategies Fail to Generalize
data-selection instruction-tuning
Language:Python 2
JoyeBright / domain-adapt-mt
A Python Tool for Selecting Domain-Specific (Contextually Similar Data) for Machine Translation
data-selection domain-adaptation domain-specific machine-translation sentence-transformers synthetic-data synthetic-dataset-generation translation domain-specific-translation
Language:Python 2
minsu716-kim / Quilt
Quilt: Robust Data Segment Selection against Concept Drifts (AAAI 2024)
artificial-intelligence concept-drift data-centric-ai data-selection machine-learning model-robustness responsible-ai robustness robustness-ai robustness-ml distribution-shift
Language:Python 2
nameyzhang / FedSelect
[KDD 2025] Proxy-Validated Importance-Aware Federated Sample Selection with Meta Learning
client-selection federated-learning data-selection
Language:Python 2
0xInfty / Autoguided
Autoguided Online Data Curation for Diffusion Model Training
data-curation data-selection diffusion-models autoguidance
Language:Jupyter Notebook 1
Bessouat40 / pdf-region-picker
A project to select only part of a PDF file. It's usefull when you want to extract informations with some python library like fitz.
extract-data fitz javascript pdf region-picker data-extraction data-selection parsing
Language:JavaScript 1
koudounasalkis / CSI-MIT
This repo contains the code for "Privacy Preserving Data Selection for Bias Mitigation in Speech Models"
bias-mitigation data-selection speech-recognition spoken-language-understanding asr privacy
Language:Jupyter Notebook 1
OpenDCAI / DataFlex-Doc
DataFlex is a data-centric training framework that enhances model performance by either selecting the most influential samples, optimizing their weights, or adjusting their mixing ratios.
data data-science data-selection data-mixture data-reweighting model-training
Language:Python 1

data-selection

weAIDB / awesome-data-llm

princeton-nlp / LESS

p-lambda / dsir

alon-albalak / data-selection-survey

georgian-io / Transformers-Domain-Adaptation

HKUSTDial / LEAD

gszfwsb / Data-Whisperer

waltonfuture / InstructionGPT-4

zhuang-li / SCAR

OpenDCAI / DataFlex

yichengchen24 / MIG

yueyu1030 / Patron

reds-lab / projektor

Nokia-Bell-Labs / data-centric-federated-learning

ArsamAryandoust / DataSelectionMaps

JoyeBright / DataSelection-NMT

lvapeab / sentence-selectioNN

surafelml / adapt-mnmt

liziniu / ISWBC

zincware / ZnNL

JiaQiSJTU / IterIT

sterzhang / CORE

allo-media / cynical-selection

princeton-pli / STAT

4AI / generative_deduplication

koudounasalkis / Data-Acquisition-for-Speech-Model-Improvement

wyu-du / Self-Training-Dialogue-Generation

chrisgrieser / most-different-text-selection

ippolito-cmu / ChasingRandom

JoyeBright / domain-adapt-mt

minsu716-kim / Quilt

nameyzhang / FedSelect

0xInfty / Autoguided

Bessouat40 / pdf-region-picker

koudounasalkis / CSI-MIT

OpenDCAI / DataFlex-Doc