ai-safety

There are 12 repositories under ai-safety topic.

jphall663 / awesome-machine-learning-interpretability
A curated list of awesome responsible machine learning resources.
fairness xai interpretability transparency machine-learning data-science python r awesome awesome-list machine-learning-interpretability interpretable-machine-learning interpretable-ml interpretable-ai explainable-ml ai-safety privacy-enhancing-technologies privacy-preserving-machine-learning reliable-ai secure-ml
3466
giskard
Giskard-AI / giskard
🐢 Open-Source Evaluation & Testing for LLMs and ML models
mlops ml-validation ml-testing ai-testing ai-safety ml-safety llmops ethical-artificial-intelligence responsible-ai fairness-ai trustworthy-ai llm-eval llm-evaluation rag-evaluation ai-security llm-security ai-red-team red-team-tools model-monitoring llm
Language:Python 3203
PKU-Alignment / safe-rlhf
Safe RLHF: Constrained Value Alignment via Safe Reinforcement Learning from Human Feedback
ai-safety alpaca beaver datasets deepspeed gpt large-language-models llama llm llms reinforcement-learning reinforcement-learning-from-human-feedback rlhf safe-reinforcement-learning safe-reinforcement-learning-from-human-feedback safe-rlhf safety transformer transformers vicuna
Language:Python 1175
tigerlab-ai / tiger
Open Source LLM toolkit to build trustworthy LLM applications. TigerArmor (AI safety), TigerRAG (embedding, RAG), TigerTune (fine-tuning)
classification fine-tuning llm llm-training rag ai-safety data-augmentation large-language-models aisafety
Language:Jupyter Notebook 386
agencyenterprise / PromptInject
PromptInject is a framework that assembles prompts in a modular fashion to provide a quantitative analysis of the robustness of LLMs to adversarial prompt attacks. 🏆 Best Paper Awards @ NeurIPS ML Safety Workshop 2022
ai-safety language-models ml-safety agi ai-alignment agi-alignment adversarial-attacks gpt-3 large-language-models machine-learning chain-of-thought prompt-engineering
Language:Python 275
ShengranHu / Thought-Cloning
[NeurIPS '23 Spotlight] Thought Cloning: Learning to Think while Acting by Imitating Human Thinking
ai-safety artificial-intelligence deep-learning imitation-learning reinforcement-learning pytorch
Language:Python 233
hendrycks / ethics
Aligning AI With Shared Human Values (ICLR 2021)
ai-safety machine-ethics ethical-ai gpt-3 ml-safety
Language:Python 213
normster / llm_rules
RuLES: a benchmark for evaluating rule-following in language models
ai-security gpt-4 ai-safety
Language:Python 194
lets-make-safe-ai / make-safe-ai
How to Make Safe AI? Let's Discuss! 💡|💬|🙌|📚
agi ai ai-safety artificial-general-intelligence artificial-intelligence ai-alignment
169
tomekkorbak / pretraining-with-human-feedback
Code accompanying the paper Pretraining Language Models with Human Preferences
ai-alignment ai-safety decision-transformers gpt language-models pretraining reinforcement-learning rlhf
Language:Python 167
Giskard-AI / awesome-ai-safety
📚 A curated list of papers & technical articles on AI Quality & Safety
ai ai-alignment ai-safety artificial-intelligence llm llmops machine-learning ml mlops natural-language-processing ai-quality ethical-ai ml-testing model-testing model-validation computer-vision awesome awesome-list ml-safety robustness
140
WindVChen / DiffAttack
An unrestricted attack based on diffusion models that can achieve both good transferability and imperceptibility.
ai-safety diffusion-models unrestricted-attacks adverarial-attacks transferable-attacks imperceptible-attacks diffusion-adversarial-attack
Language:Python 108
microsoft / SafeNLP
Safety Score for Pre-Trained Language Models
ai-safety fairness-ai nlp
Language:Python 89
ryoungj / ToolEmu
A language model (LM)-based emulation framework for identifying the risks of LM agents with tool use
agent ai-safety language-agent language-model large-language-models prompt-engineering
Language:Python 87
PKU-YuanGroup / Hallucination-Attack
Attack to induce LLMs within hallucinations
adversarial-attacks ai-safety deep-learning hallucinations llm llm-safety machine-learning nlp
Language:Python 81
megvii-research / FSSD_OoD_Detection
Feature Space Singularity for Out-of-Distribution Detection. (SafeAI 2021)
ood-detection anomaly-detection ai-safety out-of-distribution-detection anomaly
Language:Python 80
PKU-Alignment / beavertails
BeaverTails is a collection of datasets designed to facilitate research on safety alignment in large language models (LLMs).
ai-safety human-feedback human-feedback-data language-model large-language-model llm llms rlhf safe-rlhf safety beaver datasets gpt llama
Language:Makefile 79
EzgiKorkmaz / adversarial-reinforcement-learning
Reading list for adversarial perspective and robustness in deep reinforcement learning.
adversarial-attacks robust-machine-learning deep-reinforcement-learning adversarial-reinforcement-learning robust-reinforcement-learning ai-safety machine-learning-safety reinforcement-learning-safety safe-reinforcement-learning adversarial-policies explainable-rl robust-adversarial-reinforcement-learning reinforcement-learning-generalization multiagent-reinforcement-learning ai-alignment adversarial-machine-learning explainable-machine-learning responsible-ai meta-reinforcement-learning safe-rlhf
76
dlmacedo / entropic-out-of-distribution-detection
A project to add scalable state-of-the-art out-of-distribution detection (open set recognition) support by changing two lines of code! Perform efficient inferences (i.e., do not increase inference time) and detection without classification accuracy drop, hyperparameter tuning, or collecting additional data.
pytorch deep-learning out-of-distribution-detection out-of-distribution machine-learning trustworthy-ai ai-safety anomaly-detection novelty-detection robust-machine-learning trustworthy-machine-learning ood ood-detection osr open-set-recognition open-set
Language:Python 75
SafeAILab / RAIN
[ICLR'24] RAIN: Your Language Models Can Align Themselves without Finetuning
ai-safety alignment large-language-models
Language:Python 67
ai4ce / FLAT
[ICCV2021 Oral] Fooling LiDAR by Attacking GPS Trajectory
deep-learning point-cloud lidar adversarial-attacks 3d-object-detection ai-safety trustworthy-ai trustworthy-machine-learning 3d-perception robotics autonomous-driving gnss
Language:Python 66
dit7ya / awesome-ai-alignment
A curated list of awesome resources for getting-started-with and staying-in-touch-with Artificial Intelligence Alignment research.
awesome awesome-list ai-safety ai-alignment
57
dlmacedo / distinction-maximization-loss
A project to improve out-of-distribution detection (open set recognition) and uncertainty estimation by changing a few lines of code in your project! Perform efficient inferences (i.e., do not increase inference time) without repetitive model training, hyperparameter tuning, or collecting additional data.
classification deep-learning machine-learning open-set-recognition out-of-distribution-detection pytorch robust-machine-learning trustworthy-ai trustworthy-machine-learning uncertainty-estimation ai-safety anomaly-detection novelty-detection ood ood-detection open-set osr out-of-distribution
Language:Python 45
wesg52 / sparse-probing-paper
Sparse probing paper full code.
ai-alignment ai-safety interpretability mechanistic-interpretability
Language:Jupyter Notebook 37
StampyAI / stampy-ui
AI Safety Q&A web frontend
ai-safety
Language:TypeScript 31
yardenas / la-mbda
LAMBDA is a model-based reinforcement learning agent that uses Bayesian world models for safe policy optimization
model-based-reinforcement-learning ai-safety machine-learning reinforcement-learning constrained-optimization deep-learning safe-reinforcement-learning
Language:Python 29
ongov / AI-Principles
Alpha principles for the ethical use of AI and Data Driven Technologies in Ontario | Proposition de principes pour une utilisation éthique des technologies axées sur les données en Ontario
government open-government ai artifical-intelligence ethical-artificial-intelligence ai-safety ml data-driven-decisions machine-learning
26
riceissa / aiwatch
Website to track people, organizations, and products (tools, websites, etc.) in AI safety
aisafety ai-safety php database dataset data-portal ai-alignment mysql
Language:HTML 20
wesg52 / universal-neurons
Universal Neurons in GPT2 Language Models
ai-safety interpretability llm mechanistic-interpretability
Language:Jupyter Notebook 19
cure-lab / ContraNet
This is the official implementation of ContraNet (NDSS2022).
adversarial-attacks ai-safety defense
Language:Python 18
lancopku / Avg-Avg
[Findings of EMNLP 2022] Holistic Sentence Embeddings for Better Out-of-Distribution Detection
ai-safety natural-language-processing ood-detection robust-machine-learning trustworthy-machine-learning
Language:Python 17
tamlhp / awesome-privex
Awesome PrivEx: Privacy-Preserving Explainable AI (PPXAI)
awesome explainable-ai explanation model-explanation xai privacy-preserving-explainable-ai privacy-preserving-explanation privacy-preserving-model-explanation privacy-preserving-xai privex ai-risk risk-assessment ai-safety
17
IQTLabs / daisybell
Scan your AI/ML models for problems before you put them into production.
bias-correction bias-detection model-poison cybersecurity ai-alignment ai-assurance ai-safety
Language:Python 11
Jakobovski / ai-safety-cheatsheet
A compilation of AI safety ideas, problems, and solutions.
agi artificial-intelligence ai-safety agi-safety alignment
11
farsight
PAIR-code / farsight
In situ interactive widgets for responsible AI 🌱
ai ai-safety chatgpt chrome-extension gemini gemini-pro gpt-4 jupyter-notebook llm notebook responsible-ai
Language:TypeScript 11
jehumtine / LAWLIA
LAWLIA is an open-source computational legal framework designed to revolutionize legal reasoning and analysis. It combines the power of large language models with a structured syntactical grammar to facilitate precise legal assessments, truth values, and verdicts. LAWLIA is the future of computational jurisprudence
agents ai computational-law computational-linguistics large-language-models law legal-agent legal-framework legal-system legal-analysis legal-automation legal-computing legal-linguistics ai-safety
Language:Python 10

ai-safety

jphall663 / awesome-machine-learning-interpretability

Giskard-AI / giskard

PKU-Alignment / safe-rlhf

tigerlab-ai / tiger

agencyenterprise / PromptInject

ShengranHu / Thought-Cloning

hendrycks / ethics

normster / llm_rules

lets-make-safe-ai / make-safe-ai

tomekkorbak / pretraining-with-human-feedback

Giskard-AI / awesome-ai-safety

WindVChen / DiffAttack

microsoft / SafeNLP

ryoungj / ToolEmu

PKU-YuanGroup / Hallucination-Attack

megvii-research / FSSD_OoD_Detection

PKU-Alignment / beavertails

EzgiKorkmaz / adversarial-reinforcement-learning

dlmacedo / entropic-out-of-distribution-detection

SafeAILab / RAIN

ai4ce / FLAT

dit7ya / awesome-ai-alignment

dlmacedo / distinction-maximization-loss

wesg52 / sparse-probing-paper

StampyAI / stampy-ui

yardenas / la-mbda

ongov / AI-Principles

riceissa / aiwatch

wesg52 / universal-neurons

cure-lab / ContraNet

lancopku / Avg-Avg

tamlhp / awesome-privex

IQTLabs / daisybell

Jakobovski / ai-safety-cheatsheet

PAIR-code / farsight

jehumtine / LAWLIA