跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-16AI 評分44

Harnessing Textual Refusal Directions for Multimodal Safety

Harnessing Textual Refusal Directions for Multimodal Safety

AI 導讀

MARS 是一種免訓練方法,利用從 LLM 主幹提取的文本拒絕方向,在無須多模態安全數據的情況下引導多模態模型提升安全性。該方法在五個 SOTA 多模態大語言模型上接受安全性、效用及影片越獄評測,提升安全性的同時保留效用,但可能出現過度拒絕。

正文

Published on Oct 16

·

Submitted by

Moreno

on Oct 20

Authors:

,

Abstract

MARS enables multimodal safety steering using only text-derived refusal directions, avoiding the need for multimodal safety data.

Safety in Multimodal Large Language Models (MLLMs) depends on unsafe multimodal data for post-training alignment or activation steering. However, such data is hard to collect, must be gathered anew for each modality, and its collection and distribution raise ethical and legal concerns. In this work, we relax this constraint and investigate whether textual refusal directions, extracted directly from the LLM backbone, generalize across modalities (i.e., image, video). Preliminary findings confirm this ability, though effectiveness is conditioned by layer selection, steering strength, and cross-modal alignment, with the latter causing safe multimodal inputs to be spuriously steered toward refusal. Building on this, we introduce Modality-Agnostic Refusal Steering (MARS), a lightweight training-free approach that injects multimodal safety without the need for multimodal safety data. MARS corrects modality misalignment via activation centering, adaptively scales steering strength within a geometrically defined trust region, and selects the optimal intervention layer, operating at the first generated token. Evaluated on five SOTA MLLMs across safety, utility, and video jailbreak, MARS achieves consistent safety gains while preserving utility up to over-refusal. MARS reveals that safety-relevant structure is shared across modalities and that textual refusal directions are a powerful and underexplored foundation for multimodal alignment.

View arXiv page View PDF GitHub Add to collection

Get this paper in your agent:

hf papers read 2606.31876

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2606.31876 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2606.31876 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2606.31876 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co