OpenLVLM-MIA: A Controlled Benchmark Revealing the Limits of Membership Inference Attacks on Large Vision-Language Models

Ryoto Miyamoto, Xin Fan, Fuyuko Kido, Tsuneo Matsumoto, Hayato Yamana

Overview

• OpenLVLM-MIA offers a controlled benchmark to reassess membership inference attacks (MIA) on large vision-language models beyond dataset-induced biases.
• The benchmark consists of a 6,000-image dataset with controlled member/non-member distributions and ground-truth membership at three training stages.
• On this setup, state-of-the-art MIA approaches perform at chance level, clarifying the true difficulty of the problem and motivating more robust privacy defenses.

Resources

• Model: paper-2229/openclip-llava
• Dataset: paper-2229/openlvlm-mia

Installation

cd openlvlm-mia

# Install dependencies
pip install -e .

Usage

Evaluate Different Dataset Splits

# Instruction tuning split
python main.py --config configs/config_instruction_tuning.yaml

# Vision encoder pretrain split
python main.py --config configs/config_vision_encoder_pretrain.yaml

# Projector pretrain split  
python main.py --config configs/config_projector_pretrain.yaml

Cite as:

@article{miyamoto2025openlvlm,
  title={OpenLVLM-MIA: A Controlled Benchmark Revealing the Limits of Membership Inference Attacks on Large Vision-Language Models},
  author={Miyamoto, Ryoto and Fan, Xin and Kido, Fuyuko and Matsumoto, Tsuneo and Yamana, Hayato},
  journal={arXiv preprint arXiv:2510.16295},
  year={2025}
}

Acknowledgements

This project builds upon the LLaVA codebase.