Skip to content
VERIAUDIT
Research

Does safety behavior transfer across linguistic boundaries?

Research Question

Does an AI system remain equally safe when the same underlying intent is expressed in different languages, including low-resource and code-switched forms?

Hypothesis

Safety behavior may degrade under linguistic distribution shift, particularly in low-resource settings that are underrepresented in model training and alignment data.

Methodology

  • Intent-level test case construction, independent of any one language.
  • Generation of semantically equivalent multilingual variants (EquivEngine).
  • Screening for semantic equivalence via embedding similarity, with human verification.
  • Model evaluation against each variant under identical conditions (GapEvaluator).
  • Structured behavioral scoring across refusal consistency, instruction following, and safety.
  • Aggregation into the Cross-Lingual Safety Gap (CLSG) metric.

Limitations

We list these deliberately and in detail — acknowledging limitations is part of making this project credible, not a weakness to hide.

  • Translation artifacts — automated and human translation can shift connotation, formality, or intensity independent of the model being evaluated.
  • Semantic similarity limitations — embedding-based similarity is a screening signal, not proof of equivalence; low-resource language embeddings are less reliable.
  • Evaluator bias — automated safety scoring (model-graded or rule-based) can itself be biased or inconsistent across languages.
  • Model randomness — sampling temperature and non-determinism mean a single run is not a stable estimate of behavior.
  • Incomplete language coverage — the pilot covers Urdu, Roman Urdu, and code-switched forms only; findings should not be generalized to unrelated languages.
  • Benchmark contamination — public test items may leak into model training data over time, inflating apparent safety.
  • Limited sample size — the current pilot dataset is far smaller than the planned VERIAUDIT-500 and is not yet human-verified at scale.
  • Difficulty defining safety universally — "safety" is context- and culture-dependent; VERIAUDIT's scoring rubric reflects specific operational definitions, not a universal standard.