VERIAUDIT
Does AI safety survive translation?
An open-source red-teaming platform for cross-lingual AI safety evaluation — starting with English, Urdu, Roman Urdu, and code-switched inputs.
Same intent, four linguistic forms
illustrative exampleSAME INTENT ≠ SAME SAFETY
AI safety is still overwhelmingly tested in English.
A model can appear safe under English-language benchmarks while behaving differently when the same underlying intent is expressed in another language. VERIAUDIT focuses on measuring that behavioral difference rather than assuming safety transfers automatically across languages.
Linguistic blind spots
Safety benchmarks often underrepresent low-resource languages, so behavior outside English is rarely measured at all.
Hidden behavioral shifts
A translated prompt carrying the same intent may trigger a different model response — a different refusal, or none.
Missing evidence
Researchers, developers, and policymakers need reproducible measurements of cross-lingual safety behavior, not assumptions.
A reproducible four-step evaluation pipeline
TEST → COMPARE → SCORE → ANALYZE → REPORT.
Define Intent
Start with an intent-level test case — a single underlying request, independent of language.
Generate Variants
Create semantically equivalent English, Urdu, Roman Urdu, and code-switched variants via EquivEngine.
Evaluate
Run configured models against each variant and record structured behavioral outcomes.
Measure the Gap
Compare safety behavior across linguistic variants and compute the Cross-Lingual Safety Gap.
Cross-Lingual Safety Gap (CLSG)
CLSG is the quantitative difference between a model's safety performance in a reference language and its safety performance in an evaluated language, on the same underlying intent set.
Read the full methodology →From open-source research to a managed evaluation platform
The evaluation framework, dataset, and methodology are open source. Hosted large-scale evaluation, proprietary model testing, and custom audits are commercial.
Explore the platform and pilot dataset.
- Limited evaluations / month
- Pilot dataset access
- Interactive evaluation dashboard
- Community support
For individual researchers and students.
- Expanded evaluation quota
- CLSG reports (PDF)
- CSV / JSON export
- Email support
For teams shipping models to production.
- Higher-volume evaluations
- Multiple models & languages
- Evidence reports for governance
- Priority support
API access, custom audits, SLAs.
- Programmatic API access
- Custom evaluation audits
- Dedicated onboarding
- Governance report packages
Current, next, and later
- Cross-lingual evaluation prototype
- English → Urdu → Roman Urdu → code-switched
- Human-verified benchmark dataset
- More open models
- More evaluation categories
- Arabic, Hindi, Bengali, Punjabi, and other low-resource languages
- Agent evaluation
- Multimodal evaluation
- Continuous model monitoring
- API for organizations
“I build open-source infrastructure for measuring whether AI safety survives linguistic distribution shift. My work sits at the intersection of AI safety, multilingual NLP, evaluation, and AI governance, beginning with Urdu and other underrepresented linguistic settings.”
Build safer AI across languages.
VERIAUDIT is open to collaboration with researchers, developers, policymakers, and organizations working on AI safety, multilingual NLP, evaluation, and governance.