Work · ACL 2025, main conference
Fairer answers, whatever English you speak
Ask a chatbot about a privacy policy in a different variety of English and its answers get worse. We built two cooperating AI agents that close most of that gap, without retraining the model.
The problem
The same question, "Do you sell my data?", gets a better answer in standard American English than in other varieties of English. People who rely on these tools to understand how their data is used get a worse explanation of their own rights.
Answer quality (F1, higher is better), GPT-4o-mini on PrivacyQA, before our framework. Standard American English scored best and the Southwest England dialect lowest: a gap of 0.093. The dialect versions of each question were generated with Multi-VALUE, a rule-based dialect translation tool.
What we built
A pipeline of two agents that works on top of any existing model.
- The questionIn any variety of English.
- Dialect agentTranslates it into standard American English, keeping what the person meant.
- Privacy policy agentAnswers using only the relevant part of the privacy policy.
- Agreement checkThe dialect agent checks that the answer addresses the original question.
↺ If it does not, the privacy policy agent revises its answer, up to 2 times.
The result
Gap between the best and the worst dialect, GPT-4o-mini on PrivacyQA, lower is better: 80% smaller, and answers improved for every dialect. Across the three models tested, the paper reports gaps up to 82% smaller. On PolicyQA, where the gap was already small, it went from 0.029 to 0.024.
Our work
With the University of Texas at San Antonio and Tecnológico de Monterrey, we designed the two-agent framework and tested it on two privacy datasets (PrivacyQA and PolicyQA), three models (GPT-4o-mini, Llama 3.1 and DeepSeek-R1) and 50 English dialects. I am a co-author of the paper.
Đorđe Klisura, Astrid R. Bernaga Torres, Anna Karen Gárate-Escamilla, Rajesh Roshan Biswal, Ke Yang, Hilal Pataci and Anthony Rios. ACL 2025, main conference.
