Every AI governance framework mentions “bias” and “fairness.” Few mention that different definitions of fairness are mathematically incompatible.
You can’t have them all simultaneously. This isn’t a limitation of current technology or a problem we’ll solve with better algorithms. It’s a mathematical impossibility, proven by researchers.
Which means when someone says “our AI is fair,” you should ask: “Fair according to which definition?” Because there are multiple definitions, they contradict each other, and you have to choose.
This isn’t just academic philosophy. It has real implications for financial services institutions deploying AI for lending, credit, and other decisions where fairness is both ethically important and legally required.
The Paradox
Quick explanation of three common fairness definitions:
Demographic parity (statistical parity): The AI makes positive decisions (approve loan, grant credit) at the same rate for all demographic groups. If 60% of Group A gets approved, 60% of Group B should also get approved.
Equal opportunity (true positive rate parity): Among qualified applicants, the AI approves at the same rate across groups. If someone from Group A would succeed with the loan, someone from Group B with similar qualifications has the same chance of approval.
Predictive parity (positive predictive value parity): Among approved applicants, the success rate is the same across groups. If 90% of approved Group A applicants successfully repay, then 90% of approved Group B applicants should also successfully repay.
These all sound reasonable. They’re all common definitions of fairness in the research literature and policy discussions.
Here’s the problem: You cannot satisfy all three simultaneously (except in trivial cases where the groups have identical underlying characteristics).
This was proven mathematically by researchers. Most famously, Chouldechova (2017) and Kleinberg et al. (2017) showed the impossibility results.
You must choose which fairness definition matters most for your use case. That choice has consequences.
A Lending Scenario
Let me make this concrete with a simplified example that shows why this matters.
Bank uses AI to predict loan default risk for small business loans. Two groups of applicants: Group A (historically higher default rate, say 20%) and Group B (historically lower default rate, say 10%).
This difference might reflect historical economic inequality, different industries, geographic factors, or other structural reasons. For this example, assume it’s a real difference in the data.
The bank wants to be “fair.” What does that mean?
If you optimize for demographic parity (same approval rate for both groups):
You’ll approve 50% of each group (or whatever rate you choose). But because Group A has higher underlying default risk, you’re approving worse candidates from Group A and rejecting better candidates from Group B.
Result:
- Approved applicants from Group A default more often than approved applicants from Group B (violates predictive parity)
- Some qualified Group B applicants get rejected to maintain equal approval rates (arguably unfair to them)
- Overall default rate increases (bad for the bank)
If you optimize for equal opportunity (same true positive rate):
Among qualified applicants (those who would repay), you approve the same percentage from each group. Sounds fair - qualified people get loans at the same rate.
Result:
- But overall approval rates differ between groups because underlying default rates differ (violates demographic parity)
- This looks like discrimination (different approval rates by group) even though you’re treating qualified applicants equally
- Regulators or advocacy groups might flag this as disparate impact
If you optimize for predictive parity (approved applicants default at same rate):
You adjust thresholds so that approved Group A and Group B applicants have the same default rate. Bank risk is equal across groups.
Result:
- You’re holding Group A to a higher standard (need higher score to get approved) than Group B
- Qualified Group A applicants get rejected while less-qualified Group B applicants get approved (violates equal opportunity)
- This feels unfair to individual Group A applicants who are penalized for group statistics
There’s no solution that satisfies everyone. Each fairness definition optimizes for something different and makes different tradeoffs.
Why This Matters for FSI
Financial services operates under regulations that care deeply about fairness, but those regulations don’t always specify which fairness definition they mean.
Fair Lending Act and ECOA (US regulations) prohibit discrimination and require equal treatment. But “equal treatment” can mean:
- Same approval rate (demographic parity)
- Same treatment of qualified applicants (equal opportunity)
- Same risk level among approved applicants (predictive parity)
Different courts and regulators have emphasized different interpretations in different contexts.
Disparate impact analysis (used by regulators to assess discrimination) often looks at demographic parity - do approval rates differ by group? But that’s just one fairness definition.
You could have different approval rates (disparate impact by one definition) while maintaining equal opportunity (fairness by another definition).
Adverse action notices (required under FCRA when denying credit) must explain why. “You were denied because we needed to maintain equal approval rates across demographic groups” is not going to satisfy regulators or applicants.
The challenge: Regulations require fairness, but don’t always specify the precise mathematical definition. Different stakeholders (regulators, advocacy groups, applicants, the bank) might have different fairness intuitions.
You need to be explicit about which fairness definition you’re optimizing for and document why you made that choice.
Practical Approach
I’m not saying fairness is impossible or that you shouldn’t try. I’m saying you need to be realistic and explicit about tradeoffs.
Here’s what actually works:
Define fairness criteria upfront: Before building the system, decide which fairness definition matters most for this use case. Document your reasoning.
For lending: Equal opportunity (qualified applicants treated equally) is often the most defensible - you’re not discriminating against individuals based on group statistics. But understand this means approval rates might differ.
Test for multiple fairness metrics: Even if you optimize for one definition, measure all of them. Understand the tradeoffs you’re making.
Build a fairness dashboard:
- Approval rates by group (demographic parity)
- True positive rates by group (equal opportunity)
- Default rates among approved applicants by group (predictive parity)
- False positive and false negative rates by group
You can’t optimize all of them, but you should monitor all of them and understand the gaps.
Document your decisions: Write down which fairness definition you chose and why. When regulators ask (and they will), you can show you made an informed, deliberate choice rather than just hoping the AI is “fair” in some undefined way.
Include legal and compliance teams in this decision. It’s not just a technical choice - it has regulatory and legal implications.
Monitor outcomes in production: Fairness isn’t static. As populations change, economic conditions shift, or the model ages, fairness properties can change.
Continuously monitor your fairness metrics. If you notice degradation, investigate and address.
Be prepared to adjust: You might choose one fairness definition initially, deploy, and then discover regulators or stakeholders care more about a different definition.
Build systems flexible enough to adjust fairness criteria if needed. This might mean retraining models, adjusting decision thresholds, or implementing different fairness constraints.
Fairness is a Tradeoff, Not a Checkbox
Anyone who tells you “our AI is fair” or “just use our fairness tool” is oversimplifying.
Fairness is a set of competing definitions. You have to choose which one(s) to prioritize. That choice involves tradeoffs between different groups, between individuals and populations, between different ethical intuitions.
This is uncomfortable. We’d prefer a simple answer: “Run this fairness algorithm, check the box, done.”
Doesn’t work that way.
The best approach is transparency and intentionality:
- Be explicit about which fairness definition you’re using
- Measure and monitor multiple fairness metrics
- Document your choices and reasoning
- Monitor continuously and adjust as needed
- Don’t pretend you’ve solved fairness - acknowledge the tradeoffs
AI bias and fairness is not a solved problem. It’s a managed problem requiring ongoing attention, measurement, and adjustment.
Banks that understand this - that treat fairness as a continuous governance concern rather than a one-time checkbox - will navigate this complexity successfully.
Those that assume “fairness” is simple or that their vendor solved it for them will be surprised when regulators ask hard questions they can’t answer.
Be in the first group.