Story · The AI Screener That Scored Names
The Score Depended on What Your Name Looked Like 😮
Arjun, 13, from Kolkata, was reading an article about a large company that used AI to screen job applications. The AI had been trained on successful applications from the past 10 years and learned to give lower scores to applicants from certain communities — based on signals in the text like where they went to school, their names, and the words they used.
The AI was not programmed to discriminate. It had learned from data that reflected past discrimination. The historical pattern was: candidates from certain backgrounds had been hired less often. The AI learned that pattern and repeated it — at scale, and with a veneer of objectivity.
"But the data was just reflecting reality," Arjun said. His teacher replied: "It was reflecting a biased reality. When AI learns from biased history and applies it to the future, it does not fix the bias — it amplifies it and makes it harder to challenge because people trust the machine."
👉 This lesson explains the different types of AI bias, where they enter the pipeline, and what individuals, developers, and society can do about them.
Section 1 of 6
🔬 Four Types of AI Bias
Historical Bias
The training data reflects past inequalities. The AI learns those inequalities and projects them into the future. Example: hiring data that under-represents women in leadership becomes a model that scores women lower for leadership roles.
Representation Bias
Some groups are underrepresented in the training data, so the model performs worse for them. Example: face recognition trained mostly on lighter-skinned faces performs worse on darker skin tones.
Measurement Bias
The feature measured as a proxy is not equally valid across groups. Example: using zip code as a proxy for creditworthiness, when zip codes correlate with caste and class backgrounds due to historical housing segregation.
Aggregation Bias
A model trained on one population is applied to a different population without adjustment. Example: medical diagnostic AI trained mostly on data from Western patients applied to Indian patients with different genetic and lifestyle profiles.
Section 2 of 6
📍 Where Bias Enters the ML Pipeline
Bias can enter at any stage of building an AI system — not just in the data:
1
Data collection: If you survey only urban smartphone users, your data will not represent rural users, elderly users, or those without smartphones.
2
Labelling: Human labellers bring their own assumptions. A labeller from one culture might classify an image as "professional" or "unprofessional" differently than one from another culture.
3
Feature selection: Choosing which features to include is a human decision. Choosing to include zip code or surname as a feature can introduce proxy discrimination even if caste or class is not explicitly in the data.
4
Model training: If the training objective optimises only for overall accuracy, the model may sacrifice accuracy for minority groups to improve the majority average.
5
Deployment: A model trained and tested in one context (urban India) may be deployed in a different context (rural India) where its performance is much worse — but no one checks.
Section 3 of 6
📰 Real Published Examples of AI Bias
| System | Observed bias | Likely cause |
|---|---|---|
| Amazon's AI hiring tool (2018) | Down-scored résumés that contained the word "women's" (e.g. women's chess club). The company scrapped the tool. | Trained on 10 years of hiring data from a male-dominated tech industry — learned to replicate that male preference. |
| COMPAS (US recidivism prediction) | Black defendants falsely flagged as high-risk at nearly twice the rate of white defendants. | Trained on historical arrest data that reflected policing disparities, not actual re-offending rates. |
| Commercial face recognition systems | Error rates for darker-skinned women up to 35% higher than for lighter-skinned men (MIT Media Lab study, 2018). | Training datasets were 77–87% male, predominantly lighter-skinned — poor representation of other groups. |
| Medical AI (pulse oximetry models) | Oxygen saturation readings less accurate for patients with darker skin — leading to under-diagnosis of hypoxia. | Calibration data used to train models came mostly from lighter-skinned patients. |
India-specific concern: Most major AI systems are built and calibrated using data from the US, UK, or China. When these systems are deployed in India, their performance may differ significantly — across regions, languages, skin tones, names, and cultural contexts — in ways that the original developers never tested for.
Section 4 of 6
✅ Fairness Checklist for AI Users and Developers
Use this checklist when evaluating or building any AI system. Click each item to tick it off:
□
Who was included in the training data — and who was left out?
□
Does the model perform equally well for all groups it will be used on?
□
What proxy features are being used, and could they introduce indirect discrimination?
□
Are the errors equally distributed — or are some groups getting worse error rates?
□
Is the training context the same as the deployment context? If not, has performance been re-tested?
□
Is there a clear process for flagging and reviewing decisions the AI gets wrong?
□
For high-stakes decisions (jobs, credit, health, justice): is a human also involved in the final decision?
Section 5 of 6
🧑🤝🧑 What Can Individuals Do?
You are not an AI developer — yet. But you already have agency as a user and citizen:
- Notice the pattern: When an AI system seems to treat you or people you know differently — ask why. Is the training data representative? Is the model being applied in the right context?
- Question "objective" AI decisions: If someone tells you "the AI said so" as if that settles the matter — push back. Ask what the model was trained on, and how its accuracy compares across groups.
- Support diverse tech teams: The people who build AI shape what it learns. Teams that include more diverse perspectives are more likely to notice and correct bias. This is one reason why more diverse engineering teams produce fairer AI.
- Participate in AI literacy conversations: Schools, community groups, and policymakers need to understand AI bias. Share what you learn — the more people understand how bias enters AI, the more pressure there is to fix it.
Looking ahead: India is building one of the world's largest AI infrastructure programmes (India AI Mission). The choices made now — about what data to use, how to audit for bias, and what legal protections to create — will shape who benefits from AI in India and who does not. This is a conversation that everyone, including students, should be part of.
Section 6 of 6
🗺️ Key Vocabulary Summary
| Term | Simple meaning |
|---|---|
| Bias (in AI) | Systematic unfair treatment of certain groups, built into an AI system through data, design, or deployment choices |
| Historical bias | Bias inherited from past patterns of discrimination in the training data |
| Representation bias | Bias from underrepresenting some groups in the training dataset |
| Proxy discrimination | Using a feature that correlates with a protected characteristic (e.g. zip code, name) as an indirect form of discrimination |
| Fairness metric | A measure of whether a model performs equally across groups — e.g. equalised error rates or equal false positive rates |
| Aggregation bias | Applying a model trained on one population to a different population without checking whether the model is still valid |
⚖️ Quiz — Lesson 7
8 questions · Click your answer · Submit for your score
1. An AI hiring tool trained on historical hiring records that reflected gender discrimination will most likely:
2. "Representation bias" in an AI system means:
3. An AI credit scoring model uses a person's home neighbourhood (zip/PIN code) as a feature. This could introduce bias because:
4. At which stage of the ML pipeline can bias be introduced?
5. The MIT Media Lab study (2018) found that some commercial face recognition systems had error rates up to 35% higher for darker-skinned women than for lighter-skinned men. The most likely reason is:
6. Why is it a problem when "the AI said so" is used to justify a high-stakes decision without further review?
7. "Aggregation bias" occurs when:
8. Which of these is something a 13-year-old student can meaningfully do to address AI bias?
📝 Worksheet — Bias Audit
Tip: in the print dialog, choose "Save as PDF" to download.In your notebook, complete this exercise:
- Choose one AI system you have read about or used (e.g. a recommendation system, face unlock, chatbot, image search).
- Using the 4 bias types from Section 1, identify which type(s) are most likely to affect this system. Give a reason for each.
- Using the fairness checklist from Section 4, go through each question for your chosen system. For each question, rate: ✅ Known to be OK / ❌ Known issue / ❓ Not publicly known.
- Write 2 sentences: What would you want the company building this AI to do to make it more fair for users in India?
📋 Note for Parents and Teachers
What this lesson covers: Four types of AI bias (historical, representation, measurement, aggregation), where bias enters the ML pipeline, real published examples of harmful AI bias, a practical fairness checklist, and what individuals can do as users and citizens. This lesson is designed to build critical thinking about AI systems, not to create fear of AI.
Discussion prompts:
- "If an AI system appears to make fair, consistent decisions — does that mean it is not biased? What more would you need to know?"
- "AI in India will affect hundreds of millions of people. What responsibilities do Indian AI engineers, policymakers, and citizens have?"