AI Through Foucault’s Knowledge-Power
A few years ago, Amazon built an experimental system for screening resumes. The idea was simple: feed it a decade of successful hiring decisions and let it identify what a good candidate looks like. The system learned quickly. Among other things, it learned that most people hired in the past had been men, and began penalizing resumes that contained signs of womanhood, such as participation in women’s chess clubs. Amazon discovered this and abandoned the project.
The disturbing detail is not the bias itself. It is that no one taught the system to discriminate. It simply learned what “fit” looked like in history, and turned that into a norm. What happened in one lab now happens everywhere: models rank credit applications, flag suspected fraud, sort candidates, assess risk. In the Netherlands, a government system flagged thousands of families as suspected welfare fraudsters, many of them with dual nationality. Families collapsed financially, and the scandal ultimately brought down the government.
In all these cases, the system did not forbid anyone from doing anything. It only measured, ranked, and marked. That is exactly what makes these systems a perfect test case for the idea of a thinker who died before the internet became public, and who described this mechanism with unsettling precision.
Power Disguised as Measurement
Michel Foucault, the French historian and philosopher, lived in twentieth-century France after the Second World War, as modern state institutions - hospitals, schools, armies, prisons - expanded into every corner of life, and trust in them began to crack around the student revolts of 1968. He devoted his life to the question of how modern power works. The common answer was: through laws, prohibitions, and punishments. Foucault argued that this misses the main point. Modern power works largely through knowledge: through the ability to measure, document, compare, and define what counts as normal.
He showed this through the history of institutions. The asylum produced the category sane versus mad. The clinic produced healthy versus sick. The prison produced citizen versus criminal. Each time, the category was presented as scientific, objective, neutral. And each time, it organized who would be confined, treated, trusted, or excluded. Knowledge was not a mirror of reality. It was a tool that shaped reality while wearing the clothes of description.
Foucault’s central mechanism is normalization: power that acts not by saying “forbidden,” but through a norm against which everyone is measured continuously. The grade at school, the medical test, the workplace evaluation. No one forces you, exactly. You are simply measured, always, against a “normal” someone defined. Whoever defines the normal quietly shapes what everyone tries to become.
The Exam That Never Ends
If Foucault was right about schools and clinics, today’s AI systems are the perfected version of what he described. The exam no longer takes place in a classroom, at a known time, before an examiner you can see. It runs constantly, on everyone, without anyone necessarily feeling examined. Every loan application, resume, insurance claim, and credit-card movement is measured against a norm. A score is assigned, the opportunity opens or closes, and the person being measured often does not know they were measured, much less why.
This applies to speech itself as well. Content-moderation systems quietly decide which words, images, and ideas will circulate and which will be stopped at the moment of publication: normalization of discourse, not only of the people moving through it.
And classification feeds itself. In predictive policing, a system sends patrols to neighborhoods where more arrests were recorded in the past. More patrols produce more arrests, which feed the system, which justifies more patrols. The category “dangerous area” proves itself by the force of its own existence. Foucault called this the production of subjects: the category does not merely describe people. It shapes the conditions in which they live, until they begin to fit it.
But there is a deep difference between the institutions Foucault studied and the systems of today, and this is the insight the essay wants to leave behind: Foucault’s institutions had authors. Someone wrote the diagnostic criteria. Someone drafted the prison rules. The institution had an address and walls. You could criticize the definition, argue with it, protest outside it. Algorithmic normalization is the first time in history when, in many cases, no one decided what normal is. The definition was computed from past data, passed through millions of parameters, and is formulated nowhere. Power with a face can be sued. Power without an author can only be endured. You cannot protest in front of a weighted average.
The Serious Objection
Here we must make room for the other side, because it is stronger than it first appears. Human judges, human recruiters, and human clerks were never neutral. They were tired, biased, inconsistent, and their biases could not be extracted and examined. Amazon’s algorithm was caught precisely because it was code: its bias left traces, could be measured, could be proven. Formalization, even when it hides power, also leaves fingerprints. Unlike a decision-maker’s gut feeling, a model can be tested on a thousand cases, corrected, and tested again. Regulation is indeed waking up; in Europe, law has already begun classifying such systems by levels of risk.
That objection is correct, and it is exactly why the conclusion is not “tear the systems down.” The conclusion is that the central question of this age is institutional, not technological: who is allowed to examine the examiner? Bias written into code can be exposed only if someone has access to the code, authority to demand answers, and power to enforce correction. Without those three, the theoretical advantage of transparency remains theoretical, and those being measured remain alone before the score.
Whose Normal?
Foucault taught that the key question facing any knowledge system is not only “is it accurate?” but “who benefits if its definitions hold?” That is the question worth taking from here, because it is immediately usable. It applies to a citizen before the state, an employee before an evaluation system, and a small business before the algorithm that decides whether it appears in results. When a system ranks you, your business, or your employees, ask: on what past data was it trained? Whose history became the norm here? Who checked it, and what would it take to challenge it?
These systems are not going away, and many are useful. But the difference between a measuring tool and a tool of control has never been in the technology itself. It has always been in whether the measured have a way to answer back. In Foucault’s world, the first step of resistance was seeing the power. In ours, the first step is both simpler and harder: remembering that behind every “objective score” stands a normal that someone, or something, learned from a past that was not objective at all.
Sources for further reading
- Michel Foucault, Discipline and Punish
- Michel Foucault, Madness and Civilization
- Cathy O’Neil, Weapons of Math Destruction