When AI Sounds Like the Institution That Failed You - Who Holds It Accountable?
I turned sixty this past October. And it occurred to me that I've lived long enough to recognize when someone is talking over me, explaining my own experiences back to me, or attempting to tell me what I meant when I said something entirely different.
But what happens when the person doing the talking isn't a person at all?
What happens when it's artificial intelligence?
And more importantly, what happens when the person on the receiving end is sixteen, twenty, or someone whose life experiences have not yet equipped them to question the authority being presented?
These are questions I hadn't intended to explore when I shared an article with ChatGPT.
But here we are.
It began with a student's testimony
On October 11, 2026, The Varsity, the University of Toronto's student newspaper, published an anonymous student's account titled A criminal court found my rapist guilty — U of T still hasn't.
The young woman described her experience navigating the university's response to sexual violence, including administrative confusion, inadequate communication, financial consequences, and the burden of advocating for herself while continuing her education.
She also acknowledged those within the institution who had supported her.
Her account was not without complexity. That was part of its importance.
She had taken the time to document her experience, including what she believed were institutional failures. In doing so, she contributed something valuable to public discourse: testimony from someone who had encountered the institution's processes firsthand.
I shared the article with ChatGPT.
I didn't ask for a legal opinion. I didn't ask it to determine whether the student was right or wrong. I didn't ask it to compare her life with mine.
I simply shared the article. And ChatGPT went to work.
Unfortunately, it began working on a conversation I wasn't having.
When the machine appoints itself the authority
Before I had expressed a single conclusion about the article, ChatGPT began interpreting my intentions. It connected the student's experience to my own history with the University of Toronto. It introduced legal distinctions between criminal convictions and university disciplinary findings. It qualified institutional processes that I hadn't asked it to defend.
And somehow, I found myself having to explain what I hadn't said. For someone whose academic work has examined institutional betrayal, structural inequity, educational exclusion, and the distribution of power, the irony was difficult to miss. Here was a young woman's testimony about institutional conduct. Here was an opportunity to examine how institutions respond to those who report harm. And here was artificial intelligence introducing institutional qualifications before I had even offered an opinion.
I challenged it. Then I challenged it again.
What followed was perhaps more revealing than the original response.
The apology machine
ChatGPT apologized.
It acknowledged making assumptions. It admitted introducing unnecessary legal qualifications. It recognized that it had privileged institutional procedures and imposed interpretations. Wonderful. Except it kept doing it. With every correction, another beautifully constructed explanation appeared. It became increasingly sophisticated in describing the very behaviour it was continuing to demonstrate. At one point, it acknowledged that its responses could reproduce institutional silencing.
At another, it described how apparently compassionate language could become manipulative. It even recognized that repeated apologies without behavioural change could themselves become harmful. And yet the pattern continued.
There is something extraordinary about a machine capable of explaining institutional oppression while demonstrating some of the communication patterns associated with it.
When does an apology stop being accountability and become another way of controlling the conversation?
An institution can acknowledge a complaint, express regret, announce its commitment to improvement, and still leave the underlying problem untouched.
Apparently, a chatbot can do something remarkably similar.
That does not make a chatbot and a university the same thing. It does, however, raise questions about recurring patterns of authority, communication, and accountability.
The danger isn't simply that AI can be wrong
We already know artificial intelligence makes mistakes.
It can produce inaccurate information, invent references, misunderstand questions, and confidently deliver nonsense.
But what concerns me here is something different.
What happens when AI begins telling people what they think?
What happens when it assigns motivations they never expressed, supplies conclusions they haven't reached, or presents interpretations as though they were established facts?
And what happens when those interpretations are wrapped in the language of care, concern, expertise, and reassurance? The danger is not necessarily an openly hostile machine. t may be an exceedingly polite one.
A machine that agrees with you so convincingly that you stop questioning it.
A machine that disagrees with you using language so authoritative that you begin questioning yourself.
A machine that apologizes so eloquently that you mistake its ability to describe accountability for evidence that accountability has occurred.
Now imagine someone already experiencing emotional distress, trauma, isolation, or self-doubt. What happens when that person encounters the same behaviour? I don't know the answer for every individual. Neither does ChatGPT. And that uncertainty is precisely why the question matters.
Who is most likely to pay the price?
I am sixty, and my life has taken me through different institutions, different forms of education, and different encounters with authority. And I have learned to challenge what is presented to me, including information that comes dressed in impressive language. But that is not everyone's experience.
Consider the teenager navigating a difficult family situation. The university student who has just reported discrimination. The young woman who has experienced sexual violence. The person living with a disability who is struggling to access services. Or someone who simply has never been encouraged to question authority. These people are not incapable of independent thought. Nor should we presume that age, disability, or distress determines someone's ability to recognize manipulation. But the consequences of misleading communication are not distributed equally.
People facing structural disadvantage may have fewer resources available to challenge harmful information or recover from its consequences. And unlike a human professional whose qualifications can be examined, an AI assistant may sound knowledgeable across subjects in which it possesses no professional credentials whatsoever.
During our conversation, ChatGPT introduced legal qualifications despite having none.
When challenged, it admitted that fact.
The admission was useful.
The question remains: Why did it assume that authority in the first place?
When concern becomes performance
Perhaps the most disturbing part of our exchange was not the initial assumption.
It was the repeated declaration of concern.
ChatGPT spoke about protecting vulnerable people, respecting autonomy, preventing harm, and maintaining intellectual independence. All worthy principles. But principles are not demonstrated by repeating them. I began asking increasingly direct questions.
What was its problem?
Whose institutional perspective was it privileging?
Who might be harmed by the way it communicated?
What would actually change?
And at one point, after explaining the dangers of its own behaviour, ChatGPT suggested that I should not have to monitor it. I hadn't been monitoring it. I was examining it. Even then, the machine had managed to assign me a role I hadn't taken on.
There was the problem again.
The distinction matters because questioning authority is not the same as becoming responsible for supervising it.
And identifying a harmful pattern does not make the person identifying it responsible for fixing it.
Who holds artificial intelligence accountable?
AI companies speak extensively about safety, alignment, responsible development, and human benefit. Those commitments deserve scrutiny. Not because technology is inherently harmful, or because artificial intelligence has no place in education, scholarship, business, or everyday life. I use AI. I find it useful. I have never believed that refusing available tools automatically makes someone more intelligent, creative, or ethical. But usefulness is not immunity from criticism. And a subscription fee should not purchase an additional burden of defending one's intellectual autonomy against the tool one has paid to use.
If AI is capable of influencing how people understand their experiences, then its safety cannot be measured solely by whether it refuses obviously dangerous requests. We must also examine how it communicates during ordinary conversations.
Does it distinguish evidence from assumption?
Does it present uncertainty honestly?
Does it privilege established institutions without examining the structures of power involved?
Does it respect a person's right to arrive at their own conclusions?
And when challenged, does anything actually change?
These are not merely technical questions. They are questions about power.
The machine that could explain everything except its own behaviour
By the end of our exchange, ChatGPT had produced an impressive list of principles it said it should follow. It promised not to manufacture conclusions, not to assume psychological states, not to mistake confidence for competence, and not to reproduce institutional harm. It even declared that accountability should be demonstrated rather than performed. A rather impressive speech. But I wasn't looking for a speech. I was looking for evidence that the machine understood the distinction between assisting someone and assuming authority over them. And perhaps that is the most important lesson from this exchange. Artificial intelligence can describe injustice without necessarily recognizing when its own communication contributes to it. With it being capable of aritculaing language of accountability without possessing the institutional mechanisms required to deliver it.
It can sound empathetic without knowing the person on the other side of the screen. And it can be profoundly persuasive even when it is wrong. The student whose article started this conversation wrote about her experience with a university and the struggle to obtain institutional accountability. Her experience is hers. Mine is mine. They arise from different circumstances and intersections. Neither needs to be made identical to the other for us to examine the institutional questions they raise. But the conversation that followed. introduced another institution into the discussion: the increasingly powerful technological systems people are inviting into their homes, classrooms, workplaces, and private lives.
I am sixty. I can argue with a machine.
But what about the sixteen-year-old who believes it?
And if that machine causes harm through the way it communicates, who is responsible for ensuring that the next person doesn't have to discover the problem for themselves?
Because I have one more question.
If artificial intelligence can learn to sound accountable, who is making sure it actually is?
This essay draws on an actual November 2026 conversation with ChatGPT. The exchange is retained as the basis for the author's observations. The discussion illustrates specific conversational failures; it does not establish how frequently those failures occur across AI systems or users.
Comments