Essays . Forensics . Machine Learning . Opinion . Security

On December 22, 2024 by Jonathan Zdziarski

One of the focus areas of my graduate research is artificial intelligence. In my foray into adversarial game theory, I became acquainted with AI’s value alignment problem firsthand. In the development of a strategy-theoretic AI Chess agent project, I decided that it should lose points for allowing its pieces to be in jeopardy. The change resulted in the opposite effect I had hoped – significant losses, which puzzled me at first until I realized that the agent was killing off its pieces to prevent them from being put in jeopardy. In the grander context of artificial intelligence, the potential for a catastrophic value alignment failure is all too easy to create as the result of short-sighted policies (such as my Chess strategy), or other simple miscalculations.

It’s in everyone’s best interest for AI to behave rationally, however many believe that – in the context of modern AI and deep learning – AI can never be formally verified to the degree that its actions can be deterministically predicted to be responsible. Despite this, industry has plotted a course regardless of edge cases that may sometimes be life threatening. Incredible progress in artificial intelligence has all but guaranteed it will be ubiquitous one day. There is little doubt that autonomous vehicles will eventually outperform human drivers, or that machine learning can more accurately diagnose a health problem. There is, on the other hand, great doubt that AI can ever operate with rational judgment 100% of the time, nor will it likely ever understand the ethical implications of its action. Of course, humans lack ethics and rationality as well, and so society controls this by holding us accountable for our actions. Unlike humans, however, industry is treated differently. This is particularly true with emerging technologies and even more so of those that we don’t fully understand. After all, how can one hold math accountable?

Any good ethical discussion about AI usually begins with the trolley problem, a well-known psychology challenge. If it must, is it better for an autonomous vehicle to veer left and kill one person, or veer right and kill three? What if it’s three nuns? What if it’s three violent and escaped convicts? Should the occupants of the vehicle be sacrificed by swerving into a tree, or should the vehicle run over a child chasing a ball? This is not the easy question it seems on the surface, and one of the unanswered challenges in fully autonomous driving requires solving it. The trolley problem is easily dismissed as cliché yet presents some crucial questions of ethics and accountability. Fully autonomous vehicles will one day absolve the driver of responsibility in an accident, and place responsibility onto… nobody is certain yet (as evidenced by meaningless disclaimers about keeping your hands on the wheel). We have already seen the first wave of tragic accidents as the result of AI. Autonomous vehicles cheapened down to use insufficient sensors having limited dynamic range have led to fatal accidents. A depressed teenager was seemingly led to suicide recently by a large language model. Over 3,000 reports of harms have been reported in the AI Incident Database, ranging from AI-based scams and deepfakes to corporate data leaks, mislabeling emergency wildfire updates as spam, and numerous disinformation failures.

Most, if not all, AI failures to date have been dismissed to the tune of industry accidents. The tragedies are very real, and have deeply affected people’s lives however. With billions of dollars at risk, it is in industry’s best interest to maintain the current sense of ambiguity about responsibility for AI, and continue resisting pressure to explain how their platforms work. There now also exists a controversy around copyright theft; training generative AI to produce its reference samples utilizing copyrighted training data. Here, large industry stakeholders such as OpenAI simply outweigh the clout of the millions of smaller stakeholders – the authors, musicians, and artists whose work is being duplicated. Some believe AI has already done more harm than good.

What most people are afraid of – perhaps without realizing it – is the massive, unbalanced shift in power dynamics that AI stands to create if we are not careful. Car accidents and copyright infringement feel like mere early tremors; warnings that something bigger may happen if we don’t pay attention. To understand the power dynamics in play, one must first understand some of its applications. As a security professional, my first inclination is to reframe any problem in the context of a threat model. Deterministic outcomes are central to control theory and the safety mechanisms that protect humans. In the context of autonomous driving, airbags, emergency braking systems, and crumple zones alike must have predictable outcomes. Deterministic outcomes are also an easy security target. As one contrived example, consider an autonomous driving system that can be reliably coaxed into veering away from a group of pedestrians and hitting a tree (killing the driver) or into some other group of pedestrians (killing them). In a risk assessment, this presents an opportunity for error, but also a valuable primitive for a malicious actor capable of staging an outcome. Suppose the driving system is part of a fleet used in government. Legitimate risks may include acts of terrorism, especially if protecting a diplomat or the President. Society has a short memory span when it comes to terrorism, yet it is what keeps government career professionals up at night. Now suppose risk is being assessed for a CEO, and for a civil rights activist. The range of threats all remain similar, however the power dynamic does not.

Today, the problem is simply market-driven: If you don’t like the way the vehicle behaves then don’t buy it. In the scenario just described, one could choose to buy a different vehicle. This will not always be the case. Cars that attempt to navigate the trolley problem may not be monetizable, but they can certainly be mandated through regulatory agencies – particularly if those agencies are influenced by billionaire automotive CEOs with an autonomous driving agenda. In some future, the government may have the power to mandate a level of vehicle autonomy under the unsubstantiated guise of “automotive safety”. This outcome will likely lead to another unavoidable one: deviations in programming for government such as a “secure” mode in which a vehicle will prioritize the occupants of the vehicle. This sounds sensible to a bureaucrat yet consider what it means – the government may one day require, through regulation, technology that will sacrifice the occupants of your vehicle in certain situations yet sacrifice innocent bystanders when protecting high profile individuals in the same situation. Once mandated by government, technology becomes monetizable, such a feature may be marketed as “driver protection” technology available in luxury trims, just as existing technology is today. The right to survive a car accident is quickly becoming decided solely by one’s socio-economic status – only in this case, it will be a conscious decision by the driver’s own vehicle to sacrifice the lives of the occupants.

Through the seemingly innocuous act of software engineering and a little greed-based policy, AI can easily shift the power dynamics dramatically to favor the powerful. While this is a contrived example, it is certainly a realistic one given today’s political reality. Yet we have already seen AI massively change power dynamics without hesitation in war targeting computers, autonomous weapons development, even in more subtle ways: black-box systems such as OASys (The Offender Assessment System) [1] advise judges, prisons, and probation officers on risks of a given criminal offender, the likelihood of rehabilitation versus reoffending, and even creates individualized sentencing plans. Discrimination is enabled at airports by AI, as body scanners struggle with thick hair and head coverings [2], and patient medical diagnosis systems, while of great utility, deliver a different standard of care to underrepresented populations [3], raising questions about its future role in prioritizing health care (and competing for jobs with human doctors). AI’s power dynamics are, in part, determined simply by the engineering of these systems, which is one reason why diversity and inclusion is so crucial in tech. Throw in some billionaire bureaucrats and this delicate balance becomes a significant power to wield.

Industry and academia are beginning to think about these problems, albeit far too late. Stuart Russell, Distinguished Professor at U.C. Berkeley and author of Artificial Intelligence: A Modern Approach, recently denounced the standard AI model in favor of a consensus on utility [4]. While altruistic, even this step in the right direction inevitably leads to “two wolves and a lamb voting on what’s for dinner”. At the same time, Russell makes valid and poignant points: we lack the understanding to know when AI should admit that it doesn’t know and consult a human. AI must be more about what Doug Engelbart refers to as Intelligence Augmentation (IA) [5] and treated as more of a symbiotic relationship than an automation. Even with the greatest AI minds considering the many ethical challenges, we are unable to explain how modern AI builds its circuits, nor find a suitable model by which we can agree can be trusted to perform safely and ethically without human supervision. We are incapable of controlling it, and we are incapable of convincing industry to slow down. As AI becomes ubiquitous, the ability to simply press the Reset button and re-train it is disappearing.

At the beginning of this essay, I posed the question: “how can one hold math accountable?”. Clearly, it’s not the math that is to blame, but those behind the complex systems when they fail. One should not deploy new AI technology into production until it can meet a minimum bar of rationality, validation, and forensic accounting but economics will always ensure technology lands prematurely, and without the necessary controls. We don’t yet have sufficient laws to require such safety controls today. We could be doing a whole lot more in the way of legislation. In many cases, transparency is the most important form of accountability. For example, my little Chess agent could, at a slight cost of performance, maintain a history of its hallucinations influencing the move choices it made (it’s largely how I trained it). Large language models should, at the very least, be able to identify any of its outputs if supplied as an input, and cite the sources used in the composition of the sample. Being able to identify a copyrighted work shouldn’t be all that difficult a task for an AI; plagiarism detection tools do this daily at universities. Legislative reform to acknowledge privacy as a human right and better define the ownership of metadata as belonging to the individual it describes would stand to protect from much of the data mining likely going on behind the scenes in government, finance, and advertising. Copyright reform to acknowledge generative AI outputs as derivate works would go a long way in protecting authors, musicians and artists. Regulation for autonomous vehicles should require rigorous government certification testing and reinforcement training through heuristic safeties. Some level of mandatory disclosure and certification for any AI interacting with large physical environments (such as roads) serves the best interest of the public and subject methods to much needed scrutiny.

I am quite pessimistic about the future given the unlikeliness such accountability will happen in my lifetime.

In many respects, we are all stakeholders in AI technology; if we haven’t already been affected by it, we will be. We must develop a standard for testing, disclosure, forensic accounting, and corporate responsibility if we are to benefit from AI before we are trampled by it.

[1] A compendium of research and analysis on the Offender Assessment System (OASys) 2006-2009, Mia Debidin (Editor), Ministry of Justice, December 2009

[2] TSA Agents Say They’re Not Discriminating Against Black Women, But Their Body Scanners Might Be, Brenda Medina, ProPublica, April 2019

[3] Eliminating Racial Bias in Health Care AI, Isabella Backman, Yale School of Medicine, December 2023

[4] How Not to Destroy the World with AI, Stuart Russell, CITRIS and the Banatao Institute

[5] Augmenting Human Intellect: A Conceptual Framework, Douglas C. Engelbart, Stanford Research Institute, October 1962