The AI Act [Part 2] Addressing the Challenge of AI Uncontrollability — Interview with Chuo University Professor Hirano [Bridging the Humanities and Sciences]

While AI (artificial intelligence) has the potential to make our lives more convenient, it also carries the risk of exerting control over us. In what situations should we embrace AI, and when should we be cautious about relying on it? Science Portal spoke with Professor Susumu Hirano at Chuo University's Faculty of Global Informatics, an expert in AI and law, who also holds a law license in New York State.

Using film in the classroom brings fresh perspectives from students

— So, one of your courses is titled "Film and Law?"

I love movies (laughs). I use science fiction movies such as "Blade Runner," "RoboCop," and "2001: A Space Odyssey" in my classes. I ask students how they interpret the replicant's line, "Dying for the right cause. It's the most human thing we can do " (Blade Runner), what they think of the idea that robot soldiers may be more humane than human soldiers because they can more accurately distinguish non-combatants in urban warfare (RoboCop), and why the AI HAL 9000 chose to eliminate the astronauts (2001: A Space Odyssey). Many students have never seen these films, so our discussions often produce fresh and unexpected perspectives.

Opacity, a lack of accountability, and trade secrets…

AI also has a number of other inherent limitations. One example is "opacity," meaning a lack of transparency. This is commonly referred to as the "black box" problem. Opacity refers to the difficulty of understanding why an AI system produces a particular prediction, recommendation, or decision. The underlying process may be too complex to interpret, or the training data may be unknown. HAL 9000 also serves as a symbol of this problem.

Closely related to opacity is what might be called a lack of accountability. This refers to the inability to adequately explain why a particular output was produced. One of the causes is opacity. If the reasoning behind an output cannot be understood, it becomes difficult, if not impossible, to explain it.

Another factor contributing to this lack of accountability is the protection of trade secrets. For example, when someone who has been harmed by an AI system seeks damages from the company that provided it, access to information such as the system's source code may be necessary to determine whether the AI made an error. However, critics in the United States have pointed out that if AI providers can withhold such information by invoking trade secret protections, it may become impossible to fully investigate what went wrong, undermining both accountability and access to justice.

AI should not be used lightly in life-changing decisions

— So trade-secret concerns also come into play.

One example of this issue arose in Houston, where a court ruled that the algorithm in question could no longer be used. In that case, the city's independent school district terminated employment of teachers who were rated the lowest by using a vendor-provided algorithm. However, the school district could not explain to the teachers why and how they were rated the lowest.

The reason the school district could not explain how the lowest rating was given was that the vendor refused to disclose the source code and other information, even to the district, on the grounds of trade secrets. The dismissed teachers and their union filed a lawsuit seeking an injunction to stop the use of the algorithm, arguing that it violated constitutional due-process protections because it could not adequately explain its decisions. The court concluded that a system incapable of providing an adequate explanation could not be used in such circumstances. As this case illustrates, when AI is used to make decisions that significantly affect people's lives or their rights, it is important to recognize that a lack of accountability may raise constitutional concerns.

By the way, the uncontrollability of AI, together with opacity, is one of the reasons why people hesitate to use AI robotic weapons. How can we entrust a weapon to something whose actions we cannot reliably predict? It may attack civilians, or friendly troops. If a system behaves improperly but its decision-making process is opaque, it may be impossible to determine how it should be improved. According to the International Committee of the Red Cross, such a system is unsuitable for life-and-death situations.

I believe AI should not be used lightly when decisions affect people's lives or their rights. On the other hand, for tasks such as sorting cucumbers by shape or making "mechanical decisions" such as whether a tennis ball is 'in' or whether a baseball pitch was a strike, you can use AI since it keeps working faster, more accurately, and tirelessly than humans. However, I believe that we should be cautious about using AI when it comes to "value judgments" such as the recruitment activities mentioned above.

Ohtani's value cannot be fully captured by data

— Can you elaborate on why you believe AI should not be used lightly in decisions that affect people's lives or rights?

For example, I do not think AI is well suited to tasks such as promotion decisions, which require evaluating factors that may not be reflected in the available data.

Today, people often talk about data-driven baseball. However, even in Major League Baseball, where data analysis is highly advanced, many argue that Shohei Ohtani's true value, including the leadership, influence, and inspiration he brings to his teammates and the organization as a whole, cannot be fully captured by data. The data clearly show that he is an extraordinary player, but they do not fully capture everything that makes him valuable.

Similarly, researchers in both Japan and the United States have criticized the use of AI in hiring decisions because there is no guarantee that it can accurately predict whether a student will contribute to a company or succeed in the future. Because we generally do not have long-term data showing how applicants who were rejected ultimately performed over the course of 30 years, it is difficult to verify whether such hiring predictions are actually accurate. In that sense, they may be closer to fortune-telling than science.

Moreover, if a service is marketed as highly accurate simply because it relies on statistics and data analysis, companies may be misled into overestimating its reliability, especially when such services are promoted as efficient and labor-saving tools for recruitment.

"Human-Centered" as a Social Principle

By the way, we sometimes hear people say that startup companies should not be expected to provide highly sophisticated services. However, even startup companies should not be permitted to provide low-quality services that unfairly affect life-changing decisions, such as employment opportunities, or that risk infringing on people's rights.

In areas like these, if companies cannot provide services of an acceptable standard, regardless of their size, then regulation is entirely appropriate. It would be difficult to accept the argument that students should simply bear the consequences of inaccurate or unfair judgments because the provider happens to be a startup.

This idea, that businesses should be held to the same safety standards regardless of their size, can also be seen in product-liability cases. Courts have held that even small, family-owned restaurants are not exempt from liability for food poisoning simply because they are smaller than large food manufacturers. From the perspective of the victims, that conclusion is only natural.

The Cabinet Office's AI guidelines are built around the concept of "Human-Centered AI Social Principles." Likewise, the OECD AI Principles, whose adoption Japan helped lead, emphasize the importance of trustworthy AI. Furthermore, the AI Act, enacted last year, stipulates that guidelines and other necessary measures should be developed in line with internationally recognized principles. In other words, AI should not harm people, and untrustworthy AI should not be used.