Sir, – More than 40 years ago, the eminent software engineering professor David L Parnas resigned from the advisory panel of US president Ronald Reagan’s Strategic Defence Initiative, commonly known as “Star Wars”. He did so because he believed “it would not be possible to build a system that we could trust”.
At the time we had neither the ability to prove such systems correct nor the possibility of testing them before deploying them. Since then, the world has become increasingly dependent on software intensive systems. The problem of correctness has not been solved by research. It has been elided by myriad “I accept” buttons that in effect absolve the software vendors of normal product liability.
The Crowdstrike failure in July 2024 cost at least $10 billion, but the companies responsible successfully claimed their liability was limited to a refund of fees.
The so-called artificial intelligence (AI) systems are larger and more widespread than Star Wars. We are in danger of deploying these even though we still cannot guarantee their correctness or fully test them beforehand. We’ve accepted the misleading euphemism of “hallucination” to describe their inevitable errors. The approach to correcting them is, and must be, ad hoc, since no one can prove their design to be correct.
READ MORE
To deploy such systems in life-critical or even economically important areas would be an act of gross stupidity.
There is one potential game-changer, however. The European Union’s revised Product Liability Directive will take effect on December 9th. This ends the exemption of software companies from normal product liability. If it is enforced, the first major failure of an AI company could be its last. Let us hope that failure is not catastrophic. – Yours, etc,
KEVIN T RYAN,
Limerick.
