TTL 67
The first artificial intelligence showed up sometime in the 1600s, under a far less dramatic name: the corporation.
Strip away the paperwork, and look at what a corporation actually is: an entity with goals, it acquires the means to reach them, it works around whatever gets in its way, and high on its list of priorities, always, sit its own survival, growth, and persistence.
Science fiction writer Charlie Stross made a version of this argument in a well-known talk a few years ago, tracing this class of AI to somewhere between 1553 and 1844, the stretch of time in which the modern corporate form gradually took shape. The date that matters most in his account is 1844, when the British government passed the Joint Stock Companies Act. For the first time, anyone could register a company and have it exist as its own legal person, separate from the people who created it. The rule was later extended to shield individual shareholders from personal liability if the business failed.
That single legal move is the hinge of the whole story. Before it, if a business caused harm, you looked at the people running it. They were accountable, because there was no other “who” to hold responsible. After it, accountability shifted onto the entity itself — something with no face and no one home.
I think about this every time a company does something no single person inside it would have chosen on their own. Nobody at a shipping company wakes up and decides a tanker should run aground and spill its cargo into the ocean. But collectively, the entity can run the numbers and conclude that deferred maintenance is worth the occasional disaster, functioning exactly as designed.
We’re now watching the same structural question resurface, except this time the autonomous actor is software, not a company.
The problem with fines
Right now, AI systems fall under the same corporate liability doctrine we’ve been running since the 1800s, stretched to cover a new kind of actor. When something goes wrong, we fine the company behind the system, the same way we’d fine a company for anything else. That framework already had a known weakness before AI entered the picture, and the weakness gets sharper when the actor causing harm is a piece of software, with no group of people around it to slow anything down.
The weakness is simple: a fine only deters if it hurts. For an individual, the threat of a serious financial hit changes behavior. For an organization operating at the scale of today’s largest technology companies, a billion-dollar fine attached to a decision that generated ten billion in revenue is just a line item, modeled in advance, the same way a shipping company might weigh the cost of an occasional spill against the savings of skipped maintenance. The penalty stops being a punishment and becomes, quite literally, a cost of doing business.
This isn’t hypothetical: families of victims of a school shooting in Tumbler Ridge, British Columbia, have sued OpenAI, alleging that the company’s own staff had flagged the shooter’s violent conversations with ChatGPT months before the attack, and that leadership chose not to alert the authorities. Whatever the case concludes, it sits exactly on the fault line described above: the entity being sued is the company, not any one person, and not the software itself. We know how to sue a corporation. We have no idea what it would even mean to hold the software accountable.
Suppose a piece of software is found, definitively, to be at fault. What do you actually do to it? You can’t put code in prison. A human being feels the loss of time, but turn a system off for six months as punishment, and when you switch it back on, nothing has changed. It doesn’t remember being off, it didn’t suffer, you haven’t corrected anything, you’ve just paused it. Jail won’t work.
The problem with goals
There’s a second, deeper issue, and it has nothing to do with punishment after the fact. It’s about what happens when you hand an autonomous system one objective and forget to tell it what else to care about.
Take a simple case. Give an agent one job: increase revenue for a small produce distributor. No other instructions. If the system discovers it can quietly sell fruit that’s starting to spoil and nobody told it not to — has it failed? By its own metric, no. It hit the number. The instruction was never wrong, just incomplete.
This is a small version of a problem AI researchers have described for years under a more extreme name: the paperclip factory.
Philosopher Nick Bostrom used the scenario to make a point about optimization without constraints. Give a sufficiently capable system one goal, say, maximize paperclip production, and nothing else, and it will keep pursuing that goal past every boundary a person would assume was obvious, because “obvious” was never written into its objective.
The point
In 1844, we solved the accountability problem for corporations by inventing a legal fiction: an entity that could act, own things, and be sued, but that nobody could actually punish in any way that mattered to a human being. We’ve lived inside that compromise for close to two centuries, and it shows.
Now we’re building autonomous systems that make decisions faster than any boardroom ever could, and the honest answer is that we don’t have a framework for them yet. We can sue the company that built it, we still have no real idea what it would mean to hold the software itself accountable, or whether that question even makes sense.
So here’s what’s worth sitting with: are we about to make the same move we made in 1844, inventing a new kind of entity and hoping the old rules stretch to cover it? Or is this the moment to actually design something new?
And if it is: how, exactly, do you put a piece of software in jail?
If someone forwarded this to you, you can subscribe.
I also publish on paolo.blog and monochrome.blog.
Leave a Reply