the stick is a suggestion
The layer that reads your intent is fifty years old. What’s new is that it talks back.
I was asking Claude about something else entirely and it stopped to say: sounds like you might need a reminder, I’ll make one for that. Then it did. In Reminders, on my phone, where I would actually see it. I hadn’t asked for a reminder. I hadn’t mentioned reminders.
I told barcode about it. Their read was bigger than mine and arrived immediately:
AI will be the layer we use machines through.
Then they reached for fighter jets. Pilots already operate the jet through the same kind of algorithm — oh, seemed like you didn’t want to crash into the ground, let me adjust your control input just a bit so you don’t. The computers can infer your intent now. Last year they only knew exactly what you told them. Now they can infer much more than what you told them.
aka, they can think
They’re right about the layer. They’re off by about fifty years on when it showed up, and I think the last line is the wrong conclusion drawn from a correct observation.
The stick stopped being a command in 1974.
The YF-16’s first flight was an accident. On January 20th, during a high-speed taxi test, the airplane started rocking on its gear badly enough that Phil Oestricher took off rather than wreck it. The official first flight came two weeks later. Either way, 1974, and either way it was the first production fighter built with relaxed static stability — an airframe deliberately made aerodynamically unstable, which stays in the sky only because a computer is correcting it dozens of times a second.
There is no cable from the stick to the control surfaces. You move the stick. A computer reads the movement as a request, decides what the airplane should do about it, and moves the surfaces. Pull past nine G or past the angle-of-attack limit and the flight control system doesn’t warn you. It just declines. The input is clipped and the airplane does something other than what you asked.
Then Auto-GCAS, fielded across the F-16 fleet starting in 2014. It runs a terrain model against your predicted flight path, and when the two are about to intersect it takes the airplane away from you, rolls wings level, pulls up, and hands it back. It exists for the pilot who’s unconscious from G-forces or disoriented over dark water, and it has pulled airplanes out of dives with seconds left on the clock.
So barcode’s line isn’t a prediction. It’s a description of a system that’s been in the fleet for over a decade. The pilot’s hand hasn’t been connected to the airplane in fifty years.
But the jet is not inferring your intent, and the difference isn’t pedantic.
Auto-GCAS has never once wondered what you want. It has a terrain database, an inertial nav solution, a performance model of the aircraft, and arithmetic. It computes where you are going to be and where the dirt is. Put a pilot in the seat who sincerely intends to fly into the ground and the system behaves identically, because intent was never an input to it. That’s a prediction about a trajectory, checked against a model of the world. Extremely good arithmetic applied to physics.
What happened with my reminder is a different animal. There is no ground truth for he wants a reminder. No terrain database of my intentions. The model read the shape of the conversation and produced the likeliest next action for someone talking like that. The jet models the world. The assistant models you. When a model of the world is wrong you can go measure the world. When a model of you is wrong there is nothing to check it against except you, later, noticing.
Don Ihde spent a career cataloguing the ways tools sit between us and things — the tool that vanishes into your body, like glasses or a keyboard you’ve stopped feeling, versus the tool you deal with as a quasi-other, like an ATM or a chatbot. Barcode’s “layer we use machines through” is the first kind. The jet is the first kind: the pilot flies through the computer and never experiences it at all until the moment it refuses. The thing that made my reminder is the second kind. It didn’t withdraw. It spoke up. Those are not the same layer, and calling both of them the layer buries the part that’s actually new.
Which brings up what it costs to be wrong about a person.
Boeing put a system on the 737 MAX that watched angle of attack and, when the reading came back high, trimmed the nose down — so the airplane would handle like the older 737s and airlines wouldn’t have to retrain crews. It took its reading from one sensor. When that sensor lied, the system was certain, and it was certain over and over: trim down, pilot trims back, trim down again.
Lion Air 610, October 2018: 189 people. Ethiopian 302, March 2019: 157. Three hundred forty-six.
MCAS inferred an intent — the airplane must want to be pushed down, because it looks like it’s stalling — and the crews had no way to argue with it, partly because they hadn’t been told it was there. The failure wasn’t that the software did something stupid. It did precisely what it was designed to do, off a wrong reading, to people who couldn’t see the reasoning and couldn’t get a word in.
That’s the shape of the risk in the layer barcode is describing. Not that the machine turns hostile. That it becomes confidently helpful about a premise you never got to see.
On aka, they can think — I’m not settling that here, and I distrust anyone who answers it quickly in either direction.
But notice which part of my story is the interesting part. It isn’t that software created a reminder. Siri shipped in October 2011 and could do that on a keyword, badly. It’s that the reminder was right. I did need it. Nobody asked, it showed up, and it was correct.
That’s the property that changes behavior, and it’s orthogonal to whether there’s thinking in there. A thing that guesses right most of the time is a thing you stop auditing. You stop reading the confirmation. You stop asking what it based that on. The competence is the mechanism — something wrong half the time would keep you sharp.
Norbert Wiener made roughly this point in Science in 1960, reaching for the sorcerer’s apprentice: if you build a machine to accomplish your purpose and you can’t reach in and stop it once it’s running, you had better be very sure the purpose you put in is the purpose you actually wanted. His worry was speed — machines acting faster than we could check them. Ours is quieter. We aren’t putting the purpose in anymore. We’re letting the machine infer it, and then not checking, because it has been right so many times in a row.
The reminder went off. I did the thing. I still don’t know what I said that made it think I’d forget.