> Automation doesn't make operators more careful. It makes them forget how to be. The more reliable the system, the less ready the human.
The entire premise of a system is that it removes the need for careful attention.
system: signal lights tell me whether or not I can pass through an intersection, so that I do not have to attend to potentially high speed traffic from a variety of directions.
system: the side my knife blade sits on my arched guide fingers, so that I do not have to attend to the edge of the blade or the location of my fingers.
>The entire premise of a system is that it removes the need for careful attention.
I think this premise is flawed or, at best, too narrow. A system is just a logical grouping of items that perform a function. Sometimes that function can be to reduce cognitive burden, but it doesn't have to be. A "vision system" like what humans use does not reduce attention, but increases/enables it, while a building automation system can reduce attention. The ability to increase/reduce attention is not the central principle of a system.
> system: signal lights tell me whether or not I can pass through an intersection, so that I do not have to attend to potentially high speed traffic from a variety of directions.
You know I noticed this... I lived in a country where people obey traffic laws, and in a country where they very much don't.
I witnessed many more traffic accidents in the country where people are used to relying on the traffic lights to tell them if it's safe or not.
Whereas in the other country, everyone correctly assumes that the other drivers are completely insane, and so they stay vigilant.
Other than the data of road fatalities that disproves this anecdote, my own anecdote is this is the false sense of security people get in other countries that don’t have traffic laws. Oh see the people have to look all the time so it’s much safer. When you start to live it for a long time you realize it’s not true. Many more fatalities.
Now I do think the science shows if you design roads and systems to make drivers more thoughtful it can improve outcomes. Size roads for the speed limit, roundabouts, etc. these can make a difference as it balances the system.
There's some documented studies of removing all the street clutter and lines from residential area intersections forcing drivers to be more careful, especially around pedestrians, reducing overall accidents. But this does reduce throughput slightly.
Not sure why this would downvoted. Go to a less developed nation where traffic laws are not important and it’s one of those sense of false security ideas.
I think part of the point of the article is that it also makes edge-cases more dangerous and catastrophic than if there was no autopilot at all. From the article:
>The argument for automation is that it frees up cognitive bandwidth. Fewer routine decisions means more headroom to think carefully about the ones that matter.
So if the expectation is that the human pilot is expected to pay attention to mitigate the dangerous edge cases "that matter", there is a contradiction: the tool that promises to free up the bandwidth for that attention creates a complacency that prevents that attention from being applied.
In other words, it makes the normal situations safer but the abnormal situations more dangerous.
Landing is an abnormal situation for an aircraft which we make SUBSTANTIALLY less dangerous through intense automation. Do you want to rip out automated landing systems?
> I don't think we believe that the automation makes the human operators safer, but that it is overall safer than the human operator alone.
If the automation is for the easy/routine stuff, then no. The automation doesn't work in exactly the most safety-critical situations, and then the human operator is thrust into fixing the situation without the full context.
Well, it's differently safer. It's impossible to make categorical statements of more or less safe with automation because it depends massively on the design.
The crew and the plane are a single system. It is meaningless to imagine pilot “skill” without the plane. Further, we killed so many pilots in the 20th century through impossible workloads which we have now automated, it’s almost cruel to be wistful for it.
I know this is an analogy to AI, but I wish we would dispense with this idea that there’s some appropriate level of machinery which was reached just a hair before right now. There is no appropriate level of machinery, no point at which the nature of the system itself will unambiguously say “that’s enough.”
Am I missing something? Is TFA only 2-3 paragraphs of a generic metaphor, with no actual data/research from aviation (or other fields) to back up the core thesis?
American Airlines captain Warran VanderBurgh once called this phenomenon "the children of the magenta"[1] in his talk on automation dependency in the 90s.
I wonder what we'd call the children today in hindsight and what line they're chasing now...
> Automation doesn't make operators more careful. It makes them forget how to be. The more reliable the system, the less ready the human.
The entire premise of a system is that it removes the need for careful attention.
system: signal lights tell me whether or not I can pass through an intersection, so that I do not have to attend to potentially high speed traffic from a variety of directions.
system: the side my knife blade sits on my arched guide fingers, so that I do not have to attend to the edge of the blade or the location of my fingers.
etc etc.
>The entire premise of a system is that it removes the need for careful attention.
I think this premise is flawed or, at best, too narrow. A system is just a logical grouping of items that perform a function. Sometimes that function can be to reduce cognitive burden, but it doesn't have to be. A "vision system" like what humans use does not reduce attention, but increases/enables it, while a building automation system can reduce attention. The ability to increase/reduce attention is not the central principle of a system.
> system: signal lights tell me whether or not I can pass through an intersection, so that I do not have to attend to potentially high speed traffic from a variety of directions.
You know I noticed this... I lived in a country where people obey traffic laws, and in a country where they very much don't.
I witnessed many more traffic accidents in the country where people are used to relying on the traffic lights to tell them if it's safe or not.
Whereas in the other country, everyone correctly assumes that the other drivers are completely insane, and so they stay vigilant.
Other than the data of road fatalities that disproves this anecdote, my own anecdote is this is the false sense of security people get in other countries that don’t have traffic laws. Oh see the people have to look all the time so it’s much safer. When you start to live it for a long time you realize it’s not true. Many more fatalities.
Now I do think the science shows if you design roads and systems to make drivers more thoughtful it can improve outcomes. Size roads for the speed limit, roundabouts, etc. these can make a difference as it balances the system.
There's some documented studies of removing all the street clutter and lines from residential area intersections forcing drivers to be more careful, especially around pedestrians, reducing overall accidents. But this does reduce throughput slightly.
Except if you look at this map: https://en.wikipedia.org/wiki/List_of_countries_by_traffic-r...
People die on road more in countries that conventionally don't follow traffic laws.
Not sure why this would downvoted. Go to a less developed nation where traffic laws are not important and it’s one of those sense of false security ideas.
I don't think we believe that the automation makes the human operators safer, but that it is overall safer than the human operator alone.
I think part of the point of the article is that it also makes edge-cases more dangerous and catastrophic than if there was no autopilot at all. From the article:
>The argument for automation is that it frees up cognitive bandwidth. Fewer routine decisions means more headroom to think carefully about the ones that matter.
So if the expectation is that the human pilot is expected to pay attention to mitigate the dangerous edge cases "that matter", there is a contradiction: the tool that promises to free up the bandwidth for that attention creates a complacency that prevents that attention from being applied.
In other words, it makes the normal situations safer but the abnormal situations more dangerous.
Landing is an abnormal situation for an aircraft which we make SUBSTANTIALLY less dangerous through intense automation. Do you want to rip out automated landing systems?
If you expect every aircraft to land, it seems to meet the very definition of "normal" operation.
An abnormal landing would be something like trying to land with a broken elevator surface.
And you contend that autopilot makes that situation more dangerous? Do you have any support for this?
> I don't think we believe that the automation makes the human operators safer, but that it is overall safer than the human operator alone.
If the automation is for the easy/routine stuff, then no. The automation doesn't work in exactly the most safety-critical situations, and then the human operator is thrust into fixing the situation without the full context.
Well, it's differently safer. It's impossible to make categorical statements of more or less safe with automation because it depends massively on the design.
ijterestingly, AI tools are improving code theoughput but also elevating catastrophic error rates.
even if the average rate goes up tor net benefit, are organizations prepared for increased carastrophic failures?
Also, ironies of automation[1], etc.
[1] https://static1.squarespace.com/static/644321e78cd2dd37613af...
In this case the only reason I'm a pilot is the existence of autopilot haha
They should get an auotpilot to fix their website to render on desktops and not just phones.
The crew and the plane are a single system. It is meaningless to imagine pilot “skill” without the plane. Further, we killed so many pilots in the 20th century through impossible workloads which we have now automated, it’s almost cruel to be wistful for it.
I know this is an analogy to AI, but I wish we would dispense with this idea that there’s some appropriate level of machinery which was reached just a hair before right now. There is no appropriate level of machinery, no point at which the nature of the system itself will unambiguously say “that’s enough.”
And that's the whole freaking point. If the humans have to obtain and maintain the exact same skills as before than what's the point of automation?
Am I missing something? Is TFA only 2-3 paragraphs of a generic metaphor, with no actual data/research from aviation (or other fields) to back up the core thesis?
Yet another AI written slop article hitting the top of HN.
Evidence: Look at the most recent article on this blog: https://julienreszka.com/blog/difficult-conversations-don-t-...
"Memory is reconstructive. When someone recalls events differently, it feels like gaslighting. It usually isn't. Document first, then negotiate."
Bleugh
[dead]
American Airlines captain Warran VanderBurgh once called this phenomenon "the children of the magenta"[1] in his talk on automation dependency in the 90s.
I wonder what we'd call the children today in hindsight and what line they're chasing now...
[1] https://www.youtube.com/watch?v=5ESJH1NLMLs