Ironies of LLMs

There’s an irony baked into Large Language Models.  

In fact, most technologies that automate or enhance processes that involve humans contain “ironies of automation”.  For example, an assembly line is created to automate a manufacturing process.  However,  when something goes wrong, the entire thing must come to a standstill.  All production may need to immediately stop to resolve the issue.  This is typically not the case when humans build things.

Another example could be automating monitoring software for a control room.  The software may reduce the cognitive burden of the humans responsible for the control room.  However, when something does go wrong, it may take significantly longer for a human to then gain the situational awareness needed to resolve the issue as they have been out of the loop for some time.

A third example are alarms meant to draw attention, such as those used by patient monitoring equipment in a hospital.  At a certain threshold, such alarms may cause alarm fatigue, which leads to a decrease in a care provider’s response rate and an increase in their response time.

In information technology, we have long depended on computers for the retrieval of accurate information.  When a database is queried, whatever was stored can be reliably extracted.  Any errors are bound to have  happened at insertion or retrieval.  This is barring a bit flip or other soft errors,  but again, there are safeguards against these.  Humans exercise their judgement but depend on the reliability of information retrieved from computer systems, often in automated ways.

The irony is that through LLMs, we’re creating computer systems that generate information in inconsistent and reliable ways, but presented with a veneer of confidence by way of grammatically correct and tone appropriate language.  

We’ve created information technology that requires fact-checking.