I disagree with the last part of this post, though (the idea that lawyers, doctors, firefighters etc. are inevitably going to be replaced with AI as well, whether we want it or not). I think this is precisely what AI grifters would want us to believe, because if they could somehow force everyone in every part of society to pay for their slop, this would keep stock prices up. So far, however, AI has mainly been shoved into our lives by a few oligopolistic tech companies (and some VC-funded startups), and I think the main purpose here is to create the illusion (!) of inevitability because that is what investors want.

Completely unrelated fact, but isn’t the prevalence of cocaine use among U. S. adults considered to be more than 1% as well?

(Referring to this, of course - especially the last part: https://pivot-to-ai.com/2025/06/05/generative-ai-runs-on-gambling-addiction-just-one-more-prompt-bro/)

Stock markets generally love layoffs, and they appear to love AI at the moment. To be honest, I’m not sure they thought beyond that.

Yes, they will create security problems anyway, but maybe, just maybe, users won’t copy paste sensitive business documents into third party web pages?

I can see that. It becomes kind of a protection racket: Pay our subscription fees, or data breaches are going to befall you, and you will only have yourself (and your chatbot-addicted employees) to blame.

At this point it’s an even bet that they are doing this because copilot has groomed the executives into thinking it can’t do wrong.

This, or their investors (most likely both).

reliably determining whether content (or an issue) is AI generated remains a challenge, as even human-written text can appear ‘AI-like.’

True (even if this answer sounds like something a chatbot would generate). I have come across a few human slop generators/bots in my life myself. However, making up entire titles of books or papers appears to be a specialty of AI. Humans would not normally go to this trouble, I believe. They would either steal text directly from their sources (without proper attribution) or “quote” existing works without having read them.

So what kind of story can you tell? A movie that perhaps has a lot of dream sequences? Or a drug trip?

Maybe something like time travel, because then it might be okay if the protagonists kept changing their appearance to some degree. But even then, there wouldn’t be enough consistency, I guess.

This has become a thought-terminating cliché all on its own: “They are only criticizing it because it is so much smarter than they are and they are afraid of getting replaced.”

I’ve noticed a trend where people assume other fields have problems LLMs can handle, but the actually competent experts in that field know why LLMs fail at key pieces.

I am fully aware of this. However, in my experience, it is sometimes the IT departments themselves that push these chatbots onto others in the most aggressive way. I don’t know whether they found them to be useful for their own purposes (and therefore assume this must apply to everyone else as well) or whether they are just pushing LLMs because this is what management expects them to do.

First, we are providing legal advice to businesses, not individuals, which means that the questions we are dealing with tend to be even more complex and varied.

Additionally, I am a former professional writer myself (not in English, of course, but in my native language). Yet, even I find myself often using complicated language when dealing with legal issues, because matters tend to be very nuanced. “Dumbing down” something without understanding it very, very well creates a huge risk of getting it wrong.

There are, of course, people who are good at expressing legal information in a layperson’s way, but these people have usually studied their topic very intensively before. If a chatbot explains something in “simple” language, their output usually contains serious errors that are very easy for experts to spot because the chatbot operates on the basis of stochastic rules and does not understand its subject at all.

Up until AI they were the people who were inept and late at adopting new technology, and now they get to feel that they’re ahead

Exactly. It is also a new technology that requires far fewer skills to use than previous new technologies. The skills are needed to critically scrutinize the output - which in this case leads to less lazy people being more reluctant to accept the technology.

On top of this, AI fans are being talked into believing that their prompting as such is a special “skill”.

That’s why I find the narrative that we should resist working with LLMs because we would then train them and enable them to replace us problematic. That would require LLMs to be capable of doing so. I don’t believe in this (except in very limited domains such as professional spam). This type of AI is problematic because its abilities are completely oversold (and because it robs us of our time, wastes a lot of power and pollutes the entire internet with slop), not because it is “smart” in any meaningful way.

But if you’re not an expert, it’s more likely that everything will just sound legit.

Oh, absolutely! In my field, the answers made up by an LLM might sound even more legit than the accurate and well-researched ones written by humans. In legal matters, clumsy language is often the result of facts being complex and not wanting to make any mistakes. It is much easier to come up with elegant-sounding answers when they don’t have to be true, and that is what LLMs are generally good at.

And then we went back to “it’s rarely wrong though.”

I am often wondering whether the people who claim that LLMs are “rarely wrong” have access to an entirely different chatbot somehow. The chatbots I tried were rarely ever correct about anything except the most basic questions (to which the answers could be found everywhere on the internet).

I’m not a programmer myself, but for some reason, I got the chatbot to fail even in that area. I took a perfectly fine JSON file, removed one semicolon on purpose and then asked the chatbot to fix it. The chatbot came up with a number of things that were supposedly “wrong” with it. Not one word about the missing semicolon, though.

I wonder how many people either never ask the chatbots any tricky questions (with verifiable answers) or, alternatively, never bother to verify the chatbots’ output at all.

FWIW, I work in a field that is mostly related to law and accounting. Unlike with coding, there are no simple “tests” to try out whether an AI’s answer is correct or not. Of course, you could try these out in court, but this is not something I would recommend (lol).

In my experience, chatbots such as Copilot are less than useless in a context like ours. For more complex and unique questions (which is most of the questions we are dealing with everyday), it simply makes up smart-sounding BS (including a lot of nonexistent laws etc.). In the rare cases where a clear answer is already available in the legal commentaries, we want to quote it verbatim from the most reputable source, just to be on the safe side. We don’t want an LLM to rephrase it, hide its sources and possibly introduce new errors. We don’t need “plausible deniability” regarding plagiarism or anything like this.

Yet, we are being pushed to “embrace AI” as well, we are being told we need to “learn to prompt” etc. This is frustrating. My biggest fear isn’t to be replaced by an LLM, not even by someone who is a “prompting genius” or whatever. My biggest fear is to be replaced by a person who pretends that the AI’s output is smart (rather than filled with potentially hazardous legal errors), because in some workplaces, this is what’s expected, apparently.

If computers become capable of mass-producing stuff other computers will like, but many humans won’t, this might also lead to a quick decline of algorithm-based search engines, social media feeds etc. (as has been discussed here before, of course).

I think most cons, scams and cults are capable of damaging vulnerable people’s mental health even beyond the most obvious harms. The same is probably happening here, the only difference being that this con is capable of auto-generating its own propaganda/PR.

I think this was somewhat inevitable. Had these LLMs been fine-tuned to act like the mediocre autocomplete tools they are (rather than like creepy humanoids), nobody would have paid much attention to them, and investors would have started to focus on the high cost of running them quickly.

This somewhat reminds me of how cryptobros used to claim they were fighting the “legacy financial system”, yet they were creating a worse version (almost a parody) of it. This is probably inevitable if you are running an unregulated financial system and are trying to extract as much money from it as possible.

Likewise, if you have a tool capable of messing with people’s minds (to some extent) and want to make a lot of money from it, you are going to end up with something that resembles a cult, an LLM or similarly toxic groups.

I think this has happened before. There are accounts of people who completely lost touch with reality after getting involved with certain scammers, cult-leaders, self-help gurus, “life coaches”, fortune tellers or the like. However, these perpetrators were real people who could only handle a limited number of victims at any given time. Also, they probably had their very specific methods and strategies which wouldn’t work on everybody, not even all the people who might have been the most susceptible. ChatGPT, on the other hand, can do this at scale. Also, it was probably trained from all websites and public utterances of any scammer, self-help author, (wannabe) cult leader, life coach, cryptobro, MLM peddler etc. available, which allows it to generate whatever response works best to keep people “hooked”. In my view, this alone is a cause for concern.

I think we don’t know how many people might be at risk of slipping into such mental health crises under the right circumstances. As a society, we are probably good at protecting most of our fellow human beings from this danger (even if we do so unconsciously). We may not yet know what happens when people regularly experience interactions that follow a different pattern (which might be the case with chatbots).

Just guessing, but the reported “90% accuracy” are probably related to questions that could be easily answered from an FAQ list. The rest is probably at least in part about issues where the company itself f*cked up in some way… Nothing wrong with answering from an FAQ in theory, but if all the other people get nicely worded BS answers (for which the company couldn’t be held accountable), that is a nightmare from every customer’s point of view.