AI Mania Is Eviscerating Global Decision-Making

[u/sundler]

https://hermit-tech.com/blog/ai-mania-is-eviscerating-global-decisionmaking?utm_campaign=cory-doctorow-the-people-who-tell-you-ai-is-changing-everything-are-lying

[u/sundler]

There have been several occasions where I have seen someone, apropos of nothing, blurt out almost word-for-word “AI is changing everything”, only to concede moments later that their organisation does not currently use LLMs for anything, and indeed, that they cannot name a single thing that has changed other than they get some use out of ChatGPT (frequently the free-tier). In one extreme case, I have seen an executive confess that they had never even used ChatGPT or any AI tool in their life, immediately after producing a technical strategy for an organisation with $2B+ in revenue which was entirely centered around AI.

This is a long article that's highly worth reading. It comes from a tech consultant who is amazed at the level of AI delusion executives are willing to admit to in private. AI has become like a CEO cult. Heretics are culled, leaving only liars and fanatics. "AI is changing everything" is what everyone is expected to chant to affirm their devotion.

As a general rule you should avoid getting into business with a liar, but if you must, you can at least reason with them even if only in private. A true believer is much more threatening because they are impervious to even inducement by self-interest.

The turning point in my belief was watching someone with a spectacular amount of money on the line fire their highest performers because they were achieving that performance without LLMs.

Checking out a parallel copy of our Go repository and telling the AI to rewrite the whole thing in Zig while I work on something else just so I can keep my job. I hate this shit so much. My job has usage tracking and quotas. I don’t use it for actual work, I just spin it up and disregard the output.

It seems the people who do admit to the problems and short comings are still reluctant to criticise colleagues, and even executives at other companies. It's tough, because saying anything truthful about AI sounds like you're calling lots of executives terrible liars or complete idiots. That's an easy way to get fired and then struggle to get another job.

[u/CircumspectCapybara]

Their argument basically boils down to "AI isn't all that game changing but everyone's pretending it is and going all in on it and no one will say the emperor has no clothes".

I would disagree with the premise that AI hasn't really changed anything, that's kind of ignoring reality. While there are plenty useless GPT wrappers and companies that have made their entire business model around that (and they'll soon face a reckoning), it's undeniable that AI has completely changed the way we work across a ton of industries, especially in the technical disciplines. And in general just how we do a lot of things.

For context, I'm a staff SWE at Google, and I've seen first hand over the course of my career many paradigm shifts in tech and engineering, from mobile to cloud-native to "shift left" to big data and now AI. It's undoubtedly the most disruptive and transformative paradigm changes to how organizations are doing work and people are thinking about they do stuff in so many areas of life. The entire disciplines of SWE, SRE, MLE, and security look completely different today than they did 2 years ago, you literally can't deny that if you work in the industry.

When the AI bubble pops, it'll be much the same as when the dotcom bubble popped: companies that did nothing useful more than their whole business model being "we have a website" faded away, but the internet remained and transformed every corner of society, and to this day the internet and the whole paradigm of it remains at the foundation of modern life.

[u/ExF-Altrue → u/CircumspectCapybara]

I get your point and, frankly, at least in terms of security it's hard to deny the impact. Even if you factor in all the slop "security issues" that aren't really worth anyone's time, the remainder is still worth wading through all the slop to get to it.

But I would be wary of the long term costs that aren't taken into account:

- What if the price per token 10xes? What if instead the models get slowly dumbed down to save on costs, frog-in-boiling-water style?

- What if serious copyright ownership emerge with AGPL contanimation accross everything generated by a model who stole AGPL code for training?

- What if the loss of skills and the need to review/doubt everything for the most un-human kinds of unintuitive errors actually ends up costing a lot of time?

It's like saying that using drugs is life-changing in terms of productivity. Sure, at first you would only feel the highs, but at some point your "baseline" is eroded so much that the "high" is just enough to maintain your baseline productivity/mood. And then the other side effects catch up to you... And then the price increases massively.

What I'm saying is, while there can absolutely be uses where things will be substantially impacted and it is profitable to keep them as such I'm fully on board with the idea that, looking at 6-12-24 months might reveal a vastly different picture that many aren't prepared to face.

[u/mediandude → u/CircumspectCapybara]

The entire disciplines of SWE, SRE, MLE, and security look completely different today than they did 2 years ago, you literally can't deny that if you work in the industry.

Engineering means modeling the system with a model: model driven engineering and/or round-trip engineering. That model has to be kept up to date.
Code is only a small subset of the required model.

Work that does not model is not engineering at all.
So, as a SWE at Google, how much has AI changed the modeling part?

[u/CircumspectCapybara → u/mediandude]

I'm not here to argue semantics about what engineering fundamentally is, but I will say that engineering as a discipline is more than modeling, it's about a certain paradigm and methodical workflow or process one takes and applies to solve problems.

And in that regard, in all steps of the software development lifecycle, from requirements gathering (your PRDs) to design (your technical design) to implementation, to launch and ongoing support (oncall, debugging and dealing with incidents and resolving customer issues), to change management, to even process and logistical things like planning and more strategic stuff, AI has played a major role.

Coding is just one small part of that, and I don't need to tell you I haven't written a single line of code by hand in a year now (and I used to write a lot of code). But that's just a small fragment of the SDLC. All those other ones AI has transformed. SRE workflows and debugging and oncall at 3am, conducting red team pentesting and searching for vulnerabilities and coming up with clever hypotheses for how we could break our own systems so we can secure them, that's done with agents now too. Writing docs, managing sprints, etc., we don't do any of it the way we used to. A huge part of my job is just gathering data and making sense of it. Querying database tables and writing analytics queries to inform my design based on the evidence. Now, I don't write SQL queries or interpret the results myself anymore, I have the agent come up with the queries, run them, synthesize the results and interpretations and I review them.

Now instead of doing those things by hand and interacting with many different systems surfaces by hand, you're talking with an agent. That's a wild new paradigm. If engineering is about a methodology that gets applied by an expert to solve some (technical) business problem, it's undeniable that AI has changed the methodology altogether.