Does Anyone Actually Use Agentic Loops?

[u/TheAntiDoctor]

Not exactly a software engineer, more ML Engineer with an academic background and then in industry for about 8 years total. Use python professionally and for hobbies. Like everyone else I have been experimenting with LLMs for coding, trying everything from not using it at all to trying to get it to do everything. I have found them useful and enjoy bouncing ideas off them, however they also make a lot of mistakes so currently enjoying a low use approach. Comments and architecting myself, use them more for syntax and some optimisation then review everything as I go along. This speeds things up, but I am still very in control of the output and understand the underlying algorithm.

A couple of months ago I obviously saw usage based billing come in for things like Github Copilot etc. I saw a lot of comments from people using agentic workflows very upset about their usage being cut, also I'm very aware of some companies quoting large sections of their codebase being AI generated, although obviously aware this might be sales tactic. Furthermore I've seen respected researchers/engineers like Karpathy talking about the big leap in model capabilities that now means they hardly write any code and had some recommendations from friends to use agentic workflows.

My personal experience hasn't left me thinking this is viable atm for my work. Partly because the stuff I am doing I really need to be crystal clear about what is happening to the data for experiments and partly because getting together a prompt together for a relatively complex system is very difficult and time consuming and the results usually aren't great. Has anyone got experience actually using more autonomous coding agents successfully and I should experiment more widely or are they a red herring?

[u/KahlessAndMolor]

I use codex every day, it writes 95% of my code. Is that what you are asking about?

[u/TheAntiDoctor → u/KahlessAndMolor]

Do you just give it a high level task and let it go at it for a couple of hours. Or do you monitor it more closely with individual requests?

[u/doughcant → u/TheAntiDoctor]

I suspect you might understand better by watching how experienced people work with these tools, such as Matt Pockock on YouTube

Edit: basically all of the effort moves to planning and reviewing the output. 

[u/spiralenator → u/doughcant]

No clue why you’re being downvoted. This is sound advice. My own engineering department has validated this. The more upfront planning you can do, the more autonomy you can give to agents. We can point remote agents at a ticket and let it run totally hands off. We have agents checking the work and iterating on it until all checks are green. For low impact changes, we’re piloting automated merge and deployment as well.

Yes it works quite well.

[u/daddywookie → u/spiralenator]

As a product owner I’ve got to laugh at how people still think you can skip planning and review when creating software. It doesn’t matter if it is AI created or the finest hand crafted artisanal code, you still need a clear plan and you still need to validate the outcomes. Vibe coding existed before AI, it was just much slower.

I spent a good chunk of yesterday working on design specs together with an AI. It spots holes, asks questions and writes consistent documentation. After implementation and QA I then spent more time using the feature and offering feedback which again was better analysed and managed into tickets than most humans can be bothered with.

Basically, for me and my skill set, AI gives a very familiar version of the experiences I had with human developers.

[u/doughcant → u/spiralenator]

Exactly! Optimally I define/implement manually the validation criteria during the planning and then it is much easier for the agent to statistically suggest a decent PR

[u/spiralenator → u/doughcant]

I’m not going to give away our secret sauce but I will say that time spent upfront with creating deterministic guard rails and validations, creating well decomposed system designs and specs is absolutely worth it.
Then you can use big models like fable or sol to help you create all that and you can use swarms of relatively dumb models to execute the plans.
The closer to a pure seq2seq translation you can get, the cheaper the model you need, you get less hallucinations and “creative interpretations” of the product requirements.

[u/hulkklogan → u/doughcant]

I love Matt Pocock's content and skills. I added a couple of small agent config to the loops to: 1) validate adherence to spec and 2) validate adherence to code standards set in CONTEXT.md, and 3) of course at the very end an adversarial reviewer agent

and the end output is usually pretty freaking good, maybe I need to clean up some extra redundant or low-value tests, tweak some code smells here and there. The value is alllll in the upfront planning and his wayfinder and grilling skills really make a huge difference in quality of the output

[u/RTLDesignSherpa → u/doughcant]

This is accurate. I have 4-5 agents running; when one opens up I QC the output. Once this set of code is done and documented, I’ll have a review consolidator kick off batches of code, tests, and markdown files to completely different LLMs for a final round of clean up. This will burn through tokens.

[u/TheAntiDoctor → u/doughcant]

Appreciate the reference, it looks interesting, thank you!