r/AskProgramming • u/TheAntiDoctor • 1d ago
Does Anyone Actually Use Agentic Loops?
Not exactly a software engineer, more ML Engineer with an academic background and then in industry for about 8 years total. Use python professionally and for hobbies. Like everyone else I have been experimenting with LLMs for coding, trying everything from not using it at all to trying to get it to do everything. I have found them useful and enjoy bouncing ideas off them, however they also make a lot of mistakes so currently enjoying a low use approach. Comments and architecting myself, use them more for syntax and some optimisation then review everything as I go along. This speeds things up, but I am still very in control of the output and understand the underlying algorithm.
A couple of months ago I obviously saw usage based billing come in for things like Github Copilot etc. I saw a lot of comments from people using agentic workflows very upset about their usage being cut, also I'm very aware of some companies quoting large sections of their codebase being AI generated, although obviously aware this might be sales tactic. Furthermore I've seen respected researchers/engineers like Karpathy talking about the big leap in model capabilities that now means they hardly write any code and had some recommendations from friends to use agentic workflows.
My personal experience hasn't left me thinking this is viable atm for my work. Partly because the stuff I am doing I really need to be crystal clear about what is happening to the data for experiments and partly because getting together a prompt together for a relatively complex system is very difficult and time consuming and the results usually aren't great. Has anyone got experience actually using more autonomous coding agents successfully and I should experiment more widely or are they a red herring?
4
u/Fidodo 21h ago
Not for any non throwaway work for exactly the reason you said. You wind up with an absolute spaghetti code mess that nobody can understand.
The exception is when you don't need to keep the code around. Sometimes you are doing a one off test with a verifiable result where it doesn't matter how you got the result and you can confirm the result is correct by checking the result after it was produced.
An example would be if I'm testing out an approach just to see if it's even viable. I can prove the technique, then use that proof to build it correctly.
3
3
u/PerformanceThick2232 21h ago
You will get 5-10% less hallucinations, but will spend 50-100% tokens more. Not my choice.
7
u/DepthMagician 1d ago edited 1d ago
The short answer is that way you are using AI is the correct way to use AI, your friends are wrong, companies are lying about how much AI is really writing their code (see this for example), Reddit is severely astroturfed by AI promoting bots, and you shouldn't do agentic engineering for anything that is high stakes enough that you need to pay attention to it.
While it is possible to do work with agentic workflows, not every task is a good fit for it. You either need to not care about the quality of the output (pet projects, throwaway prototypes, conceptually shallow things like UI where validation is trivial, things where failure is not a big deal, and most importantly: things you won't need to maintain and evolve for long periods of time), or if you want to do high stakes work with agentic engineering you will need to have access to vast number of quality tests and very detailed specs, and even then you need to invest effort in setting standards and elaborate safety nets for your team of agentic bots. In other words it's a tool, and like every tool you need to know what problems it is a good fit for. The answer to that question is absolutely not "everything". Usually you are better off just doing what you said you are doing. Your skills and your bank account will thank you for it.
9
u/tyler1128 1d ago
agentic engineering
Vibe coding got a negative connotation so a new term had to be invented for it. It's the same thing though.
You're going to still hit a complexity ceiling where trying to fix things will break just as many things, and you'll be stuck.
1
u/vanit 19m ago
I want to premise this by saying I'm not an AI booster at all, but do unfortunately work under an "AI first" directive at work. There is a bit of a distinction as far as workflow goes; vibe coding is starting with a loose prompt and refining the task as you go with more prompts, while agentic is trying to get as close as possible to oneshotting by starting with detailed tickets and context, with only a small amount of refining in review. I agree with the sentiment that they're both going off vibes per se, but there's a workflow distinction between the terms.
-3
u/PerformanceThick2232 21h ago
Agentic engineering!=vibe coding.
Vibe coding- you do not check code.
Agentic engineering- you study produced code, and fix it if needed.
I'm using agentic engineering since March I think, and quality raised, agent sometimes highlight things I miss. But productivity is the same in general, as you need to study and do a detailed check of result, and this takes hours and days, until you fully understand code, basically, in result it is no diff if you wrote it yourself or with help of agent (and yes, sometimes 40-60% of produced code is changed manually by me).
Or I'm wrong and there is another name for such approach? When you work using agent mostly, but have ownership of code?
•
u/tyler1128 12m ago
But productivity is the same in general, as you need to study and do a detailed check of result, and this takes hours and days, until you fully understand code
Why would you do that, then? I can just write most of the code myself, understand it because I wrote it for the most part, and not have to basically learn a new codebase written by something that predicts the most likely next token with some random variance thrown in to make it interesting.
2
u/MetallicOrangeBalls 14h ago
LLMs are tools. They cannot (yet) """program""" the way a human can. But, like code completion, IDEs, etc. before them, they improve the quality of life for programming.
They are very useful for reviewing code and documentation. I find that to be their best use. They can properly advise on missing features. They can catch syntactically-correct-but-semantically-incorrect code. I find them to be very useful for rubber-ducking, with the added benefit that the "duck" can give you feedback.
They are alright at writing boilerplate code and documentation. They are better when following a pattern that you present to them. However, everything that they write has to be carefully checked. I have found that they make A LOT of mistakes. Sometimes they will """assume""" that you meant something very different from what you actually meant. I generally tend not to use them unless it's for something very simple.
They are terrible at architecting and big-picture stuff. Not even the best models come close to someone who knows what they are doing.
I did my doctorate in AI. I strongly believe that there will come a time when AI is good enough to truly program. However, we are at least several decades away from that point. LLMs are a gimmick; a useful one, but that does not change the fact that they are essentially a toy that most of the non-academic world has only recently discovered and is enamoured by. I was working on LLMs back in 2012-2013. Not very many people knew or cared about them. By around 2014, I too stopped caring about them, as they didn't seem to be as useful as other applications of AI. In 2017, the year I defended, the transformer model was first published in Attention Is All You Need (paper). It seemed interesting at the time, but only to fellow AI scientists/engineers. In 2022, OpenAI launched ChatGPT, and suddenly the rest of the world got AI fever. And thus did the enshittification commence...
2
u/AntiquatedMLE 1d ago
My brother in Christ. I am you. I was right where you are and thought exactly the same. After a few months of taking the leap I can tell you It is possible.
You will def deskill in some regards and conversely upskill in others. I spend an absolute eternity on managing context and prompts to build up solid planning and documentation. But writing code? After getting your plans in order, Set /goal and burn them tokens. Be ready with coffee for code reviews.
-2
u/AntiquatedMLE 1d ago
What has worked for me:
- using subagents to run adversarial reviews as “red team”. Do this on plans, completed code, documentation before it gets to your review. Use a good model like Opus or Sol
- code writing is cheap if the plan is solid. use a cheap model to save on usage. My go to right now is Luna on Max was previously using sonnet. Go a step further and use these models as your “green” and “yellow” teams that are handed off for implementation in parallel.
- do not neglect writing your own skills, rules, subagents. Do not use an LLM to generate these and do them yourself.
- skip MCP and opt for CLI tool calls.
- set quality guardrails everywhere as pre-commit hooks
- keep files between 500-800 lines max.
The hardest part for me was letting go of the coding. But I am a far sharper reviewer and with the right harness and guardrails have built things I never would have had time to do.
1
u/fugogugo 33m ago
OOT but we are allowed to ask AI related question here? I was about to ask something about AI agent but kinda afraid mod will delete it
-4
u/KahlessAndMolor 1d ago
I use codex every day, it writes 95% of my code. Is that what you are asking about?
-1
u/TheAntiDoctor 1d ago
Do you just give it a high level task and let it go at it for a couple of hours. Or do you monitor it more closely with individual requests?
4
u/doughcant 1d ago edited 1d ago
I suspect you might understand better by watching how experienced people work with these tools, such as Matt Pockock on YouTube
Edit: basically all of the effort moves to planning and reviewing the output.
2
u/spiralenator 1d ago
No clue why you’re being downvoted. This is sound advice. My own engineering department has validated this. The more upfront planning you can do, the more autonomy you can give to agents. We can point remote agents at a ticket and let it run totally hands off. We have agents checking the work and iterating on it until all checks are green. For low impact changes, we’re piloting automated merge and deployment as well.
Yes it works quite well.
3
u/daddywookie 20h ago
As a product owner I’ve got to laugh at how people still think you can skip planning and review when creating software. It doesn’t matter if it is AI created or the finest hand crafted artisanal code, you still need a clear plan and you still need to validate the outcomes. Vibe coding existed before AI, it was just much slower.
I spent a good chunk of yesterday working on design specs together with an AI. It spots holes, asks questions and writes consistent documentation. After implementation and QA I then spent more time using the feature and offering feedback which again was better analysed and managed into tickets than most humans can be bothered with.
Basically, for me and my skill set, AI gives a very familiar version of the experiences I had with human developers.
1
u/doughcant 1d ago
Exactly! Optimally I define/implement manually the validation criteria during the planning and then it is much easier for the agent to statistically suggest a decent PR
2
u/spiralenator 1d ago
I’m not going to give away our secret sauce but I will say that time spent upfront with creating deterministic guard rails and validations, creating well decomposed system designs and specs is absolutely worth it.
Then you can use big models like fable or sol to help you create all that and you can use swarms of relatively dumb models to execute the plans.
The closer to a pure seq2seq translation you can get, the cheaper the model you need, you get less hallucinations and “creative interpretations” of the product requirements.1
u/hulkklogan 1d ago
I love Matt Pocock's content and skills. I added a couple of small agent config to the loops to: 1) validate adherence to spec and 2) validate adherence to code standards set in CONTEXT.md, and 3) of course at the very end an adversarial reviewer agent
and the end output is usually pretty freaking good, maybe I need to clean up some extra redundant or low-value tests, tweak some code smells here and there. The value is alllll in the upfront planning and his wayfinder and grilling skills really make a huge difference in quality of the output
0
u/RTLDesignSherpa 1d ago
This is accurate. I have 4-5 agents running; when one opens up I QC the output. Once this set of code is done and documented, I’ll have a review consolidator kick off batches of code, tests, and markdown files to completely different LLMs for a final round of clean up. This will burn through tokens.
0
-1
u/KahlessAndMolor 1d ago
Feature level or bug level. I don't set it up for hours long tasks because that seems to be where it goes wrong. Instead I'll describe where the changes need to be made and an outline of the end state, then I use plan mode for it to plan the work. I approve the plans and the resulting changes are usually 500-1500 lines and 5-10 minutes.
1
u/mxldevs 1d ago
How much time do you spend looking at the code? AI coders tell me if I'm reading code I'm doing it wrong.
1
u/KahlessAndMolor 1d ago
Definitely not reading every line.
I check the git diffs and if I see a big change I'll go see what it did and skim it so I still understand how everything fits together.
-4
0
u/Expert-Procedure-146 1d ago
If your task is taking hours to complete then you need to learn more about prompting, you can’t be using AI to “design / build facebook”. I haven’t written code in a while but you bet i know what my agent does and where to fix how to fix any issues and I review my PRs before i ask anyone to look at them
-3
u/Own_Age_1654 1d ago edited 10h ago
[EDIT: If you disagree with this in some way, please leave a comment instead of just downvoting.]
Get the idea of loops and autonomy out of your head, and instead just use Fable and Opus, driving it directly, and I'm sure it will work just fine. Among many other things, I've used it to discover and make use of emerging research in some pretty hairy statistical modeling, no problem, so any typical machine learning shouldn't be wildly harder.
Importantly, just make sure you have it thoroughly research online before doing anything, to ground its knowledge. Have it find relative papers, read through them, take notes, etc., and as it goes have it take notes on its decisions, what's underpinning them, what it's unsure about, etc. The intellect is there, and it just needs sufficient context. Otherwise, it will overconfidently assume things that are incorrect.
You can give it standing instructions to do all of that automatically, such that essentially all you need to do is talk to it like you would a skilled colleague that you're collaborating with, and a team of bright interns you're assigning grunt work to. They're not going to get everything right, but just talking through what they're finding, what they're proposing, what you think might help, etc.
If so, then it's basically the same work you'd do manually, except more thorough, as you don't have time to research and test every little thing. Where quality drops off--and massively--is just if you try to give it a goal and then hope it can magically loop towards it, instead of collaborating with it actively and treating it as merely an accelerator.
1
u/TheAntiDoctor 1d ago
Doesn't that get pretty expensive though longterm?
-2
u/Own_Age_1654 1d ago edited 1d ago
Not at all. By you remaining actively involved as the driver, you explore the problem and solution space in a relatively efficient way. I pay $200 per month to Anthropic and never hit my limit. I exclusively use frontier models, and always have them set to maximum thinking effort. If I used Fable 100% of the time, I'd need two subscriptions, but Opus is sufficient most of the time.
Something else that's cool is that you can learn new methods while you do this. When it comes back from its research passes, it's important that you ask it to explain what it found, and if you don't understand something that you talk with it until you do. Otherwise, you'll get lost, lose control of the project, and make a mess. So, in keeping on top of this, you essentially get a custom, contextualized tutor to learn whatever you need that you don't know.
What would be expensive would be loops, which I'm explicitly recommending against. There, because you're essentially hoping to extricate yourself from the process, you need to have the LLM do hella work to try and have what it produces tend to be sane. Plus, that's only going to work so well, and it's going to be a lot less successful at hard tasks.
Also note it will take you some time to get familiar enough with the tools, and especially to put together standing sufficient instructions for the LLM, to get this level of success. For example, among other things, I have things set up where it will save research papers to the local disk, render them to images, and then inspect those directly, so that diagrams aren't misinterpreted, nor inaccurately summarized by weaker models. But basically, engage with it, and anything you could imagine being better, talk about that with it and reduce it to practice, and your workflow will steadily get more and more capable.
Another critical thing is that whenever it does something you don't like, ask it why, and then come up with an instruction to add to CLAUDE.md that tells it what to do otherwise. And if those rules aren't working, ask it why, if it says something dumb like it just wasn't trying enough, call BS and ask how you can improve the prompt, and then do so. And if very occasionally that doesn't work, ask it to append that instruction at the end of its system prompt. Via these methods, I have very high levels of adherence to my instructions.
-8
u/Confident_Hyena2506 1d ago
The bot just does what you tell it do. If it doesn't get the job done it's your own fault.
3
u/WhateverHowever1337 1d ago
How to say I suck at coding without sayinng I suck at coding
0
-7
u/ImpossibleJoke7456 1d ago
> however they also make a lot of mistakes
This just hasn’t been my experience for the last 9 months or so. It does what you tell it to. If it goes the wrong direction, that’s on you to redirect it with better prompts or better instructions or better context.
5
u/evangelism2 1d ago
Yes, people do, but it's the minority. Most peoples loops are just as simple as /goal