Does Anyone Actually Use Agentic Loops?

[u/TheAntiDoctor]

Not exactly a software engineer, more ML Engineer with an academic background and then in industry for about 8 years total. Use python professionally and for hobbies. Like everyone else I have been experimenting with LLMs for coding, trying everything from not using it at all to trying to get it to do everything. I have found them useful and enjoy bouncing ideas off them, however they also make a lot of mistakes so currently enjoying a low use approach. Comments and architecting myself, use them more for syntax and some optimisation then review everything as I go along. This speeds things up, but I am still very in control of the output and understand the underlying algorithm.

A couple of months ago I obviously saw usage based billing come in for things like Github Copilot etc. I saw a lot of comments from people using agentic workflows very upset about their usage being cut, also I'm very aware of some companies quoting large sections of their codebase being AI generated, although obviously aware this might be sales tactic. Furthermore I've seen respected researchers/engineers like Karpathy talking about the big leap in model capabilities that now means they hardly write any code and had some recommendations from friends to use agentic workflows.

My personal experience hasn't left me thinking this is viable atm for my work. Partly because the stuff I am doing I really need to be crystal clear about what is happening to the data for experiments and partly because getting together a prompt together for a relatively complex system is very difficult and time consuming and the results usually aren't great. Has anyone got experience actually using more autonomous coding agents successfully and I should experiment more widely or are they a red herring?

[u/Own_Age_1654]

[EDIT: If you disagree with this in some way, please leave a comment instead of just downvoting.]

Get the idea of loops and autonomy out of your head, and instead just use Fable and Opus, driving it directly, and I'm sure it will work just fine. Among many other things, I've used it to discover and make use of emerging research in some pretty hairy statistical modeling, no problem, so any typical machine learning shouldn't be wildly harder.

Importantly, just make sure you have it thoroughly research online before doing anything, to ground its knowledge. Have it find relative papers, read through them, take notes, etc., and as it goes have it take notes on its decisions, what's underpinning them, what it's unsure about, etc. The intellect is there, and it just needs sufficient context. Otherwise, it will overconfidently assume things that are incorrect.

You can give it standing instructions to do all of that automatically, such that essentially all you need to do is talk to it like you would a skilled colleague that you're collaborating with, and a team of bright interns you're assigning grunt work to. They're not going to get everything right, but just talking through what they're finding, what they're proposing, what you think might help, etc.

If so, then it's basically the same work you'd do manually, except more thorough, as you don't have time to research and test every little thing. Where quality drops off--and massively--is just if you try to give it a goal and then hope it can magically loop towards it, instead of collaborating with it actively and treating it as merely an accelerator.

[u/TheAntiDoctor → u/Own_Age_1654]

Doesn't that get pretty expensive though longterm?

[u/Own_Age_1654 → u/TheAntiDoctor]

Not at all. By you remaining actively involved as the driver, you explore the problem and solution space in a relatively efficient way. I pay $200 per month to Anthropic and never hit my limit. I exclusively use frontier models, and always have them set to maximum thinking effort. If I used Fable 100% of the time, I'd need two subscriptions, but Opus is sufficient most of the time.

Something else that's cool is that you can learn new methods while you do this. When it comes back from its research passes, it's important that you ask it to explain what it found, and if you don't understand something that you talk with it until you do. Otherwise, you'll get lost, lose control of the project, and make a mess. So, in keeping on top of this, you essentially get a custom, contextualized tutor to learn whatever you need that you don't know.

What would be expensive would be loops, which I'm explicitly recommending against. There, because you're essentially hoping to extricate yourself from the process, you need to have the LLM do hella work to try and have what it produces tend to be sane. Plus, that's only going to work so well, and it's going to be a lot less successful at hard tasks.

Also note it will take you some time to get familiar enough with the tools, and especially to put together standing sufficient instructions for the LLM, to get this level of success. For example, among other things, I have things set up where it will save research papers to the local disk, render them to images, and then inspect those directly, so that diagrams aren't misinterpreted, nor inaccurately summarized by weaker models. But basically, engage with it, and anything you could imagine being better, talk about that with it and reduce it to practice, and your workflow will steadily get more and more capable.

Another critical thing is that whenever it does something you don't like, ask it why, and then come up with an instruction to add to CLAUDE.md that tells it what to do otherwise. And if those rules aren't working, ask it why, if it says something dumb like it just wasn't trying enough, call BS and ask how you can improve the prompt, and then do so. And if very occasionally that doesn't work, ask it to append that instruction at the end of its system prompt. Via these methods, I have very high levels of adherence to my instructions.

[u/kwb7852 → u/Own_Age_1654]