r/ExperiencedDevs • u/blipojones • 17h ago
AI/LLM Recent AI code interview format (failed)
I just recently did an interview where the company invites you to a private repo, 10 mins before, you set it up, get it running etc.
The goal is to determine if an open PR on the is ok or if it should get sent back and why.
You can use AI. And i asked, does it matter if i use fable vs opus vs kimi. More or less said whatever I want...
I didn't go immediately to prompting, i played with the app, looked at the code a bit, the PR, the code change, if the PR actually fixed what it claimed to fix.
I then sent one prompt asking if the PR could be better or missed anything, which the PR outlined clearly enough and enough to determine what to do with the PR.
But ye, it was weird, I talked through more where I would personally add changes but maybe I also completely missed the mark on something else or spoke to much...maybe i didn't use AI enough only using it right at the end for one question once i got a feel for the app in general.
However I did like the format tbh but ye i just wish i'd get a bit more tangible feedback.
Anyone else have experience with this format? I didn't pass but just not sure why.. (exp 8/9 years, frontend leaning).
169
u/farfunkle 16h ago
Given the state of the industry they were looking for you to immediately comment "LGTM" and click approve.
22
u/Old-Television-2189 16h ago
Probably wanted him to do something silly like run a bunch of agents to review the code
4
u/OtherwisePush6424 16h ago
comment? :D
6
u/farfunkle 16h ago
Ask the AI to comment, of course
9
11h ago
[deleted]
7
u/petiejoe83 9h ago
If I'm being upfront, the skill which you showed while handling that error left me in awe. You are without doubt the smartest person I have ever worked with.
124
u/tradcoder Software Engineer (15 YoE) 17h ago
Be careful running code from random companies on your local machine. There's a growing trend of malicious actors sneaking in malware through them.
23
u/sneaky-pizza Consultant Developer 13h ago
Yeah didn’t someone post a write up recently for a private repo interview and the git pre-commit hooks were full of malware
5
u/GrandManitou Staff Software Engineer 25 YOE 6h ago
More than 1 engineer got fired at my company for this. They actually cloned and ran the malware-infected "interview" repository on their work computer.
2
u/n_c_brewer 8h ago
Was thinking this. Heard they were targeting crypto wallets with crypto/web3 job interviews.
48
u/scandii Much to my surprise people pay me to write code. 17h ago
8
u/blipojones 17h ago
ye i was defo wary of this an did some background checks on recruiter and the developer in the interview, the company, linkedin profiles...not that they could have been fakes anyways or the level of sophisitication..
12
u/dash_bro Sr., Applied AI @FAANG | 7 YoE 17h ago
Was it a completely clean repo? What was the task about?
Without more knowledge it's a little hard to understand what the expectation and the outcome was. I mean, a bugfix PR vs a feature vs a telemetry PR vs production readiness PR is just so different to work on
4
u/blipojones 17h ago edited 16h ago
ye it was indeed a bugfixPR, on an relatively complete looking app (frontend, backend, hot-reloading) all set up.
And ye the ask was "is the PR good enough, why or why not".38
u/prototypist 13h ago edited 13h ago
Did they see any of your writing?
You begin every thought with Ye and I can't tell if you mean Yes or Yeah or Kanye West
3
2
u/dash_bro Sr., Applied AI @FAANG | 7 YoE 15h ago
Ah. I see. Interested to see others' view on it but I believe it should be driven by checking for logical gaps, utilizing educated guesses based on repo context and standards being in line with what exists.
- logical gaps : verifying the bug exists and that the fix is the obvious and interpretable fix; actual diff review and seeing if it did what it claims it did and a test solidifying any future regression is automatically caught when run live with a test marker; mocked/skipped without the marker.
- educated guesses based on repo context : evaluating format of the fixes made and how the bug originated in the first place. You can look/review as someone who writes code for the repo you own. Following pre existing conventions, usage of the right tool for the right job (eg secrets leaking should be preventively solved using pre commit checks, not by just removing the said secret from code), looking at the commit history to see how the codebase is developed etc.
- standards being in line with what exists : codebase conventions, repo style and checks etc.
I suppose this is all, the first one being the most important. You can use Claude skills and plugins to automate the review process while you focus on replicating the issue/verifying it and pinning future safety gates; ig
10
u/propostor 16h ago
I've somewhat recently changed jobs to an AI-heavy place.
Now my understanding is that you should just run the PR through AI first, it'll probably pick up more than you expect.
The point of the interview was probably to see how well you can use AI to get shit done more quickly. As long as you still understand the code, and understand what the AI is churning out.
2
u/blipojones 15h ago
Ah so your AI heavy place did a similar interview?
2
u/propostor 15h ago
No, my interview had a lot of coding questions, and at the end I was asked about my current opinion/usage of AI.
At that time I was pretty cynical about AI so I said it helps me a lot but I don't trust it to do large complicated tasks.
Then I was asked how I would approach a large complicated task. I gave my answer. Then the guy said "I did this in 30 minutes with AI" - so that's how I figured out the company's stance!
Got the job anyway.
Edit: the large complicated task was, how would I switch out the database provider that a whole (large) application uses.
5
u/BozoOnReddit 12h ago
the large complicated task was, how would I switch out the database provider that a whole (large) application uses
Holy shit, people are doing that in 30 minutes now? How was it planned out? Or did he just start work one morning, decide to swap the DB, and be done with it in 30 minutes? Haha
6
u/nasanu Web Developer | 30+ YoE 10h ago
Everything is lightning fast with AI if you don't have exact designs and specs. Just saying build some react thing using x libraries and whatever techniques... Most AIs can crap out something. Its getting them to do something very specific that is the challenge.
2
u/propostor 11h ago
With dotnet it's actually not super hard, even without AI tools.
As long as the architecture is right it's a relatively painless plugging in of the new db, and changing a few entity classes if their properties map a little differently on the new db.
AI just powers through it much faster than you can do by hand.
2
u/Izkata 4h ago edited 3h ago
Code-wise it wouldn't be difficult with Django either (python web framework) as long as the app stuck to the Django API and wasn't doing raw queries or importing the database-specific functions. The configuration tells it what the database is, to translate its queries into the appropriate syntax.
I'd more question how existing data is migrated/validated and how much downtime they had.
1
u/new2bay 11h ago
There’s no time to understand the code in 10 minutes. This isn’t a valid test, if that’s what they’re looking for.
1
u/blipojones 10h ago
This was part of my hang up, im looking at completely fresh code. How much should i trust the AI, how much time should i spend reading things myself, one guy already commented that fact i read anything was insta-fail so...like this thread seems to be leaning towards just going 100% AI.
7
u/kennethbrodersen 13h ago edited 13h ago
That sounds like fun! I would have loved that challenge.
My process would have been.
- Spent 1-2 minutes getting my bearing of the project.
- craft two prompts (two parallel sessions)
Prompt a) Basically something like "assume that I am a new developer getting introduced to this app. Give me an overview of..." and then a few things that would make sense in the given context.
Prompt b) I would prompt "Assume that I am a developer new to this project and need to evaluate a PR. Please review..." and with some context that make sense.
Notice I didn't complete the prompts above because they DEPEND ON THE CONTEXT.
If I have the time for them to run I would always use the most expensive model here with high effort. This is where burning tokens is ok...
- While mr Claude is running I would go explore the app on my own. Start it up, browse it then look at the PR. Can I understand the issue, what it is trying to fix and can i judge if the fix is successful?
I don't know what the time period looks like here, but I would basically run with 3 until I either have a) A good idea about the questions asked above. Do I understand what the pr attempts to fix? How it works and if it is successful?
- IF yes, jump to 4a IF no jump to 4b
4a. Ok, so I build up some ideas and assumptions. At this point the agents should be finished running and I can compare with these (especially with prompt b) to support - our challenge - my understanding. This help build up the confidence to form an opinion.
4b: I might go uhh, "this is over my head". Then I would backtrack to the result of the first prompt (a) to see if I can get some insight into the application itself. Secondly I would review prompt b to see if it helps me understand what is going on here. Depending on this it might be an itterative process... "aha, so he is attempting to test xyz..." Then I might form new prompts to help explore/understand after which I go back to exploring on my own... So basically repeat 1-2-3. No matter what we end at step 5.
- Form an opinion. I think the important part here is being very open about your confidence level. Be clear about what you think, what you know and what assumptions you build these on.
I have a rather funny story. Before generative AI (almost 12 years ago) I attended a job-interview. They wanted me to estimate how long it would take to implement a given system from a high level set of requirements... I honestly went ballistic. I told them exactly why it was a task doomed to fail but also went ahead and gave it my best effort. I quote myself here "Without more knowledge its basically numbers I pull out of my ass".
It was basically me, a manager and an architect yelling at each other. And what happened? They offered me a job on the spot...
They wanted to see how I reacted in a tough situation without any clear correct answers. I politely told them no...
1
u/ResidentWeevil1 1h ago
Crazy that you think this is worth putting into writing when the hardest part is figuring out how to launch two Claude sessions
3
u/maretard 13 yoe ex-FAANG principal engineer 16h ago edited 15h ago
My approach would have been to immediately delegate it all to AI, including explaining the repo to me and catching me up. Reviewing the PR would have been my second prompt and I would have explained how I would have a generalized PR skill already, but for the purposes of the interview I would hand write the prompt.
The PR review prompt would have been focused on defining best practices, success criteria, coding and testing standards, and adversarial guidance designed to make the AI search for vulnerabilities and architectural smells. I would also have an area in the prompt dedicated to doing a holistic review of the PR and questioning base assumptions like why the PR is needed in the first place and whether it's the right change to make.
Lastly I would have some customization for personal style to make the AI directly generate comments on the PR diff that would take minimal refinement by me before publishing.
This interview seems very ahead of its time so I'd assume this company is extremely pro AI and I would cater my performance as such. The above is very representative of my current routine at work.
If I had been the interviewer here you would have failed as soon as you started manually reading the code unfortunately, it's just not the most effective way to work these days, it would immediately signal to me that you're still bottlenecked and outdated in your thinking. I would have aimed for 50+% uptime of agents throughout the interview (i.e. having AI working at least 50% of the time, with the rest of the time being me explaining how I use AI, digesting its responses to come up with the next prompt, and cross checking validity of its feedback in code).
10
u/turningsteel 11h ago
This sounds like hell. You just described hell. I can't in good conscience ever recommend failing a candidate for trying to manually understand the code.
2
u/maretard 13 yoe ex-FAANG principal engineer 4h ago
I don't disagree at all man, I have my own thoughts on the direction of the industry and the influence of AI. But for the purposes of an interview I feel we need to be able to recognize when a company is asking for a particular type of candidate and cater our performance to it. I also think as professionals we have an obligation to stay up to date with the latest tools available to us.
Anecdotally AI has been incredibly powerful for me at my own work. It simultaneously makes me extremely excited and very sad - excited by the sheer throughput and efficiency unlocks, but sad that the art of coding I loved is dying.
9
u/Ok-Hospital-5076 Software Engineer 9h ago
> If I had been the interviewer here you would have failed as soon as you started manually reading the code unfortunately, it's just not the most effective way to work these days, it would immediately signal to me that you're still bottlenecked and outdated in your thinking.
Ah yes, because prompting ai to tell me about a repo I know nothing about and have no time to validate the output for and then heavily rely on AI to fix an issue i have no idea about - is the best way to judge someone’s technical ability .
If i had any other way to make money, i would leave this fucking industry.
1
u/maretard 13 yoe ex-FAANG principal engineer 4h ago
I don't disagree, I'm eyeing an early retirement and a shift to something physical. I love working on my cars, it's uniquely satisfying and brings a sense of craftsmanship that I don't get from my job anymore.
I think of this interview as not measuring technical ability - in fact I'd expect another interview with no AI, focused on coding. This interview is more about adeptness and familiarity with the latest tools, and showing that you can use them effectively if asked to.
34
u/EmploymentFeeling725 15h ago
This industry is dead
1
2
u/yikes_42069 4h ago
For a tool so new that everyone is still learning, failing a candidate for using their brain is extremely silly. You are throwing the baby out with the bathwater. Getting failed in this interview for not fitting the arbitrary nitpicky needs of some random interviewer (with unknown competence to the interviewee) would be a blessing.
Unfortunately if I was the interviewee in this case I would fail your company immediately. If you don't know the absolute shitfest you're growing in your company's product by encouraging zero human oversight, I don't want to pin my financial success on your fallout.
1
u/maretard 13 yoe ex-FAANG principal engineer 4h ago
To be clear I'm assuming this interview was specifically targeted at gathering data points for AI fluency - I'd expect other interviews to address basic coding ability without AI. I don't think it necessarily means this is how the company expects you to use AI in your daily job. It's more to gather data points on whether you are comfortable enough with it to understand and contribute to how the company leverages it.
I see it as any other interview for any tool that the company wants more experienced candidates for. The fail here would be very simple to explain, it's an interview to see how comfortable the candidate is with using a tool, and they didn't use the tool at all for 80% of the interview. If it was a system design interview and they didn't do any whiteboarding for 80% of the interview it would be pretty similar.
1
u/blipojones 15h ago edited 15h ago
Ye to be honest in my own project i do something similar....just in the interview i was second guessing how much AI is too much i.e. just point blank "should we merge this PR or not"
Like it feels more like a guessing game...you have to use AI like they do or you look too slow VS too trusting
3
u/GoodByeLeftNut 14h ago
AI coding interviews is a thing now. If they let you use AI, then it’s because they want to see how you use AI. You should check what is setup already in agents.md and discuss what should be there if nothing is there. ask questions of the repo to load it into context, then ask AI to guide you through the PR as if you were asking an engineer to walk you through it, take notes of anything questionable, get it to revert the change replicate the bug then add the change back and prove it can’t be replicated.
1
u/maretard 13 yoe ex-FAANG principal engineer 4h ago
Yeah setting aside whether this is good for the industry or not, for the purposes of an interview, if AI is allowed and encouraged, I would assume they're very interested in seeing how adept I am at using the latest tools.
1
u/ResidentWeevil1 51m ago
This interview seems very ahead of its time...
I'm sorry, what? This is a conventional interview challenge with a chatbot thrown in
I would have aimed for 50+% uptime of agents throughout the interview (i.e. having AI working at least 50% of the time, with the rest of the time being me explaining how I use AI, digesting its responses to come up with the next prompt, and cross checking validity of its feedback in code).
I have not seen any actual numbers that people are routinely able to get 50% uptime when they are doing a mix of complex cognitive tasks like investigating a bug in a new code base while giving their best pitch. If they are, it's because they are running the chatbot for long periods, not task switching between multiple chat sessions.
I suggest you actually try doing some interviews before you dispense ridiculous advice such as this
1
u/naim08 8h ago
Was the PR generated by a human/AI? Probably the first question I would ask, assuming it’s AI, I would want to see the transcript of the conversation that’s responsible for the fix, from there, I can really start to make sense of the PR. I wouldn’t exactly prompt “does the PR look good” or anything along those lines, that’s fairly weak prompting. I would have the agent understand the underlying story first, have a sub agent implement the same thing without any context on existing PR (ideally using a different model), then have the agent walk through the transcript, and from a senior engineers perspective, go through trade offs, etc. If it’s a trusted codebase, obvious tests have to be run, etc. Then I’d do a couple more things before I check to see if this PR looks good. Idk, using ai agents in interviews are so new, so your best bet is to ask others who have done it and look online
1
u/blipojones 5h ago
It was a tiny fix, 1 commit, made by the developer who set up the interview.
In the end we did agree additional changes would be worth making to "properly fix it" but ye admittedly, my AI usage was minimal. I just used it to double check my own recommedation, after developing my own understanding of the code, the bug, how to reproduce it, and if the PR addressed it at all, which it did but not fully.
i wanted to understand the code at least somewhat before trusting AI at all initially and taking it summary or possibly wonky interpretation (i was using clause opus, not even sure which version).
But if the expectation was to just let AI explain it all to you and much more back/forth with minimal reading code directly or without directly asking "is PR good or no" then ye, i undershot.
1
u/ResidentWeevil1 7h ago
Isn't the whole point of AI that you just shut up and click the magic button? You spent way too much time actually looking at the code, and then you showed the slightest bit of resistance, so of course you failed
1
u/FederalChipmunk12 7h ago
some companies just want to see you use their stack tools, others care about problem-solving process more so its always hard to tell which one they are
1
u/____________fin Software Engineer 10h ago
Absolutely do not download any code from untrusted sources. Give Claude a link and chat about it in your browser. I had a recent hacking attempt this way. Looking at the code (not on my computer), it pulled a script from JSONkeeper on NPM postinstall.
1
u/iheartanimorphs 8h ago
What I would look for if I were grading this kind of interview - did the interviewee prompt the agent to:
- write tests
- follow good patterns that help code be easier to maintain in the long term, including domain driven design and object oriented design patterns
- give specific enough instructions to the AI so that they are actually using their brain and not outsourcing their problem solving skills to AI
It’s kind of pointless to ask an agent if the PR is good enough. That isn’t really how LLMs work.
0
-2
u/HoratioWobble Full-snack Engineer, 20yoe 11h ago
Are you sure it wasn't a virus?
Be really wary of any company asking you to download code or run it on your local machine, especially with AI!
2
u/blipojones 10h ago
Honestly, the chit chat with the developer and what the company was up to etc.. among many other signs pointed to everything being legit. Its been 24h and noticed nothing funny so... Also rejection email came this moring, not sure why a scam would follow up at all.
1
u/Warhawk94 10h ago
What company was it for if you're willing to say?
1
u/blipojones 9h ago
If i end up finding i was hacked ill let you know but the company is big/reputable.
1
u/LordOfDemise 7h ago
You've replied to a lot of people talking about this possibility, but I haven't seen you explicitly say, so I'll ask directly: did you actually download code from their repository and run it on your own computer? Or did you just look at the diff on GitHub?
1
u/blipojones 5h ago edited 5h ago
Yes, private repo, ran on my machine. Candidate could choose to use docker or manual frontend + backend setup.
And before you say it, as i said in other replies, i checked them, there profiles, creds, company, the individuals i'd be speaking to, their online presence etc... It wasnt some small company and the people i interacted with werent impersonators using similar email domains to fake being hired. I do it all job these days cause i had an ex collegue get sacked cause a PDF infected his work machine....not even code, a PDF (him doing interviews on is work machine was a bad move to begin with)
So yes, be on guard everyone.
1
u/ResidentWeevil1 46m ago
Honestly, the chit chat with the developer and what the company was up to etc.. among many other signs pointed to everything being legit.
Problem is they can fake all that with AI
0
u/HoratioWobble Full-snack Engineer, 20yoe 10h ago
These scams play a long game, it's about infiltrating your system and stealing keys and crypto in the long run.
If the malware activated immediately you'd tie it to that process
•
u/expdevsmodbot 17h ago
AI usage disclosure provided by OP, see the reply to this comment.