I'm curious what other folks are seeing in a post-AI hiring landscape and how you are approaching candidates that tell you they are fully agent-pilled when it comes to their development process. Should I keep doing the conventional leetcode and design interviews? Or just give up and expect everyone is going to use Claude Code no matter what?
- engineers on the call as pairing assistants - google, ai tools, whatever for libraries, syntax, etc. we tell the candidate directly that it's impossible for us to gsther signal on how they think about problems if they ask Claude to just whip them up a solution - the expected output - their own unit test suite
Every candidate that has submitted an ai submission thus far has failed because they have literally no idea where to go. I've interviewed dozens at this point. I'm not saying "they're unfamiliar with the structure", I'm saying "they cannot actually break down the problem even verbally".
It doesn't matter their pedigree or past experience on their resume, if they used AI to generate they don't seem to be able to resurrect the skills that actually matter for the thing, engineering and product work.
Note because I know folks hate code submissions. It's not hard. We give a CSV with 3 columns, 10 lines. Do some basic mapping and some structuring, some basic data modeling. We only expect about 1 actual class or struct. Then unit tests and it should run in the terminal. Max submission length with verbosity has been a java program at something like a hundred lines total if that, most folks complete the submission in an hour or two. Extension is that we modify one of the rules and extend the CSV by 5 lines.
I seem to still be getting good signal from this, since the engineers that I've hired off of this have been fantastic with or without AI tooling immediately in their hands during the day.
So far it provided good signals
On the interview side, I have gone through a phase which required do an architectural design task on a whiteboard. I realized how stressful and not very helpful at evaluating it might be when I myself had to sit on the interviewee's side several times. It's difficult to get into this play-design mode.
Currently, I just make natural conversations, ask some technical questions on the level where I estimate the candidate is, try to understand the level of their knowledge and whether I would like to work with them in one team as a person. Maybe it wouldn't work for highly specialized roles, but for a general backend work I have been happy with the results.
For your question, I assume you want people who can solve problems, can explain their thought process and reasons of decisions. I did interview devs recently, and my questions were related to their experience in their CV. Something like, tell me about project X in Y company, what did you do, what did you use, why, what did you learn from it.
My personal opinion is that LLMs are no difference from developers who over engineer and over complicate things. You can get them to do good work with the right steering and right understanding of the whole system within the context of the company.
Honestly one of the most fun interviews I've ever done - it involved really understanding the problem, and building an interesting solution. AI was totally allowed and expected, but you had to prove you understood the problem and what you'd made in the interview. I felt it was actually way better than traditional interviews, because it gave a chance to show the actual quality and thought of my work rather than a glorified logic puzzle.
As for the questions you’re asking, instead of asking old-school coding interview questions ask them stuff that is actually relevant to agentic programming.
Ask them to build a harness, have them explain what skills are, have them build a self-made implementation of Claude Code as take-home assignment.
The times have changed, you need to update interviewing questions to meet them. Start with agentic questions and ramp them up into more complex scenarios. This is how you gauge your candidates, see how they think and solve their way out of it.
Try to determine how creative, adaptable, motivated the candidate is. Figure out what there actual skill-sets are that they can bring to you, not whether they know some CS pedantry
Give them a time-constrained challenging problem that is wide in scope and see how well a candidate can decompose it before feeding it to AI. You can get pretty high signal within the first 2 prompts. Are they able to effectively steer the model or do they just ride with the flow and accept every AI suggestion?
How well do they know their models and their limitations? Are they able to switch tools/models effectively on the fly depending on the task? Or are they just using cursor auto mode and copy-pasting the task description into their IDE? Do they have a custom harness/workflow? What skills are they using, if any?
Judge them on quality of the output first. And pay attention to their taste.
Maybe this sounds a little strange, but this has worked exceptionally well, as we have now several juniors working with us, that do still get checked by seniors. The harness itself which holds several levels of standards, rules as well as quality gates, allows these juniors to ship out a huge amount of high quality code. We still reject about 80% of the people who interview with us, because even though they know how to code with AI, their AI interaction patterns just aren't on par with what we need. I really started to like this process. Also to clarify, the harness isn't some Coding agent setup, but rather a whole setup that works well with Codex, Claude or Cursor; its well maintained versioned and constantly improved upon, with a huge amount of automations, rules and hooks. The moment someone starts working on anything at all, this is immediately connected and tracked with project management.
I tried 2 flavors of a technical interview and this round was 1 hour in total.
The first was a traditional interview problem where we give the candidate a codebase and a docker image to run against. The problem itself involves using the docker image to post and get responses from but the candidate is expected to write a new helper/service in any language they choose. We ask for no agentic workflows for this but any other resource is fair game. This has been the most successful for us to find candidates and generally provides the best experience.
Our full agentic interview involves an existing fullstack typescript codebase that we give to the candidate a week or two before. The candidate is free to use any agentic workflows they want during the interview and we give them a ticket that we went to implement during the interview. I find a ton of variability in this interview and a lot of people will just take the entire ticket and one shot into Claude without prepping. This is lead to the biggest disparity in results of candidates so we stopped doing it.
Agentic engineering is the new microservices craze. Everyone's doing it, and some will do it exceptionally badly. This is always how these trends go. The anecdote you give here of some people just one shotting it in Claude Code without thinking about it and this producing extreme variability matches my experience. Basically, the longer you go down this road you realise some people will still actually think about things and some people will just cede cognitive control and stop thinking much at all. If you're the former, working with the latter is exceptionally draining over the long-term. I have in recent times become pretty bearish on agentic engineering for this reason.
Are they interesting to talk to? Are they passionate, curious, authentic, opinionated? Do they have a product mindset? Do they need to be told what to do? Would you hang out with them after work? Can they explain themselves in a way that’s easy to understand?
In my experience these are much better predictors of success than leetcode.
This is all fine and good at my current job, but were I to ever go back on the market. What an uncomfortable conversation!
How does one approach this weird state we're in?
I think that was a nice test, but the platform is too buggy to recommend.
Then 1-2 day paid work trial.
Interestingly…no one has said no to the trial.
It turns out that people want to derisk their decision just as much, if not more.
Still, it does benefit the hiring company, just not the applicant in these situations.
Not sure it benefits the hiring company when they are artificially reducing the pool of applicants to only people who don't already have jobs.
Not that there's shame in not having a job, especially in today's market (shit happens, I'm sure lots of very qualified people are currently looking), but there are solid statistical reasons why social proof of already being employed is seen as a positive.