The State of Interviewing in Tech

Published: 2022-07-05

The tech industry often debates the best way to interview candidates. We tend to agree that the popular formats are broken, and some of us even go an extra step – suggesting alternatives. So, let’s talk about it. First, we should talk about what a technical interview today typically looks like, specifically for software engineering/developer roles.

Most companies follow what is commonly called “leetcode-style” interviews. In such interviews, candidates are asked to solve one or more problems using data structures and algorithms the would-be-employer expects them to know… for some reason. It’s a popular opinion that most of these questions have little to do with the day-to-day job of being a developer. Additionally, there is the time pressure of trying to solve these problems in approximately 40 minutes. Actually… just “solving” the problem isn’t always good enough; the optimal solution is often required to pass. This means discussing time/space complexity, edge cases, trade-offs, etc.

Story Time

I like to contemplate how interviewing got to this state. Maybe, in another parallel universe, wholly divorced from the reality we live in, it could’ve happened something like this.

Imagine it’s the early 2000s, and there is this company called fa… ma… goo… lix… soft. Yeah - Famagoolixsoft Corp. The name was chosen in some obscure raffle.
Famagoolixsoft is a big software company. They initially became successful by creating a couple of revolutionary technologies, then leveraging their first-mover advantage to stomp out any competition. But after some close brushes with anti-trust regulation, the leaders at the company realized there may be more money to be had from anticipating users’ wants/needs. If possible, before the users themselves even know. That could take many forms: serving content, using data about the user to target them with relevant ads, and even selling the goods/services the user will need.

Anyway, Famagoolixsoft had dreams. They didn’t just want to be “Big Tech”; they wanted to be a behemoth. To do that, though, they would need a ton of software engineers. Fortunately for them, a recent stock market crash saw many budding tech companies go under, creating a buyer’s market for anyone searching for tech talent. Additionally, universities were still churning out CS graduates, and the company was more than willing to take on new grads and train them.
So the question then became, “How do we go about hiring the right folks?”
Thought leaders at Famagoolixsoft held a roundtable to discuss hiring strategies. Here’s an extract from the stenographer’s meeting minutes:

Bill: Alright, guys. We have a lot of people we will need to hire rapidly over the coming years. How do we go about doing this?

Fatima: How fast are we thinking of growing? How many candidates are we thinking of interviewing? We must remember that every moment our engineers spend interviewing is a moment they’re not building features.

Sunny: Those specifics can be worked out later. I think it’s more pertinent to discuss what our interview process will look like.

Larry: What’s wrong with our current process of asking brain teasers and trivia? If it ain’t broke…

Fatima: Except that it is broken. How does knowing “how many marbles can fit in a Geo Metro” relate to coding?

Larry: It allows me to see how they approach problem-solving. The ones who can solve it are usually brilliant.

Fatima: They’re usually also a pain to work with.

Larry: Pfft! Anecdotal evidence proves nothing. My org ships!

Bill: Larry, you can continue hiring like that in your org if you’d like. Fatima, did you have an alternative process to suggest?

Fatima: We could give them take-home projects. Give them a week to build something that should only take a few hours.

Jeff: How would we know if it only took them a few hours versus the entire week?

Fatima: Does it matter?

Larry: Only if you care about delivering features on time.

Fatima: We hire engineers, expecting them to be unproductive for months while we train them on tools and practices. Their speed will increase as they get familiar.

Jen: I think the optics may be poor: “Famagoolixsoft exploits candidates through unpaid labor.”

Fatima: It’s a few hours of work, and we wouldn’t be asking candidates to build anything we’d use.

Jen: You know that won’t matter to the media. It’s a branding thing.

Fatima: Fine. Then pay them.

Bill: Excuse me?

Fatima: Again. We hire engineers and pay them top dollar for months, expecting nothing in terms of productivity. A few dollars is a rounding error in the grand scheme of finding the right employees.

Jeff: But you still have an issue. How many projects are you gonna think of? The first few times you ask it, you’ll get submissions from persons who genuinely built it. But then someone will put the question/solution on Sourceforge or some site that collects info about interviews. How do you account for copycats?

Fatima: Of course, the candidate would have to present the project; explain their thought process, decisions, obstacles faced, and how they overcame them. That gives interviewers a standard bar against which to grade candidates.

Bill: Your org’s headcount is small enough that I’m willing to let you hire like that – for now, at least – paying candidates. Is that okay with you, Jen and Jeff?

Jen: I still think it’s a bit wild to pay for useless work, but I like Fatima’s evaluation criteria. Instead of giving the candidates tasks, we could ask them to present a project they’ve worked on. We get the same insights, the candidate gets to talk about something they (hopefully) liked creating, and we don’t burn cash. No need to worry about copycats of project difficulty. The burden is on the applicant to talk about a project that’s personal/unique to them since cookiecutter, simple projects won’t stand out.

Bill: Good thinking, Jen! Fatima, do that instead. It saves our money, image, and engineering effort.

Jeff: I’m not sure this style would scale–

Bill: We’ll cross that bridge if we come to it.

Sunny: Okay, now we can work out some specific figures–

Bill: Sorry, that’s all the time we have for now. I have a lunch meeting.

With that high-level meeting done, Famagoolixsoft overhauled their interviewing process with most organizations hiring by evaluating work that candidates had previously done. It seemed to work. They were able to hire fast. The new employees came in at different skill levels, but Famagoolixsoft had also invested resources in onboarding and training new staff on the tech stack, custom tooling, and processes. The task of building an empire was well underway.

Fast-forward to the early 2010s. The aforementioned interview process has been in place for almost a decade. There are now evident growing pains. The thought leaders gather for another summit on hiring:

Bill: Alright, folks. Thanks for being here. We have a lot to discuss, so let’s dive right in. First, let’s level-set and outline the challenges we face with our current process. Then we can craft a targeted response.

Jen: We may be able to modify the current process. It worked well when we had 2500 engineers, but at 10,000+, it becomes… overwhelming.

Sunny: Actually, the size of the workforce by itself isn’t the issue. The problem is that the number of applications we receive eclipses our available interviewers by a couple orders of magnitude.

Larry: Yeah. We’ve invested engineering effort in automated screening tools to filter out applications. But, say we’re looking to hire 3,000 people this year. We’re getting around half a million applications. Even if our tools can weed out 80-90% – which, today, it doesn’t – it still leaves too many candidates to interview. At this point, “time” eliminates more candidates than our interviewers. Getting an interview doesn’t mean you were the most qualified candidate in the pool. It just means you were lucky enough to get picked.

Bill: That isn’t specifically a fault with our interview process, though. You’re basically saying we’re victims of our success; everyone wants to work here.

Jeff: Bill has a point. We have a finite capacity of interviewing hours. Saying that the demand far outstrips the supply says nothing of how effectively we use the supply.

Larry: Well, how about the fact that our interviewing process appears optimized for hiring great storytellers?

Jeff: Meaning?

Larry: We allow them to present projects. They get to craft stories, and it’s not tricky to prep answers for “What challenges did you face?” or “What would you do differently?”

Jeff: Sounds like another behavioral interview. With props. Has that affected the quality of the candidates hired?

Fatima: You end up with a mixed bag. Some engineers need more guidance or time to onboard, which can cause team friction.

Jen: But we budget for that. If they can’t ramp up in 6 months, they will go on PIP, and the established procedures will be followed.

Fatima: Yeah - so the question is: can we evolve the process, so it’s less of a mixed bag?

Bill: Any other specific issues?

Sunny: We need a standardized, company-wide process for two reasons:

• Leaving hiring to a team/org themselves can be overly burdensome. With our growth, we’re constantly creating new lines of business. When a new team is spun up, they have a significant headcount and need to hire fast. It could mean months of the new team doing nothing except interview, hire, and onboard. Meanwhile, some teams are responsible for stable products and have no headcount. They tend to have experienced engineers who would be invaluable to the interviewing pipeline. We can efficiently use resources by having everyone interview candidates to join the company instead of a specific team. Once you clear the bar to be an engineer at Famagoolixsoft, you can be matched wherever there’s a good fit.
• We value ease of mobility between teams. Without uniformity, candidates can effectively pick their preferred process, spend a few months on the team, and then switch. For example, many candidates don’t like brain teasers and trivia questions that Larry’s org tends to ask. But Larry’s org has exciting projects. A significant portion of his headcount gets filled by internal transfers. It leaves other orgs in a perpetual hiring loop.

Larry: It works for me. My org gets proven employees who’ve already ramped up.

Fatima: Don’t forget that most of your headcount comes from persons leaving your org due to your practices and ridiculous mantra.

Jeff: Speaking of which, Larry, the data says your interview style does not yield better employees than our other methods.

Larry: Are they worse?

Bill: Larry, we’re gonna discontinue brain teasers and “gotcha” questions. We could weigh the pros/cons if it was a better style. But it isn’t. It is better at getting employees with poorer soft skills, like teamwork and communication.

Larry: This is BS. My org has built some bedrock technology that helped this company become what it is today.

Bill: That’s true. And we’re focused on building the company of tomorrow. The time of “mavericks” and “tortured geniuses” is ending. Evolve. Don’t be like Stan.

Jen: Soo… we’ve discussed the pain points. Ideas?

Fatima: Are we just gonna scrap the current presentation/discussion process?

Jeff: I don’t think so. I think it’s still a good style for the system design interviews we typically need for senior engineers and architects. It makes sense at that higher level as we can dig deeper and expect richer answers – whether the candidate discusses a project they developed or a hypothetical problem/solution given in the interview.

Sunny: It’s also a much smaller fraction of our total applicant/interviewing load.

Fatima: So what about the rest? How do we interview for lower/entry-level roles?

Jeff: We can think of this as a classification problem with a confusion matrix. We have a set of candidates. Some are qualified; some aren’t. We’re devising a process to separate the groups. But we don’t care about the accuracy; only the precision matters. Meaning that the false-negative rate can be high. We only need to optimize to keep the false-positive rate as low as possible.

Jen: In lay terms?

Fatima: He means it doesn’t matter if we reject many qualified candidates, as long as we don’t accept unqualified ones. He has a point.

Jen: But shouldn’t we be optimizing to hire the “best” candidates?

Fatima: Yes… and no. That might matter at more senior levels where candidates bring expertise that we don’t have. But at lower/entry-level, there is no “best” candidate. Provided the candidate is above some qualification bar, the best one is whichever one we hire. We invest in onboarding and training new hires. After about ⁶⁄₁₂ months, we have the best candidate for the job because they’re capable engineers, trained to think and operate a certain way, and comfortable with our custom stack/tools. Nothing outside of Famagoolixsoft can prepare candidates for life here. We just need to ensure the ones we hire have a foundation we can build on.

Jen: I see. So we just need to define what that “bar” is.

Larry: For sure, we need them to do some actual coding as part of the process.

Sunny: How about we go through interesting bugs/issues we’ve had to solve in the past and see how they think and code the situation?

Fatima: But it would take so much time to provide context

Bill: Not to mention, we’d have to vet those questions to ensure there’s no security risk, no IP being leaked.

Fatima: We should be able to extract the core of the problem and pull in any dependencies as simplifying assumptions.

Jeff: And we can leave the initial question a little ambiguous to see how many assumptions/edge cases the candidate makes vs. verifies.

Bill: How many such questions do you think we can create?

Fatima: Each team can look through their bug history. Each time they found inefficiencies, had incidents, switched data structures, hit some limit or edge case… We should be able to curate a few hundred scenarios altogether.

Bill: Sounds good. Any objections?

Larry: I’m okay with it.

Sunny: What about standardizing the process across the company?

Bill: Yeah - we’ll try that as well. It won’t solve the ballooning application load, but it should ensure the load is well spread.

The stenographer, being extremely good at their job, also sketched a diagram of the confusion matrix which Jeff was describing:

Interestingly, in this parallel reality, there is a phenomenon referred to as “imposter syndrome”, where many developers who make it into companies like Famagoolixsoft feel like they don’t belong there. Perhaps, they see others who didn’t get in and think, “But that person is at least as good as me… How come I got in and they didn’t?”
They think they’re in that red box. If only they knew the lengths companies went to to ensure no one gets in through that red box. If only they knew that most folks are probably in the blue box. If only they knew that the difference between folks in the blue and green boxes is primarily… luck. If they knew this, then maybe they wouldn’t feel like imposters. Or perhaps nothing would be different because they have to work with people and, in this parallel reality, people aren’t always kind to each other.

After that meeting, Famagoolixsoft set about evolving its hiring practices again. After a year, it wasn’t (yet) clear whether the new hiring strategy was better than the last. But it was clear that the interviewers were in better moods. They were happier discussing issues they had faced in the past and hearing candidates’ thought processes and problem-solving approaches. The candidate had to explore a problem domain, develop a workable solution, and implement it.

By the mid-2010s, there’s more data on how new hires under the new process go on to perform at Famagoolixsoft. Turns out there was a marked improvement! Almost no one from the “red box” gets hired. Teams are confident that new hires can do the job; they ramp up and become productive within the expected timeframe. In short, new hires meet expectations.

But then 2017 rolls around, and leaders have started noticing a drop in the process’s precision (i.e., false positives increasing). Aiming to arrest the decline, they hold another roundtable.

Bill: Hey, team. Thanks for coming. So… do we need to overhaul the process again, or what?

Jen: I don’t think so.

Larry: Our process was clearly working for a while. What happened?

Fatima: Innovation.

Larry: Eh?

Jen: We offer six-figure salaries. To new grads. To bootcampers. People are going to try and sell pathways to landing a job here.

Jeff: Persons have always done that, though. Resume gurus. Interview coaches. That’s not new.

Fatima: What’s new is that it’s being done at scale. Now, companies specialize in curating the questions we ask, crowd-sourced from candidates. They build communities that solve these problems in public to help future candidates.

Larry: It sounds like we just need candidates to sign NDAs before interviews.

Jen: That may stop the few, not the many. Questions are posted anonymously.

Fatima: So now there are candidates that can study our interview questions and come in prepared.

Jeff: How do we stop that? Using new questions will only lead to more leaked questions and just put us in a loop.

Fatima: Right. But I think we can use it to our advantage if we make the loop big enough. There’s a finite number of unique incidents, scenarios, data structures, etc., that we can ask about. But there’s a nearly infinite number of variations to each problem. Average candidates can maybe memorize 50… 100 problems? No one is going to remember how to perfectly respond to a thousand. And if someone does, they’re exceptional, and we’d want to hire them anyway.

Larry: I see… so you turn it into a pattern matching issue. To see who can recognize the underlying problem and apply the appropriate solution.

Fatima: Exactly!

Jeff: But the community will figure that out and start categorizing problems.

Fatima: That’s okay. The candidates can prepare by studying every category, data structure, and algorithm approach. They still must recognize the relevant ones in the interview and discuss the trade-offs.

Larry: It’s beginning to sound like a college exam.

Jen: That does match the background of most of our candidates. It also has the added advantage that you don’t need a degree to become good at recognizing problems. With the emergence of such companies and how accessible information is, aspirants can attend a boot camp, learn to code, and then study for our interviews. It’s much shorter than a four-year degree, and we expand our pool of applicants.

Larry: Why would we need to expand our applicant pool? Last I checked, we were having issues with the current volume, which is only getting worse.

Jen: You know as well as I do that many people are coming into tech, switching careers, and carrying invaluable experience and perspectives.

Jeff: True. Many leaders and engineers working on our health tech products transitioned from that industry.

Larry: Okay… But do we not care that the questions are divorced from what the day-to-day job looks like for engineers?

Fatima: Are they any more different than the questions we ask today? By the time we strip out the specifics and insert some assumptions for ease of explanation, the questions are already entirely foreign. Many interviewers ask questions that require a trie to solve optimally. Have most ever used a trie on the job? Nope. But a few engineers in the company needed it at some point to solve a problem that only Famagoolixsoft had. Then they “sanitized” the scenario and distilled it into a question general enough to add to our curated bank. The “day-to-day” of our engineers is impossible to replicate. We use a custom version control system, code editor, repository host, search engine, review tool, monitoring system, and CI/CD pipelines. We use internal-only languages and resources. We can train employees on all of that. We need them to already have some foundational ability to code, interpret the complexity/efficiency of the code they write, think critically, and check their assumptions with attention to detail. These questions test for that.

Bill: Agreed.

Jeff: Now, if only we could solve the issue of application volume.

Sunny: This process may actually give us a solution. If we’re asking algorithm-based questions, we can develop screeners. We create a portal and allow candidates to submit answers to time-boxed questions. We can perform analyses on the solution to automatically gain insight over the time and space efficiency of the answer. That would help whittle down the pool tremendously! Sure, some may cheat, but most will probably recognize that if they can’t pass the screening on their own, they likely won’t pass the interview on their own.

Larry: Interviewing at scale. I like it!

My Thoughts

If you read through this far, good job! Feel free to yell at me on Twitter if you’re upset and feel I wasted your time 😅. I’d like to reiterate: the events described above never happened. It’s just a story to make a few points. And not points I necessarily agree with either. If those points are clear to you, great. If not… 🤷🏾‍♂️. But here are a few of my thoughts.

What does it mean for a question to be a “leetcode” question?

In public discourse, when I hear questions – or a company’s hiring practices – described as “leetcode” questions, it’s generally in a derogatory manner. So what does a “leetcode” question mean to you? Does it mean trivia questions? Brain teasers? Data-structures & Algorithms (DSA) questions? Any questions you don’t think you’d see in a day-to-day job? Any question in an interview that blocks you from using the internet as a resource?
It seems to me that the most correct answer is: A “leetcode” question is any question that can end up on Leetcode. While DSA makes up the core of what Leetcode has to offer, Leetcode is not just a company; it’s a community. That community is not built around how best to study/solve DSA questions. It’s built around clearing technical interviews at tech companies.
So, what questions end up on Leetcode? Those asked by companies with a critical mass of people who want to work there and have taken interviews there. That second part is vital.

While looking at the trends in hiring among tech companies, I perused nowhiteboard.org – a site that curates job opportunities from companies with alternative hiring practices. As expected, there were mostly smaller companies there. But, I was happy to see the presence of well-known names in the industry, such as Slack, Github, and Stripe. I focused on Stripe, being the largest by headcount at 4,000+. I generally think that past the 3,000-person mark, an exciting company – working on cool stuff, with a good brand, bright future, looking to grow quickly – will start having their interview questions/process leaked regularly. The idea is that, even though applicants want to give back to the job-seeking community, they also don’t want to be identified. Identification is easier if the company interviews 10 candidates per week vs. 100.
So I hopped on Leetcode to see what, if anything, the community had to say about interviewing there. I found a few insights into their questions and process, some broken links, and a thread inquiring why Stripe-related posts were being deleted. Curious, I hopped over to Glassdoor, and sure enough, the interview reviews confirmed that Stripe employed the use of NDAs. But many reviews still went into detail about their interview questions/methods. NDAs will slow the bleed but not stop it. As the company grows and accelerates interviewing, it will grapple with the issue. From the reviews, I personally think Stripe’s approach is wonderful! I’m rooting for them. I’ll check on them after the 10,000-employee mark to see how well they’re faring.

Why Leetcode is a thing, and why you (probably) shouldn’t mind it as much as you do.

That’s not my title. While organizing this section, I stumbled across a Reddit post with this title. It is three years old but represents my opinions quite well. So, instead of restating things, I’d rather you go and read that post. Don’t worry; I’m almost done here.

To close out this post, I have a question: If you’re a small company/startup that asks leetcode-style questions… why? What problem are you optimizing for?