Redefining the Online Tech Interview for the AI Era

Online interviews take over: Pandemic forces a sudden, unprepared shift

The goal of a software engineering interview is to find the best candidate for the job based on their ability to show the skills and expertise needed to succeed. This helps employers make better hiring decisions and ensure they’re bringing on the best possible candidates for their team.

Pre-pandemic vs. post-pandemic interview processes are vastly different. The typical interview process for software engineers is as follows:

1. Phone or video screening: Brief phone or video call with the candidate to ask some initial questions and get a sense of their background and experience.
2. Technical screening: Technical assessment, which may involve solving coding challenges or writing code in real-time.
3. In-person or video interview: More in-depth interview with the candidate, which may involve additional coding challenges, discussions of the candidate’s experience and qualifications, and other questions to evaluate their fit for the position.

Step 3 shifted drastically and suddenly to be based on online interviews due to the pandemic.

And now, ChatGPT changes everything!

The arrival of ChatGPT has changed everything, including coding interviews. Take this example of how it tackles a typical coding question.

Question: Given an array of integers and a value, determine if there are any two integers in the array whose sum is equal to the given value. Return true if the sum exists and return false if it does not.

ChatGPT first suggests a brute force O(N^2) solution.

When prompted to do it more efficiently, it has no trouble coming up with an O(N) solution using a hash table.

ChatGPT-able LEETCode style interviews broken and need serious overhaul

Asking easily ChatGPT-able / Google-able questions in interviews can create an unfair advantage and may not accurately reflect a candidate’s skills. Instead, try:

1. Asking open-ended, scenario-based questions that require candidates to think critically and solve problems, rather than asking questions that can be easily answered with a simple search.
2. Providing clear guidelines and instructions to candidates before the interview, and emphasizing the importance of integrity and honesty in the hiring process.
3. Implementing a multi-stage interview process that includes both technical and behavioral assessments, to provide a more comprehensive evaluation of a candidate’s abilities and skills.

There are many different types of questions that can be used to evaluate a candidate’s true abilities in a software engineering interview. Some examples of effective questions include:

1. Ask the candidate to describe a difficult problem they have faced in their previous work, and how they approached and solved it. This can help you understand their problem-solving abilities and how they think through complex challenges.
2. Ask the candidate to explain a concept or technology that they are familiar with in detail. This can help you assess their depth of knowledge and understanding of key concepts in software engineering.
3. Ask the candidate to work through a design problem or a systems architecture question. This can help you understand their ability to design and implement complex software systems.
4. For coding assignments, give a take-home exercise which involves doing some research to identify how to tackle a problem and provide a solution.

Breaking down big problems: the key to successful problem-solving

We have to ask higher level questions that evaluate the candidate’s ability to break down an ambiguous problem into smaller pieces and tackle them. Here are some examples:

1. Describe a software system that you have designed and implemented in the past. What were the key challenges you faced, and how did you overcome them?
2. How do you ensure that the design of a software system is maintainable and scalable? Can you give an example of a design decision you made with these considerations in mind?
3. Given a set of requirements for a new software system, how would you approach the design and implementation of that system?
4. What are the key considerations you take into account when designing a software system? How do you balance the trade-offs between different design choices?
5. Describe a situation where you had to make a difficult design decision, and how you went about making that decision.

The power of collaboration: harnessing the forces of teamwork for success

Work is a team-sport, so it is important to focus on the candidate’s ability to collaborate effectively with others. To evaluate a candidate’s ability to collaborate with others effectively, you could ask them questions that focus on their experience working in team environments and their approach to collaboration. Some examples of questions that you could ask include:

1. Can you give an example of a time when you had to work closely with others to achieve a common goal? What was your role in the team, and how did you contribute to the success of the project?
2. What do you consider to be the key ingredients for a successful team? How do you foster collaboration and communication within a team?
3. How do you handle feedback from others, both positive and negative? Can you give an example of a situation where you received constructive feedback and used it to improve your work?
4. Can you give an example of a project where you had to work with people who had different skill sets and backgrounds? How did you ensure that everyone’s contributions were valued and integrated into the project?
5. Describe a situation where you had to resolve a conflict within a team. How did you approach the situation, and what was the outcome?

Empathy in disagreements: the key to finding win-win solutions

You could ask them questions that focus on their experience dealing with conflicts and their approach to resolving disputes. Some examples of questions that you could ask include:

1. Describe a time when you had to navigate a complex or contentious situation involving multiple stakeholders. How did you ensure that everyone’s concerns and perspectives were taken into account, and what was the outcome?
2. Can you give an example of a situation where you had to deal with a disagreement or conflict with a colleague or team member? How did you approach the situation, and what was the outcome?
3. Can you give an example of a time when you had to apologize or take responsibility for a mistake or error? How did you handle the situation, and what did you learn from the experience?
4. How do you handle situations where you are not able to reach a resolution or agreement with someone else? Can you give an example of a time when you were able to agree to disagree and still maintain a positive working relationship?
5. What do you consider to be the key ingredients for resolving conflicts and disputes in a fair and constructive way? How do you balance the need to address the underlying issues with the need to maintain relationships and avoid escalation?

Interview needs to focus on the candidate’s ability to tackle ambiguity effectively.

You could ask them questions that focus on their experience dealing with uncertainty and their approach to solving complex, open-ended problems. Some examples of questions that you could ask include:

1. Describe a project or situation where the goals and objectives were not clearly defined. How did you navigate the ambiguity and ensure that you were making progress towards a successful outcome?
2. Can you give an example of a situation where you had to make a decision without having all the necessary information? How did you approach the situation, and what was the outcome?
3. How do you approach problem-solving when faced with a complex, open-ended challenge? Can you give an example of a time when you had to come up with a creative solution to a difficult problem?
4. What do you consider to be the key ingredients for successfully navigating ambiguity and uncertainty? How do you balance the need for action with the need to gather more information?
5. How do you handle situations where the goals or expectations for a project change mid-way through the process? Can you give an example of a time when you had to adapt to changing circumstances and still achieve a successful outcome?

What does the candidate do when they are stuck? Growth mindset is critical.

You could ask them questions that focus on their approach to learning and development, their ability to adapt to change, and their willingness to take on new challenges. Some examples of questions that you could ask include:

1. How do you approach setbacks and failures? Can you give an example of a time when you faced a challenge and were able to learn and grow from it?
2. Can you give an example of a situation where you had to learn something new and challenging? How did you approach the learning process, and what was the outcome?
3. What do you consider to be the key ingredients for personal and professional growth? How do you cultivate a growth mindset in yourself and others?
4. Describe a time when you had to adapt to a significant change in your work environment. How did you handle the change, and what did you learn from the experience?
5. Can you give an example of a time when you took on a challenge that was outside of your comfort zone? What did you learn from the experience, and how did it impact your development?

The drive for results is a key ingredient for success

You could ask them questions that focus on their approach to achieving goals and their ability to deliver high-quality work on time. Some examples of questions that you could ask include:

1. Describe a situation where you had to overcome a significant obstacle or challenge in order to achieve a goal. How did you approach the situation, and what was the outcome?
2. Can you give an example of a time when you had to make a difficult decision in order to achieve a goal? What was the decision, and how did it impact the outcome of the project?
3. What do you consider to be the key ingredients for achieving success? How do you motivate yourself and others to deliver high-quality work on time?
4. Can you give an example of a time when you had to deliver a complex project on a tight deadline? What was your approach to managing the project, and what was the outcome?
5. How do you handle situations where the goals or expectations for a project are not clearly defined? Can you give an example of a time when you were able to set clear goals and deliver successful results in a challenging environment?

Conclusion / TLDR

• Asking ChatGPT-able questions may not reflect the candidate’s true abilities.
• Give take-home assignments instead that involve some research.
• Get signals on their skills related to:
  • Breaking a complex problem into smaller tractable pieces
  • Collaboration
  • Handling disagreements
  • Growth mindset
  • Drive for results