The emergence of AI in the tech industry has raised important questions about how we can future-proof our hiring processes. At N-iX, we conducted an experiment to explore the possibilities and limitations of AI-powered language models in the recruitment industry. We invited ChatGPT, an AI language model, to assist a fake “Candidate” during a Senior .NET Engineer interview. The result came out quite shocking.
The preparation process: how it all started
To add a little backstory: last year, during our .NET Month, we organized a quiz for the tech community at N-iX. Surprisingly, a Data Engineer with no previous experience in .NET got the most points for correct answers. This led us to think about the possibility of someone hacking our most important quiz – the one where you get the job as a result.
After doing some research, Yaroslav Mota, the .NET Community Lead at N-iX, suggested hosting an open interview for a Senior .NET Engineer role, where the “Candidate” with no expertise in .NET, would answer all interview questions with the help of ChatGPT. At this staged interview, N-iX’ers would judge whether the “Candidate” should get the job. They would also be unaware that the “Candidate” was actually an AI language model until the very end.
So we took our typical .NET interview questions to the ChatGPT and got the answers. Most of the responses needed shortening due to the excessive amount of info provided. Shockingly for us, both the code review and system design tasks we used previously in our interviews were easily cracked by ChatGPT.
The interview: welcome ChatGPT to the stage
The interview was ready, and the “Candidate” prepared for the challenge. Over 100 N-iX’ers gathered to judge in our fake interview. As the interview progressed, it became clear that ChatGPT was not just an ordinary AI language model. It provided insightful and creative answers that showed a deep understanding of the .NET framework and the programming language. Until the very end, our interviewer, Yaroslav, having conducted hundreds of tech interviews in his career, couldn’t still believe these were the responses of an AI.
The audience was impressed, and as a result of collective voting, the “Candidate” was deemed suitable for the Senior .NET Engineer role. It seemed that ChatGPT had passed the ultimate test – a job interview – with flying colors.
However, when we revealed the truth, the audience was shocked, but during the following discussion, they quickly realized that this was not a bad thing.
While we ultimately decided not to hire ChatGPT as a Senior .NET Engineer, the experiment taught us a valuable lesson. AI can be a game-changer in the recruitment process, as well as new AI tools could assist with daily programming tasks, and it’s up to us to explore its possibilities and limitations.
According to N-iX internal research (which is not generalizable and representative), tasks like doing new feature research, boilerplate code, and test generation could be accelerated with ChatGPT by up to 40% on average.
Summing up: the lessons learnt
During the discussion with our staged interview audience, we’ve come up with a few things that could help AI-proof the process:
- AI-powered language models are not perfect. Yet 🙂.
Though they can provide accurate and relevant answers to certain questions, they require specific prompts and may miss important nuances and context. - Interviewers should rely on their own judgment and experience when evaluating candidates.
- Prepare to ask questions that cannot be answered by a simple Google search or language model. This means asking open-ended questions that require candidates to demonstrate their problem-solving and critical-thinking skills.
- Ask specific questions based on the candidate’s CV, and focus on the challenges of a candidate’s previous experience.
- Interrupt the candidate and add context to the discussion based on information outside the AI-powered language model’s knowledge base.
- Assess cultural fit: It’s important to remember that the best candidate for the job isn’t necessarily the one with the most technical knowledge. Consider personality, work style, and values.
In conclusion, while the use of ChatGPT and other technology tools may be surprising to some interviewers, it’s important to remember that the ability to utilize these resources effectively is an important skill for any software engineer. By focusing on problem-solving skills, collaborative abilities, and cultural fit, interviewers can ensure they select the best candidate for the job.
Europe, Ukraine