Fairness and validity of AI interviews

Many organizations are adopting AI recruiting solutions to increase fairness in the hiring process. However, the difficulty of understanding the logic behind AI's decisions raises questions about its reliability. According to a 2022 survey, about 37% of organizations use or consider AI recruiting solutions. With the changes this technology has brought, questions about the fairness and validity of the hiring process are becoming more prevalent. While AI technology has made great strides, the complexity of the logic underlying its assessment results makes full validation difficult.

In 2020,13, public organizations that used AI recruitment solutions were involved in freedom of information lawsuits, which revealed a lack of management and oversight of certain AI recruitment solutions. The lack of explanation, fairness, and validity in the evaluation results of AI recruitment solutions was a significant problem, suggesting that the problem is not with the AI recruitment solution itself but with the way it is applied.

As AI technology rapidly advances and the AI recruitment solution industry is expected to grow, it's crucial for recruiters to grasp the intricacies of this technology. This understanding will enable them to make informed decisions about its application, leading to the development of a more sophisticated hiring process and the realization of the full potential of AI recruiting solutions.

Design systematic and scientific AI model training.

AI technology doesn't just appear out of nowhere; its progress is largely driven by researchers' intentions and the design of training data. If recruiters don't understand AI's logic directly, learning about how it was developed can help them understand how it works.

When evaluating AI recruiting solutions, it's important to be clear about what the AI is assessing. For example, factors like eye movement or speech rate analyzed in an AI interview may not be directly relevant to hiring, so it's important to understand how the technology is actually useful for hiring. Make sure you know what competencies or traits the AI model is actually assessing and that the assessment is theoretically and empirically valid.

It's also important to consider the quality and reliability of the data the AI model is training on. AI may only favour specific candidates if the data only reflects past successes. When designing an AI model, consider the characteristics of different jobs and include data from applicants with a wide range of performance, not just high performers.

By gaining a comprehensive understanding of the limitations and possibilities of AI recruiting solutions, recruiters can steer clear of potential pitfalls and design more effective hiring processes. The key to successful AI adoption lies not just in its implementation, but in its strategic utilization.

How do you define a high performer?

• Are cover letter questions the same now as they were then?
• High performers have different characteristics in different roles, so did the training process consider job classification?
• Is data from low performers included in the training?

Through these questions and concerns, recruiters must understand the features and limitations of various AI recruitment solutions and build a more sophisticated recruitment process by considering multiple aspects together.

How to ensure fairness in AI

Ensuring fairness is a critical aspect of designing an AI recruiting solution. This means preventing the AI from making biased evaluations based on applicant characteristics like gender or age. While AI is subject to the same fairness requirements as traditional blind hiring, it demands a higher level of fairness. Given the risk of AI learning human biases, recruiters have a crucial role in actively verifying AI vendor claims, rather than passively accepting them.

AI recruiting solution providers should have processes to minimize holes in the training data to ensure fairness. The development process for AI models should be transparent and include performance evaluation for bias and noise. The data trained by the AI should be unbiased, and data labelling should be accurate and consistent. Care should also ensure the AI does not overfit or overgeneralize to a particular population. For example, when evaluating interview videos, the opinions of multiple raters should be incorporated to minimize bias, with a facilitator available to intervene if necessary.

By proactively identifying and addressing bias and noise in AI models, recruiters can increase the credibility of the hiring process. This reassures both recruiters and candidates that AI recruiting solutions can be used as a truly fair and objective tool, instilling confidence in the process.

While recruiters must check for bias and noise in AI recruiting solutions, it often has many limitations and challenges. To simplify this process, trusted organizations like the Korea Telecommunications Technology Association (TTA) organize AI reliability assessments. These organizations provide certification by evaluating AI models' bias, noise, explainability, and transparency. Recruiters can use these certifications to ensure that AI recruiting solutions have addressed bias and noise and verified their reliability. This plays a vital role in ensuring the fairness of AI and paves the way for more effective use of AI in the hiring process.

How to justify AI

Recruiters looking to adopt AI recruiting solutions expect efficiency and cost-effectiveness. This notion includes not only saving time and money on hiring but also the contribution of the hired person to organizational performance. If an AI recruiting solution doesn't fit your organization's talent profile and results in inefficient hiring, it's not worth it.

AI recruiting solution validity is determined by whether the AI accurately assesses real-world candidate skills. Each AI recruiting solution provider claims validity in different ways to ensure this, but these are often based on their data and do not apply to all organizations. As with traditional recruiting tools, independent validation of AI recruiting solutions is essential.

Organizations that have already implemented an AI recruiting solution can analyze their accumulated hiring data to assess how well the AI's assessments match actual hiring outcomes. Organizations that are new to AI can validate the accuracy of their AI by comparing AI results to those of human evaluators, such as through cover letter analysis or AI interviews. This is a great way to validate the practicality of an AI recruiting solution, which is directly related to hiring more efficiently.

Adopting AI recruiting solutions plays a vital role in increasing the objectivity and fairness of hiring, as well as efficiency and cost-effectiveness. However, the successful integration of these systems depends on the technology's validation. Validation involves ensuring that the AI accurately assesses a candidate's true abilities. When selecting an AI recruiting solution, organizations should consider whether it supports such validation.

The continuous validation and retraining of AI recruitment solutions should be tailored to the organization, improving the hiring process's quality and contributing to hiring better talent. This is done in collaboration with recruiters, enabling digital HR transformation.

Advancements in AI technology are rapidly progressing in HR. Moving beyond the awe and fantasy of the past, today's recruiters need to understand the real value and limitations of AI technology and how to leverage it appropriately to maximize organizational performance. AI recruiting solutions could be better, but we need to harness their potential and continuously improve them. This approach will be a critical factor in determining your organization's future.