The use of AI in hiring has been criticized for the way it automates and entrenches existing racial and gender biases. AI systems that evaluate candidates’ facial expressions and language have been shown to prioritize white, male, and abled-bodied candidates. The problem is massive, and many companies use AI at least once during the hiring process. US Equal Employment Opportunity Commission chair Charlotte Burrows said in a meeting in January that as many as four out of five companies use automation to make employment decisions.
NYC’s Automated Employment Decision Tool law, which came into force on Wednesday, says that employers who use AI in hiring have to tell candidates they are doing so. They will also have to submit to annual independent audits to prove that their systems are not racist or sexist. Candidates will be able to request information from potential employers about what data is collected and analyzed by the technology. Violations will result in fines of up to $1,500.
Proponents of the law say that it’s a good start toward regulating AI and mitigating some of the harms and risks around its use, even if it’s not perfect. It requires that companies better understand the algorithms they use and whether the technology unfairly discriminates against women or people of color. It’s also a fairly rare regulatory success when it comes to AI policy in the US, and we’re likely to see more of these specific, local regulations. Sounds sort of promising, right?
But the law has been met with significant controversy. Public interest groups and civil rights advocates say it isn’t enforceable and extensive enough, while businesses that will have to comply with it argue that it’s impractical and burdensome.
Groups like the Center for Democracy & Technology, the Surveillance Technology Oversight Project (S.T.O.P.), the NAACP Legal Defense and Educational Fund, and the New York Civil Liberties Union argue that the law is “underinclusive” and risks leaving out many uses of automated systems in hiring, including systems in which AI is used to screen thousands of candidates.
What’s more, it’s not clear exactly what independent auditing will achieve, as the auditing industry is currently so immature. BSA, an influential tech trade group whose members include Adobe, Microsoft, and IBM, filed comments to the city in January criticizing the law, arguing that third-party audits are “not feasible.”
“There’s a lot of questions about what type of access an auditor would get to a company’s information, and how much they would really be able to interrogate about the way it operates,” says Albert Fox Cahn, executive director of S.T.O.P. “It would be like if we had financial auditors, but we didn’t have generally accepted accounting principles, let alone a tax code and auditing rules.”
Cahn argues that the law could produce a false sense of security and safety about AI and hiring. “This is a fig leave held up as proof of protection from these systems when in practice, I don’t think a single company is going to be held accountable because this was put into law,” he says.
Importantly, the mandated audits will have to evaluate whether the output of an AI system is biased against a group of people, using a metric called an “impact ratio” that determines whether the tech’s “selection rate” varies across different groups. The audits won’t have to seek to ascertain how an algorithm makes a decision, and the law skirts around the “explainability” challenges of complex forms of machine learning, like deep learning. As you might expect, that omission is also a hot topic for debate among AI experts.
In the US we’re likely to see much more AI regulation of this sort—local laws that take on one particular application of the technology—as we wait for federal legislation. And it’s in these local fights that we can understand how AI tools, safety mechanisms, and enforcement are going to be defined in the decades ahead. Already, New Jersey and California are considering similar laws.
© Copyright 2023 Technology Review, Inc. Distributed by TRIBUNE CONTENT AGENCY, LLC.
