Data Science Investment Counter —Funding raised by UK data science companies in 2018.
£ 5.640 Million

Amazon scraps AI recruiting tool that showed bias against women

An algorithm that was being developed as a recruitment tool by online giant Amazon between 2014 and 2017 was sexist and had to be scrapped, Reuters reported.

Amazon has been using automation substantially, as part of their e-commerce market strategy. The company’s experimental hiring tool used artificial intelligence to give job candidates scores based on one to five stars ratings.

“Everyone wanted this holy grail,” one of the engineers who spoke to Reuters said. “They literally wanted it to be an engine where I’m going to give you 100 resumes, it will spit out the top five, and we’ll hire those.”

Amazon had trained the system on the previous ten years of resumes they had received, which were mostly men. The algorithm, therefore, mirrored the hiring biases towards men characterised the company in the past. It even penalised the word “women” on a resume, as in a women’s club or sport, and downgraded all-women’s colleges as less desirable.

Reportedly, the algorithm was also far from efficient, in general: It would allegedly recommend jobs for applicants where they had no expertise or applicable skills.

Amazon supposedly shut down the project after realising the algorithm was not functioning as they had intended. They would also have tried to fix it but unsuccessfully.

This is not the first example of an artificial intelligence system reflecting the biases found in its training data. In May last year, a report claimed that an AI program used by a US court was biased against black people.

More recently, IBM launched a tool aimed at tackling AI bias in machine learning decision-making for this same reason.

While there is still not an infallible way to erase biases in AI, it is promising that companies like Amazon are taking down those experimental programs that don’t work as they should. As for the future, we will have to wait and see.


Co-working space and blog dedicated to all things data science.

Subscribe to our newsletter