Women with darker skin are more than twice as likely to be told their photos fail UK passport rules when they submit them online than lighter-skinned men, according to a BBC investigation.
One black student said she was wrongly told her mouth looked open each time she uploaded five different photos to the government website.
This shows how “systemic racism” can spread, Elaine Owusu said.
The Home Office said the tool helped users get their passports more quickly.
“The indicative check [helps] our customers to submit a photo that is right the first time,” said a spokeswoman.
“Over nine million people have used this service and our systems are improving.
“We will continue to develop and evaluate our systems with the objective of making applying for a passport as simple as possible for all.”
Skin colour
The passport application website uses an automated check to detect poor quality photos which do not meet Home Office rules. These include having a neutral expression, a closed mouth and looking straight at the camera.
BBC research found this check to be less accurate on darker-skinned people.
More than 1,000 photographs of politicians from across the world were fed into the online checker.
Photos of women with the darkest skin were four times more likely to be graded poor quality, than women with the lightest skin.
Ms Owusu said she managed to get a photo approved after challenging the website’s verdict, which involved writing a note to say her mouth was indeed closed.
“I didn’t want to pay to get my photo taken,” the 22-year-old from London told the BBC.
“If the algorithm can’t read my lips, it’s a problem with the system, and not with me.”
But she does not see this as a success story.
“I shouldn’t have to celebrate overriding a system that wasn’t built for me.”
It should be the norm for these systems to work well for everyone, she added.
Other reasons given for photos being judged to be poor quality included “there are reflections on your face” and “your image and the background are difficult to tell apart”
Cat Hallam, who describes her complexion as dark-skinned, is among those to have experienced the problem.
She told the BBC she had attempted to upload 10 different photographs over the course of a week, and each one had been rated as “poor” quality by the site.
“I am a learning technologist so I have a good understanding of bias in artificial intelligence,” she told the BBC.
“I understood the software was problematic – it was not my camera.
“The impact of automated systems on ethnic minority communities is regularly overlooked, with detrimental consequences.”