According to a story from The New York Times, a worried parent claims that after using his Android smartphone to capture pictures of an infection on his toddler’s crotch, Google identified the pictures as child sexual abuse material (CSAM). The incident highlighted the challenges of trying to distinguish between potential abuse and an innocent photo once it becomes part of a user’s digital library, whether on their personal device or in cloud storage, and the company closed his accounts, filed a report with the National Center for Missing and Exploited Children (NCMEC), and sparked a police investigation.
When Apple unveiled its Child Safety initiative last year, worries about the implications of blurring the boundaries of what should be deemed private were raised. In accordance with the idea, Apple would locally scan photographs on Apple devices before uploading them to iCloud, and then compare the images to the NCMEC’s hashed database of recognized CSAM. A human moderator would next analyze the material and lock the user’s account if there were enough matches identified that indicated it included CSAM.
A nonprofit digital rights organization called the Electronic Frontier Foundation (EFF) denounced Apple’s proposal, claiming it might “create a backdoor to your private life” and would “reduce privacy for all iCloud Photos customers, not enhance it.”
The saved picture scanning functionality was ultimately put on hold by Apple, but with the release of iOS 15.2, the company went ahead and included an optional capability for kid accounts that are part of a family sharing plan. The Communications program “analyzes picture attachments and detects whether a photo includes nudity, while retaining the end-to-end encryption of the messages,” provided parents give their consent, on a child’s account. If nudity is found, the picture is blurred, a warning is sent to the youngster, and they are given links to resources for internet safety.
When certain medical offices were still closed due to the COVID-19 epidemic in February 2021, the key event that The New York Times highlighted occurred. The Times said that Mark (whose last name was withheld) detected swelling in his child’s genital area and, at a nurse’s request, supplied pictures of the problem prior to a video consultation. Ultimately, the doctor prescribed medications to treat the illness.
Just two days after snapping the pictures, the NYT reports that Mark got a warning from Google informing him that his accounts had been frozen because of “harmful material” that was “a grave breach of Google’s policy and possibly be unlawful.”
Similar to many other online businesses, such as Facebook, Twitter, and Reddit, Google has been using Microsoft’s PhotoDNA to hash-match submitted photographs in order to find instances of recognized CSAM. It resulted in the arrest of a guy who used Gmail to communicate pictures of a young girl and was a registered sex offender in 2012.
With the aim of “proactively identifying never-before-seen CSAM images so it may be inspected and, if verified as CSAM, deleted and reported as fast as feasible,” Google announced the debut of its Content Safety API toolset in 2018. It makes use of the technology for its own services and also makes it available for use by others along with a video-targeting CSAI Match hash matching solution created by YouTube developers.
According to a Google spokesman who spoke to the Times, Google only scans users’ private photos when those users take “affirmative action,” which may include uploading their photos to Google Photos. The Times reports that when Google detects exploitative photographs, it is compelled by federal law to inform the CyberTipLine at the NCMEC of the prospective perpetrator. Google submitted 621,583 CSAM incidents to the NCMEC’s CyberTipLine in 2021, and the NCMEC warned the police about 4,260 possible victims, one of which, according to the New York Times, is Mark’s kid.
As a result of using Google Fi’s mobile service, Mark lost access to his emails, contacts, images, and even his phone number, according to the Times. Google rejected Mark’s request to appeal the ruling even though he did so right away. In December 2021, the San Francisco Police Department, where Mark resides, began an inquiry against him and obtained all the data he had saved with Google. According to the NYT, the case’s investigator finally concluded that “no crime happened” and that the incident “did not match the ingredients of a crime.”
Child sexual abuse material (CSAM) is repugnant, and Google is dedicated to stopping its dissemination on its platforms, according to a statement sent to The Verge by a representative of the company. “We employ a mix of hash matching technology and artificial intelligence to detect it and remove it from our platforms. We define CSAM in accordance with US law. Additionally, to assist us recognize situations where users could be looking for medical advice, our team of child safety specialists evaluates flagged material for accuracy and interacts with physicians.
While safeguarding children from abuse is unquestionably vital, others claim that scanning a user’s images intrudes on their privacy in unreasonable ways. In a comment to the NYT, Jon Callas, director of technology programs at the EFF, referred to Google’s actions as “invasive.” According to Callas, “This is exactly the nightmare that we are all frightened about.” “I’m going to go into problems because they’re going to scan my family book.”