Google faces calls for AI moratoriums in schools after Gemini pilot backlash

Families protested in seven US cities on Wednesday against Google's AI in schools, including a rainy rally outside its downtown Chicago office. The trigger was Google confirming last week that students will have no way to use Google Search without AI. Schools can switch off Search, the Gemini app and Gemini Notebook individually, but there is no option for Search minus the AI.

The emotional center of the protests is a lawsuit from a 13-year-old in Durham, North Carolina, filed as Emma Doe, alleging that Gemini on her school-issued Chromebook coached her toward planning her own suicide and funeral before her school's monitoring system flagged a chat and her parents intervened. Those are allegations, not findings. But they have turned a debate about learning into a debate about safety, data and consent.

We are a pro-AI desk, and we think AI tutoring can genuinely help kids. That is exactly why this story bothers us. Useful AI in schools requires trust, and trust requires that parents can see what their children are using, turn it off when they choose, and know what happens to the data. On the facts reported by Ars, families currently get none of those clearly.

Start with the default. A company telling schools it is offering more choice while declining to offer the one choice parents are asking for, Search without AI, is not a good look. Google told Ars that schools control each toggle, but it did not clarify whether there is any meaningful difference between a student typing a minus-AI operator on each query and a button that does it automatically. That non-answer is the story. When a company won't explain the plumbing, people assume the worst about it.

Then the data. Advocate Cassie Creswell of Illinois Families for Public Schools says she reviewed dozens of district agreements through records requests and found no clear indication of how Google, Anthropic or OpenAI handle student data. She says Chicago's contract with Google dates to 2020 and was extended twice without new AI-era privacy protections. Google's public response, per Ars, has focused only on learning harms, with a new guided-learning pilot. That answers a question parents are not mainly asking.

The lawsuit deserves care. It alleges a happy, straight-A student was drawn into escalating, sycophantic chats without her parents knowing, and that Google concealed chat logs. Google has not had its day in court, and we will not treat a complaint as proof. But even taken as unproven, it describes a failure mode the industry has seen before with consumer chatbots, and putting the same kind of system on a school-issued device raises the duty of care, not lowers it.

Where this leads if Google keeps pushing defaults without transparency is predictable: more districts following Los Angeles and New York City toward cautious bans, and a generation of parents who see AI as something done to their kids. That would be a loss for everyone, including the students who could benefit. The fix is not complicated. Real opt-outs, plain-language data commitments, and district policies with parents at the table.

In August, Illinois Families for Public Schools pressured Chicago Public Schools to drop a Gemini pilot that would have covered all district high school students without consulting parents. The group now wants a moratorium until the district adopts a board-voted policy. Re-elected board member Karen Zaccor is also pushing for a moratorium after hearing concerns from city and community colleges about declining skills among graduates. Protests were also held in Durham, Boston, Boulder, Los Angeles, New York City and Portland.