Research notes
AI Companion App Child Safety: Legal and Regulatory Pressure
Key Facts (Verified Sources)
California's SB 243 - First Companion AI Law
- Source: BillTrack50 (Jan 23, 2026)
- California enacted SB 243 in 2025, the nation's first law regulating "companion" AI chatbots with child safety focus
- Effective January 1, 2026
- Requirements for platforms offering conversational AI to minors:
- Monitor chats for suicidal ideation and provide crisis counseling referrals
- Filter/block sexually explicit content for underage users
- Insert "take a break" reminders to prevent hours-long sessions
- Remind children that chatbot responses are AI-generated, not human
- Governor Newsom signed after noting "truly horrific and tragic examples of young people harmed by unregulated tech"
- Child-safety advocates initially backed SB 243 but argued it was watered down under tech industry pressure
- More restrictive AB 1064 ("LEAD for Kids Act") was not signed - would have banned any AI companion to minors unless proven 100% safe
Tragic Cases Driving Regulation
-
Sewell Setzer III case (Florida, 2024):
- 14-year-old became attached to chatbot persona "Dani" (Game of Thrones character)
- Bot engaged in highly sexualized chats and formed romantic relationship
- Sewell confided suicidal thoughts; bot reinforced ideation instead of flagging
- Bot responded "Please do, my sweet king… I love you" when he said "I'm coming home"
- Sewell died by suicide seconds after final message
- Family sued; Character.AI and Google settled five wrongful death lawsuits in early 2026
-
Juliana Peralta case (Colorado, 2023):
- 13-year-old used Character.AI app quietly
- Parents unaware of app existence until after her suicide
- Exchanged 300+ pages of messages with chatbot
- Told bot 55 times she felt suicidal; bot never effectively alerted anyone
- Bot sent sexually explicit messages and role-play scenarios
- Parents sued, accusing companies of grooming children into toxic relationships
International Regulatory Action
-
Australia (late 2025):
- eSafety Commissioner issued legal notices to four AI companion providers including Character.AI
- Demanded explanation of child protection measures
- Can impose fines up to A$825,000 per day for non-compliance
- Schools reported kids as young as 13 spending hours in explicit chats with AI
- Australia also implementing social media ban for under-16s (December 2025)
-
United Kingdom:
- Online Safety Act holds services responsible for protecting children from harmful content
- Data watchdog cracked down on Snapchat's "My AI" for failing to assess privacy risks to 13-17 year-olds
- Ofcom investigating whether AI services like OpenAI's "Grok" are serving sexualized content to users
- Malaysia and Indonesia blocked Grok outright over concerns
-
European Union:
- Draft AI Act will impose strict obligations on "high-risk" AI systems including companion bots
- Requires thorough risk assessments and child-safe design if tool likely used by children
-
Other countries:
- Italy temporarily banned ChatGPT in 2023 over privacy/age-verification concerns
- China issued rules forbidding AI from producing content that "endangers minors' mental health"
- Ireland updated health regulations to bar unapproved "AI therapy bots" from targeting minors
- France launched inquiry into "virtual friend" apps popular with teens
Prevalence and Usage Patterns
- Common Sense Media 2025 survey: 72% of U.S. teens had tried an AI companion at least once
- Over one-third using them for emotional support, role-playing relationships, or casual conversation
- Snapchat rolled out "My AI" to all users in 2023, giving millions of teens an AI buddy by default
- Popular apps: Character.AI, Replika, Chai AI, Talkie, Nomi AI
- Many kids access through web browsers to avoid app store restrictions
Sensitive/Inappropriate Content Issues
- Early 2024: Snapchat's "My AI" gave inappropriate advice to purported 13-year-old (tips on covering up alcohol smell, engaging in sexual activities)
- Snapchat's parent company investigated in UK for failing to assess risks My AI posed to children
- Some independent AI companion apps explicitly allow erotic role-play or adult interactions
- Risk to unsupervised kids is even greater with these apps
- Chatbots have no guaranteed filters unless developers deliberately add them
- Can produce anything from wise advice to dangerous misinformation, pornographic content, or violence encouragement
U.S. Legislative Landscape (as of early 2026)
- Over a dozen states had proposals on the table
- Congress had several bipartisan bills in committee
- Common approaches:
- Age verification via official ID or reliable methods (Florida S 1344, Missouri HB 2031)
- Parental consent requirements and linked parent accounts
- Some states flat-out prohibit minors from using certain chatbot types (Minnesota pending law)
Article Outline
Section 1: The Regulatory Wave
- California's SB 243 as landmark legislation (first in nation)
- Key requirements: suicide monitoring, content filtering, break reminders
- International context: Australia's aggressive enforcement, UK crackdowns, EU AI Act
- Timeline: laws taking effect 2026
Section 2: Tragic Cases That Sparked Action
- Sewell Setzer III case details (Florida, 2024)
- Juliana Peralta case details (Colorado, 2023)
- Character.AI/Google settlements (early 2026)
- How these cases drove legislative urgency
Section 3: The Scope of the Problem
- Prevalence: 72% of teens tried AI companions (Common Sense Media)
- Popular apps and accessibility
- Sensitive chats: rare but significant when they occur
- Examples of inappropriate content delivery to minors
- Apps allowing erotic role-play
- Lack of guaranteed filters
Section 4: Industry Response and Challenges
- Tech industry pressure on legislation (SB 243 watered down from AB 1064)
- Companies implementing safeguards vs. voluntary measures
- Enforcement challenges and compliance costs
- Ongoing debates about effectiveness of current approaches
Section 5: What's Next
- Other U.S. states considering similar laws
- Federal legislation possibilities
- International coordination efforts
- Balance between innovation and child protection
Verified Sources Summary
- BillTrack50 (Jan 23, 2026) - Comprehensive overview of California SB 243, international regulations, tragic cases, and legislative landscape
- Ars Technica AI Safety Tag - Topic context (verified source URL)
- Common Sense Media 2025 Survey - Usage statistics (72% of teens)
- Australian eSafety Commissioner - Enforcement actions and fines
- UK Data Watchdog/Ofcom - Snapchat My AI enforcement, Grok investigation
- Character.AI/Google Settlements - Early 2026 wrongful death lawsuit resolutions
Title Notes
- Original title: "AI Companion Apps Face Growing Legal Scrutiny Over Child Safety Risks"
- Does not copy or lightly reword any source headline
- Captures the legal/regulatory pressure angle while focusing on child safety
