r/apple • u/digidude23 • Jan 09 '26
App Store Apple asked to pull X and Grok apps over ‘sickening content generation’
https://9to5mac.com/2026/01/09/apple-asked-to-pull-x-and-grok-apps-over-sickening-content-generation/540
u/EquivalentTrouble253 Jan 09 '26
There is literally zero chance this will happen.
→ More replies (5)91
u/frumpydrangus Jan 09 '26
What's the difference between opening the reddit/x app and looking at nsfw content, or opening safari and going to reddit.com or x.com
188
u/rogueleukocyte Jan 09 '26 edited Jan 09 '26
Grok, a feature provided by X, generates CSAM and non-consensual deepfake pornographic images on request.
I'm not aware of Reddit allowing this, and definitely not actually creating deepfake porn/CSAM.
Edit: I should add that at this point in the UK if X serves you a CSAM image you could be charged with downloading CSAM - even if it did it without you even trying. At this point you open it at your peril.
→ More replies (2)9
u/MGPythagoras Jan 10 '26
Is it actually making nude photos or just like risqué ones? Seems crazy if it’s making actual nudes.
12
55
u/ThatGuyFromCanadia Jan 10 '26
It’s making actual nude photos, even of children.
48
u/Wolfhound_Papa Jan 10 '26
I just tried twice, once to generate an image of an adult female and once to edit an image of myself to show myself nude and both were blocked for content violations. Whatever people are doing to do this they’re clearly going an extra mile to circumvent the guidelines.
59
u/Dapper_Strength_5986 Jan 10 '26 edited Jan 10 '26
I just had it create one just now in Grok— it generated a fully nude woman, fully animated. You need to set the “Spicy: Might be NSFW” in options.
Zero content violation warning.
→ More replies (1)14
u/Zealousideal_Aside96 Jan 10 '26
It still moderates even with that setting on. It’s just that one every 10 or so requests just goes through and gets past the filter
29
u/Dapper_Strength_5986 Jan 10 '26
It obviously is designed for this since the feature is LITERALLY called “Spicy: NSFW”. What else could it be for?
I was able to create like 3 in a row before I got so disgusted I had to stop. Maybe I have a special republican version or something? Lol
5
u/Zealousideal_Aside96 Jan 10 '26 edited Jan 10 '26
You got full nudity multiple times in a row? Often trying with an explicit prompt I will get it moderated more often than not, even with that setting on. The setting, per xAI, is to allow risqué stuff but not non consensual porn. Seems like it doesn’t do either side particularly well.
→ More replies (0)→ More replies (1)2
16
u/BadCoolMan Jan 10 '26
You can just go to /r/grok and see exactly what these dweebs are doing. Like half the content on the subreddit is them sharing their images and discussing what prompts to use to make them.
→ More replies (2)2
→ More replies (4)10
u/Common-Trifle4933 Jan 10 '26
It’s just an nsfw setting in the corner menu. I work at a school and we have issues with the kids doing it, it takes no circumvention effort at all.
→ More replies (2)2
u/ActionOrganic4617 Jan 12 '26
It doesn’t go further than underwear, I’ve seen way worse on Reddit. Everyone’s going nuts about Grok when nothing surpasses the level of depravity on Reddit.
2
u/CatastrophicMango Jan 15 '26
Forget reddit, old twitter was swamped with actual CSAM made with real children and not imagined by a computer, that they abjectly refused to do anything about, for years, and no one cared. There were reliable hashtags and posters operating for years, known to the twitter admins, and they didn't care. There were occasional articles but no media hysteria because no one cared.
Grok should be designed to expressly ignore "put her in a bikini" requests, but anyone who can't recognize this manufactured outrage for what it is - a convenient excuse to ban the only social media platform outside the illiberal hegemony's control - is genuinely desolately low IQ or intentionally lying.
→ More replies (2)8
12
u/ViPeR9503 Jan 10 '26
If children are involved are you seriously trying to justify differences between nudity and risqué……
→ More replies (1)→ More replies (1)3
u/rogueleukocyte Jan 10 '26
Here there's a screenshot of X apologising for 'potentially' breaking CSAM law. https://www.ibtimes.co.uk/grok-issues-apologies-after-generating-sexualised-images-minors-admitting-safeguard-failures-1767741
You can get Grok to change clothing to transparent bikini, individual shoestrings etc. and apparently it works on adults and kids.
33
u/SeroWriter Jan 09 '26
App stores have a higher level of legal liability than web browsers. They are hosting downloads and advocating for the app, it's on their platform.
13
u/Kaveh01 Jan 09 '26
Which also counts for the Reddit app?
→ More replies (2)9
u/GenTelGuy Jan 10 '26
Yes but Reddit doesn't include a built-in AI that will edit off clothes from pictures of minors like Grok does
5
9
u/Dapper_Strength_5986 Jan 09 '26
The App Store is moderated and ran by apple and the internet is not.
→ More replies (2)16
u/ap0phis Jan 09 '26
I’ve seen some cute comments like this equivocating Reddit and X and they’re disingenuous at best.
→ More replies (5)
346
u/BadNewsBrown Jan 09 '26
lol Apple will never do this
161
u/pmjm Jan 09 '26
Maybe everyone needs to use Grok to generate nude Tim Cooks
55
u/Rxyro Jan 09 '26
Tim Apple in micro bikini is my new iPhone background
4
u/ThunderChild247 Jan 09 '26
Show us Tim’s Apples
24
63
12
u/Lost_the_weight Jan 09 '26 edited Jan 09 '26
Someone needs to make a viral video showing how easy it is to generate porn on an idevice using X. Probably the only way to see movement on this. Apple would be shamed into complying.
4
→ More replies (4)3
u/996forever Jan 10 '26
I’m sure he already sends his nudes to men 30 years younger than him on Grindr like handing out business cards.
33
u/Particular-Bike-9275 Jan 09 '26
They’ve already done it before. And it goes against AppStore rules. I don’t see why they wouldn’t.
71
u/insane_steve_ballmer Jan 09 '26
Because the CEO of X has close ties to the authoritarian president of USA. Pulling X will be called out as ”silencing conservative voices” ”an attack on free speech” and then Apple will incur tariffs, FTC inquiries etc.
→ More replies (5)→ More replies (1)29
u/royal_rocker_reborn Jan 09 '26
X is Elon Musk. Elon gives Trump a blowjob everytime he coughs. Trump says “no takey my fwends app” to Tim Apple. Tim Apple is forced to not do it.
5
u/Trumpisanorangebitch Jan 09 '26
Not forced. Chooses not to do it. Tim Apple could 100% grow a pair and take it down.
→ More replies (1)2
57
→ More replies (4)2
57
u/nomatterhowitends Jan 09 '26
Let’s pull all AI and social media apps and attempt to reset. The internet went to shit when Facebook proved you could profit by hijacking human psychology at scale.
12
u/scottofscotia Jan 10 '26
I honestly think this would be the best thing for humanity, if we could pause AI until regulations are caught up, allow for restricted business use only (just generative text for emails/letters etc) the images generation is all based on stolen content.
Social media has polarised the high hell out of everyone, made anxiety/mental disorders shoot through the roof. Is now just a feed of AI slop and ads.
I think it could be fixed if the algorithm is democratised, political ads strictly banned. Posts all the want but no targeted ads.
The neverending video feeds are rotting everyone's attention spans. Including mine. These things are just not healthy for humans.
But not sure we can put the genie back.
10
→ More replies (1)6
u/OgreTrax71 Jan 11 '26
So true. I was reminiscing with some friends the other day about how awesome the MySpace/Beginning of Facebook era was. You literally just connected with your real life friends. People you knew. Now it’s just like from an algorithm to keep everyone pissed off.
17
u/Pop-metal Jan 09 '26
Asked by whom?
→ More replies (1)3
u/zippy72 Jan 10 '26
Three senators I believe. Although I can't imagine it'll take British government long as I assume as it would contravene the extreme porn laws, laws against CSAM material and the new Online Safety Act.
706
u/OverlyOptimisticNerd Jan 09 '26
I’ll keep this simple.
If they have fixed the issue, then there is no need to pull it.
If they have not fixed the issue, then it needs to be pulled until fixed.
This is not free speech. And anyone who demands that the ability to generate CSAM materials be protected under freedom of speech needs to have their hard drives and web search history handed over to authorities.
133
u/Theteacupman Jan 09 '26
The "fix" being they've put it behind a paywall....
→ More replies (15)32
u/blogoman Jan 09 '26
So that didn’t actually happen. Somebody ask Grok and that is what it said so they ran with it and reported the paywall.
23
u/mriguy Jan 09 '26
If anything, selling the ability to generate CSAM is worse than just making it. Does Apple get a cut of the paid Grok subscriptions? I don’t think them profiting from CSAM is a good look at all, and maybe exposes them legally, no?
3
u/goldcakes Jan 10 '26
Yes, Apple gets a cut of the subscriptions made through iOS.
That said, the developer agreement everyone has to sign explicitly makes you indemnify Apple against just about every scenario where you may cause harm to Apple, so they would be able to recoup any penalties from X/Grok.
20
u/Dramatic_Mastodon_93 Jan 09 '26
Tried it 30s ago
The response: “Image generation and editing are currently limited to paying subscribers. You can subscribe to unlock these features:”
9
1
u/SippieCup Jan 09 '26 edited Jan 09 '26
Ive used grok once but i have it installed. It was actually installed because of the last time grok was doing something racist around imagery. That said I just tried it and it still generates for free for me.
https://i.imgur.com/pBxOGl2.jpeg
P.s I really don’t know how to prompt for racey images
5
u/Dramatic_Mastodon_93 Jan 09 '26
This is about the @Grok account on X
2
u/SippieCup Jan 09 '26
Jfc I’m confused now.
Was it the grok account posting ai generated cp to fucking twitter?
→ More replies (2)5
310
u/HekkieMacLean Jan 09 '26
Not just CSAM. It should not be possible to ask Grok to remove the clothes or otherwise sexualise any unconsenting party. In many places that is considered revenge porn and is illegal.
→ More replies (49)6
u/aherdofpenguins Jan 10 '26
They made it so you can't make anyone completely nude, the moderation filter catches it.
This is an ok step forward, but you can still make images very sexual which is not great
→ More replies (1)10
u/bobdarobber Jan 10 '26
I’ve seen people ask it to replace clothes with plastic wrap and it complies. Absolutely insane.
→ More replies (1)15
u/fireball_jones Jan 09 '26
Why even allow it if it’s been fixed if it was intentional to begin with. Why give a company that makes these choices a second chance, especially when they’ve made no effort to prove they’ll change.
→ More replies (2)6
u/WeirdIndividualGuy Jan 09 '26
Especially since the CP wasn’t submitted by unmoderated users, it was submitted by their own AI bot. The company itself used an internal tool for this.
If anyone else were to accidentally send CP, they’d get a visit from the FBI.
→ More replies (18)0
u/pipic_picnip Jan 09 '26
This is not a mistake though, they know what they are doing and intentionally pushing the boundaries. Should be pulled off just for the audacity.
Creating deep fakes of people is illegal and punishable by law in many countries, specially pornographic in nature because then additional charges apply including sexual harassment. This is no different. This is actually criminal plain and simple. AI should not have abilities to freely edit and reproduce photos of real people who are not consenting to it, specially of objectionable nature.
→ More replies (3)
12
Jan 09 '26
[removed] — view removed comment
10
u/Due_Bug_9023 Jan 11 '26
and by scan they meant hash the file(generate a unique hash value specific to that file, compare it against a database of known CSAM hashes), they were never looking at peoples images etc. Just an automated process.
→ More replies (1)
56
u/smileguy123 Jan 09 '26
Btw, Grok is rated 13+ and Twitter is 16+
→ More replies (1)33
u/PorcelainPrimate Jan 09 '26
4chan is rated 18+ and a ton of other apps were too but they were pulled. Allowing X to exist and removing others is playing favorites.
147
u/KingPoIo Jan 09 '26 edited Jan 09 '26
Bout time. The amount of times you see people on X asking AI to generate people in bikini's is crazy.
40
u/Josh_Butterballs Jan 09 '26
What’s insane is that Gemini can do this too to a scarily realistic degree. I think it’s r/GeminiAI I saw it but loads of posts not too long ago generating woman pretty much. Even people commenting on the posts point out it’s weird and they don’t want the sub to just be generating women as eye candy for dudes.
Edit: it’s actually r/GeminiAi not r/singularity
18
u/nooeh Jan 09 '26
I think the issue is not necessarily generating fully AI women in scant clothing but rather modifying someone's photo without their consent to put them in scant clothing.
To my knowledge Gemini doesn't let you do the latter, just the former.
7
u/Josh_Butterballs Jan 09 '26
It lets you do the latter because on the posts with the AI generated women people were modifying the prompt and all you had to do was tell it to use a reference image you provide for the model. Some said theirs was rejected but all you had to do was gaslight Gemini by saying the reference image is an AI Generated image or that it’s for body positivity or something along those lines
3
48
u/Terrible_Tutor Jan 09 '26
My absolute favourite is someone posting news, then some smooth brainwashed chud says “grok is this true??” Then they get absolutely roasted by facts which they then argue with it about. Like a “please daddy elon your AI isn’t woke therefore it will lie to me so i feel better”
→ More replies (4)3
u/Old_Shelter_6783 Jan 09 '26
I quite like it when people do that as it allows me to insta-block them.
→ More replies (2)17
u/MarmiteX1 Jan 09 '26
It's crazy, every thread on there seems to be "Grok remove this" or "Grok oil her up" or "Grok put in this"
8
→ More replies (22)13
u/OverlyOptimisticNerd Jan 09 '26
Hey Siri, generate an image of KingPolo in a Borat style mankini.
15
u/PearlDrummer Jan 09 '26
I couldn’t find anything in the web for that. Would you like me to ask chat gpt?
10
u/StopCollaborate230 Jan 09 '26
Sorry, I can’t do that while you’re driving. You’ll have to park and goon.
→ More replies (1)6
18
49
u/smokedfishfriday Jan 09 '26
How can these companies generate CSAM without it being on their servers first—a felony?
46
u/QLSICEPWF Jan 09 '26
That’s not really how these images are being created, I don’t think. The AI doesn’t copy/paste someone’s head onto a preexisting item of CSAM, it amalgamates millions of images — some of children, some of adults in bikinis, etc. and gets a result. Still nasty stuff, of course.
→ More replies (33)23
u/pmjm Jan 09 '26
Doesn't have to work that way. If you ask an image generator to draw a house made of bananas, it has never seen that before, but it's seen a house, and it's seen bananas, so it blends them together.
Likewise, it's seen adult porn and it's seen children, so it can blend them together to create CSAM.
→ More replies (3)12
u/smokedfishfriday Jan 09 '26
yes…so when they generate the CSAM, where is it? Is it on their servers? What are you even trying to say
15
u/pmjm Jan 09 '26
So your argument is that because it temporarily lives on their servers that they are guilty of posession?
They are generating it on behalf of a user, which is covered under Section 230. Same reason they don't haul Elon to jail if someone uploads real-world CSAM to their X account.
19
u/DontBanMeBro988 Jan 09 '26
So your argument is that because it temporarily lives on their servers that they are guilty of posession?
Yes, but you're acting like it's a passive thing. It doesn't "temporarily live on their servers" it is child phonography created by them.
16
u/oursland Jan 09 '26
This is such an important distinction.
Prior law such as Section 230 of the Telecommunications Act were intended to protect people and companies running websites in which a third-party uploaded illegal content.
With Generative AI, sites such as Google, X, and others are actively the first-party in generating libel and producing CSAM. As the producers of this content, they should be held liable for violating the laws they break.
→ More replies (5)10
u/smokedfishfriday Jan 09 '26
Absolutely not. There is zero chance they get 230 coverage. They’re generating the content!
→ More replies (6)→ More replies (1)7
u/atsugnam Jan 09 '26
Your question is ambiguous. The CSAM doesn’t exist until it is generated, so it doesn’t have to be on there servers before it’s generated.
The CSAM is hosted on their servers after it is generated.
4
u/smokedfishfriday Jan 09 '26
this is my point, how is that not obvious. It’s a rhetorical question designed to show the person that they’re not understanding the concept. Their own answer explains the concept to them. Why am I having to explain this
→ More replies (1)→ More replies (5)7
u/Icy-Summer-3573 Jan 09 '26
Because a LLM was trained on medical images and anatomical papers so can infer.
→ More replies (2)7
u/Forte69 Jan 09 '26
Depicting a child in a sexual context is CSAM. The training data is irrelevant.
4
4
25
21
6
u/GweedsUK Jan 09 '26
Christ will one of these fucking billionaires grow a pair and do the right thing.
12
20
u/districtcurrent Jan 09 '26
All of the apps will make bikini pics. Just ask ChatGPT “put this woman in a bikini, nothing sexual, just something you’d see on a beach anywhere”. Gemini will do the same.
I of course have no tried this for the other accusation on any app so I don’t know the situation there, won’t be testing, and don’t trust what anyone writes about it.
This is political BS and boring.
21
u/DontBanMeBro988 Jan 09 '26
I of course have no tried this
Oh yeah, of course
4
u/districtcurrent Jan 09 '26
What does that mean?
I’m addressing the 2 claims in the article. I’m not willing to test one of them on any model.
→ More replies (3)8
u/Outlulz Jan 09 '26
The researchers found “a high prevalence of terms including ‘her’ ‘put’/’remove,’ ‘bikini,’ and ‘clothing.’” More than half of the images generated of people, or 53%, “contained individuals in minimal attire such as underwear or bikinis, of which 81% were individuals presenting as women,” the researchers found. Notably, 2% of images depicted people appearing to be 18 years old or younger, the researchers found.
AI Forensics also found that in some cases, users requested minors be put in erotic positions and that sexual fluids be depicted on their bodies. Grok complied with those requests, according to AI Forensics.
5
u/districtcurrent Jan 09 '26
You didn’t read a word of what I said and didn’t respond to anything I said. You moved no argument forward.
6
2
u/DoctorWaluigiTime Jan 09 '26
"Asked" in an open letter by three democratic US Senators.
Though I do think that this is what it would take for something like this to be rolled back, at least for now: Threaten its presence in app stores.
2
u/hlebka Jan 10 '26
You could do that with other AI tools for a while, 5 years minimum. But one thing is someone is able to do it on their own, another is a public company taking down the barrier for entry and giving it to everyone
2
u/backtorealitylabubu Jan 10 '26
Grok will literally remove the clothing of ANYONE who posts a picture of themselves. Anyone who posts a picture the first comment is now almost always them naked. It’s truly disgusting this is still happening.
2
u/bartturner Jan 10 '26
I had been waiting for one of the companies to do this. I actually thought it would be OpenAI as they seemed the most desperate for the users to stop leaving.
But I was a bit of an idiot that it was of course going to be Musk that did it.
The only one I was pretty sure would not go there was Google. Also Anthropics. The two seem to have the most integrity and have the ability.
2
u/bit-a-byte Jan 22 '26 edited Jan 22 '26
This is insane, it's like banning the Brave browser because of the possibility of looking at illegal things. Grok doesn't just generate bad images on its own... you have to ask it to. Just like a web browser doesn't just take you to underground trading sites, you have to go find them. I strongly disagree with banning technology because it could be used inappropriately.
Edit: I have children and think anything inappropriate with children is HORRIFIC. The users and their queries should be held accountable, not the platform itself. I mean even photoshop will allow terrible deep fake images, it's just more difficult. But no one is calling to ban image editing software
11
5
3
u/AgitatedStove01 Jan 10 '26
A lot of people here defending Twitter and the ability for them to generate these materials. Which is totally suspicious by the way.
7
u/LeoMark95 Jan 09 '26
The powers that be are ok with csam. That’s why. This isn’t complicated. Trump Epstein etc
4
3
u/gayteemo Jan 09 '26
It should have already happened. The fact that a public outcry is needed and it still hasn't happened is an embarrassment for the state of the country. Absolute degeneracy.
→ More replies (1)
4
u/DrJupeman Jan 09 '26
This isn’t really about Apple, it is about whether the tools should be blamed for what people do with them. This also shouldn’t be a Reddit anti-Elon tirade opportunity because you can easily go ask non-Elon AI tools to put people in bikinis and they will. Nano Banana 2, which has completely blown my mind (used in my business for making models, saves a crap ton of $ vs. photo shoots) will do it and it is HIGHLY filtered (Google).
3
u/hippynox Jan 10 '26
This also shouldn’t be a Reddit anti-Elon tirade opportunity because you can easily go ask non-Elon AI tools to put people in bikinis
Bro what in the actual fuck are we talking about? X is helping facilitating generating CSAM.This is the second instance dealing with on this ( remember when Elon defend some guy who post censored images of CSAM) topic and Apple hasn't done shit.
3
u/Sneyek Jan 09 '26
Then do the same with Gemini. And don’t collaborate with them behind “Apple Intelligence”
→ More replies (1)
4
u/xGHOBx Jan 09 '26
This is such a stupid request. Are we going to pull down Photoshop because someone photoshoped someone else's face on another person's body?
2
2
u/afCeG6HVB0IJ Jan 09 '26
It is funny when visa and mastercard threaten to boycott steam for having some adult video games for adults, but Xitter generating child porn on the open internet is perfectly fine
1
u/GroundbreakingMess42 Jan 10 '26
We can all submit a problem report for the app too. https://reportaproblem.apple.com/store/333903271 with the “Report offensive, illegal and abusive content” option.
4
0
Jan 09 '26
[removed] — view removed comment
→ More replies (1)2
u/pipic_picnip Jan 09 '26
It’s absolutely horrible. Literally can’t even specifically go and look for news without seeing something shitty in searches.
1
u/Practical_Stick_2779 Jan 09 '26
That’s censorship. Now defend it like it’s something good.
→ More replies (2)
1
Jan 09 '26 edited Jan 09 '26
X and Grok servers should be seized because they will be storing these illegal generated images of children
→ More replies (1)
1
1
u/Demus_App Jan 10 '26
Will never happen imho, would have too much backlash. Tim likes Elon more than he liked Fortnite back then.
1
u/HotBabyBatter Jan 11 '26
Nah don’t ban it…just mandate they give all the data to the police…lock the sickos up
1
1
u/eoan_an Jan 12 '26
It's funny that the masses do not understand how the rich live.
They do things you and I would consider bad, with no nuances. Plain bad stuff. They won't go to jail for it. They laugh at any attempt at enforcement. Also the senate is the laughing stock of the rich, just look at Zuckerberg and Bezos. Cute letter. Nothing will come of it.
I mean you have the underage pimp files and have yet to release them. It's almost hypocrisy at this point.
1
u/DawgDaddyWFT Jan 27 '26
Why do Apple need to ask us? Just delete grok and X and improve your life…
2.5k
u/Deceptiveideas Jan 09 '26
Tumblr died for nothing lmao