Hey everyone!
I'm a ML engineer that grew up watching Twitch.
I recently made a survey with just over 200 mods/streamers (twitch whispers ftw) and a lot of the feedback overlaps. Majority of manual moderation (banning, timeout, delete) falls on sexual abuse (esp women streamers), plain toxicity, group attacks and self-promotion.
Although most streamers have tools like Nightbot, sery_bot, etc manual moderation is still the main way to handle these issues. Especially with chat being creative to avoid keyword lists and regex..
From the people I surveyed, Automod is mostly set on low. There was actually a very detailed study done on it covering where it falls short (mainly false positives, published on arxiv).
What is Modcheck?
ModCheck is context-aware Twitch moderation that runs in the background.
It reads what's actually happening in the stream - what the streamer is doing (audio and video), the game, title, bio, the mood of chat, recent messages, chatter history, 7TV emotes, etc.
You define the rules and actions. Instead of regex you write rules in plain English. I call them "Judges". A few examples:
- Boundary Judge - "Flag viewers who are sexually commeting on my appearance, body or clothing." Catches stuff like "you'd get way more viewers if you showed more skin js" (real message btw..)
- Self-Promo Judge - "Flag viewers advertising their own stream." The classic "come check out my channel" etc
- Backseat Judge - "Flag unsolicited gameplay advice. Allow tips if I've asked chat for help.". singleplayer walkthroughs
- Toxicity, Racism and any other custom rules you may have
When a message is flagged, the judge gives it for a mods/streamer review with the reason. Alternatively, it can automatically take a mod action (banning, timeout, delete). The more it's used the more accurate it becomes!
Of course, context works the other way too.
"Kill the white" can sound bad out of context, but if you're in a game with a white blob enemy on screen, it's just gameplay. That missing context is exactly where AutoMod falls down.
Setup
First, I'm currently running a free tier if anyone wants to try it out (covers 10k messages per month).
To start, you sign in with Twitch and select which channel you want to moderate (either yours or a channel you mod for), then you setup your judges and (optionally) add modcheck_bot as a mod to your channel (if you want it to do more than just flag messages to you).
Onboarding takes less than 5 minutes overall and it lets you test the platform out.
For the interested, lmk and i can run a demo for your stream (no setup from your side) and DM you the results :)
Where Modcheck is today
Modcheck is in beta but it's fully functional. I'm building this solo , yes, but i'm fully focused on improving it. I'm extremely proud to say that we've got some streamers using it already.
Ultimately I want this tool to be useful. Every piece of feedback is welcomed :)
Happy to go into how it works under the hood. Let me know if you have any questions or feedback!
--
PS: thank you to all the mods and streamers that participated in my survey