A Preliminary Study into Reddit Moderation Effectiveness

I. Introduction

Community moderation plays a pivotal role in building healthy, high-engagement communities, yet can frequently became an overlooked, under-resourced operation. Reddit is an international, social news website with more than 430 million members. Unlike similiar scale social media platforms like Twitter and Facebook, Reddit incorporates thousands of individual, niche communites who independently handle their moderation efforts.

Struggles to contain hate speech on online platforms are well documented. The unique nature of Reddit, however, creates the potential for high variance in moderation quality based on each individual forum (subreddit). To investigate cross-subreddit moderation differences and better understand the current state of Reddit community moderation, we conduct the following study.

II. Implementation

For our purpose, we define "effectiveness" as combination of the perecentage of all hateful, rule-breaking comments removed and the speed in which they are removed at.

A five-member sample of the 50 most popular subreddits by member count was taken and r/news, r/worldnews, r/AskReddit, r/blog, and r/food were selected. Using Reddit’s Stream API, three batches of 1000 comments were collected for each subreddit. Distinct time intervals averaging seven hours apart were used to account for timezone activity differences, and batches were taken randomly across a period of ten days from 9/17/2022 to 9/27/2022.

Each comment in a batch of 1000 comments was run through our moderation API endpoint (https://api.moderatehatespeech.com/api/v1/moderate/) and comments >= 90% confidence level were flagged. For each flagged comment, the removal status of the comment was repeatedly checked every hour over a 48 hours interval in order to track the removal time and status, and results were aggregated into categorical percentages. Removal by a subreddit moderator, Reddit administrator, and self-deletion were all considered “removal” for our purposes.

To verify the accuracy of flagged comments, each flagged comment was manually reviewed and compared with past moderator removal actions for the subreddit, to determine if the comment met the criteria for removal. Comments considered falsely flagged were expunged from the final data.

III. Analysis

From the cleaned data, we find that as both an aggregated count and percentage, r/worldnews had the highest number of hateful comments (94 across 3000 comments, ~3%). Given the polarizing nature of news and politics, this might be expected, but we also find r/news has a significantly lower number of flagged comments (33 across 3000, ~1%). We believe this difference is attributable to the more comprehensive subreddit rules enforced by r/news and a higher rate of automated comment removals.

r/worldnews84 r/technology45r/news33r/AskReddit16 r/food10

As a percentage of all hateful comments removed, r/AskReddit fares the best, removing 50% of flagged comments, whereas r/technology comes in last at 11% removed. Accounting for comment value (above), we observe a slight downward trend between # of flagged comments and % of comments removed (r = -0.48), however P = 0.4132 @ 95% CI, suggesting an area for further study.

r/AskReddit50% r/food30%r/worldnews23%r/news18% r/technology11%

Finally, looking at the speed of removal (time removed - time posted), we find that r/AskReddit has the fastest average removal time, at 3.375 hours online before a comment is removed.

r/AskReddit3.4 hrs r/worldnews5.0 hrsr/news5.5 hrsr/food12.6 hrs r/technology17.6 hrs

Of the five subreddits studied here, we find that r/AskReddit is the most effective in removing hateful, rule-breaking comments, doing so in the highest percentage of time. While its not feasible to derive causation from our data, r/AskReddit has, by far, the highest number of comments per day, suggesting that either r/AskReddit has more experience moderating at a larger scale and thus a more streamlined pipeline, or more user activity which leads to more user reports. Topically, the percentage of flagged comments are expected, with casual, conversational subreddits like r/food and r/AskReddit having the lowest percentage, and more polarizing subreddits like r/worldnews having a much higher percentage of flagged comments.

While a pending, large-scale study is conducted, we find that these preliminary results show high variance in the moderation effectiveness between subreddits.