> ## Documentation Index
> Fetch the complete documentation index at: https://social-b97141fb-feat-user-history-moderation-detail.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.
# AI Content Moderation
> Leverage AI-powered moderation tools to create safer communities with automated content filtering and intelligent threat detection
Create safer online communities with intelligent, automated content moderation. social.plus leverages advanced AI to scan and filter inappropriate content across text, images, and video, ensuring community standards are maintained without constant manual oversight.
Block inappropriate content before it's published with proactive AI scanning
Monitor and review published content with intelligent flagging and automated actions
## Overview
social.plus offers two complementary AI moderation approaches:
**Proactive Content Filtering**
* Content is scanned before publication
* AI generates confidence scores for detected violations
* Content blocked if confidence exceeds configured threshold
* User must modify content to proceed with posting
**Reactive Content Review**
* Content is scanned after publication
* Uses `flagConfidence` and `blockConfidence` thresholds
* Automatically flags content for review or removes violations
* Maintains community safety without blocking legitimate content
## Getting Started
Contact our [support team](mailto:support@social.plus) to enable AI content moderation for your application.
Set up confidence levels and moderation categories through the social.plus Console.
Test with sample content and monitor moderation effectiveness through analytics.
## AI Pre-Moderation
Prevent inappropriate content from reaching your community with proactive AI scanning. Pre-moderation ensures all content meets your standards before publication.
**Current Availability**: Pre-moderation is currently available for image content, with text and video support coming soon.
### Image Content Detection
Our AI pre-moderation scans all uploaded images for inappropriate content across four key categories:
* **Nudity**: Detection of explicit or inappropriate nudity
* **Suggestive Content**: Sexually suggestive or provocative imagery
* **Violence**: Violent or graphic content detection
* **Disturbing Content**: Content that may be psychologically disturbing
### Configuration
Navigate to **Moderation > Image Moderation** in your social.plus Console and toggle "Enable image moderation" to **ON**.
Configure confidence thresholds for each category based on your community standards.
Upload test images to verify your confidence settings work as expected.
### Understanding Confidence Levels
**Important**: Confidence levels significantly impact moderation accuracy. Default settings may produce false positives.
Confidence levels represent the AI's certainty in detecting specific content types:
* **Low Confidence (0-30)**: High sensitivity, may block legitimate content
* **Medium Confidence (40-70)**: Balanced approach for most communities
* **High Confidence (80-100)**: Conservative filtering, may miss some violations
**Recommendation**: Start with medium confidence levels (40-60) and adjust based on your community's needs and false positive rates.
### Retaining Rejected Uploads
When pre-moderation refuses an image, the upload fails and the file is gone. There is nothing to look at if the uploader disputes the decision, nothing to tune your thresholds against, and no record that the attempt happened at all.
Turning on **reserved for review** changes that: social.plus keeps the refused original and tells your backend, over a webhook, the moment a rejection happens. Your moderators can then look at what the classifier actually blocked.
Retention is off by default and applies to images, the only content type pre-moderation covers today. The upload response is identical whether it is on or off, so your apps need no changes.
Send `reservedForReview: true` to [Update file configuration](/api-reference/network-setting/update-file-configuration) with an admin token.
Add [`file.didReject`](/api-reference/webhook-event/file-didreject-webhook-event) to your webhook subscription. It carries the `fileId` of the retained file — the upload response never does, so this event is the only way to learn it.
Fetch the retained image with [Download file](/api-reference/file/download-file), using the `fileId` from the event. Remove it when you are done with [Delete file](/api-reference/file/delete-file).
#### What the uploader sees
Nothing changes. A refused image still returns `403` with error code `400314` (`UnsafeContent`) and no `fileId`, whether or not retention is on:
```json theme={null}
{
"status": "error",
"code": 400314,
"message": "Image uploading failed: Nudity content is not permitted"
}
```
The message names the first category that breached its threshold: `Nudity content is not permitted`, `Suggestive content is not permitted`, `Violence content is not permitted`, or `Disturbing content is not permitted`.
#### What is retained
A retained file is an ordinary file document with `status` set to `rejected`. It uses the same `fileId`, the same file endpoints, and the same `network` access type as any other file your network owns.
Two differences are worth knowing:
* **Only the original is kept.** No resized variants are generated, so a download returns the original whatever `size` you ask for.
* **The rejection details are not on the file.** Reading the file never returns them. They reach you through the `file.didReject` webhook only.
Retention covers images, matching pre-moderation's coverage. A file that is deleted is deleted normally, whether or not it was retained.
#### Rejection and upload events
Three file lifecycle events are available to webhook subscribers:
| Event | Fires when |
| - | - |
| [`file.didUpload`](/api-reference/webhook-event/file-didupload-webhook-event) | A user uploads a file successfully |
| [`file.didReject`](/api-reference/webhook-event/file-didreject-webhook-event) | Pre-moderation refuses an image |
| [`file.didDelete`](/api-reference/webhook-event/file-diddelete-webhook-event) | A file is deleted |
`file.didReject` is useful on its own. With retention off, you still receive every rejection — with the classifier's verdict but no `file`, because no file was kept. A rejected upload raises this event instead of `file.didUpload`.
Its `rejection` object lists every category that breached its threshold with the confidence the classifier assigned, echoes your network's configured thresholds, and carries the message the uploader saw — enough to tell a marginal call from a clear one, and to tune your thresholds from real traffic.
**The rejection details are sensitive.** They pair your configured thresholds with the classifier's scores. Treat the `file.didReject` payload as admin-only data and do not forward it to end users. This is why reading the file never returns it.
## AI Post-Moderation
Monitor and moderate published content with intelligent detection and automated response workflows. Post-moderation provides comprehensive scanning across all content types while maintaining user experience.
All AI post-moderation results are surfaced through the **Moderation Feed** in the social.plus Console, located under **Moderation > Moderation feed**.
Detect inappropriate language, hate speech, and harmful text content
Scan visual content for policy violations and harmful imagery
AI moderation for display names, avatars, and descriptions — see dedicated page
### Moderation Feed
The Moderation Feed is the central hub for reviewing AI-flagged content. It is organized into two main workflow tabs:
The **To review** tab displays all content that requires moderator attention, organized into sub-tabs:
* **Posts and comments** — Flagged posts and comments from communities and user timelines
* **Messages** — Flagged messages from channels and direct conversations
* **Users** — Flagged user profiles — see [AI User Profile Moderation](/analytics-and-moderation/console/ai-user-profile-moderation)
Each flagged item displays:
* The AI moderation label and detected categories (e.g., "AI Mod: Harassment or bullying")
* PII detection results when applicable (e.g., URLs, person types)
* The number of user reports (e.g., "1 user", "4 users")
* The last flagged timestamp
* Available moderation actions
**Available actions for posts and messages:**
* **Delete post / Delete message** — Remove the content
* **Clear flag** — Dismiss the flag and approve the content
The **Reviewed** tab shows content that has already been moderated, organized into the same three sub-tabs (Posts and comments, Messages, Users).
Each reviewed item shows the action taken and who performed it:
* **Flag cleared by \[moderator]** — Flag was dismissed
* **Deleted by \[moderator]** — Content was removed
* **Reviewed by \[moderator]** — Content was reviewed without a specific action
The Reviewed tab includes an additional **Select moderator** filter to view actions taken by a specific team member.
Use the filter dropdowns (All feeds / All Channels, Select creator / Select sender) to narrow down the moderation queue by community, channel, or content creator.
### Content Coverage
* Text, images, videos, clips, files, and livestream content
* Posts across all communities and user timelines
* Comments and reply chains on posts
* Filter by specific community feed or content creator
* Text, image, video, audio, and file messages
* Messages from group channels, direct messages, and live chat
* Filter by specific channel or message sender
### Text Content Detection
The AI text moderation identifies and handles various types of inappropriate text content. Each category is scored on a severity scale from 0 to 7.
Each name below is a value for the `category` field. To set thresholds for one, add an entry like this to your configuration:
```json theme={null}
{ "category": "SelfHarm", "flagConfidence": 2, "blockConfidence": 5, "moderationType": "text" }
```
* **`Hate`**: Hate speech targeting protected groups
* **`Sexual`**: Adult content and explicit sexual references
* **`ViolenceText`**: Violent threats, graphic descriptions, or dangerous activities
* **`SelfHarm`**: Content related to self-harm or suicidal ideation
See [API Configuration](#api-configuration) for the full request, and [Understanding Confidence Scores](#understanding-confidence-scores) for the default thresholds.
In addition to content policy categories, the AI performs **Personally Identifiable Information (PII) detection** to flag sensitive data:
* **URL** — Links and web addresses embedded in content
* **PersonType** — References to specific person types or identities
Content that passes all AI checks displays **"AI Mod: Passed"** in the moderation feed. Content with detected violations shows the specific category labels.
### Multimedia Content Detection
**Comprehensive Scanning**: Our AI analyzes both static images and video content frame-by-frame for maximum protection.
Advanced visual content analysis covers extensive categories, each scored on a confidence scale from 0 to 100. Images and videos share the same set of categories.
Every name in the groups below is a value for the `category` field. To set thresholds for one, add an entry like this to your configuration:
```json theme={null}
{ "category": "Explicit Nudity", "flagConfidence": 40, "blockConfidence": 80, "moderationType": "media" }
```
Categories are grouped here for readability only — the grouping has no effect on configuration, and each name is set independently. See [API Configuration](#api-configuration) for a request containing every category.
* `Explicit Nudity`, `Exposed Female Genitalia`, `Exposed Male Genitalia`
* `Explicit Sexual Activity`, `Sexual Activity`, `Sex Toys`
* `Non-Explicit Nudity of Intimate parts and Kissing`, `Implied Nudity`, `Kissing on the Lips`
* `Exposed Male Nipple`, `Barechested Male`
* `Swimwear or Underwear`, `Female Swimwear or Underwear`, `Male Swimwear or Underwear`
* `Graphic Female Nudity`, `Graphic Male Nudity`, `Illustrated Explicit Nudity`
* `Violence`, `Physical Violence`, `Weapon Violence`, `Weapons`
* `Blood & Gore`, `Explosions and Blasts`
* `Self-Harm`, `Corpses`, `Emaciated Bodies`
* `Visually Disturbing`, `Air Crash`
* `Alcohol`, `Alcoholic Beverages`, `Drinking`
* `Drugs & Tobacco`, `Drugs & Tobacco Paraphernalia & Use`, `Drug Products`
* `Pills`, `Smoking`
* `Hate Symbols`, `Extremist`, `Nazi Party`, `White Supremacy`
* `Rude Gestures`, `Middle Finger`
* `Gambling`
**User Profile Moderation**: AI moderation for user profiles (display names, avatars, descriptions) is covered in a dedicated page. See [AI User Profile Moderation](/analytics-and-moderation/console/ai-user-profile-moderation) for setup, admin reset workflows, blocklist configuration, and moderation feed details.
### Understanding Confidence Scores
Each category has two thresholds. Content is removed when its score reaches the block threshold, and flagged for review when it reaches the flag threshold.
**Flag Confidence-Media** (Default: 40)
* Content scoring at or above this level gets flagged for review
* Lower values = more content flagged (higher sensitivity)
* Recommended range: 30-60 depending on community standards
**Block Confidence-Media** (Default: 80)
* Content scoring at or above this level gets automatically removed
* Higher values = fewer false positives
* Recommended range: 70-90 for balanced protection
**Flag Confidence-Text** (Default: 3, or 2 for `SelfHarm`)
* Content scoring at or above this level gets flagged for review
* Lower values = more content flagged (higher sensitivity)
* Recommended range: 3-4 depending on community standards
**Block Confidence-Text** (Default: 5)
* Content scoring at or above this level gets automatically removed
* Higher values = fewer false positives
* Recommended range: 4-5 for balanced protection
**Media Moderation (From 0 to 100 Scale)**
* **0-39**: Content passes moderation (approved)
* **40-79**: Content flagged for human review
* **80-100**: Content automatically blocked/removed
**Text Moderation (From 0 to 7 Scale)**
* **0-2**: Content passes moderation (approved)
* **3-4**: Content flagged for human review
* **5-7**: Content automatically blocked/removed
*Note: These ranges use default thresholds and can be customized. For text moderation, thresholds should be integers from 0 to 7. Because the classifier produces integer severity levels, a fractional threshold is effectively rounded up to the next integer—for example, 4.5 behaves as 5. There is no finer-grained score between integer levels.*
**Default Configuration**: Media categories start with `flagConfidence: 40` and `blockConfidence: 80`. Text categories start with `flagConfidence: 3` and `blockConfidence: 5`, except `SelfHarm`, which flags from 2. Monitor your community's content patterns and adjust these values to optimize for your specific needs.
### Configuration Parameters
| Parameter | Type | Description |
| - | - | - |
| `enabled` | Boolean | Turns AI post-moderation on or off for the network |
| `commentDeletionBehavior` | String | `delete_whole_comment` (default) or `delete_attachment_only` |
| `confidences` | Array | One entry per category, 1 to 100 entries. Sending this field replaces the entire stored list |
| `category` | String | Exact name of the moderation category, case-sensitive |
| `flagConfidence` | Number | Threshold for flagging content (0-100 for Media and 0-7 for Text) |
| `blockConfidence` | Number | Threshold for blocking content (0-100 for Media and 0-7 for Text) |
| `moderationType` | String | Type of content: `text` or `media`. Images and videos both use `media` |
| `userModerationExclusionPatterns` | Array | Regex patterns matching user IDs that should never be moderated |
| `avatarCustomUrlEnabled` | Boolean | Moderates externally hosted avatar URLs. Off by default |
**Sending `confidences` replaces the whole list**: the API does not merge your entry into the stored configuration. Any category you leave out is removed and stops being moderated. Always send the complete list, as shown in the example below.
### API Configuration
Select the appropriate API endpoint for your region to ensure optimal performance:
| Region | API Endpoint |
| - | - |
| **Europe** | `https://apix.eu.amity.co` |
| **Singapore** | `https://apix.sg.amity.co` |
| **United States** | `https://apix.us.amity.co` |
Both requests use the same path:
```
GET /api/v3/network-settings/enhance-moderation
PUT /api/v3/network-settings/enhance-moderation
```
The `PUT` requires an admin access token. The `GET` can be read with any authenticated user token.
*Note: The settings API is designed with eventual consistency, so a subsequent GET may temporarily return the previous configuration. The successful PUT response is the authoritative acknowledgement of the accepted update and can be used for programmatic verification.*
```bash Retrieve current configuration theme={null}
curl 'https://apix.eu.amity.co/api/v3/network-settings/enhance-moderation' \
-H 'Authorization: Bearer '
```
```bash Update all categories theme={null}
curl -X PUT 'https://apix.eu.amity.co/api/v3/network-settings/enhance-moderation' \
-H 'Authorization: Bearer ' \
-H 'Content-Type: application/json' \
-d '{
"confidences": [
{ "category": "Hate", "flagConfidence": 3, "blockConfidence": 5, "moderationType": "text" },
{ "category": "Sexual", "flagConfidence": 3, "blockConfidence": 5, "moderationType": "text" },
{ "category": "ViolenceText", "flagConfidence": 3, "blockConfidence": 5, "moderationType": "text" },
{ "category": "SelfHarm", "flagConfidence": 2, "blockConfidence": 5, "moderationType": "text" },
{ "category": "Air Crash", "flagConfidence": 40, "blockConfidence": 80, "moderationType": "media" },
{ "category": "Alcohol", "flagConfidence": 40, "blockConfidence": 80, "moderationType": "media" },
{ "category": "Alcoholic Beverages", "flagConfidence": 40, "blockConfidence": 80, "moderationType": "media" },
{ "category": "Barechested Male", "flagConfidence": 40, "blockConfidence": 80, "moderationType": "media" },
{ "category": "Blood & Gore", "flagConfidence": 40, "blockConfidence": 80, "moderationType": "media" },
{ "category": "Corpses", "flagConfidence": 40, "blockConfidence": 80, "moderationType": "media" },
{ "category": "Drinking", "flagConfidence": 40, "blockConfidence": 80, "moderationType": "media" },
{ "category": "Drug Products", "flagConfidence": 40, "blockConfidence": 80, "moderationType": "media" },
{ "category": "Drugs & Tobacco", "flagConfidence": 40, "blockConfidence": 80, "moderationType": "media" },
{ "category": "Drugs & Tobacco Paraphernalia & Use", "flagConfidence": 40, "blockConfidence": 80, "moderationType": "media" },
{ "category": "Emaciated Bodies", "flagConfidence": 40, "blockConfidence": 80, "moderationType": "media" },
{ "category": "Explicit Nudity", "flagConfidence": 40, "blockConfidence": 80, "moderationType": "media" },
{ "category": "Explicit Sexual Activity", "flagConfidence": 40, "blockConfidence": 80, "moderationType": "media" },
{ "category": "Explosions and Blasts", "flagConfidence": 40, "blockConfidence": 80, "moderationType": "media" },
{ "category": "Exposed Male Genitalia", "flagConfidence": 40, "blockConfidence": 80, "moderationType": "media" },
{ "category": "Exposed Male Nipple", "flagConfidence": 40, "blockConfidence": 80, "moderationType": "media" },
{ "category": "Exposed Female Genitalia", "flagConfidence": 40, "blockConfidence": 80, "moderationType": "media" },
{ "category": "Extremist", "flagConfidence": 40, "blockConfidence": 80, "moderationType": "media" },
{ "category": "Female Swimwear or Underwear", "flagConfidence": 40, "blockConfidence": 80, "moderationType": "media" },
{ "category": "Gambling", "flagConfidence": 40, "blockConfidence": 80, "moderationType": "media" },
{ "category": "Graphic Female Nudity", "flagConfidence": 40, "blockConfidence": 80, "moderationType": "media" },
{ "category": "Graphic Male Nudity", "flagConfidence": 40, "blockConfidence": 80, "moderationType": "media" },
{ "category": "Hate Symbols", "flagConfidence": 40, "blockConfidence": 80, "moderationType": "media" },
{ "category": "Illustrated Explicit Nudity", "flagConfidence": 40, "blockConfidence": 80, "moderationType": "media" },
{ "category": "Implied Nudity", "flagConfidence": 40, "blockConfidence": 80, "moderationType": "media" },
{ "category": "Kissing on the Lips", "flagConfidence": 40, "blockConfidence": 80, "moderationType": "media" },
{ "category": "Male Swimwear or Underwear", "flagConfidence": 40, "blockConfidence": 80, "moderationType": "media" },
{ "category": "Middle Finger", "flagConfidence": 40, "blockConfidence": 80, "moderationType": "media" },
{ "category": "Nazi Party", "flagConfidence": 40, "blockConfidence": 80, "moderationType": "media" },
{ "category": "Non-Explicit Nudity of Intimate parts and Kissing", "flagConfidence": 40, "blockConfidence": 80, "moderationType": "media" },
{ "category": "Physical Violence", "flagConfidence": 40, "blockConfidence": 80, "moderationType": "media" },
{ "category": "Pills", "flagConfidence": 40, "blockConfidence": 80, "moderationType": "media" },
{ "category": "Rude Gestures", "flagConfidence": 40, "blockConfidence": 80, "moderationType": "media" },
{ "category": "Self-Harm", "flagConfidence": 40, "blockConfidence": 80, "moderationType": "media" },
{ "category": "Sex Toys", "flagConfidence": 40, "blockConfidence": 80, "moderationType": "media" },
{ "category": "Sexual Activity", "flagConfidence": 40, "blockConfidence": 80, "moderationType": "media" },
{ "category": "Smoking", "flagConfidence": 40, "blockConfidence": 80, "moderationType": "media" },
{ "category": "Swimwear or Underwear", "flagConfidence": 40, "blockConfidence": 80, "moderationType": "media" },
{ "category": "Violence", "flagConfidence": 40, "blockConfidence": 80, "moderationType": "media" },
{ "category": "Visually Disturbing", "flagConfidence": 40, "blockConfidence": 80, "moderationType": "media" },
{ "category": "Weapon Violence", "flagConfidence": 40, "blockConfidence": 80, "moderationType": "media" },
{ "category": "Weapons", "flagConfidence": 40, "blockConfidence": 80, "moderationType": "media" },
{ "category": "White Supremacy", "flagConfidence": 40, "blockConfidence": 80, "moderationType": "media" }
]
}'
```
The example above sends every available category at its default thresholds, so it is a safe starting point to copy and edit. Change the `flagConfidence` and `blockConfidence` values you want to tune, and remove any category you do not want moderated.
The optional fields `enabled`, `commentDeletionBehavior`, `userModerationExclusionPatterns`, and `avatarCustomUrlEnabled` can be sent in the same request body, and are written only when included.
## API Reference
For detailed administration workflows, see the [Moderation Overview](/analytics-and-moderation/console/moderation/overview) and analytics export documentation.
## Best Practices
* **Start Conservative**: Begin with moderate confidence levels and adjust based on results
* **Monitor Performance**: Track false positive and false negative rates
* **Community-Specific**: Tailor settings to your community's content standards
* **Regular Review**: Periodically review and update thresholds as your community evolves
* **Review Queue Management**: Ensure consistent review of flagged content
* **Moderator Training**: Train team on community standards and edge cases
* **Appeal Process**: Provide clear paths for users to contest moderation decisions
* **Transparency**: Communicate moderation policies clearly to users
* **Batch Processing**: Handle high-volume content efficiently
* **Regional APIs**: Use geographically appropriate endpoints
* **Webhook Integration**: Implement real-time event handling for flagged content
* **Monitoring**: Set up alerts for unusual moderation patterns
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.