Project Avril is a community data archive for machine learning. It gathers conversational data from the general and introduction channels of Discord servers and publishes it all as one big free archive that any ML enthusiast can access.
Most conversation on Discord never leaves the servers it lives in. Messages are locked inside communities, and almost none of it is available for research or training. Project Avril aims to collect public conversation, strip it down to clean data, and releases the result so anyone can study it, train on it, or build on it.
- Servers that you scrape must be non-NSFW and family friendly
- Please verify by checking in the search bar, that no major slurs are present in the chat that you will be exporting from, prior to export
- Desired channels are:
- Introduction channels (were people state who they are and etc.)
- General chats, lounges, chats, human longform conversations
- Chats with links to books, PDF's and other informatory content, or containing informatory content themselves
- Undesired channels include:
- Chats with high amounts of media (images are not included by the scraper)
- Chats from servers that are poorly or completely not moderated
- Any other channels will most likely not be taken into account, unless they have a special, desirable content
There is a submissions channel in our Discord server
where anyone can chip in. Run the scraper yourself on your general and introduction channels,
then upload the exported .parquet files to the submissions channel.
Every file that chips in makes the archive bigger and more useful for everyone.
Bot tokens are recommended. Add the bot to each server with View Channel and
Read Message History; enable the Message Content privileged intent if message
content is required. Discord does not expose a bot's access to a user's normal
DM and group DM history, so those sources require a user token.
Automating a normal Discord user account may violate Discord's terms and may
put that account at risk. Tokens are sent only to discord.com, we do not collect them.
Token saving is disabled by default, when enabled, the token is stored by the operating
system's credential manager rather than in a settings file or outside server.
To get the token for your personal account:
- Open Discord in your web browser and login
- Open any server or direct message channel
- Press Ctrl+Shift+I to show developer tools
- Navigate to the Network tab
- Press Ctrl+R to reload
- Switch between random channels to trigger network requests
- Search for a request that starts with messages
- Select the Headers tab on the right
- Scroll down to the Request Headers section
- Copy the value of the authorization header
The token is generated during bot creation. If you lost it, generate a new one:
- Open Discords developer portal
- Open your application's settings
- Navigate to the Bot section on the left
- Under Token click Reset Token
- Click Yes, do it! and authenticate to confirm
Remember:
- Integrations using the previous token will stop working until updated
- Your bot needs to have the Message Content Intent enabled to read messages
I will start off by saying that the preferred format is PARQUET please use it for the sake of ease of submission and to save yourself computer space.
Only text useful for conversational training is retained. Empty messages and attachment or embed only events are skipped. IDs, timestamps, replies, reactions, attachments, embeds, pins, and Discord message types are discarded.
TOON keeps the human readable source and channel names so each file remains identifiable. Every message has exactly two fields:
source: "Example server"
channel: "general"
messages[1]:
- author: "snek"
content: "hello"
Parquet contains only two string columns, author and content. The human
readable source and channel names are stored in schema metadata.
See THIRD_PARTY_NOTICES.md for attribution.
Thank you to everyone who joins us in building the archive. More data means better models for everyone, and every contribution gets us there.
See LICENSE