🔖 Community Archive
Deployed at: https://www.community-archive.org/
We believe there is immense value in preserving the history of subcultures on twitter. While Twitter charges exorbitantly for convenient access to our own data, we are getting people to request and download their data, and upload it to a common archive!
_from Xiq's Towards data sovereignty, and a backup of the twitter canon_.
The goals of this project are (1) create a public domain dataset so that we can analyze & build apps on top of our own data, commercial or otherwise (2) develop an open workflow for archival that people can self host or create private archives for their communities if they wish.
Bulk data
For bulk analysis, use the current Parquet export with tweets, profiles, and a manifest. Check the manifest timestamp for freshness and resolve the latest pointer each time; older download links expire. See the data access guide for downloads and targeted API queries.
How to use the API (from your own app)
See API docs for the keyed API v1, the bulk Parquet export, the open semantic search API, and the legacy Supabase REST API.
See also this Google colab notebook for an example you can run in your browser that fetches tweets from the archive for a specific user and runs some basic analysis.
Development Instructions
See local setup.
Contributing
We welcome all contributions! Contribute to the code, or upload your archive, clone it & try it locally, join our discord & give us feedback or ideas.
Privacy
We take your privacy seriously. When you upload your Twitter archive:
- Data is processed locally on your device
- Only essential information is sent to our servers
- No personal messages or sensitive data are uploaded