767bd4a947 | ||
---|---|---|
.vscode | ||
backend | ||
frontend | ||
gephi | ||
.dokku-monorepo | ||
.gitignore | ||
.gitlab-ci.yml | ||
BILL-OF-MATERIALS.md | ||
CHANGELOG.md | ||
LICENSE | ||
README.md | ||
docker-compose.yml | ||
example.env | ||
netlify.toml | ||
nlnet-logo.png | ||
screenshot.png |
README.md
fediverse.space 🌐
The map of the fediverse that you always wanted.
Read the latest updates on Mastodon: @fediversespace
Requirements
Though dockerized, backend development is easiest if you have the following installed.
- For the scraper + API:
- Elixir
- Postgres
- For laying out the graph:
- Java
- For the frontend:
- Node.js
- Yarn
Running it
Backend
cp example.env .env
and modify environment variables as requireddocker-compose build
docker-compose up -d phoenix
- if you don't specify
phoenix
, it'll also startgephi
which should only be run as a regular one-off job
- if you don't specify
- Create the elasticsearch index:
iex -S mix app.start
Elasticsearch.Index.hot_swap(Backend.Elasticsearch.Cluster, :instances)
Frontend
cd frontend && yarn install
yarn start
Commands
Backend
./gradlew shadowJar
compiles the graph layout program. java -Xmx1g -jar build/libs/graphBuilder.jar
runs it.
If running in docker, this means you run
docker-compose build gephi
docker-compose run gephi java -Xmx1g -jar build/libs/graphBuilder.jar
lays out the graph
Frontend
yarn build
creates an optimized build for deployment
Privacy
This project doesn't crawl personal instances: the goal is to understand communities, not individuals. The threshold for what makes an instance "personal" is defined in the backend config and the graph builder SQL.
Deployment
You don't have to follow these instructions, but it's one way to set up a continuous deployment pipeline. The following are for the backend; the frontend is just a static HTML/JS site that can be deployed anywhere.
- Install Dokku on your web server.
- Install dokku-postgres, dokku-monorepo, dokku-elasticsearch, and dokku-letsencrypt.
- Create the apps
dokku apps:create phoenix
dokku apps:create gephi
- Create the backing database
dokku postgres:create fediversedb
dokku postgres:link fediversedb phoenix
dokku postgres:link fediversedb gephi
- Set up ElasticSearch
dokku elasticsearch:create fediverse
dokku elasticsearch:link fediverse phoenix
- Update the backend configuration. In particular, change the
user_agent
in config.exs to something descriptive. - Push the apps, e.g.
git push dokku@<DOMAIN>:phoenix
(note that the first push cannot be from the CD pipeline). - Set up SSL for the Phoenix app
dokku letsencrypt phoenix
dokku letsencrypt:cron-job --add
- Set up a cron job for the graph layout (use the
dokku
user). E.g.
SHELL=/bin/bash
0 2 * * * /usr/bin/dokku run gephi java -Xmx1g -jar build/libs/graphBuilder.jar
- (Optional) Set up caching with something like dokku-nginx-cache
Before the app starts running, make sure that the Elasticsearch index exists -- otherwise it'll create one called
instances
, which should be the name of the alias. Then it won't be able to hot swap if you reindex in the future.
Acknowledgements
Many thanks to NLnet for their support and guidance of this project.