Gotosocial on Hyteck

Postmortem - how to completely screw up an update

2025-10-19T12:05:10+02:00

The fediverse instance gay-pirate-assassins.de was down for a couple of days. This postmortem will outline what went wrong and what I did to prevent things from going that wrong in the future.

Timeline

2025-10-05 17:26: Update announcement
2025-10-05 ~17:45: Update started
2025-10-05 ~18:00: Services restart
2025-10-05 ~18:00: GoToSocial doesn’t come up
2025-10-12 ~10:00: Issue is found
2025-10-12 10:30: Issue is fixed
2025-10-12 10:31: GoToSocial is started, migrations start
2025-10-12 15:38: Migrations finished successfully
2025-10-12 15:38: Service available again
2025-10-12 18:36:Announcement sent

All times are given in CEST.

The beginning: An update goes wrong

I run a small fediverse server with a few users called. gay-pirate-assassins which is powered by GoToSocial. The (amazing) GoToSocial devs released v0.20.0-rc1 and v0.20.0-rc2. As the new features seemed pretty cool, I’m inpatient and the second release candidate seemed stable, I decided to update to v0.20.0-rc2. So I stared a backup (via borgmatic), waited for it to finish and confirmed it ran successfully. Then I changed the version number in the mash-ansible playbook I use. Then I pulled the newest version of the playbook and it’s roles because I wanted to update all services that run on the server. I checked the Changelog, didn’t see anything and then started the update. It went through and GoToSocial started up just fine.

But the instance start page showed me 0 users, 0 posts and 0 federated instances. Something has gone horribly wrong!

Migrations

It was pretty clear to me, that the migrations went wrong. The GoToSocial Migration notes specifically mentioned long-running migrations that could take several hours. I assumed that somehow, during the running database migration, the service must have restarted and left the DB in a broken state. This issue happened to me before.

Well, that’s what backups are for, so let’s pull it.

Backups

Backups for this server are done two ways:

via postgres-backup: Backups of the database are written to disk
via borgmatic: Backups via borg are written to backup nodes, one of them at my home

They run every night automatically, monitored by Healthchecks. I triggered a manual run before the update so that is the one I mounted using Vorta.

And then the realization.


mash-postgres:5432 $ ls -lh
total 2.1M
-r-------- 1 moanos root 418K Oct 05 04:03 gitea
-r-------- 1 moanos root 123K Oct 05 04:03 healthchecks
-r-------- 1 moanos root 217K Oct 05 04:03 ilmo
-r-------- 1 moanos root 370K Oct 05 04:03 notfellchen
-r-------- 1 moanos root 67K Oct 05 04:03 oxitraffic
-r-------- 1 moanos root 931 Oct 05 04:03 prometheus_postgres_exporter
-r-------- 1 moanos root 142K Oct 05 04:03 semaphore
-r-------- 1 moanos root 110K Oct 05 04:03 vaultwarden
-r-------- 1 moanos root 669K Oct 05 04:03 woodpecker_ci_server

Fuck. The database gay-pirate-assassins is not there. Why?

To explain that I have to tell you how it should work: Services deployed by the mash-playbook are automatically wired to the database and reverse proxy by a complex set of Ansible variables. This is great, because adding a service can therefore be as easy as adding

healthchecks_enabled: true
healthchecks_hostname: health.hyteck.de

to the vars.yml file.

This will then configure the postgres database automatically, based on the group_vars. They look like this

mash_playbook_postgres_managed_databases_auto_itemized:
- |-
{{
({
'name': healthchecks_database_name,
'username': healthchecks_database_username,
'password': healthchecks_database_password,
} if healthchecks_enabled and healthchecks_database_hostname == postgres_connection_hostname and healthchecks_database_type == 'postgres' else omit)
}}

Note that a healthchecks database is only added to the managed databases if healthchecks_enabled is True.

This is really useful for backups because the borgmatic configuration also pulls the list mash_playbook_postgres_managed_databases_auto_itemized. Therefore, you do not need to specify which databases to back up, it just backs up all managed databases.

However, the database for gay-pirate assassins was not managed. In the playbook it’s only possible to configure a service once. You can not manage multiple GoToSocial instances in the same vars.yml. In the past, I had two instances of GoToSocial running on the server. I therefore followed the how-to of “Running multiple instances of the same service on the same host”.

Basically this means that an additional vars.yml must be created that is treated as a completely different server. Databases must be created manually as they are not managed.

With that knowledge you can understand that when I say that the database for gay-pirate-assassins was not managed, this means it was not included in the list of databases to be backed up. The backup service thought it ran successfully, because it backed up everything it knew of.

So this left me with a three-month-old backup. Unacceptable.

Investigating

So the existing database needed to be rescued. I SSHed into the server and checked the database. It looked completely normal. I asked the devs if they could me provide me with the migrations as they already did in the past. However, they pointed out that the migrations are too difficult for that approach. They suggested to delete the oldest migration to force a re-run of the migrations.

Here is where I was confused, because this was the bun_migrations table:

gay-pirate-assassins=# SELECT * FROM bun_migrations ORDER BY id DESC LIMIT 5;
id | name | group_id | migrated_at
-----+----------------+----------+-------------------------------
193 | 20250324173534 | 20 | 2025-04-23 20:00:33.955776+00
192 | 20250321131230 | 20 | 2025-04-23 19:58:06.873134+00
191 | 20250318093828 | 20 | 2025-04-23 19:57:50.540568+00
190 | 20250314120945 | 20 | 2025-04-23 19:57:30.677481+00

The last migration ran in April, when I updated to v0.19.1. Strange.

At this point I went on vacation and paused investigations, not only because the vacation was great, but also because I bamboozeld by this state.

After my vacation I came back, and did some backups of the database.

$ docker run -e PGPASSWORD="XXXX" -it --rm --network mash-postgres postgres pg_dump -U gay-pirate-assassins -h mash-postgres gay-pirate-assassins > manual-backup/gay-pirate-assassins-2025-10-13.sql

Then I deleted the last migration, as I was advised

DELETE FROM bun_migration WHERE id=193;

and restarted the server. While watching the server come up it hit me in the face:

Oct 12 08:31:29 s3 mash-gpa-gotosocial[2251925]: timestamp="12/10/2025 08:31:29.905" func=bundb.sqliteConn level=INFO msg="connected to SQLITE database with address file:/opt/gotosocial/sqlite.db?_pragma=busy_timeout%281800000%29&_pragma=journal_mode%>
Oct 12 13:38:46 s3 mash-gpa-gotosocial[2304549]: timestamp="12/10/2025 13:38:46.588" func=router.(*Router).Start.func1 level=INFO msg="listening on 0.0.0.0:8080"

The server is starting from a completely different database! That explains why

the last migration was never done
the server showed me 0 users, 0 posts and 0 federated instances even though the postgres database had plenty of those

All of a sudden a SQlite database was configured. This happened because of this commit which introduced SQlite support and set it as default. This was not mentioned in the Changelog.

So what happened is, that the config changed and then the server was restarted and an empty DB was initialized. The postgres DB never started to migrate.

Fixing

To fix it, I did the following

Configure the playbook to use postgres for GoToSocial:

# vars.yml
gotosocial_database_type: postgres

Run the playbook to configure GoToSocial (but not starting the service)

just run-tags install-gotosocial

Check the configuration is correct
Start the service

The migrations took several hours but after that, everything looked stable again. I don’t think there are any lasting consequences. However, the server was unavailable for several days.

Learnings

I believe the main issue here was not the change in the config that went unnoticed by me. While I’d ideally notice stuff like this, the server is a hobby, and I’ll continue to not check every config option that changed.

The larger issue was the backup. Having a backup would have made this easy to solve. And there are other, less lucky problems where I’d be completely lost without a backup. So to make sure this doesn’t happen again, I did/will do the following:

1. Mainstream the config

As explained, I used a specific non-mainstream setup in the ansible playbook because, in the past, I ran two instances of GoToSocial on the server. After shutting down one of them, I never moved gay-pirate-assassins to be part of the main config. This means important parts of the configuration had to be done manually, which I botched.

So in the past week I cleaned up and gay-pirate-assassins is now part of the main vars.yml and will benefit from all relevant automations.

2. Checking backups

I was confident in my backups because

they run every night very consistently. If they fail e.g. because of a network outage I reliably get a warning.
I verified successfully run of the backup job prior to upgrading

The main problem was me assuming that a successful run of the backup command, meant a successful backup. Everyone will tell you that a backup that is not tested is not to be trusted. And they are right. However, doing frequent test-restores exceeds my time and server capacity. So what I’ll do instead is the following:

mount the backup before an upgrade
tail the backup file as created by postgres-backup and ensure the data is from the same day
check media folders for the last changed image

This is not a 100% guarantee, but I’d argue it’s a pretty good compromise for now. As the frequency of mounting backups increases and therefore becomes faster, I’ll re-evaluate to do a test-restore at least semi-regulary.

Conclusion

I fucked up, but I was lucky that my error was recoverable and no data was lost. Next time this will hopefully be not due to luck, but better planning!

Any questions? Let me know!

I did something naughty: Circumventing Authorized-Fetch as implemented by GoToSocial

2024-12-11T06:10:10+02:00

Yes the title is correct, but I had nothing malicious in mind!

What this is about

For @qzt@queereszentrumtuebingen.de we include the public feed in a sidbar on the homepage. Initially this was done using the standard API to fetch statuses /api/v1/accounts/{account_id}/statuses and worked like a charm. The problem started when GoToSocial (the fediverse server we use, similar to mastodon) implemented authorized fetch. This is a a good thing! Authorized fetch means, that every call to a endpoint needs to be authorized by an access_token. You get an access token from a fedi account. It’s what fediverse clients like Tusky or Phanpy do on your behalf to get the posts that make up you timeline.

Authorized fetch has major advantages as

data scraping can only be done by other fediaccounts
blocking can not be circumvented by using the public API

and much more. Sadly it also broke our website integration.

Possible Solutions

So what now? I initially wanted to turn of authorized fetch for @qzt@queereszentrumtuebingen.de by messing with the GoToSocial code and turning it off for the whole server. This would have been possible as this is the only user on the server. The GoToSocial devs helped me manage to find where to do that. But it’s not ideal and would make me build a custom docker image fore each update.

Next idea: The whole point of authorized fetch is, that only fedi-accounts (and apps they authorized) can access the API. So lets do that! Set up a new account, add app and authorize it as described in the GoToSocial documentation. I used #Bruno for that, that was much more comfortable than using curl for me. With that authorization code you can now get an access token for your app. Put that in the Javascript that loads posts and we are good right? Sadly no. It would totally work. But it would also allow anyone to read and post on behalf of the account. That calls for malicious actors using this for scraping or spamming.

So instead, we need a proxy that stores the access token securely and restricts the actions.

The proxy

Such a proxie must

offer the endpoint that provides the same data as the FediverseAPI
authorize itself to the FediverseAPI via access_token
restrict to read access of consenting accounts

The last point is really important, as we don’t want to allow others to use this endpoint to scrape data unauthorized.

I wrote a short FastAPI server that offers this. It only implements one method

@app.get("/api/v1/accounts/{account_id}/statuses")
async def fetch_data(account_id):
if account_id not in ALLOWED_ACCOUNTS:
raise HTTPException(status_code=401, detail="You can only use this proxy to access configured accounts")
headers = {"Authorization": f"Bearer {ACCESS_TOKEN}"}
response = requests.get(f"{EXTERNAL_API_BASE_URL}/api/v1/accounts/{account_id}/statuses", headers=headers)
return response.json()

Basically this is the whole API code, I only trimmed a few checks and error handling.

Deployment

To deploy, I put it in a docker container and started it via docker-compose. Reverse proxing is handled by Traefik, I won’t go into detail here.

services:
fediproxy.example.org:
image: docker.io/moanos/fediproxy
container_name: "fediproxy.example.org"
restart: unless-stopped
environment:
EXTERNAL_API_BASE_URL: ${EXTERNAL_API_BASE_URL}
ACCESS_TOKEN: ${ACCESS_TOKEN}
ALLOWED_ACCOUNTS: ${ALLOWED_ACCOUNTS}
labels:
- "traefik.enable=true"
- "traefik.docker.network=traefik"
- "traefik.http.routers.fediproxy.rule=Host(`fediproxy.example.org`)"
- "traefik.http.routers.fediproxy.service=fediproxy-service"
- "traefik.http.routers.fediproxy.entrypoints=web-secure"
- "traefik.http.routers.fediproxy.tls=true"
- "traefik.http.routers.fediproxy.tls.certResolver=default"
- "traefik.http.services.fediproxy-service.loadbalancer.server.port=8000"
networks:
- traefik
networks:
traefik:
name: "traefik"
external: true

I added a short .env to configure:

ACCESS_TOKEN=VERYSECRETTOKENTHATISDEFINETLYREAL
EXTERNAL_API_BASE_URL=https://gay-pirate-assassins.de
ALLOWED_ACCOUNTS=ZGGZF4G8NNOTREAL81Z8G7RTC

Results

Now I can again use something like the wordpress plugin Include Mastodon Feed just by pointing to the proxy: [include-mastodon-feed instance="fediproxy.example.org.de" account="ZGGZF4G8NNOTREAL81Z8G7RTC"]

Hope you enjoyed the read. Source code for the proxy can be found here: https://git.hyteck.de/moanos/FediProxy If you want to play around a bit you can use https://git.hyteck.de/moanos/include-fedi

Sloth logo of GTS by Anna Abramek, Creative Commons BY-SA license.