docs/troubleshooting: cleanup upgrade instructions for postgres k8s (#14773)
* docs/troubleshooting: cleanup upgrade instructions for postgres k8s * website/troubleshooting: upgrade pg on k8s: use lowercase for headers * Update website/docs/troubleshooting/postgres/upgrade_kubernetes.md Signed-off-by: Tana M Berry <tanamarieberry@yahoo.com> * Update website/docs/troubleshooting/postgres/upgrade_kubernetes.md Signed-off-by: Tana M Berry <tanamarieberry@yahoo.com> * Update website/docs/troubleshooting/postgres/upgrade_kubernetes.md Signed-off-by: Tana M Berry <tanamarieberry@yahoo.com> * Update website/docs/troubleshooting/postgres/upgrade_kubernetes.md Signed-off-by: Tana M Berry <tanamarieberry@yahoo.com> * Update website/docs/troubleshooting/postgres/upgrade_kubernetes.md Signed-off-by: Tana M Berry <tanamarieberry@yahoo.com> * Update website/docs/troubleshooting/postgres/upgrade_kubernetes.md Signed-off-by: Tana M Berry <tanamarieberry@yahoo.com> * Update website/docs/troubleshooting/postgres/upgrade_kubernetes.md Signed-off-by: Tana M Berry <tanamarieberry@yahoo.com> * bump build * tweak --------- Signed-off-by: Tana M Berry <tanamarieberry@yahoo.com> Co-authored-by: Tana M Berry <tanamarieberry@yahoo.com> Co-authored-by: Tana M Berry <tana@goauthentik.io>
This commit is contained in:
Dominic R
@ -1,98 +1,161 @@
|
|||||||
---
|
---
|
||||||
title: Upgrade PostgreSQL on Kubernetes
|
title: Upgrading PostgreSQL on Kubernetes
|
||||||
---
|
---
|
||||||
|
|
||||||
## Preparation
|
This guide walks you through upgrading PostgreSQL in your authentik Kubernetes deployment. The process requires a brief downtime period while the database is migrated.
|
||||||
|
|
||||||
- `authentik-postgresql-0` is the Kubernetes Pod running PostgreSQL.
|
:::note
|
||||||
|
For this guide, we assume the PostgreSQL pod is named `authentik-postgresql-0`, which is the default name in the authentik Helm chart.
|
||||||
|
:::
|
||||||
|
|
||||||
### Prerequisites
|
## Prerequisites
|
||||||
|
|
||||||
This migration requires some downtime, during which authentik must be stopped. To do this, run the following command:
|
- `kubectl` access with permissions to `scale` deployments and `exec` into pods
|
||||||
|
- Your existing `values.yaml` file used for authentik deployment
|
||||||
|
- Basic understanding of Kubernetes and Helm commands
|
||||||
|
|
||||||
|
## Overview of workflow
|
||||||
|
|
||||||
|
The basic steps to upgrades PostgreSQL on Kubernetes are:
|
||||||
|
|
||||||
|
1. Stop authentik services
|
||||||
|
2. Back up the database
|
||||||
|
3. Prepare the data directory
|
||||||
|
4. Upgrade PostgreSQL
|
||||||
|
5. Restore database content
|
||||||
|
6. Restart authentik services
|
||||||
|
|
||||||
|
## Stop authentik services
|
||||||
|
|
||||||
|
Begin by scaling down authentik services to prevent database access during the migration:
|
||||||
|
|
||||||
```shell
|
```shell
|
||||||
kubectl scale deploy --replicas 0 authentik-server
|
kubectl scale deploy --replicas 0 authentik-server
|
||||||
kubectl scale deploy --replicas 0 authentik-worker
|
kubectl scale deploy --replicas 0 authentik-worker
|
||||||
```
|
```
|
||||||
|
|
||||||
### Dump the current database
|
## Back up the database
|
||||||
|
|
||||||
Run `kubectl exec -it authentik-postgresql-0 -- bash` to get a shell in the PostgreSQL pod.
|
Connect to your PostgreSQL pod:
|
||||||
|
|
||||||
Run the following commands to dump the current data into a `.sql` file:
|
|
||||||
|
|
||||||
```shell
|
```shell
|
||||||
# This is the path where the PVC is mounted, so we'll place the dump here too
|
kubectl exec -it authentik-postgresql-0 -- bash
|
||||||
|
```
|
||||||
|
|
||||||
|
After you are connected, execute these commands to create a database backup:
|
||||||
|
|
||||||
|
```shell
|
||||||
|
# Navigate to the PostgreSQL data directory
|
||||||
cd /bitnami/postgresql/
|
cd /bitnami/postgresql/
|
||||||
# Set the postgres password based on the `POSTGRES_POSTGRES_PASSWORD` environment variable
|
|
||||||
|
# Set the PostgreSQL password from environment variable
|
||||||
export PGPASSWORD=$POSTGRES_POSTGRES_PASSWORD
|
export PGPASSWORD=$POSTGRES_POSTGRES_PASSWORD
|
||||||
# Dump the authentik database into an sql file
|
|
||||||
pg_dump -U $POSTGRES_USER $POSTGRES_DB > dump-11.sql
|
# Create a full database dump
|
||||||
|
pg_dump -U $POSTGRES_USER $POSTGRES_DB > /bitnami/postgresql/dump.sql
|
||||||
```
|
```
|
||||||
|
|
||||||
### Stop PostgreSQL and start the upgrade
|
:::tip
|
||||||
|
Consider copying the dump file to a safe location outside the pod:
|
||||||
To upgrade, change the following entries in your `values.yaml` used to deploy authentik:
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
postgresql:
|
|
||||||
diagnosticMode:
|
|
||||||
enabled: true
|
|
||||||
image:
|
|
||||||
tag: 15.2.0-debian-11-r26
|
|
||||||
```
|
|
||||||
|
|
||||||
Now run `helm upgrade --install authentik authentik/authentik -f values.yaml` to apply these changes. Depending on your configuration, you might have to repeat the steps from [Prerequisites](#prerequisites).
|
|
||||||
|
|
||||||
After the upgrade is finished, you should have a new PostgreSQL pod running with the updated image.
|
|
||||||
|
|
||||||
### Remove the old data
|
|
||||||
|
|
||||||
Because the PVC mounted by the PostgreSQL pod still contains the old data, we need to remove/rename that data, so that PostgreSQL can initialize it with the new version.
|
|
||||||
|
|
||||||
Run `kubectl exec -it authentik-postgresql-0 -- bash` to get a shell in the PostgreSQL pod.
|
|
||||||
|
|
||||||
Run the following commands to move the old data:
|
|
||||||
|
|
||||||
```shell
|
```shell
|
||||||
# This is the path where the PVC is mounted
|
# From a separate terminal
|
||||||
cd /bitnami/postgresql/
|
kubectl cp authentik-postgresql-0:/bitnami/postgresql/dump.sql ./authentik-db-backup.sql
|
||||||
# Move Postgres' data folder to data-11, which is the version we're upgrading to.
|
|
||||||
# The data folder can also be deleted; however it is recommended to rename it first
|
|
||||||
# in case the upgrade fails.
|
|
||||||
mv data data-11
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### Restart PostgreSQL
|
This ensures you have a backup even if something goes wrong with the pod or storage.
|
||||||
|
:::
|
||||||
|
|
||||||
In the step [Stop PostgreSQL and start the upgrade](#stop-postgresql-and-start-the-upgrade), we enabled the _diagnostic mode_, which means the PostgreSQL pod is running, but the actual Postgres process isn't running. Now that we've removed the old data directory, we can disable the diagnostic mode.
|
## Prepare the data directory
|
||||||
|
|
||||||
Once again, change the following entries in your `values.yaml` used to deploy authentik:
|
While still connected to the PostgreSQL pod, prepare the data directory for the upgrade:
|
||||||
|
|
||||||
|
```shell
|
||||||
|
# Ensure you're in the PostgreSQL data directory
|
||||||
|
cd /bitnami/postgresql/
|
||||||
|
|
||||||
|
# Verify the SQL dump exists and has content
|
||||||
|
ls -lh dump.sql
|
||||||
|
|
||||||
|
# Preserve the existing data by renaming the directory
|
||||||
|
mv data data-old
|
||||||
|
```
|
||||||
|
|
||||||
|
:::caution
|
||||||
|
Do not delete the old data directory immediately. Keeping it as `data-old` allows for recovery if the upgrade encounters issues.
|
||||||
|
:::
|
||||||
|
|
||||||
|
## Upgrade PostgreSQL
|
||||||
|
|
||||||
|
Now update your `values.yaml` to specify the new PostgreSQL version:
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
postgresql:
|
postgresql:
|
||||||
image:
|
image:
|
||||||
tag: 15.2.0-debian-11-r26
|
tag: <NEW_VERSION>
|
||||||
```
|
```
|
||||||
|
|
||||||
And once again run `helm upgrade --install authentik authentik/authentik -f values.yaml` to apply these changes. Depending on your configuration, you might have to repeat the steps from [Prerequisites](#prerequisites).
|
Apply these changes using Helm to deploy the updated configuration.
|
||||||
|
|
||||||
After the PostgreSQL pod is running again, we need to restore the data from the dump we created above.
|
This will restart the PostgreSQL pod with the new image. When the pod starts, PostgreSQL will initialize a new, empty data directory since the previous directory was renamed.
|
||||||
|
|
||||||
Run `kubectl exec -it authentik-postgresql-0 -- bash` to get a shell in the PostgreSQL pod.
|
## Restore database content
|
||||||
|
|
||||||
Run the following commands to restore the data:
|
Connect to the PostgreSQL pod again:
|
||||||
|
|
||||||
```shell
|
```shell
|
||||||
# This is the path where the PVC is mounted
|
kubectl exec -it authentik-postgresql-0 -- bash
|
||||||
cd /bitnami/postgresql/
|
|
||||||
# Set the Postgres password based on the `POSTGRES_POSTGRES_PASSWORD` environment variable.
|
|
||||||
export PGPASSWORD=$POSTGRES_POSTGRES_PASSWORD
|
|
||||||
psql -U $POSTGRES_USER $POSTGRES_DB < dump-11.sql
|
|
||||||
```
|
```
|
||||||
|
|
||||||
After the last command finishes, all of the data is restored, and you can restart authentik.
|
Restore your database from the backup:
|
||||||
|
|
||||||
### Restarting authentik
|
```shell
|
||||||
|
# Navigate to the PostgreSQL directory
|
||||||
|
cd /bitnami/postgresql/
|
||||||
|
|
||||||
Run `helm upgrade --install authentik authentik/authentik -f values.yaml` once again, which will restart your authentik server and worker containers.
|
# Verify your dump file is still there
|
||||||
|
ls -lh dump.sql
|
||||||
|
|
||||||
|
# Set the PostgreSQL password
|
||||||
|
export PGPASSWORD=$POSTGRES_POSTGRES_PASSWORD
|
||||||
|
|
||||||
|
# Import the database dump
|
||||||
|
psql -U $POSTGRES_USER $POSTGRES_DB < dump.sql
|
||||||
|
```
|
||||||
|
|
||||||
|
## Restart authentik services
|
||||||
|
|
||||||
|
After the database restoration completes successfully, restart authentik using Helm with your updated configuration.
|
||||||
|
|
||||||
|
This will scale your authentik server and worker deployments back to their original replica counts.
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
If you encounter issues during the upgrade process:
|
||||||
|
|
||||||
|
- Check PostgreSQL logs:
|
||||||
|
```shell
|
||||||
|
kubectl logs authentik-postgresql-0
|
||||||
|
```
|
||||||
|
- Verify the values in your `values.yaml` file match the recommended settings
|
||||||
|
- Ensure you have sufficient storage available for both the database dump and the database itself
|
||||||
|
|
||||||
|
### Dump file not found
|
||||||
|
|
||||||
|
If your dump file is missing after upgrading:
|
||||||
|
|
||||||
|
- You may need to restore from the external backup if you copied it out of the pod
|
||||||
|
- The volume might have been recreated if you're using ephemeral storage
|
||||||
|
|
||||||
|
### Restoring the original database
|
||||||
|
|
||||||
|
For persistent problems, you can restore from the `data-old` directory if needed:
|
||||||
|
|
||||||
|
```shell
|
||||||
|
kubectl exec -it authentik-postgresql-0 -- bash
|
||||||
|
cd /bitnami/postgresql/
|
||||||
|
mv data data-new-failed
|
||||||
|
mv data-old data
|
||||||
|
```
|
||||||
|
|
||||||
|
Then restart PostgreSQL with the original version in your `values.yaml`.
|
||||||
|
|||||||
Reference in New Issue
Block a user