Files
authentik/lifecycle/migrate.py
Marc 'risson' Schmitt abc0c2d2a2 root: Multi-tenancy (#7590)
* tenants -> brands, init new tenant model, migrate some config to tenants

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* setup logging for tenants

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* configure celery and cache

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* small fixes, runs

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* task fixes, creation of tenant now works by cloning a template schema, some other small stuff

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* lint

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* fix-tests

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* upstream fixes

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* fix-pylint

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* lint

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* fix tests

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* fix avatar tests

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* migrate config reputation_expiry as well

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* fix web rebase

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* lint

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* fix migrations for template schema

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* fix migrations for template schema

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* fix migrations for template schema 3

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* revert reputation expiry migration

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* fix type

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* fix some more tests

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* website: tenants -> brands

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* try fixing e2e tests

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* start frontend :help:

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* add ability to disable tenants api

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* delete embedded outpost if it is disabled

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* make sure embedded outpost is disabled when tenants are enabled

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* management commands: add --schema option where relevant

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* store files per-tenant

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* fix embedded outpost deletion

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* lint

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* fix files migration

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* add tenant api tests

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* add domain tests

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* add settings tests

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* make --schema-name default to public in mgmt commands

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* lint

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* sources/ldap: make sure lock is per-tenant

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* fix stuff I broke

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* fix remaining failing tests

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* lint

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* try fixing e2e tests

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* much better frontend, but save does not refresh form properly

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* update django-tenants with latest fixes

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* lint

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* i18n-extract

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* review comments

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* move event_retention from brands to tenants

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* wip

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* root: add support for storing media files in S3

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* use permissions for settings api

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* lint

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* blueprints: disable tenants management

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* fix tests

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* fix embedded outpost create/delete logic

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* make gen

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* make sure prometheus metrics are correctly served

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* makefile: don't delete the go api client when not regenerating it

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* tenants api: add recovery group and token creation endpoints

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* fix startup

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* fix prometheus metrics

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* fix tests

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* lint

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* fix web stuff

Signed-off-by: Jens Langhammer <jens@goauthentik.io>

* fix migrations from stable

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* fix oauth source type import

Signed-off-by: Jens Langhammer <jens@goauthentik.io>

* Revert "fix oauth source type import"

This reverts commit d015fd0244.

* try with setting_changed signal

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* try with connection_created signal

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* fix scim tests

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* fix web after merge

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* fix enterprise settings

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* Revert "try with connection_created signal"

This reverts commit 764a999db8.

* Revert "try with setting_changed signal"

This reverts commit 32b40a3bbb.

* lib/expression: refactor expression compilation

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* fix django version

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* fix web after merge

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* relock poetry

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* fix reconcile

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* try running tenant save in a transaction

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* black

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* test: export postgres logs for debugging and use failfast

Signed-off-by: Jens Langhammer <jens@goauthentik.io>

* test: fix container name for logs

Signed-off-by: Jens Langhammer <jens@goauthentik.io>

* do not copy tenant data

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* Revert "try running tenant save in a transaction"

This reverts commit da6dec5a61.

* Revert "do not copy tenant data"

This reverts commit d07ae9423672f068b0bd8be409ff9b58452a80f2.

* Revert "Revert "do not copy tenant data""

This reverts commit 4bffb19704.

* fix clone with nodata

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* why not

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* remove failfast

Signed-off-by: Jens Langhammer <jens@goauthentik.io>

* remove postgres query logging

Signed-off-by: Jens Langhammer <jens@goauthentik.io>

* update reconcile logic to clearly differentiate between tenant and global

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* fix

Signed-off-by: Jens Langhammer <jens@goauthentik.io>

* fix reconcile app decorator

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* enable django checks

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* actually nodata was unnecessary as we're cloning from template and not from public

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* pylint

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* update django-tenants with sequence fix

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* actually update

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* fix e2e tests

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* add tests for settings api

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* add tests for recovery api

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* lint

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* recovery tests: do them on a new tenant

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* web: fix system status being degraded when embedded outpost is disabled

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* fix recovery tests

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* fix tenants tests

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* lint-fix

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* lint-fix

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* update UI

Signed-off-by: Jens Langhammer <jens@goauthentik.io>

* add management command to create a tenant

Signed-off-by: Jens Langhammer <jens@goauthentik.io>

* add docs

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* release notes

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* more docs

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* checklist

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* self review

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* spelling

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* make web after upgrading

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* remove extra xlif file

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* prettier

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* Revert "add management command to create a tenant"

This reverts commit 39d13c0447.

* split api into smaller files, only import urls when tenants is enabled

Signed-off-by: Jens Langhammer <jens@goauthentik.io>

* rewite some things on the release notes

Signed-off-by: Jens Langhammer <jens@goauthentik.io>

* root: make sure install_id comes from public schema

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* require a license to use tenants

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* lint

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* fix tenants tests

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* fix files migration

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* release notes: add warning about user sessions being invalidated

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

* remove api disabled test, we can't test for it

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>

---------

Signed-off-by: Marc 'risson' Schmitt <marc.schmitt@risson.space>
Signed-off-by: Jens Langhammer <jens@goauthentik.io>
Co-authored-by: Jens Langhammer <jens@goauthentik.io>
2024-01-23 14:28:06 +01:00

116 lines
3.9 KiB
Python
Executable File

#!/usr/bin/env python
"""System Migration handler"""
from importlib.util import module_from_spec, spec_from_file_location
from inspect import getmembers, isclass
from os import environ, system
from pathlib import Path
from typing import Any
from psycopg import Connection, Cursor, connect
from structlog.stdlib import get_logger
from authentik.lib.config import CONFIG
LOGGER = get_logger()
ADV_LOCK_UID = 1000
LOCKED = False
class CommandError(Exception):
"""Error raised when a system_crit command fails"""
class BaseMigration:
"""Base System Migration"""
cur: Cursor
con: Connection
def __init__(self, cur: Any, con: Any):
self.cur = cur
self.con = con
def system_crit(self, command: str):
"""Run system command"""
LOGGER.debug("Running system_crit command", command=command)
retval = system(command) # nosec
if retval != 0:
raise CommandError("Migration error")
def fake_migration(self, *app_migration: tuple[str, str]):
"""Fake apply a list of migrations, arguments are
expected to be tuples of (app_label, migration_name)"""
for app, _migration in app_migration:
self.system_crit(f"./manage.py migrate {app} {_migration} --fake")
def needs_migration(self) -> bool:
"""Return true if Migration needs to be run"""
return False
def run(self):
"""Run the actual migration"""
def wait_for_lock(cursor: Cursor):
"""lock an advisory lock to prevent multiple instances from migrating at once"""
LOGGER.info("waiting to acquire database lock")
cursor.execute("SELECT pg_advisory_lock(%s)", (ADV_LOCK_UID,))
# pylint: disable=global-statement
global LOCKED
LOCKED = True
def release_lock(cursor: Cursor):
"""Release database lock"""
if not LOCKED:
return
cursor.execute("SELECT pg_advisory_unlock(%s)", (ADV_LOCK_UID,))
if __name__ == "__main__":
conn = connect(
dbname=CONFIG.get("postgresql.name"),
user=CONFIG.get("postgresql.user"),
password=CONFIG.get("postgresql.password"),
host=CONFIG.get("postgresql.host"),
port=CONFIG.get_int("postgresql.port"),
sslmode=CONFIG.get("postgresql.sslmode"),
sslrootcert=CONFIG.get("postgresql.sslrootcert"),
sslcert=CONFIG.get("postgresql.sslcert"),
sslkey=CONFIG.get("postgresql.sslkey"),
)
curr = conn.cursor()
try:
for migration_path in Path(__file__).parent.absolute().glob("system_migrations/*.py"):
spec = spec_from_file_location("lifecycle.system_migrations", migration_path)
if not spec:
continue
mod = module_from_spec(spec)
spec.loader.exec_module(mod)
for name, sub in getmembers(mod, isclass):
if name != "Migration":
continue
migration = sub(curr, conn)
if migration.needs_migration():
wait_for_lock(curr)
LOGGER.info("Migration needs to be applied", migration=migration_path.name)
migration.run()
LOGGER.info("Migration finished applying", migration=migration_path.name)
release_lock(curr)
LOGGER.info("applying django migrations")
environ.setdefault("DJANGO_SETTINGS_MODULE", "authentik.root.settings")
wait_for_lock(curr)
try:
from django.core.management import execute_from_command_line
except ImportError as exc:
raise ImportError(
"Couldn't import Django. Are you sure it's installed and "
"available on your PYTHONPATH environment variable? Did you "
"forget to activate a virtual environment?"
) from exc
execute_from_command_line(["", "migrate_schemas"])
execute_from_command_line(["", "migrate_schemas", "--schema", "template", "--tenant"])
finally:
release_lock(curr)