Breaking
Single Container Image
The different versions have been combined into a single container image. The image with the -lite extension does not exist anymore and all deployments can be done with just the base image. Since Postgres was the default before, you need to change your image name when you do not use Postgres as your database, just remove the -lite.
Dropped sqlx SQLite in favor of Hiqlite
From this version on, Rauthy will not support a default SQLite anymore. Instead, it will use Hiqlite, which under the hood uses SQLite again and is another project of mine.
Hiqlite will bring lots of advantages. It will use a few more resources than a direct, plain SQLite, but only ~10-15 MB of memory for small instances. In return, you will get higher consistency and never blocking writes to the database during high traffic. It also reduces the latency for all read statements by a huge margin compared to the solution before. Rauthy always enables the dashboard feature for Hiqlite, which will be available over the Hiqlite API port / server.
The biggest feature it brings though is the ability to run a HA cluster without any external dependencies. You can use Hiqlite on a single instance and it would "feel" the same as just a SQLite, but you can also spin up 3 or 5 nodes to get High Availability without the need for an external database. It uses the Raft algorithm to sync data while still using just a simple SQLite under the hood. The internal design of Hiqlite has been optimized a lot to provide way higher throughput as you would normally get when you just use a direct connection to a SQLite file. If you are interested more about the internals, take a look at the hiqlite/README.md or hiqlite/ARCHITECTURE.md.
With these features, Hiqlite will always be the preferred database solution for Rauthy. You should really not spin up a dedicated Postgres instance just for Rauthy, because it would just use too many resources, which is not necessary. If you have a Postgres up and running anyway, you can still opt-in to use it.
This was a very big migration and tens of thousands of lines of code has been changed. All tests are passing and a lot of additional checks have been included. I could not find any leftover issues or errors, but please let me know if you find something.
If you are using Rauthy with Postgres as database, you don't need to do that much. If however you use SQLite, no worries, Rauthy can handle the migration for you after adopting a few config variables. Even if you do the auto-migration from an existing SQLite to Hiqlite, Rauthy will keep the original SQLite file in place for additional safety, so you don't need to worry about a backup (as long as you set the config correctly of course). The next bigger release will maybe do cleanup work when everything worked fine for sure, or you can do it manually.
New / Changed Config Variables
There are quite a few new config variables and some old ones are gone. What you need to set to migration will be explained below.
#####################################
############## BACKUPS ###############
#####################################
# When the auto-backup task should run.
# Accepts cron syntax:
# "sec min hour day_of_month month day_of_week year"
# default: "0 30 2 * * * *"
HQL_BACKUP_CRON="0 30 2 * * * *"
# Local backups older than the configured days will be cleaned up after
# the backup cron job.
# default: 30
#HQL_BACKUP_KEEP_DAYS=30
# Backups older than the configured days will be cleaned up locally
# after each `Client::backup()` and the cron job `HQL_BACKUP_CRON`.
# default: 3
#HQL_BACKUP_KEEP_DAYS_LOCAL=3
# If you ever need to restore from a backup, the process is simple.
# 1. Have the cluster shut down. This is probably the case anyway, if
# you need to restore from a backup.
# 2. Provide the backup file name on S3 storage with the
# HQL_BACKUP_RESTORE value.
# 3. Start up the cluster again.
# 4. After the restart, make sure to remove the HQL_BACKUP_RESTORE
# env value.
#HQL_BACKUP_RESTORE=
# Access values for the S3 bucket where backups will be pushed to.
#HQL_S3_URL=https://s3.example.com
#HQL_S3_BUCKET=my_bucket
#HQL_S3_REGION=example
#HQL_S3_PATH_STYLE=true
#HQL_S3_KEY=s3_key
#HQL_S3_SECRET=s3_secret
#####################################
############# CLUSTER ###############
#####################################
# Can be set to 'k8s' to try to split off the node id from the hostname
# when Hiqlite is running as a StatefulSet inside Kubernetes.
#HQL_NODE_ID_FROM=k8s
# The node id must exist in the nodes and there must always be
# at least a node with ID 1
# Will be ignored if `HQL_NODE_ID_FROM=k8s`
HQL_NODE_ID=1
# All cluster member nodes.
# To make setting the env var easy, the values are separated by `\s`
# while nodes are separated by `\n`
# in the following format:
#
# id addr_raft addr_api
# id addr_raft addr_api
# id addr_raft addr_api
#
HQL_NODES="
1 localhost:8100 localhost:8200
"
# Sets the limit when the Raft will trigger the creation of a new
# state machine snapshot and purge all logs that are included in
# the snapshot.
# Higher values can achieve more throughput in very write heavy
# situations but will end up in more disk usage and longer
# snapshot creations / log purges.
# default: 10000
#HQL_LOGS_UNTIL_SNAPSHOT=10000
# Secrets for Raft internal authentication as well as for the API.
# These must be at least 16 characters long and you should provide
# different ones for both variables.
HQL_SECRET_RAFT=SuperSecureSecret1337
HQL_SECRET_API=SuperSecureSecret1337
# You can either parse `ENC_KEYS` and `ENC_KEY_ACTIVE` from the
# environment with setting this value to `env`, or parse them from
# a file on disk with `file:path/to/enc/keys/file`
# default: env
#HQL_ENC_KEYS_FROM=env
#####################################
############ DATABASE ###############
#####################################
# Max DB connections for the Postgres pool.
# Irrelevant for Hiqlite.
# default: 20
#DATABASE_MAX_CONN=20
# If specified, the currently configured Database will be DELETED and
# OVERWRITTEN with a migration from the given database with this variable.
# Can be used to migrate between different databases.
#
# !!! USE WITH CARE !!!
#
#MIGRATE_DB_FROM=sqlite:data/rauthy.db
#MIGRATE_DB_FROM=postgresql://postgres:123SuperSafe@localhost:5432/rauthy
# Hiqlite is the default database for Rauthy.
# You can opt-out and use Postgres instead by setting the proper
# `DATABASE_URL=postgresql://...` by setting `HIQLITE=false`
# default: true
#HIQLITE=true
# The data dir hiqlite will store raft logs and state machine data in.
# default: data
#HQL_DATA_DIR=data
# The file name of the SQLite database in the state machine folder.
# default: hiqlite.db
#HQL_FILENAME_DB=hiqlite.db
# If set to `true`, all SQL statements will be logged for debugging
# purposes.
# default: false
#HQL_LOG_STATEMENTS=false
# The size of the pooled connections for local database reads.
#
# Do not confuse this with a pool size for network databases, as it
# is much more efficient. You can't really translate between them,
# because it depends on many things, but assuming a factor of 10 is
# a good start. This means, if you needed a (read) pool size of 40
# connections for something like a postgres before, you should start
# at a `read_pool_size` of 4.
#
# Keep in mind that this pool is only used for reads and writes will
# travel through the Raft and have their own dedicated connection.
#
# default: 4
#HQL_READ_POOL_SIZE=4
# Enables immediate flush + sync to disk after each Log Store Batch.
# The situations where you would need this are very rare, and you
# should use it with care.
#
# The default is `false`, and a flush + sync will be done in 200ms
# intervals. Even if the application should crash, the OS will take
# care of flushing left-over buffers to disk and no data will get
# lost. If something worse happens, you might lose the last 200ms
# of commits (on that node, not the whole cluster). This is only
# important to know for single instance deployments. HA nodes will
# sync data from other cluster members after a restart anyway.
#
# The only situation where you might want to enable this option is
# when you are on a host that might lose power out of nowhere, and
# it has no backup battery, or when your OS / disk itself is unstable.
#
# `sync_immediate` will greatly reduce the write throughput and put
# a lot more pressure on the disk. If you have lots of writes, it
# can pretty quickly kill your SSD for instance.
#HQL_SYNC_IMMEDIATE=false
# The password for the Hiqlite dashboard as Argon2ID hash.
# '123SuperMegaSafe' in this example
#
# You only need to provide this value if you need to access the
# Hiqlite debugging dashboard for whatever reason. If no password
# hash is given, the dashboard will not be reachable.
#HQL_PASSWORD_DASHBOARD=JGFyZ29uMmlkJHY9MTkkbT0xOTQ1Nix0PTIscD0xJGQ2RlJDYTBtaS9OUnkvL1RubmZNa0EkVzJMeTQrc1dxZ0FGd0RyQjBZKy9iWjBQUlZlOTdUMURwQkk5QUoxeW1wRQ==
Migration (Postgres)
If you use Rauthy with Postgres and want to keep doing that, the only thing you need to do is to opt-out of Hiqlite.
HIQLITE=false
Migration (SQLite)
If you use Rauthy with SQLite and want to migrate to Hiqlite, you can utilize all the above-mentioned new config variables, but mandatory are the following ones.
Backups
Backups for the internal database work in the same way as before, but because I moved the backup functionality directly into Hiqlite, the variable names have been changed so they make sense if it is used in another context.
BACKUP_TASK->HQL_BACKUP_CRONBACKUP_NAMEdoes not exist anymore, will be chosen automatically depending on the cluster leader nameBACKUP_RETENTION_LOCAL->HQL_BACKUP_KEEP_DAYS_LOCALRESTORE_BACKUP->HQL_BACKUP_RESTORES3_URL->HQL_S3_URLS3_REGION->HQL_S3_REGIONS3_PATH_STYLE->HQL_S3_PATH_STYLES3_BUCKET->HQL_S3_BUCKETS3_ACCESS_KEY->HQL_S3_KEYS3_ACCESS_SECRET->HQL_S3_SECRET
S3_DANGER_ALLOW_INSECURE stayed as it is.
BACKUP_RETENTION_LOCAL is new, and it will actually handle the backup cleanup on the S3 storage for you, without defining retention rules for the whole bucket.
Hiqlite Dashboard
Rauthy comes with the dashboard feature from Hiqlite enabled. If you want to make use of it, you need to set the password for logging in. This is a static config variable, and it will only be a single password, no users / accounts. The main idea behind the dashboard is to have debugging capabilities in production, which is usually hard to do with a SQLite running inside a container.
You need to generate a random password (at least 16 characters), hash it with Argon2ID, and then base64 encode it. You can do all this manually, or use the hiqlite cli to generate a complete hiqlite config where you copy it from.
Manual:
Use an online tool like for instance https://argon2.online to generate an Argon2ID hash.
Set the following options:
- Salt: Random
- Parallelism Factor: 2
- Memory Cost: 32
- Iterations: 2
- Hash Length: 32
- Argon2id
Then copy the Output in Encoded Form and base64 encode it, for instance using https://www.base64encode.org.
hiqlite cli:
Currently, you can only install the hiqlite cli via cargo:
cargo install hiqlite --features serverhiqlite generate-config -p YouRandomSecurePasswordAtLeast16Chars- grab the password from the config, location written in the output
Migration
Unless you specified a custom target path on disk for SQLite(HQL_DATA_DIR)) before, you should be good with the configuration now. If you start up Rauthy now, it will be like a fresh install, which you most probably don't want. To migrate your current SQLite to Hiqlite at startup, you need to set the MIGRATE_DB_FROM once at startup. If you used the default path before, you need to set:
MIGRATE_DB_FROM=sqlite:data/rauthy.db
For a custom path, just adopt the value accordingly. This works as well by the way, if you want to migrate from Postgres to Hiqlite.
!!! CAUTION !!!
You must remove this variable after Rauthy has been started successfully! Otherwise, it would do the migration again and again with each following restart and therefore remove everything that has happened in between!
User Registration - Redirect Hint
As an additional hardening, the open redirect hint for user registrations has been locked down a bit by default. If you used this feature before, you should update Client URIs via the Admin UI, so all possible redirect_uris you are using will still be considered valid, or opt-out of the additional hardening.
# If set to `true`, any validation of the `redirect_uri` provided during
# a user registration will be disabled.
# Clients can use this feature to redirect the user back to their application
# after a successful registration, so instead of ending up in the user
# dashboard, they come back to the client app that initiated the registration.
#
# The given `redirect_uri` will be compared against all registered
# `client_uri`s and will throw an error, if there is no match. However,
# this check will prevent ephemeral clients from using this feature. Only
# if you need it in combination with ephemeral clients, you should
# set this option to `true`. Otherwise it is advised to set the correct
# Client URI in the admin UI. The `redirect_uri` will be allowed if it starts
# with any registered `client_uri`.
#
# default: false
#USER_REG_OPEN_REDIRECT=true
Optional User Family Name
Since I received quite a few questions and requests regarding the mandatory family_name for users, I decided to change it and make it optional. This change should not affect you in any way if you only consumed id_tokens, because the family_name inside them has been optional before already. Its existence in the id_token depends on allowed and requested claims.
However, if you used to communicate with the Rauthy API directly, you should be aware of this change. The User.family_name is now optional in all situations.
Changes
Efficiency and speed improvements
During the migration to Hiqlite, I stumbled about a few DB queries in different places that were low haging fruits for efficiency and speed improvements. I did these while migrating the code base. There were a few ones, for instance in situations like session invalidation, for password reminder cron jobs, roles / groups / scopes name updates, and so on. These do not affect the behavior, just the handling under the hood has been improved.
SVG Sanitization
Rauthy allows you to upload SVGs as either client or upstream IdP logos. This is an action, that only an authorized rauthy_admin role can do. However, as an additional defense in-depth and protection against an evil admin, Rauthy now sanitizes all uploaded SVGs, no matter what.
Latin-1 Extended A added
The allowed characters for input validation for:
- user
given_name - user
family_name - client name
have been expanded and will also allow characters from Latin-1 Extended A.
Bugfix
- When Rauthy sent E-Mails, the name of the recipient has not been set correctly
#602 - The banner should not be logged as plain text when
LOG_FMT=jsonis set
#605 - When requesting a single user by its ID using an API key, you would get an invalid session error response.
This was due to an earlier migration a few versions back. The session check should be done only when no API key
is present to make this request work.
#609