Containerized mariadb:10.11 healthcheck broken after recovery

[masiton]

Docker container running in compose has a volume mounted at var/lib/mysql This volume is periodically backed up as tarball. Upon recovery, the tarball is unpacked and mounted as same volume. Database works, healtcheck fails.

Compose:

services:
  db:
    image: mariadb:10.11
    restart: unless-stopped
    environment:
      - MYSQL_ROOT_PASSWORD=${INIT_MYSQL_ROOT_PASSWORD}
      - MYSQL_LOG_CONSOLE=true
      - MARIADB_AUTO_UPGRADE=1
    healthcheck:
      test:
        [
          "CMD",
          "/usr/local/bin/healthcheck.sh",
          "--connect",
          "--mariadbupgrade",
          "--innodb_initialized",
        ]
      interval: 20s
      start_period: 30s
      timeout: 5s
      retries: 10
    volumes:
      - disk-dbnfs:/var/lib/mysql

Log:

2025-11-12 01:37:45+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.11.14+maria~ubu2204 started.

2025-11-12 01:37:46+00:00 [Warn] [Entrypoint]: /sys/fs/cgroup///memory.pressure not writable, functionality unavailable to MariaDB

2025-11-12 01:37:46+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'

2025-11-12 01:37:46+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.11.14+maria~ubu2204 started.

2025-11-12 01:37:46+00:00 [Note] [Entrypoint]: MariaDB upgrade not required

2025-11-12  1:37:46 0 [Note] Starting MariaDB 10.11.14-MariaDB-ubu2204 source revision 053f9bcb5b147bf00edb99e1310bae9125b7f125 server_uid l+k2bxj0COsQFq99aPFDs/q81tA= as process 1

2025-11-12  1:37:46 0 [Note] InnoDB: Compressed tables use zlib 1.2.11

2025-11-12  1:37:46 0 [Note] InnoDB: Using transactional memory

2025-11-12  1:37:46 0 [Note] InnoDB: Number of transaction pools: 1

2025-11-12  1:37:46 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions

2025-11-12  1:37:46 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts)

2025-11-12  1:37:46 0 [Warning] mariadbd: io_uring_queue_init() failed with EPERM: sysctl kernel.io_uring_disabled has the value 2, or 1 and the user of the process is not a member of sysctl kernel.io_uring_group. (see man 2 io_uring_setup).

create_uring failed: falling back to libaio

2025-11-12  1:37:46 0 [Note] InnoDB: Using Linux native AIO

2025-11-12  1:37:47 0 [Note] InnoDB: innodb_buffer_pool_size_max=128m, innodb_buffer_pool_size=128m

2025-11-12  1:37:47 0 [Note] InnoDB: Completed initialization of buffer pool

2025-11-12  1:37:47 0 [Note] InnoDB: Buffered log writes (block size=512 bytes)

2025-11-12  1:37:47 0 [Note] InnoDB: End of log at LSN=48031870

2025-11-12  1:37:47 0 [Note] InnoDB: 128 rollback segments are active.

2025-11-12  1:37:47 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...

2025-11-12  1:37:47 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.

2025-11-12  1:37:47 0 [Note] InnoDB: log sequence number 48031870; transaction id 187612

2025-11-12  1:37:47 0 [Note] Plugin 'FEEDBACK' is disabled.

2025-11-12  1:37:47 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool

2025-11-12  1:37:47 0 [Warning] You need to use --log-bin to make --expire-logs-days or --binlog-expire-logs-seconds work.

2025-11-12  1:37:47 0 [Note] Server socket created on IP: '0.0.0.0', port: '3306'.

2025-11-12  1:37:47 0 [Note] Server socket created on IP: '::', port: '3306'.

2025-11-12  1:37:47 0 [Note] mariadbd: ready for connections.

Version: '10.11.14-MariaDB-ubu2204'  socket: '/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution

2025-11-12  1:37:47 0 [Note] InnoDB: Buffer pool(s) load completed at 251112  1:37:47

2025-11-12  1:37:50 3 [Warning] Access denied for user 'root'@'localhost' (using password: NO)

2025-11-12  1:37:50 4 [Warning] Access denied for user 'root'@'::1' (using password: NO)

2025-11-12  1:37:55 5 [Warning] Access denied for user 'root'@'localhost' (using password: NO)

2025-11-12  1:37:55 6 [Warning] Access denied for user 'root'@'::1' (using password: NO)

2025-11-12  1:38:00 7 [Warning] Access denied for user 'root'@'localhost' (using password: NO)

2025-11-12  1:38:01 8 [Warning] Access denied for user 'root'@'::1' (using password: NO)

MariaDB [(none)]> SELECT user,host,plugin FROM mysql.user WHERE user='root';
+------+-----------+-----------------------+
| User | Host      | plugin                |
+------+-----------+-----------------------+
| root | localhost | mysql_native_password |
| root | %         | mysql_native_password |
+------+-----------+-----------------------+
2 rows in set (0.071 sec)

Running

ALTER USER 'root'@'localhost' IDENTIFIED VIA unix_socket;
FLUSH PRIVILEGES;

will cause the container to run in "starting" state until the healthcheck timeout and then report unhealthy again, without the log warnings about access denied, but healthcheck will still fail.

[grooverdan]

This volume is periodically backed up as tarball.

Unless you're using some volume snapshot or doing a process suspend of the container, this isn't safe. A consistent read of the files are required. You may have got lucky so far in not relying on a backup while InnoDB flushing is occurring, but its not safe to copy of the files of a running server as a backup.

https://mariadb.com/docs/server/server-management/automated-mariadb-deployment-and-administration/docker-and-mariadb/docker-official-image-frequently-asked-questions#how-do-i-create-a-mariadb-backup-of-the-data

MYSQL_LOG_CONSOLE - not a mariadb docker config option - it was ignored.

MYSQL_ROOT_PASSWORD only has an effect on initialisation of the datadir.

If you remove mysql_upgrade_info in the datadir and restart with MARIADB_AUTO_UPGRADE=1 the healthcheck users will be recreated along with the configuration file.

ALTER USER 'root'@'localhost' IDENTIFIED VIA unix_socket;

The user used for healthchecks is in .my-healthcheck.cnf. While a root user can be used, if you're relying on unix_socket for healthcheck use --su=root, but there's far too many grants on the root user by default to consider it for healthcheck.sh.

FLUSH PRIVILEGES;

Is not required after any alter user or user manipulation statement.

[masiton]

Unless you're using some volume snapshot or doing a process suspend of the container, this isn't safe. A consistent read of the files are required. You may have got lucky so far in not relying on a backup while InnoDB flushing is occurring, but its not safe to copy of the files of a running server as a backup.

Yes, the container is stopped before the backup is taken and then restarted when the backup has finished. I'll come back and edit after trying the suggested. Thanks.

[masiton]

I realized that if recovery is done onto a NFS mounted volume, there is no way for the database to work. MariaDB requires certain files and directories to have specific ownership and permission model (not world-writable, namely), meaning NFS is not usable. Either the files are as MariaDB requires, in which case the system will not be able to mount the NFS share, because the host is neither the owner (MariaDB is) and the permission is not open (not world-readable/writable), or you open the NFS share so it's mounted successfully, but MariaDB fails to run because permission checks are not satisfied. Additionally, maproot or mapall filters will not work here either for the same reason.

When you try to mount MariaDB like this, it will boot, realize the security checks have failed and revert the root user from unix_socket to native_password auth mode, which is what breaks the healthcheck script.

I was convinced I could somehow get this to work, since NFSv4 allows for parameters that set order of operations, block sizes and other tweaks required for a DB engine, and it was that much convenient to manage a NFS share host's backups/replication, but I knew from the start I'm going against best practice here and I just abandoned the idea altogether and I just deploy the DB in-container, which allows for the permission checks, ownership and block-device access to all work as intended.