Healthcheck issue with Galera use

[zguig52]

I am working on Galera deployment with your container (I was previously using Bitnami all in one image).

I managed to configure all the nodes and secure the bootstrap. I have issues with nodes joining the cluster and healthcheck.

From what I understood, healthcheck user is generated locally on each node and credentials are stored in DATA folder. I saw that it should not be synced to other nodes and managed each time locally.

With Galera, it seems that the healthcheck user is propagated as they are totally in sync. So I tried to synchronize first node credential files to other nodes and it works, BUT:
Any new node joining the cluster, is clearing the content of the datadir:
WSREP_SST: [INFO] Cleaning the existing datadir and innodb-data/log directories

As the configuration file is stored there, it will be cleared each time there is a full sync performed on any node (after a restart and failure to perform an incremental sync for example).

At minimal, I was thinking that it would be nice to avoid storing this configuration file in the data directory to avoid potential removal of it during folder clear (maybe using another dedicated volume writeable in /etc/mysql folder?).

It would also be nice to expose env vars to allow defining the user and password of healthcheck (as there is already a var for MARIADB_HEALTHCHECK_GRANTS, there could be MARIADB_HEALTHCHECK_USER and MARIADB_HEALTHCHECK_PASSWORD).

Thus, if somebody wants to preform healthchecks, it is quite simple, he uses these variables and at entry point, the user is created if not existing + the configuration file is generated based on this during boot, without having to rely on a configuration file locally stored in data directory.

What do you think of all this?

[grooverdan]

Thanks for writing this up.

At some point yes, it was foreseen that the healthcheck user would propagate with at least rsync as a SST mechanism. It might not on mysqldump and mariabackup.

I see the "Cleaning the existing datadir and innodb-data/log directories" is exclusively in the mariabackup SST. In this case the .my-healthcheck.cnf isn't part of the backup/restore. So its this effect that is leaving the joiner without a config file.

I evidently thought about this but have yet to implement this - https://jira.mariadb.org/browse/MDEV-31717, and I certainly could for next release.

Is having a consistent set of working healthcheck users across the cluster acceptable?

On proposals:

I was explicitly storing the config file in the datadir as it is strongly tied to the data there. I was avoiding too many persistent volumes.

Is /etc/mysql an option? Not sure - the documentation reference recommends that this is mounted read only, mainly because permission mismatches can be considered by the server as insecure and it will not read the file. The current healthcheck.sh has options --defaults-file=*|--defaults-extra-file=*|--defaults-group-suffix=* so creating a file there and using these options in your HEALTHCHECK . Its the --defaults-extra-file that defaults to $datadir/.my-healthcheck.cnf if present. Note however that MARIADB_AUTO_UPGRADE=1 will auto-recreate the default file and health check users. So the TLDR is /etc/mysql is an option if, --defaults-extra-file=/etc/mysql/xx.cnf as part of HEALTHCHECK script args (first arg), you create this file, and you create users != 'healthcheck'.

On env MARIADB_HEALTHCHECK_USER and MARIADB_HEALTHCHECK_PASSWORD, yes using these as a default creation of the user/file could work. I think HEALTHCHECK execution can access env variable, so there's a direct path there without a file. I was avoiding creating more variables unless really needed. The interaction between all of them is getting complicated.

Alternate:

Use the healthcheck.sh with --su-mysql and env MARIADB_MYSQL_LOCALHOST_USER=1 and MARIADB_MYSQL_LOCALHOST_GRANTS as your healthcheck user which will function without a config file. With MARIADB_AUTO_UPGRADE=1 , you'll probably need --defaults-extra-file to a dummy config file (e.g) /etc/mysql/my.cnf (syntax parseable by not offering a [mariadb-client] group used by the mariadb executable to override the .my-healthcheck.cnf default.

So my TLDR conclusion, would be use alternative above, and I'll endeavour to get MDEV-31717 fixed so it works in an almost default scenario.

Acceptable?

[grooverdan]

While I think of it, are you (and anyone reading this) willing to do a 5-10minute MariaDB survey this month (October 2025): https://mariadb.typeform.com/survey-2025?utm_source=doiissue

[zguig52]

At some point yes, it was foreseen that the healthcheck user would propagate with at least rsync as a SST mechanism. It might not on mysqldump and mariabackup.

I am using mariabackup as wsrep_sst_method and it propagates (as I was able to make the healthcheck working on other nodes by copying the conf file from the primary node).

I evidently thought about this but have yet to implement this - https://jira.mariadb.org/browse/MDEV-31717, and I certainly could for next release.

It cloud be a solution, just be carefull on the SST mechanism compatibility (I don't give SSH/RSYNC access between nodes, so how to get this information and share it through mariabackup, is the tool able to directly read and write plain files and not only database related content)?

Is having a consistent set of working healthcheck users across the cluster acceptable?

As all other users and permissions are consistent across the cluster, it seems to me like a more simple solution to manage and understand, don't you think? It took me some time to understand why healthcheck was working on the first node and not on the others - I hadn't seen that there was a specific user for this as there were no associated env user var like for backup or replication / I thought primarily that it was a local socket passwordless permission config (I did not properly read associated doc).

Is /etc/mysql an option? Not sure - the documentation reference recommends that this is mounted read only.

I totally agree regarding permissions on this existing volume not to be changed. I was more thinking of a dedicated subfolder in /etc/mysql (as it is a config file). But I think that you are right, it is better to avoid writing there at all (and mixing mount permissions). Let's keep etc as a read only stuff :)

The current healthcheck.sh has options --defaults-file=|--defaults-extra-file=|--defaults-group-suffix=* so creating a file there and using these options in your HEALTHCHECK . Its the --defaults-extra-file that defaults to $datadir/.my-healthcheck.cnf if present. Note however that MARIADB_AUTO_UPGRADE=1 will auto-recreate the default file and health check users. So the TLDR is /etc/mysql is an option if, --defaults-extra-file=/etc/mysql/xx.cnf as part of HEALTHCHECK script args (first arg), you create this file, and you create users != 'healthcheck'.

I will try to implement a workaround with custom user / password and conf file as suggested, thanks for the idea!

On env MARIADB_HEALTHCHECK_USER and MARIADB_HEALTHCHECK_PASSWORD, yes using these as a default creation of the user/file could work. I think HEALTHCHECK execution can access env variable, so there's a direct path there without a file. I was avoiding creating more variables unless really needed. The interaction between all of them is getting complicated.

Maybe this would even simplify the healthcheck process, without having to deal with a configuration file and modifying the database replication script due to deployments / official container image constraints (but this file/copy option might still be an independent feature of SST scripts / based on an additional option).

Another point to take in account in favour of this basic env strategy is that somebody that do not want healthcheck usage will have a user created that he does not want (and there might be some cases where there is already a user in the target deployed system called healthcheck, being used by an application). This could create headaches to some people (until I investigate the issue, I did not knew that there was this healthcheck local user in the story)!

Sidenote: I love having many ENV variables, so I don't need to handle conf files (and any changes are detected automatically by the container that can reboot if any changes are presents)!

Use the healthcheck.sh with --su-mysql and env MARIADB_MYSQL_LOCALHOST_USER=1 and MARIADB_MYSQL_LOCALHOST_GRANTS as your healthcheck user which will function without a config file. With MARIADB_AUTO_UPGRADE=1 , you'll probably need --defaults-extra-file to a dummy config file (e.g) /etc/mysql/my.cnf (syntax parseable by not offering a [mariadb-client] group used by the mariadb executable to override the .my-healthcheck.cnf default.

I prefer the first option you suggested on reading it (as this might look more like a "ugly hack" to pass a dummy conf file). But I keep this idea in case I am getting troubles with implementing first idea.

Acceptable?

Thanks a lot for your reply and your time! I will keep you informed if/how I managed to make it work.

[zguig52]

So here are my feedbacks after working on it:

• solution 1 / KO: using a dedicated additional external user managed was not possible. I use the health status of the container to ensure that I can pass any commands to the database. I thought I would use the default healthcheck user during bootstrap phase and change it later, but this makes the cluster bootstrap linked to a specific node and not being able to start it from another node, not having this automatically created user config file.
  ==> Another option would be to change the way I ensure I can send commands to MariaDB, but only TCP port available is not enough as it might be not being ready to receive commands. I though healthcheck.sh --galera_ready was the best way to ensure it. Maybe there are other options I did not consider?

• solution 2 / KO: copy the initially created healthcheck conf file, mount it to all containers (in /etc/mysql/conf.d with other conf files as read only / putting it inside data dir will make discussed SST dir cleanup fail) and passed it as --defaults-extra-file to the healthcheck script
  ==> I have checked the entrypoint script to ensure that it will not create troubles with the MARIADB_AUTO_UPGRADE=1 you spoke and it seems that there will be different password regenerated on nodes having not this file present, even with --defaults-extra-file in use, as healthcheck user + random password and config file creation is based on this specific file.

• solution 3 / OK: using MARIADB_MYSQL_LOCALHOST_USER=1 / localhost only user without any password and local socket use + healthcheck.sh --su-mysql --galera_ready (the second solution you have proposed)
  ==> except that there is a useless healtcheck user created / updated from time to time on any node starting, the fact that there is no password and only local socket use makes it fine to be used on any node at any time.
  ==> I did not get why there was a need to use --defaults-extra-file to a dummy config file with MARIADB_AUTO_UPGRADE=1

Thanks again for your advices and I look forward if I can give any help on simplifying the healthcheck usage.