KV store connection TLS client certs not reloaded after renewal #148
Description
To connect to KV store it is recommended to use authentication such as TLS client certificates. These certificates have a limited validity duration. So they are renewed before expiration. Currently modelmesh does not detect and reload the client certificates from disc if they have been updated/renewed. Only a restart of modelmesh loads the new certificates. This results in connection issues to the KV store and eventually failures, because state cannot be synced anymore.
When using modelmesh with kubernetes and using cert-manager to issue the client certificates, the certs are provided as secrets which are mounted into the container and cert-manager automatically updates them before expiration. However modelmesh does not use the new certificat.
Related to #135, can also function as workaround because it restart the pod on kv connection failures