Notifications: Dispatcher pod fails on startup due to rabbitmq not ready.

[scblack321]

Recently in DEV environment, observed that the dispatcher pod failed to start the first time with the below error.
Kubernetes did restart the pod and the 2nd time it came up okay:

tapisdev@tapisdeploy notifications]$ knotifp 
notifications-api-85d78b49c4-zhtpz                     1/1     Running                  0                2m48s
notifications-dispatcher-6bb695cb54-c69xs              1/1     Running                  1 (2m43s ago)    2m48s
notifications-init-db-dfbj2                            0/1     Completed                0                3m4s
notifications-pgadmin-5f49f957db-kbncb                 1/1     Running                  1 (7d18h ago)    19d
notifications-postgres-16-787cb5c499-cqtrl             1/1     Running                  0                3m26s
notifications-rabbitmq-7f4c8c558f-6rlpr                1/1     Running                  0                2m56s

Apparently rabbitmq is not coming up quickly enough during a full burndown/up. Is there a way to make the dispatcher more robust?

2025-10-30 03:09:43.185 INFO [main/] e.u.t.t.n.service.DispatchService:122 - NTFLIB_DSP_INIT_MSGBRKR Dispatch service initializing message broker.
Exception in thread "main" edu.utexas.tacc.tapis.shared.exceptions.runtime.TapisRuntimeException: QMGR_UNINITIALIZED_ERROR The QueueManager must be initialized before being used.
at edu.utexas.tacc.tapis.notifications.service.MessageBroker.initRabbitVHost(MessageBroker.java:373)
at edu.utexas.tacc.tapis.notifications.service.MessageBroker.(MessageBroker.java:95)
at edu.utexas.tacc.tapis.notifications.service.MessageBroker.init(MessageBroker.java:136)
at edu.utexas.tacc.tapis.notifications.service.DispatchService.initService(DispatchService.java:123)
at edu.utexas.tacc.tapis.notifications.DispatchApplication.main(DispatchApplication.java:71)
Caused by: java.net.ConnectException
at java.net.http/jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:955)
at java.net.http/jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:133)
at edu.utexas.tacc.tapis.sharedq.VHostManager.hasVHost(VHostManager.java:99)
at edu.utexas.tacc.tapis.sharedq.VHostManager.initVHost(VHostManager.java:76)
at edu.utexas.tacc.tapis.notifications.service.MessageBroker.initRabbitVHost(MessageBroker.java:368)
... 4 more
Caused by: java.net.ConnectException
at java.net.http/jdk.internal.net.http.common.Utils.toConnectException(Utils.java:1065)
at java.net.http/jdk.internal.net.http.PlainHttpConnection.connectAsync(PlainHttpConnection.java:227)
at java.net.http/jdk.internal.net.http.PlainHttpConnection.checkRetryConnect(PlainHttpConnection.java:280)
at java.net.http/jdk.internal.net.http.PlainHttpConnection.lambda$connectAsync$2(PlainHttpConnection.java:238)
at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1773)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.nio.channels.ClosedChannelException
at java.base/sun.nio.ch.SocketChannelImpl.ensureOpen(SocketChannelImpl.java:202)
at java.base/sun.nio.ch.SocketChannelImpl.beginConnect(SocketChannelImpl.java:786)
at java.base/sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:874)
at java.net.http/jdk.internal.net.http.PlainHttpConnection.lambda$connectAsync$1(PlainHttpConnection.java:210)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:571)
at java.net.http/jdk.internal.net.http.PlainHttpConnection.connectAsync(PlainHttpConnection.java:212)
... 9 more