Request for tutorial/example: Proper usage in a multi-threaded environment?

[saurabhnanda]

I'm trying to write a job-queue backed by an RDBMS and have everything up & running, but not designed perfectly. I'm happy to share the entire design/source-code, because I plan to open-source it once it's battle-tested in production, but for now, here's the short version of the question --

How is the following going to behave:

forever $ withResource dbPool $ \conn -> async $ doSomething conn

Will pool be able to track usage of a resource in an async thread and put it back in the pool only when the thread has exited? Under no circumstance, do I want multiple threads sharing the same DB connection!

[ramirez7]

I am pretty sure that code will result in multiple threads sharing connections.

• withResource will hand over a connection
• async will spawn a thread & that thread will have a reference to the conn
• async will return immediately, so withResource will return the conn to the pool

It's better to spawn the thread first and then have the thread ask for a resource:

forever $ async $ withResource dbPool $ \conn -> doSomething conn

Here's a gist that illustrates the difference between your pasted use of withResource and the one I suggested above:

https://gist.github.com/ramirez7/ede1bc97d851d01e4c603d17da5f738d

The code you pasted in the issue can give currently-allocated resources to new workers, which is exactly what you don't want.

---

Since this issue is about using pools in concurrent environments in general, I'd like to share an example of a deadlock that can occur with subtly wrong use of resource-pool:

https://gist.github.com/ramirez7/17602611ba15a74c50a4aaffeebca25a

Nested calls to withResource are what cause the issue. If the first call successfully allocates resources in multiple threads but then exhausts the pool, the nested calls will block forever.

---

When using a pool concurrently, if you have more threads than resources, some threads will end up waiting. You can then control the number of concurrent workers by changing your pool size.

---

A poorly configured pool can easily become the bottleneck in a concurrent system. This resource-pool PR exposes some useful metrics about pools that can be useful for debugging & scaling.