Possible race condition in callbacks when server and client processes start at same time

[s-e-martin]

This is apparent in the DSTC++ tests where occasional hangs were being noticed.

The issue appears to occur when a server and client process are started up at virtually the same time. It appears as though the server code gets executed (using examples/callback, we see the double_value(223344) called with a callback message). However, the client callback never gets invoked. As best as I can tell, the DSTC library should be unloaded before the test execution occurs

Reproduction steps:

Clone and build RMC / DSTC into the same folder:

git clone git@github.com:PDXostc/dstc.git
cd dstc
make
cd ..
git clone git@github.com:PDXostc/reliable_multicast.git
cd reliable_multicast
make
cd ..

Build callback example

cd dstc/examples/callback
make

Execute the script in this tarball: test_script.tar.gz, which ensures that the previous callback_client/server have exited, and that there are no open handles to libdstc.so. This script also sets LD_LIBRARY_PATH to ensure that the libdstc.so object that is being loaded is the one you just built.

The test will hang as early as a handful to as many as a 100+ iterations.

./test_script.sh