Skip to content

TCPTransport: multiple "dictionary changed size during iteration" issues #68

@rayrapetyan

Description

@rayrapetyan

opensearch-benchmark 0.0.2
thespian 3.10.1

While running opensearch-benchmark (based on thespian) I'm sporadically getting following errors in logs:

2022-03-27 00:35:36.463023 p374 ERR xmit UNcaught exception dictionary changed size during iteration; aborting intent.
Traceback (most recent call last):
File "/opt/sm/python/lib/python3.9/site-packages/thespian/system/transport/TCPTransport.py", line 801, in _nextTransmitStep
return getattr(self, 'next_XMIT%s' % intent.stage,
File "/opt/sm/python/lib/python3.9/site-packages/thespian/system/transport/TCPTransport.py", line 830, in _next_XMIT_1
if any(T for T in self._transmitIntents.values()
File "/opt/sm/python/lib/python3.9/site-packages/thespian/system/transport/TCPTransport.py", line 830, in
if any(T for T in self._transmitIntents.values()
RuntimeError: dictionary changed size during iteration

2022-03-26 23:28:41.846563 p844 ERR Actor osbenchmark.worker_coordinator.worker_coordinator.Worker @ ActorAddr-(T|:19163) transport run exception: Traceback (most recent call last):
File "/opt/sm/python/lib/python3.9/site-packages/thespian/system/actorManager.py", line 87, in run
r = self.transport.run(self.handleMessages)
File "/opt/sm/python/lib/python3.9/site-packages/thespian/system/transport/wakeupTransportBase.py", line 74, in run
rval = self._run_subtransport(incomingHandler, max_runtime)
File "/opt/sm/python/lib/python3.9/site-packages/thespian/system/transport/wakeupTransportBase.py", line 80, in _run_subtransport
rval = self._runWithExpiry(incomingHandler)
File "/opt/sm/python/lib/python3.9/site-packages/thespian/system/transport/TCPTransport.py", line 1098, in _runWithExpiry
partition(TCPTransport._waitForSendable,
File "/opt/sm/python/lib/python3.9/site-packages/thespian/system/utilis.py", line 207, in partition
return foldl(appendLeftOrRight, inp_iterable, (output_type(), output_type()))
RuntimeError: dictionary changed size during iteration

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions