Process hangs up after installing flash-llm-rl

[zzfoutofspace]

Hello, I successfully installed flash-rl following your instructions, and finished the test to ensure the correctness.

Then I directly ran the script you provided
bash recipe/flash_rl/gsm8k_qwen0_5b_int8.sh flash-int8-TIS-2 2

The process hangs up after the ray instance is started:

2025-08-22 10:54:24,273 - numexpr.utils - INFO - Note: detected 384 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
2025-08-22 10:54:24,273 - numexpr.utils - INFO - Note: NumExpr detected 384 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 16.
2025-08-22 10:54:24,273 - numexpr.utils - INFO - NumExpr defaulting to 16 threads.
DEBUG 08-22 10:54:24 [init.py:28] No plugins for group vllm.platform_plugins found.
DEBUG 08-22 10:54:24 [init.py:34] Checking if TPU platform is available.
DEBUG 08-22 10:54:24 [init.py:44] TPU platform is not available because: No module named 'libtpu'
DEBUG 08-22 10:54:24 [init.py:52] Checking if CUDA platform is available.
DEBUG 08-22 10:54:24 [init.py:72] Confirmed CUDA platform is available.
DEBUG 08-22 10:54:24 [init.py:100] Checking if ROCm platform is available.
DEBUG 08-22 10:54:24 [init.py:114] ROCm platform is not available because: No module named 'amdsmi'
DEBUG 08-22 10:54:24 [init.py:122] Checking if HPU platform is available.
DEBUG 08-22 10:54:24 [init.py:129] HPU platform is not available because habana_frameworks is not found.
DEBUG 08-22 10:54:24 [init.py:140] Checking if XPU platform is available.
DEBUG 08-22 10:54:24 [init.py:150] XPU platform is not available because: No module named 'intel_extension_for_pytorch'
DEBUG 08-22 10:54:24 [init.py:158] Checking if CPU platform is available.
DEBUG 08-22 10:54:24 [init.py:180] Checking if Neuron platform is available.
DEBUG 08-22 10:54:24 [init.py:187] Neuron platform is not available because: No module named 'transformers_neuronx'
DEBUG 08-22 10:54:24 [init.py:52] Checking if CUDA platform is available.
DEBUG 08-22 10:54:24 [init.py:72] Confirmed CUDA platform is available.
INFO 08-22 10:54:24 [init.py:239] Automatically detected platform cuda.
2025-08-22 10:54:26,762 - flash_rl.vllm_patch - DEBUG - Successfully patched the process_weights_after_loading function of vllm
2025-08-22 10:54:26,762 - flash_rl.vllm_patch - DEBUG - Successfully patched kv_cache process_weights_after_loading
2025-08-22 10:54:26,762 - flash_rl - DEBUG - Patching vllm process_weights_after_loading... status: True
2025-08-22 10:54:26,762 - flash_rl.vllm_patch - DEBUG - Successfully patched vllm
2025-08-22 10:54:26,762 - flash_rl - DEBUG - Patching the vllm LLM to enable flash_rl quantization... status: True
2025-08-22 10:54:27,353 - hydra.core.utils - DEBUG - Setting JobRuntime:name=UNKNOWN_NAME
2025-08-22 10:54:27,353 - hydra.core.utils - DEBUG - Setting JobRuntime:name=main_ppo
2025-08-22 10:54:34,306 INFO worker.py:1832 -- Started a local Ray instance. View the dashboard at http://127.0.0.1:8265

The experiments is conducted on 8 GPUs on one node, with vllm==0.8.4.

Is there any idea why the process keeps hanging up?

[LiyuanLucasLiu]

Thanks for reporting! For debugging purpose, can you try running the bf16 script as in https://github.com/yaof20/verl/blob/flash-rl/recipe/flash_rl/gsm8k_qwen0_5b_bf16.sh to see whether it also hangs? Also, can you try changing trainer.n_gpus_per_node from 4 to 8?

[zzfoutofspace]

Thanks for your quick reply.

During replicating the error on another machine, another error appears when trying to evaluate the installation following step 2 as suggested:

FLASHRL_TEST_RELOAD="Qwen/Qwen2.5-0.5B-Instruct,LiyuanLucasLiu/Qwen2.5-0.5B-Instruct-VERL" \ FLASHRL_CONFIG="fp8" \ FLASHRL_LOGGING_LEVEL=DEBUG \ VLLM_LOGGING_LEVEL=DEBUG \ CUDA_VISIBLE_DEVICES=0 \ torchrun --standalone --nnodes=1 --nproc-per-node=1 --no-python lm_eval --model vllm \ --model_args pretrained=Qwen/Qwen2.5-0.5B-Instruct,tensor_parallel_size=1,dtype=auto,gpu_memory_utilization=0.8,data_parallel_size=1 \ --tasks gsm8k \ --batch_size auto

---

Error:
INFO 08-22 17:22:47 [core.py:61] Initializing a V1 LLM engine (v0.8.4) with config: model='Qwen/Qwen2.5-0.5B-Instruct', speculative_config=None, tokenizer='Qwen/Qwen2.5-0.5B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='auto', reasoning_backend=None), observability_config=ObservabilityConfig(show_hidden_metrics=False, otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=1234, served_model_name=Qwen/Qwen2.5-0.5B-Instruct, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"level":3,"custom_ops":["none"],"splitting_ops":
["vllm.unified_attention","vllm.unified_attention_with_output"],"use_inductor":true,"compile_sizes":[],"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":512}
DEBUG 08-22 17:22:47 [decorators.py:109] Inferred dynamic dimensions for forward method of <class 'vllm.model_executor.models.llama.LlamaModel'>: ['input_ids', 'positions', 'intermediate_tensors', 'inputs_embeds']
WARNING 08-22 17:22:48 [utils.py:2444] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x7f74b15b61d0>
DEBUG 08-22 17:22:48 [config.py:3908] enabled custom ops: Counter()
DEBUG 08-22 17:22:48 [config.py:3910] disabled custom ops: Counter()
ERROR 08-22 17:22:48 [core.py:387] EngineCore hit an exception: Traceback (most recent call last):
ERROR 08-22 17:22:48 [core.py:387] File "/usr/local/lib64/python3.11/site-packages/vllm/v1/engine/core.py", line 378, in run_engine_core
ERROR 08-22 17:22:48 [core.py:387] engine_core = EngineCoreProc(*args, **kwargs)
ERROR 08-22 17:22:48 [core.py:387] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 08-22 17:22:48 [core.py:387] File "/usr/local/lib64/python3.11/site-packages/vllm/v1/engine/core.py", line 320, in init
ERROR 08-22 17:22:48 [core.py:387] super().init(vllm_config, executor_class, log_stats)
ERROR 08-22 17:22:48 [core.py:387] File "/usr/local/lib64/python3.11/site-packages/vllm/v1/engine/core.py", line 67, in init
ERROR 08-22 17:22:48 [core.py:387] self.model_executor = executor_class(vllm_config)
ERROR 08-22 17:22:48 [core.py:387] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 08-22 17:22:48 [core.py:387] File "/usr/local/lib64/python3.11/site-packages/vllm/executor/executor_base.py", line 52, in init
ERROR 08-22 17:22:48 [core.py:387] self._init_executor()
ERROR 08-22 17:22:48 [core.py:387] File "/usr/local/lib64/python3.11/site-packages/vllm/executor/uniproc_executor.py", line 46, in _init_executor
ERROR 08-22 17:22:48 [core.py:387] self.collective_rpc("init_device")
ERROR 08-22 17:22:48 [core.py:387] File "/usr/local/lib64/python3.11/site-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
ERROR 08-22 17:22:48 [core.py:387] answer = run_method(self.driver_worker, method, args, kwargs)
ERROR 08-22 17:22:48 [core.py:387] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 08-22 17:22:48 [core.py:387] File "/usr/local/lib64/python3.11/site-packages/vllm/utils.py", line 2378, in run_method
ERROR 08-22 17:22:48 [core.py:387] return func(*args, **kwargs)
ERROR 08-22 17:22:48 [core.py:387] ^^^^^^^^^^^^^^^^^^^^^
ERROR 08-22 17:22:48 [core.py:387] File "/usr/local/lib64/python3.11/site-packages/vllm/worker/worker_base.py", line 604, in init_device
ERROR 08-22 17:22:48 [core.py:387] self.worker.init_device() # type: ignore
ERROR 08-22 17:22:48 [core.py:387] ^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 08-22 17:22:48 [core.py:387] File "/usr/local/lib64/python3.11/site-packages/vllm/v1/worker/gpu_worker.py", line 103, in init_device
ERROR 08-22 17:22:48 [core.py:387] torch.cuda.set_device(self.device)
ERROR 08-22 17:22:48 [core.py:387] File "/usr/local/lib64/python3.11/site-packages/torch/cuda/init.py", line 476, in set_device
ERROR 08-22 17:22:48 [core.py:387] torch._C._cuda_setDevice(device)
ERROR 08-22 17:22:48 [core.py:387] File "/usr/local/lib64/python3.11/site-packages/torch/cuda/init.py", line 305, in _lazy_init
ERROR 08-22 17:22:48 [core.py:387] raise RuntimeError(
ERROR 08-22 17:22:48 [core.py:387] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
ERROR 08-22 17:22:48 [core.py:387]
CRITICAL 08-22 17:22:48 [core_client.py:359] Got fatal signal from worker processes, shutting down. See stack trace above for root cause issue.
E0822 17:22:48.181000 6826 torch/distributed/elastic/multiprocessing/api.py:869] failed (exitcode: -9) local_rank: 0 (pid: 6893) of binary: lm_eval
Traceback (most recent call last):
File "/usr/local/bin/torchrun", line 8, in
sys.exit(main())
^^^^^^
File "/usr/local/lib64/python3.11/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 355, in wrapper
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/torch/distributed/run.py", line 918, in main
run(args)
File "/usr/local/lib64/python3.11/site-packages/torch/distributed/run.py", line 909, in run
elastic_launch(
File "/usr/local/lib64/python3.11/site-packages/torch/distributed/launcher/api.py", line 138, in call
return launch_agent(self._config, self._entrypoint, list(args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/torch/distributed/launcher/api.py", line 269, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

lm_eval FAILED
Failures:
<NO_OTHER_FAILURES>

Root Cause (first observed failure):
[0]:
time : 2025-08-22_17:22:48
host : xxxxxxxx.site
rank : 0 (local_rank: 0)
exitcode : -9 (pid: 6893)
error_file: <N/A>
traceback : Signal 9 (SIGKILL) received by PID 6893

---

Handling this error I added another env variable:

export VLLM_WORKER_MULTIPROC_METHOD=spawn

Then the process keeps hanging up like:

INFO 08-22 17:24:13 [core.py:61] Initializing a V1 LLM engine (v0.8.4) with config: model='Qwen/Qwen2.5-0.5B-Instruct', speculative_config=None, tokenizer='Qwen/Qwen2.5-0.5B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='auto', reasoning_backend=None), observability_config=ObservabilityConfig(show_hidden_metrics=False, otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=1234, served_model_name=Qwen/Qwen2.5-0.5B-Instruct, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"level":3,"custom_ops":["none"],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"use_inductor":true,"compile_sizes":[],"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":512}
DEBUG 08-22 17:24:13 [init.py:28] No plugins for group vllm.general_plugins found.
DEBUG 08-22 17:24:13 [decorators.py:109] Inferred dynamic dimensions for forward method of <class 'vllm.model_executor.models.llama.LlamaModel'>: ['input_ids', 'positions', 'intermediate_tensors', 'inputs_embeds']
WARNING 08-22 17:24:13 [utils.py:2444] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x7f001493ae10>
DEBUG 08-22 17:24:13 [config.py:3908] enabled custom ops: Counter()
DEBUG 08-22 17:24:13 [config.py:3910] disabled custom ops: Counter()
DEBUG 08-22 17:24:14 [parallel_state.py:822] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://29.127.50.145:42031 backend=nccl
DEBUG 08-22 17:24:17 [core_client.py:421] Waiting for 1 core engine proc(s) to start: {0}
DEBUG 08-22 17:24:27 [core_client.py:421] Waiting for 1 core engine proc(s) to start: {0}
DEBUG 08-22 17:24:37 [core_client.py:421] Waiting for 1 core engine proc(s) to start: {0}
DEBUG 08-22 17:24:47 [core_client.py:421] Waiting for 1 core engine proc(s) to start: {0}
DEBUG 08-22 17:24:57 [core_client.py:421] Waiting for 1 core engine proc(s) to start: {0}
DEBUG 08-22 17:25:07 [core_client.py:421] Waiting for 1 core engine proc(s) to start: {0}
DEBUG 08-22 17:25:17 [core_client.py:421] Waiting for 1 core engine proc(s) to start: {0}
DEBUG 08-22 17:25:27 [core_client.py:421] Waiting for 1 core engine proc(s) to start: {0}
DEBUG 08-22 17:25:37 [core_client.py:421] Waiting for 1 core engine proc(s) to start: {0}
DEBUG 08-22 17:25:47 [core_client.py:421] Waiting for 1 core engine proc(s) to start: {0}
DEBUG 08-22 17:25:57 [core_client.py:421] Waiting for 1 core engine proc(s) to start: {0}
DEBUG 08-22 17:26:07 [core_client.py:421] Waiting for 1 core engine proc(s) to start: {0}
DEBUG 08-22 17:26:17 [core_client.py:421] Waiting for 1 core engine proc(s) to start: {0}

Do you think this might also be the same reason of the hanging up in the original question? Any ideas on the bug?

[zzfoutofspace]

And for the script from the original problem, the bug now shows up as
2025-08-22 18:13:11,826 INFO worker.py:1832 -- Started a local Ray instance. View the dashboard at http://127.0.0.1:8265 [2025-08-22 18:13:45,119 E 11753 11753] core_worker.cc:499: Failed to register worker to Raylet: IOError: [RayletClient] Unable to register worker with raylet. Failed to read data from the socket: End of file worker_id=01000000fffffffffffffffffffffffffffff fffffffffffffffffff

[sharonyu-115]

I had the same problem as well when running the test on single node.

And for the script from the original problem, the bug now shows up as 2025-08-22 18:13:11,826 INFO worker.py:1832 -- Started a local Ray instance. View the dashboard at http://127.0.0.1:8265 [2025-08-22 18:13:45,119 E 11753 11753] core_worker.cc:499: Failed to register worker to Raylet: IOError: [RayletClient] Unable to register worker with raylet. Failed to read data from the socket: End of file worker_id=01000000fffffffffffffffffffffffffffff fffffffffffffffffff

And I'm able to recreate the problem with the following test program to initialize ray:

#!/usr/bin/env python3
"""
Simple test script to initialize and test Ray.
"""

import ray
import time
import sys


def test_ray_initialization():
    """Test Ray initialization and basic functionality."""
    
    print("Starting Ray initialization test...")

    try:
        # Initialize Ray
        print("Initializing Ray...")
        ray.init()
        
        print(f"Ray initialized successfully!")
        print(f"Ray cluster resources: {ray.cluster_resources()}")
        print(f"Ray available resources: {ray.available_resources()}")
        
        # Test a simple remote function
        @ray.remote
        def hello_ray(name):
            return f"Hello {name} from Ray worker!"
        
        # Execute the remote function
        print("Testing remote function execution...")
        future = hello_ray.remote("World")
        result = ray.get(future)
        print(f"📨 Remote function result: {result}")
        
        # Test multiple parallel tasks
        print("Testing parallel execution...")
        futures = [hello_ray.remote(f"Worker-{i}") for i in range(3)]
        results = ray.get(futures)
        for result in results:
            print(f"{result}")
        
        print("All Ray tests passed successfully!")
        return True
        
    except Exception as e:
        print(f"Ray initialization failed: {e}")
        return False
        
    finally:
        # Shutdown Ray
        print("Shutting down Ray...")
        ray.shutdown()
        print("Ray shutdown complete")


def main():
    """Main function to run the Ray test."""
    
    print("=" * 50)
    print("Ray Initialization Test Script")
    print("=" * 50)
    
    success = test_ray_initialization()
    
    if success:
        print("\nRay test completed successfully!")
        sys.exit(0)
    else:
        print("\nRay test failed!")
        sys.exit(1)


if __name__ == "__main__":
    main()

I guess the problem is the patch is activated byusercustomize.py when python interpreter starts. import flash_rl with FLASHRL_CONFIG=fp8 triggers import vllm, which for some reason causes conflict and the ray init error. But I'm not yet able to identify the root cause. Simply add an import vllm instruction in the __init__.py as below can also reproduce the error:

if 'FLASHRL_CONFIG' in os.environ and check_vllm_installed():
    
    from .vllm_patch import patch_vllm_llm, patch_vllm_process_weights_after_loading
 ...
else:
    import vllm
    logger.debug("Skipping the patching of vllm")

[LiyuanLucasLiu]

Thanks all for reporting!!! @sharonyu-115 @zzfoutofspace I have a brief understanding of what happened now. As a temporary fix, you can do:

1. remove FlashRL import in usercustomize.py;
2. manually add import flash_rl before import vllm in every file that imports vLLM.

As to the cause, likely the issue comes from import from vllm (e.g., flash-infer), which would setup the distributed environment & could interfere with ray etc. We are advised by the vLLM people to further wrap our package as a plug-in module for vLLM, and we are working on that (unfortunately no ETA for this).

[LiyuanLucasLiu]

@sharonyu-115 @zzfoutofspace Can you provide more information on reproducing the hanging? I tested all four environments as below and couldn't reproduce the issue. Also, can you try using one of the following docker image to help debugging the source of the bug? Many thanks!

hiyouga/verl:ngc-th2.6.0-cu126-vllm0.8.3-flashinfer0.2.2-cxx11abi0
hiyouga/verl:ngc-th2.6.0-cu126-vllm0.8.4-flashinfer0.2.2-cxx11abi0
hiyouga/verl:ngc-th2.7.0-cu12.6-vllm0.9.1
hiyouga/verl:ngc-th2.7.1-cu12.6-vllm0.10.0

[sharonyu-115]

Hi @LiyuanLucasLiu Thank you for looking into this. I'm using the hiyouga/verl:ngc-th2.6.0-cu126-vllm0.8.4-flashinfer0.2.2-cxx11abi0 image. As I'm on a environment managed by slurm, so I converted the image to .sqsh file, then:

1. Launch the container:
   srun --gres=gpu:8 --nodes=1 --ntasks=1 --container-image=... --pty bash
2. Install FlashRL. I installed from source pip install --no-deps -e .. pip install flash-llm-rl did not get the usercustomize.py updated for me, which I don't know why.
3. Run the python program I have in above comment to test ray initialization.

With out FLASHRL_CONFIG set, it worked fine:

python test_ray_init.py
2025-08-23 08:40:45,399 - flash_rl - DEBUG - Skipping the patching of vllm
==================================================
Ray Initialization Test Script
==================================================
🚀 Starting Ray initialization test...
📡 Initializing Ray...
2025-08-23 08:40:45,939 - filelock - DEBUG - Attempting to acquire lock 23456177321680 on /tmp/ray/session_2025-08-23_08-40-45_938783_1416845/node_ip_address.json.lock
2025-08-23 08:40:45,939 - filelock - DEBUG - Lock 23456177321680 acquired on /tmp/ray/session_2025-08-23_08-40-45_938783_1416845/node_ip_address.json.lock
2025-08-23 08:40:45,939 - filelock - DEBUG - Attempting to release lock 23456177321680 on /tmp/ray/session_2025-08-23_08-40-45_938783_1416845/node_ip_address.json.lock
2025-08-23 08:40:45,939 - filelock - DEBUG - Lock 23456177321680 released on /tmp/ray/session_2025-08-23_08-40-45_938783_1416845/node_ip_address.json.lock
2025-08-23 08:40:45,939 - filelock - DEBUG - Attempting to acquire lock 23456177322112 on /tmp/ray/session_2025-08-23_08-40-45_938783_1416845/ports_by_node.json.lock
2025-08-23 08:40:45,939 - filelock - DEBUG - Lock 23456177322112 acquired on /tmp/ray/session_2025-08-23_08-40-45_938783_1416845/ports_by_node.json.lock
2025-08-23 08:40:45,939 - filelock - DEBUG - Attempting to release lock 23456177322112 on /tmp/ray/session_2025-08-23_08-40-45_938783_1416845/ports_by_node.json.lock
2025-08-23 08:40:45,939 - filelock - DEBUG - Lock 23456177322112 released on /tmp/ray/session_2025-08-23_08-40-45_938783_1416845/ports_by_node.json.lock
2025-08-23 08:40:45,939 - filelock - DEBUG - Attempting to acquire lock 23456177321680 on /tmp/ray/session_2025-08-23_08-40-45_938783_1416845/ports_by_node.json.lock
2025-08-23 08:40:45,939 - filelock - DEBUG - Lock 23456177321680 acquired on /tmp/ray/session_2025-08-23_08-40-45_938783_1416845/ports_by_node.json.lock
2025-08-23 08:40:45,940 - filelock - DEBUG - Attempting to release lock 23456177321680 on /tmp/ray/session_2025-08-23_08-40-45_938783_1416845/ports_by_node.json.lock
2025-08-23 08:40:45,940 - filelock - DEBUG - Lock 23456177321680 released on /tmp/ray/session_2025-08-23_08-40-45_938783_1416845/ports_by_node.json.lock
2025-08-23 08:40:45,940 - filelock - DEBUG - Attempting to acquire lock 23456177321488 on /tmp/ray/session_2025-08-23_08-40-45_938783_1416845/ports_by_node.json.lock
2025-08-23 08:40:45,940 - filelock - DEBUG - Lock 23456177321488 acquired on /tmp/ray/session_2025-08-23_08-40-45_938783_1416845/ports_by_node.json.lock
2025-08-23 08:40:45,940 - filelock - DEBUG - Attempting to release lock 23456177321488 on /tmp/ray/session_2025-08-23_08-40-45_938783_1416845/ports_by_node.json.lock
2025-08-23 08:40:45,940 - filelock - DEBUG - Lock 23456177321488 released on /tmp/ray/session_2025-08-23_08-40-45_938783_1416845/ports_by_node.json.lock
2025-08-23 08:40:45,940 - filelock - DEBUG - Attempting to acquire lock 23456177314480 on /tmp/ray/session_2025-08-23_08-40-45_938783_1416845/ports_by_node.json.lock
2025-08-23 08:40:45,940 - filelock - DEBUG - Lock 23456177314480 acquired on /tmp/ray/session_2025-08-23_08-40-45_938783_1416845/ports_by_node.json.lock
2025-08-23 08:40:45,940 - filelock - DEBUG - Attempting to release lock 23456177314480 on /tmp/ray/session_2025-08-23_08-40-45_938783_1416845/ports_by_node.json.lock
2025-08-23 08:40:45,940 - filelock - DEBUG - Lock 23456177314480 released on /tmp/ray/session_2025-08-23_08-40-45_938783_1416845/ports_by_node.json.lock
2025-08-23 08:40:45,940 - filelock - DEBUG - Attempting to acquire lock 23456177319952 on /tmp/ray/session_2025-08-23_08-40-45_938783_1416845/ports_by_node.json.lock
2025-08-23 08:40:45,940 - filelock - DEBUG - Lock 23456177319952 acquired on /tmp/ray/session_2025-08-23_08-40-45_938783_1416845/ports_by_node.json.lock
2025-08-23 08:40:45,940 - filelock - DEBUG - Attempting to release lock 23456177319952 on /tmp/ray/session_2025-08-23_08-40-45_938783_1416845/ports_by_node.json.lock
2025-08-23 08:40:45,940 - filelock - DEBUG - Lock 23456177319952 released on /tmp/ray/session_2025-08-23_08-40-45_938783_1416845/ports_by_node.json.lock
2025-08-23 08:40:47,386	INFO worker.py:1832 -- Started a local Ray instance. View the dashboard at 127.0.0.1:8265
✅ Ray initialized successfully!
📊 Ray cluster resources: {'GPU': 8.0, 'accelerator_type:H100': 1.0, 'node:10.65.15.77': 1.0, 'node:__internal_head__': 1.0, 'memory': 1951104470016.0, 'object_store_memory': 200000000000.0, 'CPU': 128.0}
🔧 Ray available resources: {'node:10.65.15.77': 1.0, 'memory': 1951104470016.0, 'GPU': 8.0, 'CPU': 128.0, 'object_store_memory': 200000000000.0, 'node:__internal_head__': 1.0, 'accelerator_type:H100': 1.0}
🔄 Testing remote function execution...
(raylet) 2025-08-23 08:41:01,587 - filelock - DEBUG - Attempting to acquire lock 23456175451264 on /tmp/ray/session_2025-08-23_08-40-45_938783_1416845/ports_by_node.json.lock
(raylet) 2025-08-23 08:41:01,587 - filelock - DEBUG - Lock 23456175451264 acquired on /tmp/ray/session_2025-08-23_08-40-45_938783_1416845/ports_by_node.json.lock
(raylet) 2025-08-23 08:41:01,587 - filelock - DEBUG - Attempting to release lock 23456175451264 on /tmp/ray/session_2025-08-23_08-40-45_938783_1416845/ports_by_node.json.lock
(raylet) 2025-08-23 08:41:01,587 - filelock - DEBUG - Lock 23456175451264 released on /tmp/ray/session_2025-08-23_08-40-45_938783_1416845/ports_by_node.json.lock
📨 Remote function result: Hello World from Ray worker!
🔄 Testing parallel execution...
📨 Hello Worker-0 from Ray worker!
📨 Hello Worker-1 from Ray worker!
📨 Hello Worker-2 from Ray worker!
✅ All Ray tests passed successfully!
🛑 Shutting down Ray...

With FLASHRL_CONFIG=fp8, it hangs:

python test_ray_init.py.
INFO 08-23 08:41:21 [__init__.py:239] Automatically detected platform cuda.
2025-08-23 08:41:23,243 - flash_rl.vllm_patch - DEBUG - Successfully patched the _process_weights_after_loading function of vllm
2025-08-23 08:41:23,243 - flash_rl.vllm_patch - DEBUG - Successfully patched kv_cache process_weights_after_loading
2025-08-23 08:41:23,243 - flash_rl - DEBUG - Patching vllm process_weights_after_loading... status: True
2025-08-23 08:41:23,243 - flash_rl.vllm_patch - DEBUG - Successfully patched vllm
2025-08-23 08:41:23,243 - flash_rl - DEBUG - Patching the vllm LLM to enable flash_rl quantization... status: True
2025-08-23 08:41:23,520 - flash_rl.vllm_patch - DEBUG - Successfully patched vllm Fp8LinearMethod create_weights
2025-08-23 08:41:23,521 - flash_rl - DEBUG - Patching the vllm fp8 linear... status: True
==================================================
Ray Initialization Test Script
==================================================
🚀 Starting Ray initialization test...
📡 Initializing Ray...
2025-08-23 08:41:23,578 - filelock - DEBUG - Attempting to acquire lock 23447456017584 on /tmp/ray/session_2025-08-23_08-41-23_577827_1426982/node_ip_address.json.lock
2025-08-23 08:41:23,578 - filelock - DEBUG - Lock 23447456017584 acquired on /tmp/ray/session_2025-08-23_08-41-23_577827_1426982/node_ip_address.json.lock
2025-08-23 08:41:23,578 - filelock - DEBUG - Attempting to release lock 23447456017584 on /tmp/ray/session_2025-08-23_08-41-23_577827_1426982/node_ip_address.json.lock
2025-08-23 08:41:23,578 - filelock - DEBUG - Lock 23447456017584 released on /tmp/ray/session_2025-08-23_08-41-23_577827_1426982/node_ip_address.json.lock
2025-08-23 08:41:23,578 - filelock - DEBUG - Attempting to acquire lock 23447456018112 on /tmp/ray/session_2025-08-23_08-41-23_577827_1426982/ports_by_node.json.lock
2025-08-23 08:41:23,578 - filelock - DEBUG - Lock 23447456018112 acquired on /tmp/ray/session_2025-08-23_08-41-23_577827_1426982/ports_by_node.json.lock
2025-08-23 08:41:23,578 - filelock - DEBUG - Attempting to release lock 23447456018112 on /tmp/ray/session_2025-08-23_08-41-23_577827_1426982/ports_by_node.json.lock
2025-08-23 08:41:23,579 - filelock - DEBUG - Lock 23447456018112 released on /tmp/ray/session_2025-08-23_08-41-23_577827_1426982/ports_by_node.json.lock
2025-08-23 08:41:23,579 - filelock - DEBUG - Attempting to acquire lock 23447456017584 on /tmp/ray/session_2025-08-23_08-41-23_577827_1426982/ports_by_node.json.lock
2025-08-23 08:41:23,579 - filelock - DEBUG - Lock 23447456017584 acquired on /tmp/ray/session_2025-08-23_08-41-23_577827_1426982/ports_by_node.json.lock
2025-08-23 08:41:23,579 - filelock - DEBUG - Attempting to release lock 23447456017584 on /tmp/ray/session_2025-08-23_08-41-23_577827_1426982/ports_by_node.json.lock
2025-08-23 08:41:23,579 - filelock - DEBUG - Lock 23447456017584 released on /tmp/ray/session_2025-08-23_08-41-23_577827_1426982/ports_by_node.json.lock
2025-08-23 08:41:23,579 - filelock - DEBUG - Attempting to acquire lock 23447456007696 on /tmp/ray/session_2025-08-23_08-41-23_577827_1426982/ports_by_node.json.lock
2025-08-23 08:41:23,579 - filelock - DEBUG - Lock 23447456007696 acquired on /tmp/ray/session_2025-08-23_08-41-23_577827_1426982/ports_by_node.json.lock
2025-08-23 08:41:23,579 - filelock - DEBUG - Attempting to release lock 23447456007696 on /tmp/ray/session_2025-08-23_08-41-23_577827_1426982/ports_by_node.json.lock
2025-08-23 08:41:23,579 - filelock - DEBUG - Lock 23447456007696 released on /tmp/ray/session_2025-08-23_08-41-23_577827_1426982/ports_by_node.json.lock
2025-08-23 08:41:23,579 - filelock - DEBUG - Attempting to acquire lock 23447456014896 on /tmp/ray/session_2025-08-23_08-41-23_577827_1426982/ports_by_node.json.lock
2025-08-23 08:41:23,579 - filelock - DEBUG - Lock 23447456014896 acquired on /tmp/ray/session_2025-08-23_08-41-23_577827_1426982/ports_by_node.json.lock
2025-08-23 08:41:23,579 - filelock - DEBUG - Attempting to release lock 23447456014896 on /tmp/ray/session_2025-08-23_08-41-23_577827_1426982/ports_by_node.json.lock
2025-08-23 08:41:23,579 - filelock - DEBUG - Lock 23447456014896 released on /tmp/ray/session_2025-08-23_08-41-23_577827_1426982/ports_by_node.json.lock
2025-08-23 08:41:23,579 - filelock - DEBUG - Attempting to acquire lock 23447456011584 on /tmp/ray/session_2025-08-23_08-41-23_577827_1426982/ports_by_node.json.lock
2025-08-23 08:41:23,579 - filelock - DEBUG - Lock 23447456011584 acquired on /tmp/ray/session_2025-08-23_08-41-23_577827_1426982/ports_by_node.json.lock
2025-08-23 08:41:23,579 - filelock - DEBUG - Attempting to release lock 23447456011584 on /tmp/ray/session_2025-08-23_08-41-23_577827_1426982/ports_by_node.json.lock
2025-08-23 08:41:23,579 - filelock - DEBUG - Lock 23447456011584 released on /tmp/ray/session_2025-08-23_08-41-23_577827_1426982/ports_by_node.json.lock
2025-08-23 08:41:29,690	INFO worker.py:1832 -- Started a local Ray instance. View the dashboard at 127.0.0.1:8265

I think a workaround could be using the ray head and worker using ray command, then submit the python job. It worked for me on the same environment with multi-node test.

[LiyuanLucasLiu]

Thanks for the additional information! I previously also met some issues when spawning the work via k8s. Will try to work on alternative ways for activating the patch.

[LZC-BELIEVER]

Thanks for the additional information! I previously also met some issues when spawning the work via k8s. Will try to work on alternative ways for activating the patch.

Hello!

I have also met the bug "core_worker_process.cc:224: Failed to register worker to Raylet: IOError: Failed to read data from the socket: End of file worker_id=01000000ffffffffffffffffffffffffffffffffffffffffffffffff" after I verified the proper install of FLASH_RL, is there a better way to fix that problem? Do I still need to manually "add import flash_rl before import vllm in every file that imports vLLM", or should I use the image that you've tested in the reply above.

Looking forward to your reply!