Skip to content

Error while generating traces for GPU_Microbenchmark #491

@PrabinKuSabat

Description

@PrabinKuSabat

While trying to generate traces from a RTX4090 card using the gpu-app-collection-partial collection I run into this error invoking nvbit on the traces. But, I'm able to generate traces for rodinia_2.0-ft app collection.
The final output with the instruction run looks like this:

(base) cab-prj@dmacs-3gpunode:~/Prabin/4Try/accel-sim-framework$ ./util/tracer_nvbit/run_hw_trace.py -B GPU_Microbenchmark
Running l1_bw_32f
------------- NVBit (NVidia Binary Instrumentation Tool v1.7.6) Loaded --------------
NVBit core environment variables (mostly for nvbit-devs):
ACK_CTX_INIT_LIMITATION = 0 - if set, no warning will be printed for nvbit_at_ctx_init()
            NVDISASM = nvdisasm - override default nvdisasm found in PATH
            NOBANNER = 0 - if set, does not print this banner
       NO_EAGER_LOAD = 0 - eager module loading is turned on by NVBit to prevent potential NVBit tool deadlock, turn it off if you want to use the lazy module loading feature
---------------------------------------------------------------------------------
         INSTR_BEGIN = 0 - Beginning of the instruction interval where to apply instrumentation
           INSTR_END = 4294967295 - End of the instruction interval where to apply instrumentation
    EXCLUDE_PRED_OFF = 1 - Exclude predicated off instruction from count
      TRACE_LINEINFO = 0 - Include source code line info at the start of each traced line. The target binary must be compiled with -lineinfo or --generate-line-info
DYNAMIC_KERNEL_RANGE =  - Specify kernel IDs or ranges to trace. Format:
  - Single ID:       "2" traces only kernel 2.
  - Range:           "5-8" traces kernels 5 through 8 (inclusive).
  - Open-ended:      "10-" traces from kernel 10 onward.
  - Multiple ranges: "2 5-8 10-" (space-separated).
  - With regex:      "5-8@kernel_a.*,kernel_b.*" traces kernels 5–8 with matching names.
If unset or empty, all kernels will be traced from the beginning.
   ACTIVE_FROM_START = 1 - Start instruction tracing from start or wait for cuProfilerStart and cuProfilerStop. If set to 0, DYNAMIC_KERNEL_RANGE options have no effect
        TOOL_VERBOSE = 0 - Enable verbosity inside the tool
       TOOL_COMPRESS = 1 - Enable traces compression
     TOOL_TRACE_CORE = 0 - write the core id in the traces
TERMINATE_UPON_LIMIT = 0 - Stop the process once the current kernel > DYNAMIC_KERNEL_LIMIT_END
USER_DEFINED_FOLDERS = 0 - Uses the user defined folder TRACES_FOLDER path environment
 TRACE_FILE_COMPRESS = 1 - Create xz-compressed tracefile
ENABLE_SPINLOCK_FAST_FORWARD = 0 - Enable spinlock fast forwarding
SPINLOCK_ITER_TO_KEEP = 1 - Number of iterations to keep for spinlock fast forwarding
----------------------------------------------------------------------------------------------------
WARNING: Do not call CUDA memory allocation in nvbit_at_ctx_init(). It will cause deadlocks. Do them in nvbit_tool_init(). If you encounter deadlocks, remove CUDA API calls to debug.
SM_NUMBER: 128
WARP_SIZE: 32
MAX_THREADS_PER_SM: 1536
MAX_SHARED_MEM_SIZE: 102400
MAX_WARPS_PER_SM: 48
MAX_REG_PER_SM: 65536
MAX_THREAD_BLOCK_SIZE: 1024
MAX_SHARED_MEM_SIZE_PER_BLOCK: 49152
MAX_REG_PER_BLOCK: 65536
L1_SIZE: 49152
L2_SIZE: 75497472
MEM_SIZE: 25386352640
MEM_CLK_FREQUENCY: 10501
MEM_BITWIDTH: 384
CLK_FREQUENCY: 1500
THREADS_PER_BLOCK: 1024
BLOCKS_PER_SM: 1
THREADS_PER_SM: 1024
BLOCKS_NUM: 128
TOTAL_THREADS: 131072
Warning: Ignoring CALL.REL.NOINC (NVBit 1.7.4 bug)
Writing results to /home/cab-prj/Prabin/4Try/accel-sim-framework/hw_run/traces/device-0/12.4/l1_bw_32f/__tpb_1024___tpsm_1024___blocks_1___ws_32/traces//kernel-1-ctx_0x55fdf5349750.trace.xz
L1 bandwidth = 0.0104964(byte/clk/SM), 0.0162763(GB/s/SM)
Total Clk number = 6393507114
Error invoking nvbit on /home/cab-prj/Prabin/4Try/accel-sim-framework/hw_run/traces/device-0/12.4/l1_bw_32f/__tpb_1024___tpsm_1024___blocks_1___ws_32

What can be the reason behind this happening and is there any work-around or solutions to this?

Thank you!

Metadata

Metadata

Assignees

No one assigned

    Labels

    accel-simIssues related to accel-sim frontend and tracing.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions