qat engine coredump in ASYNC_get_wait_ctx()

[wanlebing]

HI all：

I met a coredump with the backtrace below. The application calls ASYNC_start_job, and the crash occurs when RSA_private_encrypt returns 0.

#0  0x000000000096ff30 in ASYNC_get_wait_ctx ()
#1  0x00007f57352194a4 in qat_wake_job (job=<optimized out>, jobStatus=2) at qat_events.c:306
#2  0x00007f5734e33138 in LacPke_MsgCallback () from //opt/QAT/build/libqat_s.so
#3  0x00007f5734e58d13 in adf_user_notify_msgs_poll () from //opt/QAT/build/libqat_s.so
#4  0x00007f5734e5216c in adf_pollRing () from //opt/QAT/build/libqat_s.so
#5  0x00007f5734e525da in icp_adf_pollInstance () from //opt/QAT/build/libqat_s.so
#6  0x00007f5734e4cc2d in icp_sal_CyPollInstance () from //opt/QAT/build/libqat_s.so
#7  0x00007f573521c9a4 in qat_timer_poll_func (ih=<optimized out>) at qat_hw_polling.c:200
#8  0x00007f5739c081ca in start_thread () from /lib64/libpthread.so.0
#9  0x00007f57372398d3 in clone () from /lib64/libc.so.6

It appears the root cause is that ASYNC_start_job did not effectively start a valid ASYNC job at the application layer, but the underlying layer still generated a QAT asynchronous task. Consequently, qat_wake_job retrieved an invalid ASYNC job, leading to the core dump.

I wonder if this is a known issue, any help would be appreciated, thanks.

[Yogaraj-Alamenda]

Async_jobs are created and maintained by application and QAT Engine checks ASYNC_get_current_job() and if it has non-NULL pointer, it gets into Async Flow. That said if there is no valid ASync Job started then I would expect failure at qat_pause_job() but here it is failing during wake job which is strange. Can you please check what is happening during qat_pause_job() ?

[wanlebing]

Async_jobs are created and maintained by application and QAT Engine checks ASYNC_get_current_job() and if it has non-NULL pointer, it gets into Async Flow. That said if there is no valid ASync Job started then I would expect failure at qat_pause_job() but here it is failing during wake job which is strange. Can you please check what is happening during qat_pause_job() ?

sorry for the delay, here's our implementation:

firstly, in the application, external calls trigger internal execution:

ASYNC_start_job(&job->asyncjob, job->waitctx, &job_ret,
                               async_job_func, &args, sizeof(struct async_args)));
RSA_private_encrypt is called within async_job_func.

The call to RSA_private_encrypt within async_job_func succeeds, this is the expected scenario:

1. The call to ASYNC_start_job is expected to return ASYNC_PAUSE and initiate an asynchronous task within the QAT engine.
2. Once the QAT engine polls and completes the task, it notifies the application layer via a callback.
3. The application layer then invokes ASYNC_start_job(&job->asyncjob, job->waitctx, &ret, NULL, NULL, 0); again; this returns ASYNC_FINISH, at which point the asyncjob is freed by OpenSSL.

The call to RSA_private_encrypt within async_job_func fails, which is an abnormal scenario:

1. The call to async_job_func fails, and ASYNC_start_job immediately returns ASYNC_FINISH, Consequently, the asyncjob is freed by OpenSSL.
2. However, the QAT engine's asynchronous task is still generated. When the QAT engine polls for job completion and triggers the callback to notify the application layer, it cannot retrieve a valid job. This results in a core dump during qat_wake_job.

From what we can observe, that seems to be the case:

1. ASYNC_start_job initiates a job and triggers a QAT asynchronous task. The underlying execution begins, and the job context is switched out.
2. When the job switches back to execute the function, it fails, and the job is subsequently freed by OpenSSL.
3. qat_wake_job cannot retrieve the valid job, resulting in a core dump.

Thanks again.

[venkatesh6911]

@wanlebing I will check and come back soon.

[venkatesh6911]

@wanlebing Can you share the debug logs of the qatengine ? We have to see whether RSA_private_encrypt() had actually passed without any failures. This function could have resulted into a failure before submission to the QAT, causing OpenSSL to free the job while QAT driver still holds a reference to it.

[wanlebing]

@wanlebing Can you share the debug logs of the qatengine ? We have to see whether RSA_private_encrypt() had actually passed without any failures. This function could have resulted into a failure before submission to the QAT, causing OpenSSL to free the job while QAT driver still holds a reference to it.

This occurred in the production environment, so it's not convenient to enable qatengine debug logs right now. I will look into how to enable the logs to reproduce the issue.

Currently, in production, the only observation is that the problem arises when the ASYNC_start_job function fails.

[venkatesh6911]

Can you track the return status codes of the async start job ? We should see ASYNC_PAUSE and then ASYNC_FINISH for the QAT offload. If there is no ASYNC_PAUSE state received, then most likely there is a failure in the QAT submission path after the async callback is registered.

`switch (ASYNC_start_job(&job, waitctx, &ret, async_job_func, &args, sizeof(args))) {
case ASYNC_PAUSE:
// Job paused - QAT is working on it
// Need to wait for QAT to complete
break;

case ASYNC_FINISH:
job = NULL;
break;
}`

Also please check the FW counters of the QAT device during the crash to know if the request was submitted to the hardware,
cat /sys/kernel/debug/<qat_endpoint_id>/fw_counters .

[wanlebing]

Can you track the return status codes of the async start job ? We should see ASYNC_PAUSE and then ASYNC_FINISH for the QAT offload. If there is no ASYNC_PAUSE state received, then most likely there is a failure in the QAT submission path after the async callback is registered.

`switch (ASYNC_start_job(&job, waitctx, &ret, async_job_func, &args, sizeof(args))) { case ASYNC_PAUSE: // Job paused - QAT is working on it // Need to wait for QAT to complete break;

case ASYNC_FINISH: job = NULL; break; }`

Also please check the FW counters of the QAT device during the crash to know if the request was submitted to the hardware, cat /sys/kernel/debug/<qat_endpoint_id>/fw_counters .

When an error occurs, the return code for async start job is only ASYNC_FINISH.

[venkatesh6911]

So, this is most likely the case: There is an error in the submission path after the callback is registered with QAT and when polled for the response, there is no valid async wait context assigned by the OpenSSL because OpenSSL already cleaned it up.

Can you share details about your QAT hardware setup and config ?

• lspci | egrep "Co-p"
• adf_ctl status
• Snapshot of your device config file

We have to make sure that the hardware is setup and configured correctly first.

[wanlebing]

So, this is most likely the case: There is an error in the submission path after the callback is registered with QAT and when polled for the response, there is no valid async wait context assigned by the OpenSSL because OpenSSL already cleaned it up.

Can you share details about your QAT hardware setup and config ?

• lspci | egrep "Co-p"
• adf_ctl status
• Snapshot of your device config file

We have to make sure that the hardware is setup and configured correctly first.

The configuration is as follows:

lspci | egrep "Co-p"
76:00.0 Co-processor: Intel Corporation 4xxx Series QAT (rev 01)
78:00.0 Co-processor: Intel Corporation Device 2710
f3:00.0 Co-processor: Intel Corporation 4xxx Series QAT (rev 01)
f5:00.0 Co-processor: Intel Corporation Device 2710

/usr/local/bin/adf_ctl status
Checking status of all devices.
There is 2 QAT acceleration device(s) in the system:
qat_dev0 - type: 4xxx, inst_id: 0, node_id: 0, bsf: 0000:76:00.0, #accel: 1 #engines: 9 state: up
qat_dev1 - type: 4xxx, inst_id: 1, node_id: 1, bsf: 0000:f3:00.0, #accel: 1 #engines: 9 state: up

cat /etc/4xxx_dev0.conf

[GENERAL]
ServicesEnabled = asym;dc

ConfigVersion = 2

CyNumConcurrentSymRequests = 512
CyNumConcurrentAsymRequests = 64

statsGeneral = 1
statsDh = 1
statsDrbg = 1
statsDsa = 1
statsEcc = 1
statsKeyGen = 1
statsDc = 1
statsLn = 1
statsPrime = 1
statsRsa = 1
statsSym = 1

ServiceChainingEnabled = 0

HeartbeatTimer = 1000

StorageEnabled = 0

PkeServiceDisabled = 0

AutoResetOnError = 0

PmIdleSupport = 1

KptEnabled = 1

KptMaxSWKPerFn = 1

KptMaxSWKPerPASID = 1

KptMaxSWKLifetime = 31536000

KptSWKShared = 0

[KERNEL]
NumberCyInstances = 0
NumberDcInstances = 0

[SIOV]
NumberAdis = 0

[SHIM]
NumberCyInstances = 2
NumberDcInstances = 2
NumProcesses = 1
LimitDevAccess = 0

Cy0Name = "SSL0"
Cy0IsPolled = 1
Cy0CoreAffinity = 1

Cy1Name = "SSL1"
Cy1IsPolled = 1
Cy1CoreAffinity = 2

Dc0Name = "Dc0"
Dc0IsPolled = 1

Dc0CoreAffinity = 3

Dc1Name = "Dc1"
Dc1IsPolled = 1
Dc1CoreAffinity = 4

[venkatesh6911]

Can you share the following:

• Build commands for QAT Engine/Provider (./configure ....)
• Version of QAT Engine
• openssl version -a
• contents of openssl.cnf

[wanlebing]

Can you share the following:

• Build commands for QAT Engine/Provider (./configure ....)
• Version of QAT Engine
• openssl version -a
• contents of openssl.cnf

QAT build command：

    ./configure --with-qat_hw_dir=/$data/$qat_dir                                       \
                --enable-qat_sw --enable-qat_plock                                      \
                --with-openssl_dir=/$data/openssl-$openssl_version                      \
                --with-openssl_install_dir=$install_dir                                 \
                --with-qat_sw_crypto_mb_install_dir=/root/intel/ippcp_$ippcp_version/

versions：
openssl: 3.0.14
engine: 1.4.0
ippcp: 2021.12.1
qat: 20.L.1.1.50-00003

[venkatesh6911]

I could reproduce the crash.

0 0x00007f9f31ae86c0 in ASYNC_WAIT_CTX_get_fd () from /home/vej/openssl-3014_install/lib64/libcrypto.so.3
[Current thread is 1 (Thread 0x7f9f2f1ff640 (LWP 2894699))]
Missing separate debuginfos, use: dnf debuginfo-install glibc-2.34-83.el9_3.7.x86_64 libgcc-11.4.1-2.1.el9.x86_64 systemd-libs-252-18.el9.x86_64
(gdb) (gdb) (gdb)
Thread 2 (Thread 0x7f9f31fa1740 (LWP 2894698)):
warning: Unexpected size of section `.reg-xstate/2894698' in core file.
#0 0x00007f9f31713975 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1 0x00007f9f31718527 in nanosleep () from /lib64/libc.so.6
#2 0x00007f9f3171845e in sleep () from /lib64/libc.so.6
#3 0x000000000040183b in main (argc=1, argv=0x7ffdd2ed5fe8) at test_async_failure_inject.c:193

Thread 1 (Thread 0x7f9f2f1ff640 (LWP 2894699)):
#0 0x00007f9f31ae86c0 in ASYNC_WAIT_CTX_get_fd () from /home/vej/openssl-3014_install/lib64/libcrypto.so.3
#1 0x00007f9f31f2e14c in qat_wake_job (job=, jobStatus=2) at qat_events.c:326
#2 0x00007f9f31956421 in LacPke_MsgCallback () from /home/vej/QAT20.L.1.1.50-00003/build/libqat_s.so
#3 0x00007f9f3197ba41 in adf_user_notify_msgs_poll () from /home/vej/QAT20.L.1.1.50-00003/build/libqat_s.so
#4 0x00007f9f31977360 in adf_pollRing () from /home/vej/QAT20.L.1.1.50-00003/build/libqat_s.so
#5 0x00007f9f319777a5 in icp_adf_pollInstance () from /home/vej/QAT20.L.1.1.50-00003/build/libqat_s.so
#6 0x00007f9f3196fb62 in icp_sal_CyPollInstance () from /home/vej/QAT20.L.1.1.50-00003/build/libqat_s.so
#7 0x00007f9f31f31759 in qat_timer_poll_func (ih=) at qat_hw_polling.c:200
#8 0x00007f9f3169f802 in start_thread () from /lib64/libc.so.6
#9 0x00007f9f3163f450 in clone3 () from /lib64/libc.so.6

The test scenario which resulted in the crash is below.

1. Submitting 50 async operations rapidly
2. Freeing ASYNC_WAIT_CTX immediately without waiting for completion
3. Creating race condition where QAT callbacks access freed memory
4. Consistently produces SIGSEGV during cleanup phase

[venkatesh6911]

I got ASYNC_PAUSE return codes for all the started async jobs.
I am not sure why did not get them at all.

The primary reason for the crash is the incorrect async job lifecycle management in the application. When ASYNC_start_job returns ASYNC_PAUSE, you should wait for the operation to complete before freeing ASYNC_WAIT_CTX resources. The crash occurs because your application frees these contexts while QAT hardware operations are still pending.

// 1. Submit all operations and track them
for (i = 0; i < NUM_ATTEMPTS; i++) {
ASYNC_start_job(&jobs[i].job, jobs[i].waitctx, &job_ret, ...);
jobs[i].submitted = true;
}

// 2. Wait for each completion using select()
for (i = 0; i < NUM_ATTEMPTS; i++) {
if (jobs[i].submitted && !jobs[i].completed) {
// Get fd from ASYNC_WAIT_CTX
ASYNC_WAIT_CTX_get_fd(jobs[i].waitctx, ...);

    // Wait for notification
    select(waitfd + 1, &waitfdset, NULL, NULL, &timeout);
    
    // Resume to get ASYNC_FINISH
    ASYNC_start_job(&jobs[i].job, jobs[i].waitctx, &job_ret, NULL, NULL, 0);
    jobs[i].completed = true;
}

}

// 3. Only NOW safe to free contexts
for (i = 0; i < NUM_ATTEMPTS; i++) {
ASYNC_WAIT_CTX_free(jobs[i].waitctx); // Safe - operations complete
}

Please check if you are calling the ASYNC_WAIT_CTX_free() while there are QAT operations in-flight and you are resuming the job immediately once you get ASYNC_PAUSE.

[wanlebing]

ASYNC_FINISH

The observed crash occurs when the async job fails and returns ASYNC_FINISH directly, skipping the expected transition from ASYNC_PAUSE to ASYNC_FINISH.