Memory leak with CRT S3 Async Client #946
Description
Describe the bug
While using aws-crt-java to retrieve object in S3 and store them on filesystem using S3 CRT Async Client, I have detected a memory leak in libaws-crt-jni.so. Java Native Memory Tracking allows me to find some lack in releasing native memory.
Regression Issue
- Select this option if this issue appears to be a regression.
Expected Behavior
I expect not to see memory usage growing while using CRT S3 Async Client
Current Behavior
As we can see on the following timeserie, the memory usage RSS is growing steadily
Reproduction Steps
The following code simulate the use case allowing me to detect the leak: retrieving multiple objects from S3 in parallel, use them then delete them and reiterate the process.
import java.io.IOException;
import java.nio.file.Path;
import java.nio.file.Files;
import java.util.UUID;
import java.util.concurrent.CompletableFuture;
import java.util.logging.Level;
import java.util.logging.Logger;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3AsyncClient;
import software.amazon.awssdk.services.s3.model.ChecksumMode;
import software.amazon.awssdk.services.s3.model.GetObjectRequest;
public class CRTLeak {
/**
* @param args the command line arguments
*/
public static void main(String[] args) {
final String BUCKET = "";
final String KEY = "";
S3AsyncClient client = S3AsyncClient.crtBuilder()
.targetThroughputInGbps(16.0)
.maxNativeMemoryLimitInBytes(1024 * 1024 * 1024 * 2L)
.forcePathStyle(true)
.region(Region.EU_WEST_1)
.build();
for (int i=0; i<10000; i++){
Path destPath = Path.of(UUID.randomUUID().toString()).toAbsolutePath();
CompletableFuture.runAsync(() -> getAndDeleteFile(client, destPath));
}
}
private static void getAndDeleteFile(S3AsyncClient client, Path destPath) {
client.getObject(GetObjectRequest
.builder()
.checksumMode(ChecksumMode.ENABLED)
.bucket(System.getenv("BUCKET"))
.key(System.getenv("KEY"))
.build(),
destPath).thenAccept(r -> {
try {
/**
* Some code here to use the result, and the file at destPath
*/
Files.delete(destPath);
getAndDeleteFile(client, destPath);
} catch (IOException ex) {
Logger.getLogger(CRTLeak.class.getName()).log(Level.SEVERE, null, ex);
}
});
}
}
Possible Solution
Using Java Native Memory Tracking allowed me to find some possible reasons of the leak. Here is a summary of that
Native Memory Tracking:
(Omitting categories weighting less than 1KB)
Total: reserved=4849469KB +562630KB, committed=3119265KB +576258KB
When digging into the detail we got 2 potential issues in libaws-crt-jni.so
[0x0000ffff628f7c4c] in AWSCRT_1292423841223742280libaws-crt-jni.so+0x55c4c
[0x0000ffff629176bc]Java_software_amazon_awssdk_crt_s3_S3Client_s3ClientMakeMetaRequest+0x2b8 in AWSCRT_1292423841223742280libaws-crt-jni.so
[0x0000ffff8873dc18]
(malloc=234275KB type=Internal +232818KB #4918161 +4887608)
[0x0000ffff62917794]Java_software_amazon_awssdk_crt_s3_S3Client_s3ClientMakeMetaRequest+0x390 in AWSCRT_1292423841223742280libaws-crt-jni.so
[0x0000ffff8873dc18]
(malloc=96058KB type=Internal +95461KB #4918161 +4887608)
As I identified some leaks I am working on a PR, I will be able to submit soon.
Additional Information/Context
No response
aws-crt-java version used
0.39.4
Java version used
Java 21 on Amazon Corretto 21.0.9-al2023-headless
Operating System and version
Amazon Corretto 21.0.9-al2023-headless docker running on Ubuntu 22.04