Skip to content

Commit d5e6e1e

Browse files
committed
[fix](rest-s3)Set AWS Request Checksum Calculation only if not set, avoid issues on non-S3 storage
AWS SDK for Java v2 uses the system property AwsSystemSetting.AWS_REQUEST_CHECKSUM_CALCULATION to control request payload checksum calculation. ↳ According to the official discussion AWS SDK v2 Discussion #5802 : ↳ By default, the SDK may automatically calculate checksums when required. ↳ For AWS S3, setting the default value "WHEN_REQUIRED" ensures checksums are calculated only when necessary, providing both compatibility and performance benefits. ↳ For non-S3 object storage (such as COS, OSS, OBS), using this default value may cause errors, for example: ↳ aws-chunked encoding is not supported with the specified x-amz-content-sha256 value because these services do not support AWS-specific chunked encoding or SHA256 checksum.
1 parent 1d8a5a1 commit d5e6e1e

File tree

1 file changed

+8
-2
lines changed

1 file changed

+8
-2
lines changed

fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/IcebergRestProperties.java

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -338,9 +338,15 @@ public boolean isIcebergRestNestedNamespaceEnabled() {
338338
*/
339339
public void toFileIOProperties(List<StorageProperties> storagePropertiesList,
340340
Map<String, String> fileIOProperties, Configuration conf) {
341+
341342
for (StorageProperties storageProperties : storagePropertiesList) {
342-
// For other storage types, just use fileIOProperties map
343-
conf.addResource(storageProperties.getHadoopStorageConfig());
343+
if (storageProperties instanceof AbstractS3CompatibleProperties) {
344+
// For all S3-compatible storage types, put properties in fileIOProperties map
345+
toS3FileIOProperties((AbstractS3CompatibleProperties) storageProperties, fileIOProperties);
346+
} else {
347+
// For other storage types, just use fileIOProperties map
348+
conf.addResource(storageProperties.getHadoopStorageConfig());
349+
}
344350
}
345351

346352
}

0 commit comments

Comments
 (0)