Skip to content

HDFS-17639. hasStorageType() allocates array unnecessarily on every call#8316

Open
balodesecurity wants to merge 1 commit intoapache:trunkfrom
balodesecurity:HDFS-17639
Open

HDFS-17639. hasStorageType() allocates array unnecessarily on every call#8316
balodesecurity wants to merge 1 commit intoapache:trunkfrom
balodesecurity:HDFS-17639

Conversation

@balodesecurity
Copy link

Problem

DatanodeDescriptor.hasStorageType() delegates to getStorageInfos() which:

  1. Acquires the storageMap lock
  2. Copies all storage values into a new DatanodeStorageInfo[] array
  3. Releases the lock
  4. Returns the array

hasStorageType() then iterates the returned array. This means every call allocates a new array and acquires the lock twice (the second time is reentrant since callers like injectStorage() already hold the lock). The same pattern exists in getStorageTypes().

On large clusters with many storages per DataNode, hasStorageType() is called frequently — during block placement, heartbeat processing, and topology updates — making this allocation and double-lock pattern a measurable source of lock contention and GC pressure.

Fix

In both hasStorageType() and getStorageTypes(), iterate storageMap.values() directly under a single synchronized (storageMap) block, eliminating the array allocation entirely. Callers that already hold the lock (e.g. injectStorage(), updateStorage()) benefit from Java's reentrant semantics — a single lock acquisition instead of two.

Testing

Added TestDatanodeDescriptor#testHasStorageTypeAndGetStorageTypes:

  • Creates a DatanodeDescriptor and verifies no type is present before any storage is injected
  • Injects a DISK storage and asserts hasStorageType(DISK) returns true and hasStorageType(SSD) returns false
  • Injects an SSD storage and asserts both types are now present via both hasStorageType() and getStorageTypes()
Tests run: 1, Failures: 0, Errors: 0
testHasStorageTypeAndGetStorageTypes — PASSED (0.058s)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants