Fix unauth-SSRC memory grow

[sirzooro]

Fix unauthenticated-SSRC memory exhaustion and add security-warning logging

Summary

This change addresses two classes of vulnerability where unauthenticated data in RTP/RTCP headers could be used to allocate unbounded per-SSRC state before the auth tag is verified.

Security Fixes

Unauthenticated SSRC state allocation (memory exhaustion)

srtp.go / srtcp.go — getSRTPSSRCState and getSRTCPSSRCState now accept a keepNew bool parameter. During decryption the SSRC is read from the unauthenticated RTP/RTCP header, so these functions are called in read-only mode (keepNew = false): a temporary state object is created but not inserted into the map. The state is committed to the map by the new setSRTPSSRCState / setSRTCPSSRCState helpers only after markAsValid() succeeds (i.e. authentication is confirmed).

session_srtp.go — decryptRTP is now called before getOrCreateReadStream. Previously an unauthenticated peer could cause getOrCreateReadStream to allocate a per-SSRC stream for an arbitrary spoofed SSRC, exhausting memory without ever providing a valid packet.

Both fixes include inline comments explaining the security rationale and the intentional ordering of the replay check relative to authentication.

What Changed

context.go

• Added an upfront validation: authTagRTPLen < 0 returns errTooShortSRTPAuthTag before the profile auth-key length is queried.
• Bug fix: errShortSrtpMasterKey error message was accidentally printing the raw masterKey slice instead of its length masterKeyLen.
• Added setSRTPSSRCState and setSRTCPSSRCState helpers for explicit post-auth state commitment.
• getSRTPSSRCState and getSRTCPSSRCState return a second bool indicating whether the state was pre-existing.
• SetROC and SetIndex updated to pass keepNew = true (they always want to create state).

session_srtp.go

• Moved decryptRTP call above getOrCreateReadStream to prevent unauthenticated stream allocation (see Security Fixes above).

srtcp.go / srtp.go

• Decryption paths updated to use getSRTCPSSRCState(ssrc, false) / getSRTPSSRCState(ssrc, false) with deferred setSRTCPSSRCState / setSRTPSSRCState after authentication.
• Encryption paths updated to use getSRTCPSSRCState(ssrc, true) / getSRTPSSRCState(ssrc, true) (no behaviour change for encrypt).

API Notes

• getSRTPSSRCState / getSRTCPSSRCState signatures changed (internal); callers outside the package are unaffected.

Tests

• All existing SRTP/SRTCP unit tests pass with go test ..

Reviewer Notes

• The core security invariant is: no per-SSRC state (map entry, replay detector, or stream) is committed until the packet's auth tag has been successfully verified.
• The replay check remains intentionally before authentication to avoid wasting CPU on AES/HMAC for flooded duplicate packets; safety is preserved because the replay detector only records an index as "seen" when markAsValid() is explicitly called.

[codecov]

Codecov Report

❌ Patch coverage is 95.45455% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.42%. Comparing base (51804b0) to head (814aebc).

Files with missing lines	Patch %	Lines
context.go	93.33%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #372   +/-   ##
=======================================
  Coverage   82.42%   82.42%           
=======================================
  Files          19       19           
  Lines        1462     1474   +12     
=======================================
+ Hits         1205     1215   +10     
- Misses        142      143    +1     
- Partials      115      116    +1     

Flag	Coverage Δ
go	82.42% <95.45%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Sean-Der]

Nice find!

@sirzooro would you mind adding a test to assert no allocations on auth failure.

IMO the warning on NULL crypto isn’t needed. If we add to much noise then maybe something real would get ignored.

[Sean-Der]

Does doing replay detection without authentication allow a DoS? I can spam RTP packets and break sessions now without keying materials

[sirzooro]

Nice find!

@sirzooro would you mind adding a test to assert no allocations on auth failure.

IMO the warning on NULL crypto isn’t needed. If we add to much noise then maybe something real would get ignored.

Added new tests. Is there some valid production use of NULL cipher beside debugging? If yes, I would add another option to silence these warnings. I would like to have at least warning if someone would accidentally use insecure configuration in production environment.

Does doing replay detection without authentication allow a DoS? I can spam RTP packets and break sessions now without keying materials

I do not see anything obvious in the code, AI scan also did not reveal anything related to this. Maybe this is caused by increased CPU usage (higher jitter) or receive UDP buffer was overflown and some frames were dropped?

[sirzooro]

@Sean-Der @JoTurk is something more needed here?

[JoTurk]

i looked yesterday and it looks good to me, but i didn't really review it, I can review it tonight, @Sean-Der what do you think?

[Sean-Der]

@sirzooro Can we reduce it down to just fix unauth-SSRC memory grow

I would like to still keep single purpose commits (even if AI can do big changes easily) especially in these security changes. Sorry for the push back :/

I think we should merge/tag that right away!

[sirzooro]

@Sean-Der I have removed logging changes. PR contains main fix for unauth-SSRC memory grow, and two other small ones in context.go (first two points in description above). I also updated PR's description to match actual changes.