newlog: sanitize non-Latin-1 chars in gzip header#5977
Merged
Conversation
When a log batch rolls over, newlogd writes the metadata header into the gzip Name field for app logs (the app's DisplayName) and the Comment field for device logs. RFC 1952 restricts both fields to NUL-free ISO-8859-1, and Go's gzip writer enforces this on Close() with the error "gzip.Write: non-Latin-1 header string". The error path is log.Fatal, so when an operator names an app with a CJK character, a smart quote, an emoji, an extended-Latin glyph like Łódź, or anything else outside Latin-1, newlogd dies the moment it tries to finalize the next gzip for that app -- which presents on the device as a reboot loop attributed to "agent newlogd". Sanitize the header before assigning. Runes outside 0x01-0xFF are rewritten to the literal text form of a Go Unicode escape (\uXXXX or \UXXXXXXXX); valid Latin-1 (including 0x80-0xFF accented characters) passes through. The substitution is reversible, so the cloud-side parser can still recover the original DisplayName if it chooses to. A warning is logged on the first sanitization so the operator can see which header was rewritten and why. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #5977 +/- ##
==========================================
+ Coverage 20.64% 21.06% +0.41%
==========================================
Files 489 499 +10
Lines 90373 92071 +1698
==========================================
+ Hits 18659 19393 +734
- Misses 70136 70918 +782
- Partials 1578 1760 +182 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
If an operator names an app instance with a character outside ISO-8859-1
(any CJK ideograph, a smart-quote, an em-dash, an emoji, an
extended-Latin glyph such as
Ł/č/ș, or anything else beyondLatin-1), newlogd panics with
the moment it tries to finalize the next per-app gzip in its
compression path. The agent reboot then surfaces on the controller as
with a fresh reboot every time the doomed app produces another batch.
Root cause:
prepareGzipToOutTempFileassigned the operator-suppliedDisplayName straight into
gzip.Writer.Name(app logs) orgzip.Writer.Comment(device logs). RFC 1952 restricts both fieldsto NUL-free ISO-8859-1, and Go's
compress/gzipenforces that onClose(). The fatal path is alog.FatalinfinalizeGzipToOutTempFile, so the single bad DisplayName takesnewlogd down on every gzip rotation, which presents to the operator
as a continuous reboot loop.
Fix: a small
sanitizeGzipHeaderhelper rewrites any rune outside0x01-0xFF to its literal-text Unicode-escape form (backslash-u-XXXX
for BMP, backslash-U-XXXXXXXX for supplementary plane, backslash-u-0000
for NUL). Valid Latin-1 characters (including 0x80-0xFF accents like
café-Ñandú) pass through unchanged. The substitution is reversible,so a controller can still recover the original DisplayName. A warning
is logged the first time a header is rewritten so the operator can
see what was changed.
Observed on a 14.5.2-lts device in the field; the offending code is
identical on master and on every active LTS branch.
How to test and validate this PR
pkg/newlog/cmd/util_test.go:TestSanitizeGzipHeadercovers ASCII, full Latin-1, CJK, smartquotes, extended-Latin (Łódź), BMP symbols, supplementary-plane
emoji, NUL bytes, and a mixed input.
TestSanitizeGzipHeaderRoundTripsThroughGzipproves a sanitizedheader is accepted by
gzip.Writer.Close()and is recoverableon read-back -- this is the direct regression test for the field
crash.
TestUnsanitizedHeaderCrashesGzippins the precondition that anunsanitized non-Latin-1
gzip.Writer.Namedoes failClose(),so the test will surface it if Go ever loosens the rule.
DisplayName contains a non-Latin-1 character (e.g. an emoji), and
confirm that newlogd no longer reboots when that app's log batch
rolls over.
Changelog notes
Fix newlogd reboot loop on devices running an app whose name contains
non-Latin-1 characters (CJK, emoji, smart quotes, extended-Latin
glyphs, etc.).
PR Backports
Checklist