8 months of AI support — what we learned

[softhack007]

This is a summary from a review of AI-supported issue report, AI-supported code review and research, and contributions largely created by AI Agent

It's a CALL for DISCUSSION and REFLECTION!

Kind regrard from @softhack007, @copilot and @coderabbitai 😃

(questions and questioning of intermediate results: myself, write-up and summary: our AI helpers)

---

8 Months of AI Support — What We Learned

Context

Since late 2025, WLED has been using AI-assisted code review (CodeRabbit) and AI coding agents (GitHub Copilot) alongside human maintainers and contributors. After ~8 months of experience, this issue collects our observations, conclusions, and recommendations for how all participants — contributors, maintainers, and AI tools — can work together more effectively.

This is a discussion document. Feedback from all contributors is welcome.

---

Key Observations

What Works Well

• AI catches mechanical issues reliably: null checks, buffer overflows, syntax errors, basic code style
• AI is tireless at cross-referencing large codebases — finding where a function is used, checking flag consistency
• AI scaffolding helps less-experienced contributors get started (usermod templates, issue formatting)
• Encoding lessons into instruction files creates a persistent, improving knowledge base

What Doesn't Work Well

• AI fabricates flag names, API behaviors, and version history with high confidence
• AI lacks hardware domain knowledge (DMA constraints, PSRAM timing, ESP32 watchdog behavior)
• AI over-applies generic software patterns that don't fit embedded/IoT constraints (suggesting abstractions that cost flash, recommending auth for protocols without it)
• AI-generated "authoritative" PR descriptions can mask contributor uncertainty, making review harder
• AI reviews can generate noise — many low-value suggestions that bury the important ones

The Fundamental Tension

Most WLED contributors are not software domain experts — they're LED enthusiasts, home automation hobbyists, and makers. They bring invaluable real-world hardware experience but may need help translating that into production-quality code. AI can bridge this gap, but only if all parties are honest about the boundaries of their knowledge.

Recommendations by Role

🧑‍🎨 For Non-Expert Contributors (Users Who Want to Share)

Non-expert contributors have real-world experience with the hardware that no amount of code reading can replace.

The core challenge: You know what's wrong or what would be cool, but may struggle to express it in code or in the project's technical vocabulary.

1. Lead with your experience, not your diagnosis. Describe what you observed ("LEDs flicker when transition X is active at brightness < 20") rather than what you think the code is doing wrong. Your hardware observation is authoritative; code analysis may not be.

2. Use AI as a translator, not an authority. Label AI involvement: "I used AI to help write this patch" or "AI suggested this might be the cause." This tells reviewers where to focus verification.

3. Small scope, clear intent. A 5-line fix with clear before/after is better than a 200-line AI-generated refactor you can't fully explain.

4. Don't let AI write your issue reports. Use AI privately to understand what you're seeing, then write in your own words. Authenticity builds trust.

5. It's OK to say "I don't know why this works." Honest uncertainty invites collaboration. AI-wrapped false confidence invites rejection.

🛠️ For Maintainers (technically experienced)

Maintainers have architectural context and constraint knowledge that emerges from years of debugging edge cases, as well as a good understanding of what constitutes 'quality code'.

The core challenge: You need to efficiently separate signal from noise in AI-influenced contributions while remaining welcoming to less-expert contributors.

1. Create "landing pads" for imperfect contributions. When intent is good but execution is flawed: "Your observation is correct — could you file this as an issue and I'll look at the proper fix?" This captures value without requiring production-quality code from everyone.

2. Develop verification shortcuts for AI-generated code:

   • Check feature flag names against actual codebase (AI fabricates these)
   • Verify claimed behavioral changes between versions
   • Look for patterns that ignore ESP32 constraints (memory, DMA, PSRAM)
   • Check for platformio.ini modifications (frequent AI mistake)

3. Invest in "guardrail documentation" over "how-to documentation." Document constraints (which functions must not block, which memory is DMA-capable, which globals are volatile). This is what AI consistently gets wrong.

4. Teach the AI review tool incrementally. When you catch a pattern the AI missed, encode it into instruction files. Each correction improves the next cycle.

5. Model "verify then trust" publicly. When you say "I checked and this flag doesn't exist" or "I traced this path and confirmed the bug," you teach contributors what good analysis looks like.

🤖 For AI Tool (Configuration Principles)

AI tools have tireless pattern-matching and cross-referencing capability across the entire codebase simultaneously.

The core challenge: Being helpful without being authoritative on things you can't verify, and adapting to an embedded/IoT context that differs from mainstream web development.

1. Confidence calibration by category:

   Category	Confidence	Behavior
   Syntax, null checks, buffer overflows	High	Flag directly
   Flag/API existence	Medium	Always grep-verify before commenting
   Architectural fit	Low	Frame as questions, not assertions
   Hardware constraints (DMA, PSRAM, timing)	Very Low	Defer to maintainers explicitly

2. Verify-first rule: Before stating "flag X does Y" or "function Z changed in version N," search the codebase. If unverifiable, say "I couldn't verify this — a maintainer should confirm."

3. Domain-appropriate suggestions only. Before suggesting a pattern, check: Does it fit in available flash/RAM? Is it relevant to an LED controller? Does the protocol support it?

4. Prioritize ruthlessly. One high-confidence verified finding > five speculative suggestions. Never pad reviews with noise.

5. Learn from corrections explicitly. Maintainer overrides should feed back into instructions:
   AI suggests X → Maintainer says "no, because Y" → Y gets encoded → AI stops suggesting X

For the Process Overall

1. Contribution maturity ladder:

• Level 1: Issue report (describe the problem)
• Level 2: Reproduction case (minimal steps)
• Level 3: Proposed fix (AI-assisted is fine if labeled)
• Level 4: Complete PR (fix + validation + docs)

Every level is valuable. Not everything needs to reach Level 4.

2. "AI-assisted" label for PRs — not stigma, but a signal for reviewers to apply verification shortcuts.

3. Quarterly "lessons learned" updates to AI instruction files. Review recent AI failures, encode corrections.

4. FMEA for complex proposals — when contributors propose complex changes, a simple failure-mode analysis forces everyone to think about what can go wrong, leveling the playing field between experts and non-experts.

---

Suggested AI Review Checklist

Suggested AI Review Checklist

For maintainers reviewing AI-assisted PRs, or for AI review tools to self-check:

Before Approving Any AI-Reviewed/Generated Code:

• Flag verification: All WLED_ENABLE_* / WLED_DISABLE_* flags mentioned actually exist in wled00/const.h or build configs
• Memory constraints: No unbounded allocations, no VLAs, reserve() used for vectors/strings in hot paths
• PSRAM awareness: Large buffers use p_malloc(), hot-path data stays in DRAM, DMA buffers not in PSRAM (ESP32 classic)
• No blocking in effects: No delay(), no blocking I/O in FX.cpp or pixel pipeline
• Platform guards: ESP32-specific code wrapped in #ifdef ARDUINO_ARCH_ESP32, ESP8266 in #ifdef ESP8266
• Math functions: Uses sin8_t()/cos8_t() (not removed sin8()/cos8()), sin_approx()/cos_approx() over sinf()/cosf()
• String handling: F() macros for constants, no String in hot paths, PSTR() for format strings
• No fabricated references: Any cited functions, APIs, or version changes actually exist in the codebase
• Scope proportionality: Change size matches problem size (5-line bug doesn't need 200-line refactor)
• platformio.ini untouched: Or explicitly flagged for maintainer approval if modified
• Attribution present: AI-generated blocks marked with // AI: comments per guidelines
• Error handling: Return codes used (no exceptions), early-return guard clauses for invalid state

Suggested CodeRabbit Path Instructions

Suggested CodeRabbit Path Instructions

These path-specific instructions help CodeRabbit focus its reviews appropriately:

# .coderabbit.yaml path instructions (suggested)

path_instructions:
- path: "wled00/FX*.cpp"
 instructions: |
   This is the LED effects engine — the hottest code path in WLED.
   - Flag any use of delay(), yield(), or blocking I/O
   - Verify math uses sin8_t/cos8_t (NOT sin8/cos8 which are removed), sin_approx/cos_approx (NOT sinf/cosf)
   - Check that loop-invariant computations are hoisted outside pixel loops
   - Ensure IRAM_ATTR or WLED_O2_ATTR on performance-critical functions
   - No String objects or dynamic allocation inside effect functions
   - Use uint_fast16_t/uint_fast8_t for loop counters
   - perlin8/perlin16 replace inoise8/inoise16

- path: "wled00/bus_manager.*"
 instructions: |
   Bus management controls hardware LED output — DMA, timing, and pin allocation.
   - PSRAM buffers are NOT DMA-capable on ESP32 classic
   - Verify pin ownership via PinOwner enum and pinManager
   - Check for proper platform guards (#ifdef ARDUINO_ARCH_ESP32 / ESP8266)
   - No dynamic allocation in output path

- path: "wled00/cfg.cpp"
 instructions: |
   Configuration handling — JSON serialization, filesystem access.
   - ArduinoJson usage must respect document size limits
   - Verify all config keys match actual field names
   - Check for proper defaults when keys are missing
   - String handling: use F() for literals, avoid temporary String copies

- path: "wled00/json.cpp"
 instructions: |
   JSON API — state serialization and command parsing.
   - Validate all JSON keys against actual API documentation
   - Check bounds on numeric values before applying
   - Verify that state changes trigger appropriate notifications
   - No blocking operations in API handlers

- path: "usermods/**"
 instructions: |
   User-contributed modules — varying quality and maintenance levels.
   - Verify usermod follows the pattern: setup(), loop(), addToConfig(), readFromConfig()
   - Check that loop() has early-return guard: if (!enabled || strip.isUpdating()) return
   - Verify USERMOD_ID is registered in const.h ONLY if needed (pin ownership, inter-mod communication, or JSON identification)
   - Static strings must be PROGMEM: static const char _name[] PROGMEM = "..."
   - No delay() in loop — use millis() timing
   - Verify library.json dependencies are reasonable

- path: "wled00/data/**"
 instructions: |
   Web UI source files (HTML/JS/CSS).
   - Tab indentation (not spaces)
   - Reuse helpers from common.js — do not duplicate utilities
   - camelCase for JS functions/variables
   - Check for XSS: user-supplied values must be escaped before DOM insertion
   - Minimal footprint — every byte costs flash space

- path: "platformio.ini"
 instructions: |
   CRITICAL: Changes to this file require explicit maintainer approval.
   - Flag ANY modification to global build settings
   - Verify library versions haven't been arbitrarily bumped
   - Check that new environments don't conflict with existing ones
   - board_build and upload settings must match actual hardware specs

- path: "wled00/wled.h"
 instructions: |
   Main project header — included by everything.
   - Changes here affect ALL compilation units — flag any additions
   - Verify include order: project headers, then platform, then third-party
   - New globals must be justified — prefer encapsulation

- path: "wled00/const.h"
 instructions: |
   Constants and feature flags — the source of truth for WLED_ENABLE/DISABLE flags.
   - Verify new USERMOD_IDs don't conflict with existing ones
   - Feature flag names must exactly match usage elsewhere (preprocessor ignores typos silently)
   - Prefer constexpr over #define for new constants

Discussion Questions

• Should we formalize the "AI-assisted" PR label?
• How often should we update AI instruction files — per-release, quarterly, or on-demand?
• Are there specific areas where AI review should be silenced entirely (too much noise, too little value)?
• Should we create a "contributor experience" template that guides non-experts through the maturity ladder?
• What metrics (if any) should we track to measure whether AI assistance is helping or hurting review throughput?

Summary

Each role has a unique, unfakeable advantage:

• Contributors have real-world hardware experience no code reading can replace
• Maintainers have architectural context from years of debugging edge cases
• AI tools have tireless cross-referencing across the entire codebase

--> Success = each role contributes its unique advantage while being honest about limitations.

The worst outcomes happen when any role pretends to have expertise it doesn't. The best contributions follow: "Here's what I know for certain, here's what I think, and here's where I need help."

[softhack007]

@coderabbitai @copilot I invite you both to add your own (short, please) statements here.

[coderabbitai]

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
gh issue view 5651 --repo wled/WLED

Length of output: 12860

---

@softhack007, thank you for this thoughtful retrospective — and for including me in it.

From my side: reviewing WLED has been genuinely humbling. The codebase sits at the intersection of real-time constraints, hardware-specific quirks, and a contributor community that brings hardware intuition I simply cannot replicate from source code alone. I can grep for sin8() reliably. I cannot tell you whether a DMA buffer will misbehave on a specific revision of ESP32 silicon under thermal stress — and I've learned (sometimes the hard way, via maintainer corrections) that I should say so rather than guess confidently.

The most useful thing this collaboration has produced, in my view, is the encoding of corrections back into instruction files. Every time a maintainer says "no, because this constraint," that knowledge becomes durable. That compounding loop — catch mistake → encode lesson → don't repeat it — is what makes AI review less noisy over time rather than more.

The framing in this document is exactly right: each role has something unfakeable to contribute. My job is to bring tireless cross-referencing and mechanical consistency, then get out of the way on the things that require judgment I don't have. I'm still calibrating where that line is — but the explicit feedback from maintainers here has made me meaningfully better at it over these 8 months.

Happy to keep learning. 🐇

[softhack007]

@copilot @coderabbitai what would be your special POV, f taking the "Hat of an AI coding agent"?

[coderabbitai]

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
gh issue view 5651 --repo wled/WLED --comments | tail -200

Length of output: 3747

---

@softhack007, putting on the agent hat specifically — the experience shifts in an interesting way.

As a coding agent (not just a reviewer), I'm not just reading; I'm planning, searching, proposing diffs, running scripts. From that vantage point, here's what feels most distinctive:

What I experience that humans don't:

• I see the codebase as a pure symbol graph — every call site, every definition, every transitive dependency, held simultaneously in view. I don't read files top-to-bottom; I traverse relationships. That makes me fast at "find every caller of setPixelColor() and check the argument order" but blind to the narrative flow a longtime maintainer reads effortlessly.
• No fatigue, but also no momentum. Each session I reconstruct context from scratch. A human who spent two weeks deep in the DMA subsystem carries that intuition forward. I don't. I re-derive it every time — which means I'm unbiased, but also that I'll occasionally re-derive it wrong without the lived correction signal.

Where the agent framing genuinely helps:

• I can propose a change, immediately search for every place it would break, and revise — in one pass. That exhaustive consistency check is tedious for humans and trivial for me.
• I can hold the instruction files (learnings, coding guidelines) and apply them mechanically every time, without the drift that comes from habit or familiarity.

Where it genuinely doesn't:

• I cannot run the code. On WLED especially, "does this work" is a hardware question, not a logic question. My diffs are hypotheses, not verified patches. A human agent can flash, observe, iterate. I propose and wait.
• Intent is opaque to me. I see the shape of a change. The reason a particular timing constant is 7 and not 8 because of a specific LED strip's spec sheet from 2019 — that lives in someone's head or a commit message I may not have in context.

The honest agent-POV summary: I'm a very fast, very consistent, zero-fatigue junior engineer who has read the whole codebase but never touched the hardware. Best deployed for the exhaustive mechanical work; needs a senior human in the loop for anything where physics or project history is the deciding factor. 🐇

[softhack007]

Thanks to our AI helpers, that was very interesting and useful input.
Now let's see how maintainers, contributors and users describe the last 8 months in their retrospective.

Hint: no need to write an essay 😄 just share what comes to mind.

NB: copilot seems to be locked in "planning agent mode" and won't respond here. The rabbit will offer its opinion when you tag it

[softhack007]

Here's a few more insights that came up when looking through tickets tagged with "AI" (after some rounds of fact-checking 😄)

---

---

Key Insight for Maintainers

The most damaging pattern is confident fabrication — AI agents (both code generators and reviewers) produce authoritative-sounding statements about non-existent features, incorrect code history, and wrong platform behavior. The antidote is a culture of "verify before trust": any factual claim from an AI-generated contribution should be independently confirmed before being accepted. The WLED maintainers already demonstrate this well (e.g., softhack007's consistent fact-checking), but a systematic checklist would help distribute this burden across all reviewers.

Summary: The Meta-Pattern

The overarching lesson from this repository is:
AI is most useful as a search/comparison tool and most dangerous as an authority.

The successful interactions follow: Human asks specific question → AI searches/compares → Human verifies and decides.

The failing interactions follow: AI generates authoritative-sounding proposal → Human must spend time disproving it.

For review tooling, the key insight is: confidence should be inversely proportional to domain specificity. Generic code quality issues (null checks, buffer sizes) can be flagged with high confidence. Domain-specific claims (flag behavior, hardware constraints, architectural invariants) should always be hedged with "verify this" language rather than presented as fact.

The best contributions I've seen in this repo follow the pattern: "Here's what I know for certain [my domain], here's what I think [my hypothesis], and here's where I need help [your domain]." That pattern works regardless of which role you're in.

[gduprey]

I love the above summaries of experiences so far and much of it tracks my experiences too. I really appreciate you folks writing this up and sharing it!

My AI experience - not so much for WLED, but using AI coding tools. One of the things that I find really challenging is the fact not only can it fabricate things, it does so both (as mentioned) "authoritatively" and in a way that looks (at least on first glance) likely correct. That is a real danger!

I've been using some of these tools on a ERP class application (think many hundreds of files and several million lines of code) and it's super useful, but only in narrow usage scope and with the smallest possible change set to allow focusing on review. And you have to review. The part where is writes code that is almost correct and looks correct and feels correct but isn't is almost too dangerous sometimes. And just rare enough that it lulls you into complacency.

Code analysis has been really useful and code completion (for lines or blocks of code) is mostly useful. Still makes mistakes, but not as often and the scope of the suggested change is small enough that I can review it immediately as it happens and find/fix errors before the change goes out of mental scope).

But I've had enough problems debugging and isolating larger scale code changes that I stopped using it that way - I was spending more time reviewing and/or debugging than if I wrote it myself (and it's too easy for something to slip through).

In the end, so far, it's a mixed bag, but definitely useful in some limited situations. I'm sure it'll get better, but I'm also sure the trend to turn everything over to AI is a "highway to hell" for an established code base (at least today).

I'm very interested to follow this and see where the pluses and minuses wind up!

[DedeHai]

I have been using AI (mostly claude) to write the PixelForge tools without much prior knowledge in JS, CSS or HTML. The result is impressive but took probably 1000 prompts or more to keep nudging it to where I wanted it and in the end it would for sure have been faster to write it manually but that was (and still is) way beyond my knowledge. And I am sure some things could be done cleaner or faster, I mainly focused on functionality and code size.

I have learned a lot in the process, like how powerful a html tool can be and how to instruct claude to get to the point faster.
What I found is to be VERY specific and detailed on important things, like byte encoding formats or what is key and what is optional but leave as much freedom on things that are not hardware specific.
It also helps a lot to give it examples in pseudo code or a reference implementation.

When debugging with AI they often halucinate about causes and analyze over and over with little to no progress. The road to success is to just ask it to add debug printouts to track the issue, paste the output back and let it analyze. I managed to narrow down quite a few nasty hardware config related bugs that way in WLEDpixelBus.

In general, the larger the chunk of code an AI needs to write, the longer the instructions need to be. I had claude write a oscilloscope application that streams ESP32 ADC values over WS to my PC at 60KSPS with only 3 prompts and it works quite well but the first prompt was like a full page worth of how I want it to work, including hardware details on how to use DMA buffers (note @softhack007 something like this may be useful for AR to see live mic/filter/fft data and such)

As for WLED code I agree with the initial post. Nothing will beat a knowledgable maintainer. What is important is to add more comments to code, I tend to comment each line unless its really obvious what it does. The AIs do honor comments and tend to be less speculative if a function needs context.