Consider migrating to sphinx-autoapi for automatic API documentation generation

[edgarrmondragon]

Summary

Consider migrating from the current manual autodoc + autosummary approach to sphinx-autoapi for automatic API documentation generation. This would reduce maintenance burden but requires careful consideration of tradeoffs.

Current State

• 165 manually listed items across 29 autosummary sections in docs/reference.rst
• Manual maintenance required when adding new classes or restructuring modules
• 6 redirect mappings in conf.py for re-exported classes
• 3 custom templates for different class types (plugin, stream, generic)
• Semantic organization by functionality (Plugin Classes, Stream Classes, Testing, etc.)

Benefits of Migration

• ✅ Zero manual listing - Classes are automatically discovered
• ✅ No forgetting to document - New classes appear automatically
• ✅ Better re-export handling - Reduces redirect complexity
• ✅ Configuration-driven - Easier to maintain inclusion/exclusion rules
• ✅ Less maintenance burden - No need to update reference.rst when restructuring

Challenges

• ❌ Semantic organization - AutoAPI generates by package structure, not by concept

  • Current: "All Authenticators", "All Test Classes" grouped together
  • AutoAPI: Organized by module hierarchy (singer_sdk.authenticators, singer_sdk.testing, etc.)
  • Mitigation: Create hand-written overview page with semantic links to autoapi-generated docs

• ❌ Custom template logic - Need to replicate 3 template types in Jinja2

  • plugin_class.rst - Shows all inherited members
  • stream_class.rst - Shows inherited members from Stream only
  • class.rst - Shows only init and call special members

• ❌ Migration complexity - Need to test thoroughly for feature parity

Migration Path

1. Install sphinx-autoapi in docs dependencies
2. Configure in parallel with current setup for testing
3. Create custom Jinja2 templates to match current behavior
4. Generate docs and compare with current output
5. Create hand-written overview page to restore semantic groupings
6. Remove manual autosummary entries once satisfied
7. Maintain redirects for backward compatibility

Configuration Starting Point

# In conf.py
extensions = [
    "sphinx.ext.napoleon",  # Keep for docstring parsing
    "autoapi.extension",    # Add autoapi
    # ... other extensions
]

autoapi_dirs = ["../singer_sdk"]
autoapi_root = "_autoapi"
autoapi_options = [
    "members",
    "show-inheritance",
    "undoc-members",
]
autoapi_ignore = [
    "**/internal/*",
    # Add other patterns as needed
]

When to Prioritize This

Consider prioritizing if:

• We're frequently adding/removing dozens of classes
• Module restructuring becomes more common
• Redirect maintenance burden grows significantly
• We're willing to reorganize documentation structure

References

• sphinx-autoapi documentation
• Current maintenance guide in docs/reference.rst (added in recent commit)
• Related: SQL module restructuring (commit f4bd041) required manual updates

Related Files

• docs/reference.rst - 409 lines of manual API listings
• docs/conf.py - Redirect mappings and Sphinx configuration
• docs/_templates/ - Custom class templates (plugin_class.rst, stream_class.rst, class.rst)