Support `Accept: text/markdown` for package documentation

[Tritlo]

Problem

Hackage serves documentation exclusively as HTML. For automated tools like LLMs, Claude Code, and bots, this means parsing HTML to extract the content, consuming a lot more tokens than required.

To make Haskell more useful for agents, we should implement a Markdown version of the docs. This allows agents (and humans!) to consume the docs directly, without having to parse the HTML. This is already implemented on Vercel

Proposal

Add content negotiation so that requests with Accept: text/markdown (the standard for agents) to return Markdown for key endpoints. Initially, we should focus on package descriptions and module documentation.

This will potentially involve having haddock generate markdown instead of HTML as it does currently. Before embarking on this project though, I would like some community feedback.

[jhrcek]

It's not markdown/content negotiable, but you might find it interesting that most packages have single file hoogle txt docs generated, e.g. https://hackage.haskell.org/package/aeson-2.2.3.0/docs/aeson.txt

Not easily discoverable either, but Imo an improvement from the point of view of LLMs.

[Tritlo]

that's a great point. maybe v1.0 for the header could just return the .txt files (they're valid markdown as well, yes?)

[phadej]

Hackage serves whatever haddock generates. hackage-server assumes a bit of structure of generated documentation tarball HTML, but technically you can upload docs tarball with whatever contents you want (it's a security loophole, really).

hackage-server doesn't really know anything about the docs. In particular, it doesn't (re)generate anything, all the (documentation) content is pregenerated by docbuilder or users. hackage-server barely serves the static content then.

[phadej]

As a human, I prefer well rendered HTMLs with links so I can browse around the documentation. Please no (unrendered) markdown (or whatever markup) for the humans.

[phadej]

This reminds me of semantic web

The goal of the Semantic Web is to make Internet data machine-readable.

It's fun and sad, how we given up of trying to provide information in structured data formats, and now instead rely on (imprecise) (natural) language models to extract information from "prose". Honestly, I think we have failed as technological society.

[Tritlo]

As a human, I prefer well rendered HTMLs with links so I can browse around the documentation. Please no (unrendered) markdown (or whatever markup) for the humans.

I agree! That's why we'd ideally want to generate this and serve only upon request

[Bodigrim]

As @phadej says, this is really a task for Haddock in the first place to produce Markdown documentation. Hackage (and Cabal) would be minor changes only.

[gbaz]

I think teaching haddock to generate markdown is a good idea, though really what that should be part of is a move to a backend-independent intermediate layer and then "markdown" as one possible target. That said, I do not thing gen AI is a good motivation. Surely soon enough the "tokens" problem will be solved and html parsing will be done as a preliminary pass. After all, most of the internet is in HTML.

[Tritlo]

they'll be solved by big corp for sure, but for oss models on consumer hardware you still want to save context