Higher-order derivatives

[Technici4n]

I am opening this issue to track support for higher-order derivatives (second order and more). In principle all 4 combinations (forward and reverse over forward and reverse) should be supported. Currently none seems to be.

Some preliminary questions would be:

• How to represent duals and coduals for higher-order differentiation? I tried to ask Mooncake what the tangent type of a Dual should be, but this seems suboptimal:

julia> Mooncake.Dual{Mooncake.Dual(1.0, 2.0),tangent_type(typeof(Mooncake.Dual(1.0, 2.0)))}
Mooncake.Dual{Mooncake.Dual{Float64, Float64}(1.0, 2.0), Tangent{@NamedTuple{primal::Float64, tangent::Float64}}}

In principle these nested duals should usually not be interacted with so an ugly representation might be OK.

• How to avoid perturbation confusion? Is this something that compiler-based AD systems need to be careful with as well?

[yebai]

@sunxd3 is working on testing and fixing forward over reverse, which ought to be the lowest hanging fruit.

[gdalle]

How to represent duals and coduals for higher-order differentiation? I tried to ask Mooncake what the tangent type of a Dual should be, but this seems suboptimal:

This is probably fine

How to avoid perturbation confusion? Is this something that compiler-based AD systems need to be careful with as well?

I think we might end up needing tags as in ForwardDiff. When you look at Mooncake forward mode, it's ForwardDiff with a mustache. But they have those in JAX too