Help wanted for implementing FindMinimum, FindMaximum with hipparchus MultivariateOptimizer

[axkr]

Can someone of the hipparchus experts please review the FindMinimum implementation and the usage of the different MultivariateOptimizers:

• https://github.com/axkr/symja_android_library/blob/master/symja_android_library/matheclipse-core/src/main/java/org/matheclipse/core/reflection/system/FindMinimum.java

The current implemented interface is testable online here:

• https://matheclipse.org/

For example for the questions:

• what are the best "default values" to set for each optimizer method?
• how can the constraint determination improved for the different methods?
• which "(multi-variate) optimizers" are the best "default (multi-variate) optimizers" to use for the different ObjectiveFunctions and constraint parameters (OptimizationData) and how to determine this?
• ...

See

• https://github.com/axkr/symja_android_library/blob/master/symja_android_library/doc/functions/FindMinimum.md
• https://github.com/axkr/symja_android_library/blob/master/symja_android_library/doc/functions/FindMaximum.md
• Improve FindMinimum, FindMaximum with more hipparchus methods axkr/symja_android_library#1150

[maisonobe]

It is difficult to come up with default values, as they highly depend on the problem.
From my experience, optimizers often work better is the variables they managed are scaled in such a way they all wander in a domain with the same order of magnitude (say all components of the state vector between -1.0 and +1.0, or between -1000.0 and +1000.0). What should be avoided is very different order of magnitude. I remember one physical problem where one component was about 4.0e14 while another one in the same vector was about 1.0e-8; 22 orders of magnitude differences was too much for the optimizer, but scaling down the larger component and scaling up the small component worked well.

Concerning constrained optimizations, we only have SQP optimization (contributed recently) that naturally handled constraints. For least squares optimization methods, what we used for domain bounds was to rely only on clipping, i.e. forcing the state vector within bounds when the optimizer attempted to go out. This was done using ParameterValidator. This works only for simple bounds. Another trick I have seen was to put penalty on the cost function, so the optimizer does not tries to go too far.

I guess selecting the proper method is really problem dependent and should remain the responsibility of users. I don't known how to determine this automatically.

[axkr]

In WMA you get this result, if the iteration limit is set "to low":

In[1]:=  FindMinimum[Exp[x], {x, 3}, MaxIterations ->3]

FindMinimum::cvmit: "Failed to converge to the requested accuracy or precision within 3 iterations"

Out[1]={1.30887, {x -> 0.269167}} 

Can hipparchus also set "the last result from the iteration" if possible?
For example as parts[1], parts[2],... of an MathIllegalStateException with LocalizedCoreFormats.MAX_COUNT_EXCEEDED ?
Or is there another way to get this information "consistently" in the optim package (also for other exceptions) ?

[maisonobe]

I don't think it is possible right now in a completely generic way. Each optimizer has its own way to manage attempts.
There is perhaps a workaround, though : to use a custom implementation of ConvergenceChecker that would store the last arguments to the converged method. Then, if the optimizer fails, one could retrieve the last point and value.