Include searchdirection in OptimizationState

[goerz]

It would be convenient if OptimizationState included a searchdirection and/or a step width. I can usually pull these quantities out of the original state, but it would be nice to have them in a well-defined location.

For simple gradient descent, the search direction would be the negative gradient, but e.g. for LBFGS it would be the direction obtained from the internal quasi-Hessian. Knowing the search direction allows to really drill down into the linesearch, see, e.g., the plots in the README of GRAPELinesearchAnalysis.jl. In principle, I can infer the search direction as the difference in u from the previous iteration, but then I don't know the step width (how far the optimizer went in the direction of the search direction). Knowing both quantities directly from the solver is useful for actually checking that the update is in the search direction, but if that's guaranteed, only one of them would be sufficient.

[goerz]

I can usually pull these quantities out of the original state

Well, I thought I could, but it turns out the state is woefully inadequate, at least for some optimizers. I'm dumping state in the callback in an optimization with NLopt.LD_LBFGS() and seeing this:

Optimization.OptimizationState{Vector{Float64}, Float64, Nothing, Nothing, Nothing}
  iter: Int64 0
  u: Array{Float64}((9,)) [0.13823471888875286, 0.30634894494361153, 0.1562189341456497, 2.6013349121108944, 2.601329761795745, 3.6818511117018, 4.997713711972462, 4.989897143702104, 4.9818589345359605]
  objective: Float64 0.01069630598006488
  grad: Nothing nothing
  hess: Nothing nothing
  original: Nothing nothing

So it's missing a lot of information, and doesn't forward any internal LD_LBFGS state.

[ChrisRackauckas]

Yes, we need to fill it in a bit more per optimizer, and not all optimizers have al linformation.