Using AI to Optimize Parameters

http://arxiv.org/pdf/1402.1694v4.pdf

I came across this interesting paper today, which describes a way of doing Markov Chain Monte Carlo in a way that speeds up optimizing the parameters of a probabilistic model up to 200x. And of course, no big surprise, it's about using intelligent approximation.

This brings to mind to possibility that one day the algorithm we use to do optimization may actually use "AI". What I mean by that is that exploring a curve in hyperspace to find the minimum is in some sense like many other problems: You can look around and collect clues as to the characteristics of the curvature, and use those clues to build a kind of mental model of the dynamics at play, and then ultimately to use what you know about the space to intelligently explore it as quickly as you can.

Imagine a "neural net" of sorts that had been trained on millions of example optimization problems, and was able to very efficiently build an internal model of the topology of a hyper-surface, using that to calculate the optimum parameters... it might be hundreds, or thousands of times faster than naive human attempts that use exact algorithms... and for problems that are just mind-blowingly complex and viewed as semi-intractable today, it might be trillions of times faster.

An amusing side realization is that optimization is a core requirement for building AIs / neural nets in the first place, and so it's conceivable that there could be a recursive benefit to using AIs to do optimization... back to our good old exponential improvements in technology game: Your now supercharged ability to do optimization allows you to train an even more capable neural net, which allows an even faster optimization algorithm, which allows even more capable neural nets, and back and forth you go.