Optimisation and local maxima

It is very easy to get fixated on a particular solution to a problem. The phase “beginners mind” is used to try to allow yourself to look at a problem with a fresh perspective.

When fixed on a particular solution optimisation experiments are done to try to get the best out of what is currently available. For any given solution there is a limit, in other words the best that solution will ever perform. As you approach this you get diminishing returns for your effort. This is a local maxima - it is not the best any solution can achieve - it is the best your current solution can achieve.

Machine learning / AI algorithms are designed to optimise and find the best outcome for their current work, and they know only too well that they might of found the best solution so far, but there might be better ones elsewhere. So every now and then they do something random, then try to optimise that. Doing this again and again allows the algorithm to search the possibility space to find a global maxima - i.e. the best solution to the current problem.

We can be inspired by this to say - when our experiments are yielding fewer and fewer returns - it is time to try something bold, something new. Our mindset shifts to new possibilities and new solutions to the original problem. The alternative is to keep trying to squeeze the last bit of image quality of out a VHS tape.

This is a tricky balance - you don’t want to give up too quickly, but you don’t want to spend too long on a solution that is the best it is ever going to get.