“I am a slow walker, but I never walk back.”

Darwin is describing the pace of his own work on the Beagle and afterwards, where the conclusions took twenty years. The sentence is not about speed but about the fact that he never took a question back to where he had started.