A historian friend of mine, erudito has looked at this question many times and actually occassionally teaches about it at high school. Don't let his conservatism fool you - he actually does have a clue...

As you've mentioned farming doesn't improve lifespan, resistance to disease or general health. The capacity of it, particularly in the early stages of development, to provide a store of food is questionable.

And herein lies the problem. We liberal minded souls, with an intrinsic distate for it, forget about war.

Farming provides a numerically superior and more concentration population. Gatherer-Hunter societies didn't stand a chance.