Since the dawn of warfare, so since the very beginning, man has sought the ability to subdue or kill his opponents with his bare hands. Weapons have changed, but not alleviated, that search. Bones and shafts break. Arrows and bullets run out. Swords and rifles get dropped and lost. Then, the warrior has only his wits and whatever barehand skills he possesses to prevail over his enemy, especially in close quarters, where he can smell him.
Hunting and fighting were intrinsically tied back in pre-history, before civilization, agriculture, religion, and armies. Indeed, fighting and hunting were probably the same skill sets. Killing to eat, be it mastodon or deer, and killing to protect, be it from neanderthal or cave bear, drew on the same or similar skills and tactics: Put down the other before he or it puts you down.
Sports were originally intended to teach the young how to hunt and fight—again, essentially the same skill sets. Special skills evolved with such game changers as the knife, the spear, and the arrow. Special operators were those who could employ such weapons effectively. There were probably early innovators of specialized hand-to-hand systems to deal with these game changers, but we have, so far, no record of that.
Civilization was born around 12,000 to 10,000 years ago, along with agriculture and religion. With the advent of these came also the advent of organized militaries. Armies quickly learned that things such as command and control, strategy and tactics, and martial skills for soldiers needed to be developed and taught. These skills were mostly focused on the sword and shield, the bow and lance, and likely little thought was given to hand-to-hand training. But, also likely, most soldiers possessed hand-to-hand skills gained at home, through sports and the rough play that is common, then and now, amongst boys and young men.