The scientists who had been cast down had been effective in their teaching of humanity. And humanity itself proved to be a more inventive and hardy race than the Anu intended.
There came a time where the military who were left in power began to fear the influence of the wandering Anu on human civilisations. They were paranoid, and concerned that the humans would get up into space and kill all of them if current developments continued.
And so, they bio-engineered a deadly disease that affected only their own kind. Rather than kill instantly, this bioweapon caused the Anu on the surface to age very quickly and become weak and confused before finally losing grip on life. Soon, there were none left.