When Manuel considered candidates for the first Viceroy of Portuguese India in 1505, the post went to Francisco de Almeida. He was succeeded by Afonso de Albuquerque and, later on, Lopo Soares de Albergaria, Diogo Lopes de Sequeira and the corrupt Duarte de Menezes before da Gama found himself back in favour in 1524.
Da Gama continued to advise Manuel on Indian matters until 1505, then settled into retirement in Évora, while he argued with the Order of Santiago over the ownership of Sines, which he had been promised, but the order refused to hand over.
In the meantime, his wife bore him six sons and a daughter, and it is possible that the fall out over Ferdinand Magellan defection to Castile in 1518 may have turned to da Gama's benefit.
He is reputed to have threatened to do the same, prompting Manuel to appoint him the first Count of Vidigueira. A royal decree on 29 December granted da Gama and his heirs all the revenues from Vidigueira and Vila dos Frades, which had been excised from the holdings of Dom Jaime, Duke of Braganza. He was the first Portuguese count not born with royal blood.
His fortunes continued to change after John III succeeded his father in 1521and set about reviewing Portuguese policy in the Indies. Under Manuel, there had been an obsession with controlling the flow of trade into the Red Sea and the Persian Gulf and ambitious crusading schemes against Mecca and the Holy Land. Now, with an emerging threat from Spanish interests and the Magellan expedition attention turned further east towards Portuguese interests in the Moluccas.