Morgan Tsvangirai, leader of the opposition Movement for Democratic Change (MDC), announced the decision on Friday, ending deadlock that deepened a political and economic crisis.
"This is an important step towards ending the political impasse in Zimbabwe," Annan, a member of a group of prominent figures known as the Elders, said in a statement.
"But it is not a guarantee that Zimbabwe’s distress is over. Rebuilding the economy and ending the people’s terrible suffering will take much more work on the part of all Zimbabweans, regional leaders and the international community."
The once-prosperous southern African country’s economy is in ruins, with half the population needing food aid. The continent’s deadliest cholera epidemic in 15 years has killed 3,100 people and infected another 60,000.
Official inflation — last recorded in mid-2008 — had soared to 231 million percent and food and fuel are in short supply. The United Nations says unemployment is 94 percent.
Fellow Elders member and former U.S. President Jimmy Carter said while the agreement was not perfect it should be supported.
"The international community should now do what it can to give this agreement the best chance of success," he said.
Under the agreement, Tsvangirai will become prime minister, with Mugabe remaining as president. But it is unclear whether the old foes will be able to work together effectively to persuade Western states to provide investment and aid.
The United States has voiced scepticism over the deal, saying implementation is key and that aid would come only when a representative government was in place.
African Union and Tanzanian President Jakaya Kikwete hailed Tsvangirai’s decision to join the government.
"The situation looks promising … That is a step forward," he told Reuters as he arrived in Ethiopia for an AU summit that begins on Sunday. "That is progress in the right direction."
Many Zimbabweans have been longing for a new leadership. The few who have jobs queue for hours outside banks to withdraw enough cash to buy a loaf of bread. Millions have fled to Zimbabwe’s neighbours, straining regional economies.
Mugabe and Tsvangirai signed a power-sharing deal in September to end a crisis after elections that were condemned around the world, but implementation had been held up by disagreements over who would get top cabinet positions.
Under pressure from regional leaders, Tsvangirai agreed at a summit this week to share control of the Home Affairs ministry, responsible for the police, with Mugabe’s ZANU-PF party for six months. The new government is set to be formed by February 11.