Addressing journalists in Harare, Foreign Minister Simbarashe Mumbengegwi said Annan had "misrepresented" facts by saying he together with the former US president Jimmy Carter and former Mozambican first lady Graca Machel had been barred from Zimbabwe.
The trio was scheduled to make an analysis of the humanitarian crisis in Zimbabwe on a November 22-23 visit. But they cancelled the trip after Harare denied them visas.
"The postponement was necessary because Mr Annan had made no prior consultations with government of Zimbabwe regarding both the timing and programme of his proposed visit as is normal practice," said Mumbengegwi.
"It is quite clear that no meaningful assessment of the humanitarian situation could be undertaken in the few hours the delegation intended to be in Zimbabwe.
"It was on that basis that Mr Annan was advised, in good time, to postpone the visit to allow for responses to be made to the consolidated appeal based on the joint assessment of the humanitarian situation by the government of Zimbabwe, the World Food Programme and the UN Zimbabwe team," Mumbengegwi added.
Zimbabwe is facing a serious humanitarian crisis with more than half the population facing starvation. A cholera outbreak in September has since claimed about 300 lives, according to the World Health Organisation.
Mumbengegwi said President Robert Mugabes government was fully aware of the humanitarian challenges facing the country and that Harare was determined to address these challenges.
"The government is willing to engage with all those of good will in an effort (to fight the humanitarian crisis.) The government takes strong exception to any exceptions to any suggestions that there are those that care more about the welfare of our people than we do," he said.
Asked if he would allow Annan and his team to come to Zimbabwe in future, Mumbengegwi said: "If we come up with a mutually agreed to date. We told them that and that is not a secret."
Earlier in Johannesburg the trio had announced their visit was being called off because Mugabe had refused them entry.
At a press conference, the three, who were due to travel to Zimbabwe on behalf of The Elders grouping of leading statesmen and women, said they were disappointed that they had been denied an opportunity to shine a light on the humanitarian crisis in the country.
South African President Kgalema Motlanthe and ex-president Thabo Mbeki, who is mediating in Zimbabwe’s power-sharing talks, had intervened on their behalf with Mugabe, but to no avail, they said.
"It seems obvious to me that the leaders of the (Mugabe) government are very immune to reaching out for help for their people," Carter said.
Machel, a well-known social rights campaigner and wife of The Elders convener, anti-apartheid icon Nelson Mandela, also said she was "extremely disappointed."
This is the first time The Elders – which was founded in 2007 to tackle conflicts around the world and also includes South African Archbishop Emeritus Desmond Tutu, an outspoken critic of Mugabe – has been refused entry to a country.
The Elders had been due to meet with aid organizations and health workers to discuss a worsening food crisis and a spiralling cholera outbreak that has claimed nearly 300 lives and infected thousands of others, according to several diplomatic and medical sources.
State media said seven members of a local Christian sect were among the victims of the water-borne disease in the Harare township of Budiriro this week. They had refused treatment on the grounds their religion bans them from all medication other than "holy water."
While stressing their focus was on humanitarian needs, including the food shortages suffered by over 3 million people, Annan, Machel and Carter had also sought meetings with Mugabe and the opposition.
They had been scheduled to meet with MDC leader Morgan Tsvangirai in South Africa on Friday. It was not clear whether the meeting took place.
The MDC is coming under pressure from Zimbabwe’s neighbours to help stem the crisis by accepting Mugabe’s terms for joining a unity government, in which Mugabe remains president and Tsvangirai becomes prime minister.
The MDC, which won the last parliamentary elections, accuses Mugabe’s Zanu-PF of keeping all the important cabinet portfolios, bar finance, for itself.
The parties are reportedly scheduled to meet again next week under Mbeki’s mediation to discuss a draft constitutional amendment that will set out Tsvangirai’s powers as prime minister relative to Mugabe’s.