The former UN secretary general, the former US president and human rights advocate Machel, who is married to Nelson Mandela, said on Saturday they were denied visas for a mission to assess the needs of Zimbabweans, many of whom are suffering from hunger and disease.
The three are members of the Elders, a group Mandela formed to foster peace.
They have insisted their visit was not related to regional attempts to get President Robert Mugabe and his rival, opposition leader Morgan Tsvangirai, to implement a stalled power-sharing agreement.
On Sunday, the three talked to men, women and children who have fled Zimbabwe and sought refugee at the Central Methodist church in downtown Johannesburg.
Carter spoke to 14-year-old Kennedy Manyani, an orphan who crossed the crocodile-infested Limpopo River into South Africa by himself three months ago.
"I came because my grandmother could not afford to buy me clothes, food," he told Carter.
More than 1 600 people are squeezed into the church with many more sleeping on the pavement outside. An estimated three million Zimbabweans live in South Africa and millions of others have fled to neighbouring countries in search of jobs and security.
While Zimbabwe’s political crisis occupies politicians, the humanitarian crisis is deepening. The health system has collapsed and a cholera outbreak has killed nearly 300 people in Zimbabwe, the United Nations said.
The three were met with cheers as they made their way through the cramped building.
"Their visit boosts morale. It shows us that we are not alone," said Herbert Nedi (24) who helps at the school and adult education centre the church has started.
Zimbabweans, who were the targets of anti-foreigner violence in South Africa this year, daily face the risk of arrest, beatings, harassment and sexual assault.
The church was raided by police this year and a number of refugees — most of whom are undocumented migrants — were deported.
"We are very vulnerable here," said Bishop Paul Verryn.
On Sunday, Carter, Annan and Machel met with Botswana President Seretse Ian Khama and representatives of aid agencies and Zimbabwean social organisations. They met with Zimbabwe’s main opposition leader on Saturday and are scheduled to meet South African President Kgalema Motlanthe on Monday.
Khama has been one of the few African leaders to openly criticise Mugabe.
"Our biggest concern is that the focus is moving away from the plight of the Zimbabwean people, who are currently suffering through starvation and diseases," he said after an earlier meeting with Motlanthe in the South African capital, Pretoria.
Annan said Saturday that Zimbabwe gave no official reason for refusing them visas for the mission.
But the Sunday Mail, a mouthpiece for the Zimbabwean government, quoted Foreign Minister Samuel Mumbengegwi as saying Annan had failed to consult with the government beforehand on the "timing and programme" of the visit.
The minister accused Annan of "misrepresenting the facts" about the aborted trip, and criticised the group for launching the mission, according to the newspaper.
"We take strong exception to any suggestions that there are those out there who care more about the welfare of our people than we do," he was quoted as saying.
Mumbengegwi also said the group would have had difficulty conducting a meaningful assessment, because the government had already completed its own "humanitarian audit" in conjunction UN agencies based in the country, the paper reported.
No details of that audit have been made available, and in the newspaper report, Mumbengegwi did not name any of the UN agencies involved.
The Elders — including 12 former world leaders and prominent rights activists — have mediated in a number of other international crises, such as Sudan and Kenya. – Sapa-AP