The former foreign minister said it would be impossible to participate in the November 7 vote after the government’s refusal to sack or suspend officials and ministers implicated in fraud.
But he added he would not "shut any doors" to his rival and his vice presidential candidate said the two were "moderate and reasonable men", with "some kind of compromise" possible.
Dr Abdullah ended his campaign in an emotional speech to thousands of supporters after his ultimatum for measures dismantling what he had called the "machinery of fraud" passed without concessions.
He stopped short of calling for a boycott and did not concede victory, leaving Mr Karzai’s campaign to insist the vote should proceed anyway.
The Independent Election Commission said it would now review the Afghan constitution to decide whether a vote must be held.
However diplomats told the Daily Telegraph there was no real desire from Mr Karzai or the international community to risk the lives of Nato troops and Afghan voters in a second round with only one candidate.
One European diplomat said the next few days would see moves to encourage Dr Abdullah to formally renounce his challenge and for the Supreme Court to rule a second vote was unnecessary.
He said: "We don’t see there will be any interest in a second round now. It will be a waste of time and resources." Dr Abdullah said he had made the "painful" decision to pull out because a "transparent election is not possible."
He made his speech after diplomats spent hours mediating between the two candidates on Saturday trying to coax them into a power-sharing deal according to one official.
Dr Abdullah had faced intense "persuasion" from diplomats keen to avoid a protracted second round that few doubted Mr Karzai would win.
While Mr Karzai had been expected to triumph, some kind of national unity government is viewed as crucial to bolster the legitimacy of the government and stop the country fracturing along ethnic lines.
They had argued Dr Abdullah should concede and not call for a potentially turbulent boycott.
The official said: "They were telling him, look you have come out of this a lot stronger than you went into it, you are a young man, you have the future ahead of you."
The talks ended without agreement between the two candidates, but Dr Abdullah’s campaign suggested negotiations would continue.
Making his speech, the 49-year-old asked his supporters "not to go to the streets, not to demonstrate."
Dr Abdullah said: "For the next few days I will let Mr Karzai work it out for himself.
"I will be in the country. I will not shut any doors, but I will stick to the principles of my campaign." Humayun Shah Asifi, Dr Abdullah’s vice presidential candidate, said: "We have two moderate people. Both sides are reasonable people. They have worked together in the past. I think some kind of compromise is possible."
Baryalai Arsalai, a former candidate, who dropped out in the first round to join the Abdullah campaign, said power sharing could lend the government legitimacy.
Gordon Brown said he hoped a government would form which reached out to "all parts of Afghan society". (Telegraph)