Brown lost the sight in his left eye when he was just 16 after he suffered a kick to the head during a school rugby game.
Clarkson’s onslaught against the PM came at the launch of a Top Gear Live stage tour in Sydney.
But politicians and disabled associations are outraged at the big-mouthed presenter’s latest gaffe, calling his comments as “unacceptable” and “insensitive”.
A spokesperson for the charity Visibility said: “It sounds as if Jeremy Clarkson was using the term ‘one eyed’ as an insult. It is very unfortunate that Mr Clarkson has used his high media profile to present sight difficulties in such a negative way.
“I would challenge Jeremy Clarkson to come and spend a day with Visibility listening to and speaking with people who overcome significant difficulties caused by sight loss on a daily basis.
“If Mr Clarkson takes up our challenge, I would hope that next time he speaks about visual impairment, he does so from a more informed position and uses his media profile to highlight visual impairment in a more positive way.”
A spokesperson for the Highland Society for the Blind said: “We are in the business to change perceptions of blind and visually impaired people.
“What Clarkson might want to do is to come along and help on one of our driving days where blind individuals get to drive along an air strip and feel the thrill of the driving experience.
“He might then change his thoughts about being so insensitive.”
The Royal National Institute of Blind People (RNIB) was also quick to condemn the “offensive” comments, saying: “Any suggestion that equates disability with incompetence is totally unacceptable.
“Most people here are proud that the Prime Minister is a Scot and believe him to be the right person to get the UK through this global economic crisis.”
Patrick Harvie, Green MSP, said: “Jeremy Clarkson’s brain seems permanently stuck in neutral. He clearly loves his role as the populist face of climate change denial, but his shameless self-promotion regularly gets in the way of the truth.
“On this occasion, his petty insults are no substitute for sensible criticism of the Prime Minister.”
Number 10 would only say that Clarkson “is entitled to his own interpretation of the economic circumstances”.
The spokesperson declined to comment on the specific insult about Mr Brown.
The controversial presenter compared Mr Brown to Australian Prime Minister Kevin Rudd, shortly after Mr Rudd had addressed the country on the severity of the global financial crisis.
According to The Australian newspaper, Clarkson said: “He [Rudd] genuinely looked terrified. The poor man, he’s actually seen the books.
“[In the UK] we’ve got this one-eyed Scottish idiot. He keeps telling us everything’s fine and he’s saved the world and we know he’s lying, but he’s smooth at telling us.”
Last November, more than 1,800 people complained to the BBC after Clarkson made a joke on Top Gear about lorry drivers killing sex workers.
Some 340 people also complained to regulator Ofcom, but it later ruled the joke did not breach the broadcasting code.