Amid the acrid smoke and dust, the cries of the injured being dragged out of the rubble, General Adnan Thabit arrived at the Hamra hotel bomb site in sunglasses, pressed fatigues and a crimson beret.

"Well, gentlemen," he said to me and another journalist who had just been blasted out of our hotel rooms by suicide bombers, "this is what happens when terrorists carry out terrorism - a lot of people dead, a lot of people hurt. Now you can see what we are up against."

The general was savouring his moment. His special forces have been accused by the media and others of carrying out the worst human rights abuses against "suspected insurgents" in what is becoming an ever more savage and dirty war.

Behind the daily reports of suicide bombings and attacks on coalition forces is a far more shadowy struggle, one that involves tortured prisoners huddled in dungeons, death-squad victims with their hands tied behind their backs, often mutilated with knives and electric drills, and distraught families searching for relations who have been "disappeared".

This hidden struggle surfaced last week when US forces and Iraqi police raided an Interior Ministry bunker only a couple of hundred yards from where we were standing. They found 169 tortured and starving captives, who looked like Holocaust victims. The "disappeared" prisoners were being held, it is claimed, by the Shia Muslim Badr militia, which controls part of the ministry. Bayan Jabr, the Minister of the Interior, is himself a former Badr commander, but the ministry's involvement does not end there: General Adnan's commandos come under its control. So does the Wolf Brigade, which vies with the commandos for the title of most feared.

Baghdad is now a city in the shadow of gunmen. As I left the Hamra to replace what was lost in my bombed room, I had to negotiate checkpoints of the Badr militia, their Shia enemies, the Mehdi Army of the radical cleric Muqtada al-Sadr, and the Kurdish peshmerga. The Iraqi police and the government paramilitaries have their own roadblocks.

And there are others: the Shia Defenders of Khadamiya - set up under Hussein al-Sadr, a cousin of Muqtada, who is an ally of the former prime minister Iyad Allawi - and the government-backed Tiger and Scorpion brigades. They all have similar looks: balaclavas or wraparound sunglasses and headbands, black leather gloves with fingers cut off, and a variety of weapons. When not manning checkpoints, they hurtle through the streets in four-wheel drives, scattering the traffic by firing in the air. Out of sight they are accused of arbitrary arrests, intimidation and extrajudicial killings.

The US and Britain, which trained many of the forces involved, and which still have ultimate responsibility for them, are implicated. But the pattern of illegality is also the continuation of a process that began with the questionable justification for the invasion. American and British forces have played their own part, from the abuses of Abu Ghraib to deaths in British military custody, from the deployment of white phosphorus as a chemical weapon in the assault on Fallujah to the wild use of overwhelming American firepower, which some have called almost as indiscriminate as the killings caused by Sunni insurgents' car bombings.

But more than two years after President George Bush officially declared a victorious conclusion to the war in Iraq, the body count continues to rise. Faced with an insurgency that shows no signs of abating, the US and Iraqi government rely more and more on the paramilitaries. Donald Rumsfeld, the US Defence Secretary, has said units such as General Adnan's commandos are among "forces that are going to have the greatest leverage on suppressing and eliminating the insurgency".

Those on the receiving end of some of this "leverage", however, describe terrifying experiences. Ahmed Sadoun (not his full name) was arrested in the middle of the night at his home in Mosul by government paramilitaries accompanied by American soldiers. He was held for seven months before being released without charge and left Iraq as soon as he could.

Speaking from Amman, Mr Sadoun, a 38-year-old engineer, said: "They kicked down our door and asked about a neighbour. When I said I did not know where the man was, they started kicking me and beating me. The soldiers had paint on their faces and did not have same uniforms as other troops. The Americans did not take part, but they saw what was happening.

"But this was nothing. When they took me to their base I was blindfolded and beaten very, very badly with metal rods. They then hung me up on hooks by my wrists until I thought they would tear off. I think that stopped because one of the Americans said something. I could hear English spoken in an angry voice. But this happened again later.

"I was in a room which was so small that not all the prisoners could even lie down properly. All night we could hear people screaming, people being hit. One day they said I could go, but after what happened to me I know they could do this to me again. So I left."

At one roadblock I met the Wolf Brigade, which was widely involved in suppressing disturbances in Mosul around the time Mr Sadoun was being held, though he does not know which paramilitaries seized him. One of them took off his balaclava and turned out to be in his late teens, belying the ferocity of the snarling wolf badges on his arms.

The young man shook his head about what happened at the Hamra. "That is bad, very bad," he said. "But you are alive, that is good - too many dead people in Baghdad." He was keen to make the point that "the people like us because we kill the people who try to kill them. Listen, mister, we are fighting bad people, you cannot treat them like normal persons."

But what about the innocent who get caught and end up being abused in detention centres? "Mister, those are just lies, you must not believe them. These people are terrorists. We are here because the police cannot do the job by themselves."

The paramilitary influence on the police is particularly overt in the British-controlled south of Iraq, where the British invited the militias to join the security forces, and then saw them take over. Nothing was done by the British authorities when police in plain clothes, along with their militia colleagues, killed Christians, claiming they sold alcohol, or Sunnis for being supposedly Baathists.

Action was only belatedly taken when a particularly menacing faction, a "force within a force" based at the Jamiat police station on the outskirts of Basra, captured two SAS soldiers who were gathering information on their mistreatment of prisoners.

British troops smashed into a police station to rescue the two soldiers and later arrested more than a dozen others. But now they more or less stay out of Basra, leaving Iraq's second city at the mercy of a police force that even its commanders say they barely control. There have been dozens of assassinations, including that of at least one foreign journalist.

Even families of fellow policemen are not exempt. Ammar Muthar, a member of the border police, knew his father, Muthar Abadi, was on the Shia militia hit list, because he had acted as a missile engineer in the war against Shia Iran. Ammar brought his father from Al-Amarah to Basra for safety. But while he was out one day, six policemen, in uniform but wearing black masks, dragged Abadi away. His body was later found, shot five times, three in the face.

"The neighbours could do nothing because it was the police who took him away," said Ammar. "They wanted to kill him, and no one could stop them."

One British officer said: "You hear about the militias infiltrating the police. But they did not have to. We invited them to join."

THE CHARGE SHEET

Endemic torture of prisoners

The discovery last week of starved and tortured prisoners in an Interior Ministry bunker emphasised that detention without due process remains endemic in Iraq, echoing the Saddam era. In Basra, militia elements in the police used cells to imprison and torture their enemies. When the Abu Ghraib abuse scandal broke, it emerged that among the prisoners being maltreated by US soldiers were "undocumented" detainees who had been kept hidden from the Red Cross. And Britain was shamed by the death of Baha Mousa, a Basra hotel clerk, in British military custody.

Use of napalm and phosphorus

Last week the Pentagon admitted using white phosporus as an offensive weapon in last year's assault on Fallujah. Its official use is to create smokescreens to shield troop movements, but if fired into trenches or foxholes it can burn victims to the bone. The legality of this use is debatable. Last year the US also admitted, after previous denials, that it had used napalm - which Britain has banned - against Iraqi forces during the invasion. There is also controversy over the deployment of cluster munitions, which Britain has said should not be used in or near civilian areas.

Extra-judicial killings

Early in the insurgency, it appeared that Sunnis loyal to Saddam were attacking defenceless Shias - an impression the coalition authorities sought to reinforce. Extra-judicial killings by Shias in the paramilitary groups operated by the Interior Ministry, or in party militias, have become increasingly open. Victims are rounded up in house-to-house raids at night and their bodies, frequently handcuffed, are found dumped later. Sunni civilians are the main target, but Shias seen as collaborators under Saddam, or connected with rival militias, have also "disappeared".

Indiscriminate 'spray and slay'

Heavy-handed tactics against the insurgency, dubbed "spray and slay", have attracted much criticism. The current American offensive in the west and north-west appears to replicate the methods used in Fallujah: the population is ordered to leave before the town is sealed off and subjected to an air and ground assault. Those killed are invariably described as insurgent fighters, even in incidents where there is strong evidence that groups of civilians, including women and children, have been caught up in airstrikes.