New York in the late 1980s and 1990s was a polarised city state, where whites and minority groups lived side by side but in no sense together. They were, to use a phrase coined at the time, “the closest of strangers”.

My arrival in New York in 1987 coincided with the notorious Tawana Brawley case. Tawana was a 15-year-old African-American girl from upstate New York who was abducted and raped by a group of white men. Tawana was subsequently found in a semi-conscious state inside a rubbish bag with racial insults, including “KKK”, “N*****” and “Bitch”, scrawled across her body.

One of her alleged assailants was a local policeman, who committed suicide the week after Tawana was found. Another worked as an assistant district attorney.

Even before Tawana’s ordeal, passions were high in New York because of an incident the previous year in Howard Beach, Queens, in which a black teenager had been killed while trying to escape a white mob. Her assault raised tensions to a whole new level. Howard Beach had been about working-class racial tensions, the kind that New Yorkers were all too familiar with. The Brawley crime implicated the white male establishment.

As a grand jury deliberated on whether to bring charges, there were riots and protest marches. Polls showed that a majority of black Americans supported Tawana — the comedian Bill Cosby was among those who donated money to her legal fund — while a majority of whites doubted her story.

Then came the bombshell in late 1988. There was overwhelming evidence that Tawana Brawley had lied. Yet a strange thing happened. Blacks and whites still retained their original positions. Even African-Americans who accepted that she had lied claimed that the larger “truth” remained valid; as Patricia Williams, a respected black legal scholar, wrote in 1991: “[Tawana] has been the victim of some unspeakable crime. No matter how she got there. No matter who did it to her — and even if she did it to herself.”

But by then many in the white community were no longer listening. A new narrative had taken over — and now the victim and villains had been reversed. In April 1989, the month that Tawana’s ex-boyfriend admitted to a newspaper that she had made the whole thing up, Trisha Meili, a white, Yale-educated banker at Salomon Brothers, was jogging in Central Park when she was allegedly attacked by up to 30 black youths who were out “wilding”. She was beaten, gang-raped and left for dead.

The crime appeared to confirm white New York’s worst fears: that the city was under attack by a malevolent underclass, that black activists were playing the race card to their advantage and that the true victims in the race war were law-abiding middle-class whites.

The police soon identified some of the rapists, and the Central Park Five, as they became known, were sentenced to between five and 15 years for the crime. White New York breathed a sigh of relief. But, as during the Brawley case, the two communities clung to their own versions of the truth. Though all whites believed the boys to be as guilty as hell, many in the black community proclaimed their innocence with equal vehemence.

Both cases helped to establish a toxic pattern in which the two communities not only clung to their own racial identities, but believed their own racial truths. Every narrative had to have a martyr and a villain. Every outcome had to be framed within a racial conspiracy or a cover-up.

In the 1991 Rodney King case, in which cops in Los Angeles were filmed beating an unarmed black motorist, the black community believed there was no indictment by a majority-white grand jury because the police were white. In the OJ Simpson case, three years later, whites believed that the famous footballer was found not guilty of murdering his white ex-wife by the majority-black jury largely because he was black.

I should say here that of the Tawana Brawley and the Central Park jogger cases, the greater lie was in the latter. The five black youths were in fact innocent. It would take 13 years, but eventually DNA evidence would prove that a lone serial rapist had attacked Trisha Meili that night.

It was only after 9/11, when all of America could unite against a common external enemy, that these racial divisions started to heal. The terrible attack on the twin towers gave the nation a single narrative in which all races could be both heroes and victims and the villains were far, far away.

What helped to destroy this rapprochement, at least among African-Americans, was the combination of a recession that hit black families harder than white, and disappointment over Barack Obama’s failure to do more for the black community. Whites did not begin to appreciate black anger until it was too late.

Now, as racial fissures are once more forming and broadening in America, the spectre of separate lives and separate truths has re-emerged. In New York the divisions are stark. The police are openly defying Bill de Blasio, the mayor, whom they accuse of pandering to racial grievance-mongers. Meanwhile protesters are mobbing mid-town restaurants, chanting, “I can’t breathe” (in reference to Eric Garner, who died last year after being held in a chokehold by police).

It seems blame narratives and race martyrdom are fashionable again. Particularly depressing is the film Selma, about the 1965 Southern civil rights march from Selma to Montgomery, Alabama.

In direct contradiction of the historical record, the film portrays the former president Lyndon Johnson as Martin Luther King’s adversary in the civil rights struggle. There was plenty of white villainy to go around in the 1960s, but tarnishing the record and motives of people such as Johnson, who fought to usher in civil rights, simply feeds white paranoia that the black community has a political agenda.

The great tragedy in all this is that while America tears itself apart again from the inside out, the villains behind 9/11, the Fort Hood shootings, the Boston bombings and the Charlie Hebdo massacre in Paris watch and wait.