First, some history. In 1883, during the middle of the Gilded Age, Alva Vanderbilt decided to force her way into the elite sector of New York society known as “Mrs Astor’s Four Hundred”.

For years, Mrs Astor had maintained her own list of acceptable blue bloods. “Old money”, a relative term compared with Europe, counted; “new money” did not. Unfortunately for Alva, the Vanderbilt family wealth — which topped $1bn (£616m) in today’s money — was considered new money.

It was perhaps not surprising that Mrs Astor fought so hard to maintain the tribal identity of New York high society. The Gilded Age was an era of sudden prosperity (the economy grew by 400% between 1860 and 1900) and gross income disparities. According to best estimates, by 1905 the top 1% held more than 50% of the country’s wealth. Yet it was also an era of unprecedented social mobility.

Alva Vanderbilt understood that the issue at stake was class versus caste. Armed with that insight, she built the showiest mansion on Park Avenue, planned history’s most expensive costume ball (costing $250,000 when the average income was $380 a year), and invited every smart person in New York — except for Mrs Astor and her daughter Carrie, then in the middle of her debutante season.

According to legend, mindful of young Carrie’s marriage prospects, Mrs Astor swallowed her pride and deigned at last to pay a formal call on Mrs Alva Vanderbilt. The visit elicited an invitation by return of post. For the sake of her daughter, Mrs Astor allowed the walls of “society” to come tumbling down.

The reason it is important to remember this story is because America has reached one of those periodic crossroads when nothing seems certain any more. This year, a slew of books and studies has confirmed that not only is there an ever-growing wage gap between the top 10% and the bottom 90%, but the middle class is both losing its share of the national income and shrinking in size. The gap between the have-some and the have-it-all has become so great that commentators are referring to the “new Gilded Age”.

Income inequality, while troubling and sometimes infuriating, has never gripped Americans in the same way that it has Europeans. Marx’s triumph of the proletariat was not and is not a vote winner. On the other hand, social mobility (and its twin sibling, equality of opportunity), may be the closest thing to a national religion.

The idea that the middle class, the standard-bearer of freedom, prosperity and optimism, is in decline appears to strike at the very heart of the American dream. After all, what sort of American “dream” rewards hard work with economic demotion? The term was first coined by the historian James Truslow Adams in his 1931 book, The Epic of America. The American dream, he wrote, was “of a land in which life should be better and richer and fuller for everyone, with opportunity for each according to ability or achievement”.

Adams was articulating an idea that had been deeply embedded in the American psyche for centuries. In 1785 Thomas Jefferson told a friend that he was determined to prevent the social evils that accrued when property was “absolutely concentered in very few hands”. Jefferson achieved his wish.

Half a century later, the French commentator Alexis de Tocqueville observed that in the US, “new families are constantly springing up, others are constantly falling away, and all that remain change their condition”.

Successive presidents have reiterated the theme of America as the land of opportunity. Theodore Roosevelt spoke of every man having “a fair chance to make of himself all that in him lies”. In this year’s state of the union address, President Barack Obama declared: “I believe, that here in America, our success should depend not on accident of birth, but the strength of our work ethic and the scope of our dreams.”

Now for some statistics. Class is fuzzier and less obvious in America than in Britain, but it still exists. Numbers pulled from the Congressional Budget Office and the Census Bureau show that the median household income in 2012 was $50,017. In 1999 it was $56,080. Home ownership has also taken a knock. In 2002, 70% of Americans owned their home; today it’s 65%. For many Americans, the last straw is the news that Canada’s middle class has become the richest in the world.

For Democrats, the plight of the middle class reflects the lack of equality in the economy. Their remedy is stronger unions and higher taxes. To Republicans, the diagnosis is lack of growth. Their remedy is deregulation and lower taxes. Both believe their prescriptions will restore the American dream; and if they don’t, for Democrats the cause will be racism and corporate cronyism. For Republicans it will be big government and over-generous welfare.

However, as the story of the Vanderbilts versus the Astors reminds us, income inequality isn’t a straight path to economic stagnation. Both are social ills but they aren’t the same ill. They need to be approached on their own terms. In the 1930s, Truslow Adams’s American dream was thought to be just that, a dream. President Franklin Roosevelt declared “equality of opportunity as we have known it no longer exists… The unfeeling statistics of the past three decades show that the independent businessman is running a losing race.”

Maybe so, but what seemed a permanent state of affairs then was in fact temporary. The key to understanding social mobility lies in deciphering the influence of institutions and individuals.

A landmark study this year led by the Harvard economist Raj Chetty found that regardless of the recent economic swings, social mobility in America hasn’t changed in half a century. Nor does being poor per se limit a person’s ability to advance up the scale. The American dream is intact. Instead, the Chetty team found that the greatest predictor of entrenched poverty was lone motherhood. The chance of a child born to a poor, single mother making it to the top echelons of society is 5%.

While not a nightmare, that’s still a tragedy. It isn’t forces that keep people down, it’s other people.