The “heirs to the skinheads and the Paki-bashers of old” were among those who took to the streets of London for the Unite the Kingdom rally, the Home Secretary said.
“While not everyone chanted racist slogans, some did. Clear that in their view of this country, I have no place,” Shabana Mahmood said.
Between 110,000 and 150,000 people turned out for the rally organised by far-right activist Tommy Robinson on September 13, according to the Metropolitan Police.
At the Labour Party conference, Ms Mahmood said that “division within this country will grow” if her party fails to address immigration concerns.
In her speech, the Home Secretary set out a series of conditions for those seeking indefinite leave to remain status.
Under proposed changes, migrants who want to remain in the UK will have to learn English to a high standard, have a clean criminal record and volunteer in their community.
They will also need to be working, paying national insurance and not be claiming benefits.
Speaking on Monday, Ms Mahmood told the conference: “Just days into this job, on September 13, 150,000 people marched through London.
“They did so under the banner of a convicted criminal and a former BNP (British National Party) member.
“While not everyone was violent, some were, 26 police officers were injured as they tried to keep the peace.
“And while not everyone chanted racist slogans, some did. Clear that in their view of this country, I have no place.
“It would be easy to dismiss this as nothing but an angry minority, heirs to the skinheads and the Paki-bashers of old, and make no mistake, some were.
“But to dismiss what happened that day would be to ignore something bigger, something broader, that is happening across this country.
“The story of who we are is contested.”
Ms Mahmood said that working-class communities will turn away from Labour and “seek solace in the false promises of (Reform UK leader Nigel) Farage” if the Government fails to act on migration concerns.
She said: “They will turn towards something smaller, something narrower, something less welcoming, and the division within this country will grow.
“So the challenge we now face is this, not just to win the next election but to keep the country together and to fight for our belief in a greater Britain, not a littler England.”
Ms Mahmood also said “patriotism, a force for good, is turning into something smaller, something more like ethno-nationalism”.
A consultation on the changes to the indefinite leave to remain status will be launched later this year.
In its white paper published in May, the Government also pledged to increase the amount of time migrants have to wait before they can apply for indefinite leave to remain in the UK, to 10 years.
Currently, most migrants who come to Britain on time-limited work visas can make an application for indefinite leave to remain after five years.
