The manager said that he still regularly thinks about his early departure from the Three Lions.
The 62-year-old took charge of just one game as England boss after taking over in July last year, during which he featured in a 1-0 win over Slovakia in September 2016.
He left by mutual agreement with the FA following a newspaper investigation claiming he offered advice on how to “get around” rules on player transfers.
The FA had accused Allardyce of “inappropriate” conduct and Gareth Southgate was placed in temporary charge, with Allardyce going on to excel in charge of Crystal Palace before leaving Selhurst Park at the end of last season.
Allardyce said on Sunday that he had numerous regrets about the whole episode and struggles to get it off his mind.
“Yeah, there’s no doubt about that (the biggest regret of his managerial career).
“Legally, it is a difficult subject for me to talk about at the minute but I think to say one thing personally: I don’t think I should have left,” he said.
Get more stories like this on Twitter & Facebook