Simon was speaking at the Music Industry Trusts Award at London’s Grosvenor House Hotel, where he was receiving the 24th edition of the prestigious music award to honour his contribution to the music industry.
He told the Press Association: “You never take a show off that’s getting 10 million viewers. It would be pointless.”
The TV show has been under fire for an apparent ratings slump, but he said: “You’ve got to understand, the way we watch TV now is so different. We had 6.5 million a week ago – rounded up, it’s nearly 10 million. That’s how many people watch the show every week.”
Simon, 56, shared a tender moment with his partner Lauren Silverman, 38, on the red carpet at the event, planting a kiss on her lips.
He said: “I said to everyone organising it, I don’t want to know anything. I like surprises. Hopefully they’re good surprises.”
Sinitta and Louis Walsh (Ian West/PA)
Talking about the music industry, he said: “It’s such a tough business, this, I’m not going to lie. When I first started, it feels like yesterday – if I’d have known at my age I was still happy, still getting a few hits, that would have been enough.”
Former X Factor host Louis Walsh, 63, who was axed from the judging panel by Simon for this year’s show, said: “It’s my first time sitting at home in 11 years, it’s great – in my pyjamas, watching X Factor. But he looks different, his face – is it the lighting, is it the make-up?”
On a potential return, he said: “Oh I don’t know, I’m keeping an eye on it. I love Louisa and I love Che. They’re the two people I love.”
Caroline Flack at the awards (Ian West/PA)
On the advice he would give to Simon, he said: “You’ve made too many changes. Listen to the public. He has to listen to the public – he will for next year.”
Explaining Simon’s appeal, he said: “He has got a huge ego, he just wants to be the best, that’s really what drives him. He’s amazing in every way. He looks different, he walks different, he dresses different, everything is different about him. His face is different!”
The X Factor host Caroline Flack, 35, said: “I did three years on the Xtra Factor, so I feel like I’m at school and now I’m in the fourth year!”
On Simon, she said: “He’s very personable. He’s actually not as scary as you think. Before I first met him I was like, ‘oh my lord, what is this going to be?’ But he actually makes you feel more comfortable than anybody else.”