The Take That star confirmed last year that the 2013 show was his last. But Simon hinted that he wasn’t Gary’s biggest fan, telling the Daily Mirror: “The truth is, and I am not being rude, I wouldn’t hire him to run my record label.
“Essentially that is what you are doing on The X Factor. You have got to be wide-eyed and Cheryl (Cole), one of the main reasons she is coming back is she is good at that.
“She is the one who really spotted Cher Lloyd.”
Asked if he and Gary could work on the same panel, he shook his head and replied: “Gary was seen as the head judge and so am I.
“It is not so much what I say on the show as what I can do. I do believe I am the best person at spotting a potential star and making them work well on the show.”
Simon had praise for Louis Walsh, who said he was bowing out of the show and then changed his mind, telling the newspaper: “I would like to think after 10 years Louis deserves a spot.
“Wouldn’t you miss him if he wasn’t there? I tried it without him once and it didn’t work. “
He said that he would like a singer, rather than a record executive, to take the last judging spot.
Simon also defended lowering The X Factor’s minimum age to 14, saying: “Can you imagine saying to Gary Barlow at 16, ‘You’re too young to be a musician’? Give me a break.”
And he said he had not given up on the US, despite X Factor USA being cancelled .
“When the time is right I will go back to America,” he told the newspaper. “I have already been offered two shows over there. They still like me.”