David said the music star was working on his tour.
Sir Elton John was approached about an X Factor judging role but was too busy, husband David Furnish has said.
Robbie Williams and his wife Ayda Field are thought to be joining the talent show, after being signed up by Simon Cowell, but full details are yet to be revealed.
David told The Sun: “There was talk about the possibility of a judging role for Elton.
“But he does 100 shows a year and he doesn’t have time for X Factor judging roles. We’re too busy with Elton’s tour”.
Robbie and Ayda are believed to be replacing long-serving judge Louis Walsh and pop star Nicole Scherzinger.
Sharon Osbourne will only appear as a judge during the live episodes, skipping the audition process of the ITV series, which returns in August.