Simon, who has been working in Los Angeles, only communicates with Scott Bruton, Austin Drage and Eoghan Quigg through his personal assistant, according to the Daily Mirror.
He will meet them again in Barbados for the first time since the judging in Barbados when he attends their studio rehearsals on Thursday.
A show source told the Mirror: “He’s not exactly been hands on. Simon is busy, but he could have called to make them feel special.”
Scott, 19, added: “He’s always in LA doing promotion for American Idol, flitting here and there.”
But new judge Cheryl Cole has been in constant contact with her singers – Diane Vickers, Alexandra Burke and Laura White – and even had them over for dinner.
Alexandra, 20, said: “She even texted me at 2am last night to see how my hair was doing.”
Judges Dannii Minogue and Louis Walsh have also put in the time with their acts.
But an X Factor spokeswoman said: “Simon has been away, but has a team of experts, who he is in constant contact with, working with his singers.”
The first round of live performances starts this Saturday, October 11, at 7:15pm.