The 34-year-old presenter - who was slammed as "utterly irrelevant" by one critic - will not have his existing contract renewed for a second season after receiving "terrible reviews".
An insider told the Daily Mirror: "Steve keeps saying that living in Los Angeles is a dream come true. He loves the weather and the women, and he loves working on X Factor - it's just a shame that the US public has not warmed to him.
"He got terrible reviews from critics when he started and things haven't got better. There have been mistakes and he just doesn't seem to have the knack for dealing with contestants.
"The executives at Fox have already decided not to renew his contract and so, short of a miracle happening, it is pretty much a done deal that he will not take part next season."
It is believed the Welsh star could be replaced by Pussycat Dolls singer Nicole Scherzinger, who took up a role on the judging panel alongside L.A. Reid, Paula Abdul and show boss Simon Cowell after Cheryl Cole was sensationally fired from the series after just four rounds of auditions.
A US X Factor spokesperson said: "No discussions have been had and no decisions will be made about any new contracts until the New Year."
Steve has previously blamed his "rude" presenting manner - which saw him fail to comfort axed 13-year-old hopeful Rachel Crow when she collapsed to the floor in tears live on air - on "screaming" producers.
He said: "I'm wearing an earpiece and there's about five people screaming, 'Shut Paula up. Shut her up now. Move on.' I'm going to have hearing damage in this ear by the end of series.
"I feel terrible because I feel on occasions I'm coming across as rude and abrupt and that's really not me. I cherish every word that comes out of Paula."