Conservative hardliners have started to float potential alternative candidates should House Speaker Kevin McCarthy be ousted, as internal frustration with the California Republican has grown, according to four GOP sources familiar with the conversations.

But the rebels still have not coalesced around a single candidate and it's not clear anyone else could get 218 votes -- the same problem they ran into during the speaker's race.

In recent days, some of the names that have been floated in House Freedom Caucus meetings and other private conversations include Oklahoma Rep. Kevin Hern, chairman of the Republican Study Committee; Rep. Mark Green of Tennessee, chairman of the Homeland Security Committee; and Rep. Tom Emmer of Minnesota, the House GOP Whip. The Washington Post was first to report Emmer's name, though he told the publication he has no interest in the job.

Sources told CNN Green was approached by some members about the idea, but Green also said he wasn't interested.

Republican Rep. Matt Gaetz of Florida has repeatedly threatened to force a removal vote, known as the motion to vacate, but he has not indicated when he might do it, other than to say he is waiting to see how the shutdown fight play outs, though a vote could be triggered at any time.

GOP sources told CNN there is no formal mobilization effort among hardliners to recruit a challenger or whip motion to vacate votes. But should McCarthy face a removal vote, the speaker is likely to fight it out on the floor, making the push to remove him more challenging.

McCarthy critics believe they'll have the votes to oust him, even as the speaker has repeatedly maintained he's not concerned about such a vote and believes he could win it.

"You know, I think that members should be looking at for stronger leadership," said Rep. Eli Crane, an Arizona Republican who told CNN he could support an effort to oust McCarthy. "And you know what, quite frankly, I know that people in my district that I represent want to see stronger leadership. So I'm all about it."