Nancy Pelosi, D-Calif.

Let’s be clear: Pelosi would like to see Rangel go. Railing against him – a tax cheat running the tax-writing committee! – is a political goldmine for the GOP (which has already tried to embarrass House Democrats by forcing floor votes on Rangel’s gavel status). But prying him from Ways and Means is trickier than you’d think.

Her first option is to convince Rangel to give up his post voluntarily. Good luck with that. Rangel is a House lifer. He won his seat in 1970, part of a generation of African-American congressmen who decided they’d have better luck climbing the internal House ladder than trying to run for higher office in their home states. The seniority system ensured his slow, steady rise and when – after 12 long years of GOP rule – Democrats won back the House in 2006, Rangel, at 76 years old, found himself in one of Washington’s most powerful perches.

For Rangel, the Ways and Means chairmanship is both a reward and a ticket to relevance. The alternative – backbencher status – is flatly unacceptable. Don’t forget: he was so fed up with being out of power that he announced in the summer of 2006 that he’d retire after that year’s election if Democrats didn’t take back the House.

Not surprisingly, Rangel already began digging in his heels on Thursday night, insisting that the forthcoming report – to be released on Friday – actually exonerates him. Expect him to stick to this script for as long as he can.

So it’s likely that Pelosi will have to play hardball if she wants to oust Rangel. Which is where things get tricky, because within the Democratic caucus, big-picture political calculations – what will help us win in November? – are often secondary to factional politics.

In Rangel’s case, the particular issue has to do with the Congressional Black Caucus, of which he is a founding member. For understandable reasons, the CBC tends to be sensitive when it comes to gavels and committee assignments. Historically, many of its members – like Rangel – have relied on the seniority system for their political status and power. The idea of removing a CBC member from a choice assignment or bypassing a CBC member for a plum opening is not to be considered lightly.

Pelosi discovered this four years ago, after the feds raided then-rep. William Jefferson’s home and discovered in his freezer $90,000 in – literally – cold, hard cash. Jefferson was a member of Ways and Means and, as with Rangel now, the Republican taunts began almost instantly.

The 2006 election was months away and Republican corruption was a key plank in the Democratic platform. Pelosi wanted to be rid of Jefferson – fast. He wouldn’t resign from Ways and Means, so she sought his expulsion. And when she did, all hell broke loose. CBC members rushed to his defense, noting that he hadn’t been convicted of anything (or even) indicted. To punish him preemptively, they argued, would violate due process – and raise the specter of a double-standard, since white Democrats under similar clouds had not been forced to give up their committee seats.

After weeks of maneuvering, Pelosi finally arranged for a vote of the full Democratic caucus on Jefferson’s fate. The meeting was heated, with CBC members – and some unlikely allies—revolting. Despite the strong case for booting Jefferson, nearly 60 House Democrats voted not to remove him – not quite enough to save him, but plenty to teach Pelosi a lesson.

It’s worth noting, too, that Pelosi received some quiet help when she moved on Jefferson from Rangel and several of his CBC allies. Jefferson was something of an outcast in the group. Out of principle, most CBC members stood with him. But some of the group’s most important leaders – like Rangel – quietly assisted Pelosi (hoping, no doubt, to gain a chit from a future Speaker).

But when it comes to Rangel now, Pelosi won’t find similar help at the top of the CBC. The Harlem Democrat is a CBC lion, a founding member who inspires deep reverence from its members. If Rangel asks them to fight for him, they will.

This is why Pelosi has been dreading this day for so long. If she pushes, she could probably muster the votes to bounce Rangel, but the price would be steep: an ugly fight that would attract national press attention, hand Republicans cannon fodder, and divide her own coalition. Or she could look the other way, which would give the GOP even more ammunition and anger swing-district Democrats, who don’t want to have to explain to their constituents why they’re “protecting” Charlie Rangel.

Maybe Rangel will have a change of heart and fall on his sword. But to hear him on Thursday night, that hardly seems likely. So now it’s Pelosi’s move.

The break had its roots in the Iraq war, and Harman’s role as the top Democrat on the House Intelligence Committee when it was launched, and as the occupation turned sour. A “select” panel, Intelligence is unique from other House committees in that it’s composed of an equal number of Democrats and Republicans; members are privy to sensitive documents and top-level security briefings, and the idea is that their work on the committee is above partisan politics.

The problem, from Pelosi’s standpoint, is that Harman’s status as the No. 1 Democrat on Intelligence essentially made her the highest-profile House Democrat in the middle of the last decade. At the time, don’t forget, Republicans had been running the House for a decade, and they also controlled the White House and Senate. To the national media, Pelosi, who was elected the Democrats’ House leader in 2002, was a bit player, especially on matters of national security and foreign policy. Her caucus had little clout, and the press mostly tuned her out. When television news producers needed a congressional Democrat to discuss Iraq, they gravitated toward Harman; at least she (theoretically) spoke with the benefit of intelligence information few others had access to. On one stretch in 2006, as the war spiraled out of control, Harman was on a Sunday morning news show four times in a six-month stretch; Pelosi’s total: zero.

“Maybe it’s my 15 minutes of fame, maybe it’s longer,” Harman said as her exposure increased. “This is why I came here.”

But Harman didn’t used her platform the way Pelosi wanted her to. Instead of aggressively challenging the Bush administration over Iraq and its intelligence-gathering operations, she developed a reputation for cooperative work with the panel’s Republicans and, in the eyes of many war critics, acquiescence to the White House’s position. By 2005, Pelosi indicated that she would force Harman out as ranking member after the 2006 elections. Technically, Pelosi cited term limits; members had traditionally rotated off the committee after four terms, but that unofficial limit was easy to get around. Harman, one of the wealthiest members of Congress who enjoys personal relationships with many elite party donors, responded by enlisting her friends and allies in a pressure campaign. That only irritated Pelosi, who dug in her heels, insisting that Harman would be forced to leave the committee.

The tension was especially notable because of how close Pelosi and Harman had once been. Pelosi had already been in the House for six years when Harman, elected to represent a Southern California district, arrived in 1993, although they’d known each other for years, dating back to Pelosi’s days as chairwoman of the California Democratic Party. Pelosi was among those who encouraged Harman when she gave up her seat to run for governor in 1998, an effort that ended with a dismal third-place showing in the Democratic primary. Pelosi then recruited Harman to seek her old House seat, which had fallen into Republican hands, in 2000. As an inducement, she and Democratic leaders promised to restore Harman’s seniority — meaning her position on the Intelligence panel, where Pelosi was then the ranking member. When Pelosi rotated off the panel in January 2003, months after winning election as minority leader, she installed Harman as her replacement as ranking member. It was a move not without controversy; members of the Congressional Black Caucus believed the spot belonged to one of their own, Georgia Rep. Sanford Bishop, who would have been in line had Harman’s old seniority not been factored in. But Pelosi gave the job to Harman, a testament to the strength of their alliance at that time.

In the 2006 midterm elections, Democrats won back the House for the first time since 1994. Harman, despite being publicly rebuffed by Pelosi for the better part of two years, still hoped to persuade the speaker-to-be to let her keep the Intelligence slot; with Democrats in the majority, Harman would be the chairwoman. Harman allies, including some friendly media members, followed up by painting Pelosi’s decision as her first major test: Would she prove she was “serious” about national security and keep Harman around? But Pelosi didn’t even nibble. Harman was given the boot and the chairmanship went to Texas Rep. Silvestre Reyes (with Pelosi passing over Alcee Hastings, the Florida congressman who was technically behind Harman and who, as a federal judge, had been impeached in the 1980s).

It’s not fair to say that Harman has been irrelevant since that moment. She is, after all, still frequently quoted in news article and is a regular on the cable news circuit. But life in the House hasn’t been the same for her, either, these past four-plus years. Pelosi’s move severely reduced her clout within the House, and her visibility has been eclipsed by Pelosi and several other House Democrats these past few years. There was speculation at the end of 2006 that Harman would leave the House without her Intelligence gig, but she stuck it out — initially, perhaps, in the hope of landing a Cabinet post (Homeland Security was mentioned in the press) in a Hillary Clinton administration (Harman endorsed the former first lady in the Democratic primaries), and then (perhaps) to use the 2010 election to win a measure of vindication after being linked to an espionage scandal involving AIPAC in 2009.

Maybe if Pelosi had walked away from Congress after last November’s Republican triumph Harman wouldn’t now be leaving Congress. A new leader might have looked differently on Harman and her credentials. But Pelosi stayed, and as long as she’s around, there’s really no room for Jane Harman to do much in the U.S. House. And so now, finally, she’s leaving.