RICHMOND, Virginia - National political figures are calling for Virginia Gov. Ralph Northam to resign over a racist picture that appeared on his medical school yearbook page. But protests in the streets and pleas from lawmakers so far haven't convinced him to step down.

The former pediatric neurologist was out of sight Monday, huddling with members of his administration. He's trying to determine if he can survive the scandal that erupted Friday when a conservative news website published a racist photo from his 1984 medical school yearbook page.

After initially apologizing and saying it was him in the photo, Northam changed his story on Saturday.

"In the hours since I made my statement yesterday, I reflected with my family and classmates from the time and affirmed my conclusion that I am not the person in that photo," he said.

Northam was already drawing national attention for comments defending late-term abortions.

After the photo surfaced, he lost virtually all support from his own party. Democratic State Sen. Louise Lucas, who has spoken with the governor twice since the scandal erupted, is stunned Northam is still in office.

"He's lost moral authority to continued to govern and I know he understands those words," Lucas said.

National Democrats like Nancy Pelosi, Hillary Clinton and several 2020 presidential candidates have also called on him to resign.

Virginia's Republican speaker of the House said he has no plans to start impeachment hearings but hopes Northam leaves on his own. Most Virginians CBS News spoke with also want Northam gone, including April Brown, who volunteered for his campaign.

"It's time to go. You've got to get out. The fact that he's dug his heels in and says, 'No, I'm not, and what do I have to do to move forward and gain your trust?' Get out, is step number one," Brown said.

If he does resign, Lt. Gov. Justin Fairfax would take over, but he too is facing controversy. The same website that exposed Northam's yearbook is reporting an allegation that Fairfax sexually assaulted a woman at the 2004 Democratic National Convention in Boston. The woman said the incident took place in Fairfax's hotel room. On Monday, he denied assaulting her and said it was a consensual encounter.

On Monday, Fairfax stoked speculation that the governor's team leaked the assault allegations but a spokeswoman for Northam said that's untrue. Fairfax has not called for the governor's resignation but the two men have not spoken in 48 hours.