Abushagur submitted plans for the establishment of 10 ministries, among them defense, interior and justice. He withdrew his first cabinet proposal after it encountered strong criticism and protests.
Abushagur had only 72 hours to get a new cabinet list approved or he would be forced to resign less than a month into his term.
By Sunday, parliamentary spokesman Omar Humeidan said he had reached the end of the 25-day period in which he must form a government, reports Fox News.
The vote to remove Abushagur from office was overwhelming. The General National Congress (GNC) voted 125 to 44 against the proposal, reports the BBC. Seventeen members abstained.
Abushagur's dismissal means that Libya's government may be without permanent, democratically-elected leadership for many weeks, reports the Washington Post.
The BBC reports that the GNC now has three to four weeks to elect a new leader.
Without a stable government, the investigation into the killing of US Ambassador J. Christopher Stevens and three other Americans may remain on the back burner.
AP reports that the incoming leader will be responsible for rebuilding Libya's army and police force and removing major pockets of support for the former regime.