I see it like this: "Fighters fight, Clerics and Magic-Users use magic, and Specialists are people who spent a lot of time developing skills people don't normally have - not everyone is supposed to be able to pick locks, fiddle with mechanisms and pick pockets." Keeping this in mind, I'd say that the classes are quite balanced in the amount of things they will get to do during an adventure.
However, if your players are complaining that the Fighter, Cleric and Magic-User 'don't have skills', ask yourself this: have you been making them roll dice to accomplish things they probably shouldn't have to roll for?
Examples:
Your players want to setup an ambush for a caravan and they have plenty of time to find a good spot. They also describe a specific plan - finding a spot with dense vegetation and hills on both sides, and jumping from the bushes / attacking with bows. In this situation, don't make them roll stealth! Instead, roll for the caravan guard's surprise with a 4 in 6 chance. And if the ambushed party is outnumbered 2 to 1 and consists of normal 0-rank humans, you could even rule that the party manages to kill a few caravan guards before even rolling the dice, as they jump out from the bushes with their weapons ready.
A Fighter wants to sneak up to a guard who is sleeping on duty. Don't make him roll the stealth skill for this unless you think there is a very good reason why he could fail (such as small, unseen objects he could trip over in the way, gravel on the ground that would make a lot of noise, etc.) and there is a also good reason why, in this situation, someone who is a specialist could pull it off instead ("being a master of stealth, you manage to avoid tripping over some cooking pots at the last second" / "being a master of stealth, you manage to move over the gravel without making a sound"). However, even if the Fighter fails the stealth skill roll, don't penalize him overmuch! He doesn't need to have failed completely. After all, he still got the drop on someone who was sleeping, and that should count for something.
A Specialist wants to approach a castle in full daylight without being seen, and there are guards on the walls who are on full alert for some reason. This is the kind of outrageous feat that only a Specialist should be able to consistently pull off, as he runs from tree to tree in the exact split seconds in which the guards get distracted. And even then, he could still fail. In this case, roll stealth.
My opinion is that removing the Specialist class is a 'bad idea' unless you're ok with changing the look & feel of the game - in which case, by all means, it's your game.
I think that the important thing to remember is that you should give everyone a fair chance to be on the spotlight. Try to think of 'scenes' that make each of them the 'main character'. If your players aren't getting any spotlight time (*and* if it is not their own fault, that is, they are not consistently avoiding every chance you give them to act like proper adventurers), ask yourself if there is anything you could add to the game that would make them feel useful to the group.