I also think four might be two low, but I've been thinking along slightly different lines.
I'm considering if I get to run LotFP to allow all characters to begin with points to spend on Specialist skills equal to 2+Intelligence Modifier, and then Specialists get an extra four or six points on top of that. But I'll also be adding some more skills, including Lore (for random knowledge that isn't covered by your class), and possibly Charm and Intimidate (or maybe just a single 'Influence' skill) to add a bit more character skill to those areas.
The basic idea is that if this is stuff that everybody can do then people should vary in their ratings at least slightly, while still allowing specialists to be just better in their chosen skills (because even when everybody begins with an average of two pips to place an additional four is an attractive boost). This way most specialists can start out good in two skills of their choice, instead of being good at one and bad at everything else.