Maybe it's as simple as the fact that jobs make more consistent demands on time, if it's just a useful hobby people might question your commitment.
But I think it ignores a tendency to underrate altruistic jobs. I would think that this is because altruistic jobs demand a lot of time, rather than flashes of brilliant insight. If you are an average intelligent and caring person, your intelligence might be used better in a "non-altruistic" job, and if your job isn't altruistic enough for your taste (or for appearances) you can spend the rest of your time 'doing good.' People will then admire you for your efficient prioritization and for your kindness.
If you make the conscious decision to enter into an altruistic job, there is the possibility that you will be classed with people who are in altruistic jobs precisely because the jobs don't require flashes of brilliant insight; then (even if your hobby involves splitting atoms) no one will care about your hobby, because you will be hanging out with people at work that can't split atoms and the folks at the Domestic Atomic Association are probably not as nice as you are.
If you are a successful software consultant and you read to disadvantaged students on the weekend, people think that you are a great guy; if you are a social worker that writes Java applets, chances are people think that you're some chick that couldn't get a private sector job (note the heavy-handed use of subliminal linguistic gender stereotypes.)