There is no objective fact of the matter regarding moral standards. Rather, we want a moral system that can be widely adopted and that when widely adopted promotes things we find good.

A moral system that said "you have to spend every waking moment curing malaria and feeding the hungry" would probably either just make people feel burned out and miserable or else be rejected outright. Many imaginable and prima facie plausible moral systems turn out to say that. A moral system that said "just do whatever the hell you want" would probably lead to few people bothering to cure malaria and feed the hungry.

It seems plausible to me that a system that says "you should be making things better for others but it's fine to devote most of your time and energy and resources to your own welfare and that of your family" does, given human nature, actually roughly maximize net good done. I expect the optimum is more demanding than the average person's actual moral system, but probably not (much?) more demanding than the average effective altruist's.

I've occasionally tried to find philosophers offering a reasoned defence of supererogation, of the idea that you are not obliged to do all the good that you could do, but I have not found much, until I looked just now. Here is an article contrasting two conceptions of what morality is: morality of law, and morality of virtue.

The morality of law rules out supererogation. The moral law says what is good, which is necessarily obligatory. Utilitarianism (I say, the article does not) is an example of law-based morality. Yes, you must always and everywhere undertake the best thing you could possibly do: that is what good means. The best possible action is compulsory; all else is forbidden. The utilitarian view is the morality that is generally asserted, if not accepted, on LessWrong and in EA. But it is not the only view.

The morality of virtue enjoins one to cultivate virtue — one's own virtue. These are certain qualities of character from which certain actions will flow, but it is the character that matters.

The paper is more concerned to argue that the morality of law excludes supererogation (starting from the initial position that the exclusion of supererogation is against our intuitions and requires explaining). It references other work arguing that the morality of virtue may be more welcoming of the concept.

Personally, I find myself more in agreement with the morality of virtue. I will not torture kittens or eat chimpanzees, but I do not much care whether Spain still has bullfights, and I have no qualms about eating meat in general. I am more interested in cultivating my own garden than in whether everyone else has a garden. A mere accumulation of more and more people living much the same lives does not strike me as morally valuable. I value the heights of an unequally distributed civilisation over a uniform mediocrity.

But I have not troubled to build a moral system around these observations of how I live and prefer to live.

My behavior is a compromise between different parts of me, and I'm good enough. “Doing good the best things rather than bad good/neutral things is generally supererogatory”. I was very relieved to learn that word and concept!