On accepting an argument if you have limited computational power.

Furthermore, we can't model anything more complicated then a hydrogen atom with QM without resorting to approximations, and by the time you get to something as complicated as bulk matter or atomic nuclei of heavy elements, we can't even verify that the predictions of QM are what we in fact observe.

From what I understand, we have more than one. We just don't know which, if any, is correct.

That specialization of labor does a lot of help doesn't mean that extreme specialization still does a lot of help. There are so many issues involved with letting someone else do something for you (finding/chosing the person, trust, explaining what you have to do, moving the person to the place, schedule/calendar issues, negotiating the price, legal issues, ...) that for many things, it's less efficient to pay someone to do it than to do it yourself, even if for the core of the task, a specialized person would be more efficient.

Also, you've to consider willpower/akrasia/enjoyment related issues. For example, many people will feel much more motivated when fixing your own house than when fixing someone else house, so even if you need more time to do it than a professional, you could still fill better doing it yourself, than working (even less) extra hours in your job and paying someone to do it. Oversimplifying things like that just doesn't work in real life.

And finally, you've to consider emergency situations. Trolley like situations are emergency situations, like if you see someone being mugged, or someone drowning of whatever, in those, you just don't have the option to pay someone to act for you.

In general, the best way to get something done is to pay someone who's better at it than you. As such, you can fairly accurately simplify it into thinking about how to earn money

I'm not sure this generalizes well - would this work if everybody was doing it? (It might).

I'd respond to that thusly: In order to extend our tools a little bit (as is necessary to deal with new situations), it helps to imagine extending them further than necessary. Otherwise multiple extensions in series will render them hopeless.

That's hardly a critique of the trolley problem. Special relativity itself stipulates that it doesn't apply to faster-than-light movement, but a moral theory can't say "certain unlikely or confusing situations don't count". The whole point of a moral theory is to answer those cases where intuition is insufficient, the extremes you talk about. Imagine where we'd be if people just accepted Newtonian physics, saying "It works in all practical cases, so ignore the extremes at very small sizes and very high speeds, they are faulty models". Of course we don't allow that in the sciences, so why should we in ethics?

I think this may have been given a down vote because to some a Singularity seems settled and rational. I take it to be the idea that eventually--inevitably--machines will become smart enough to improve themselves, that once they become just a bit smarter than human programmers and with the capacity of self-modification, that it will be a runaway phenomenon.

And that's why FAI is such a big deal: precisely because it's "far into the fog of future" in the sense that we don't have any control on the outcome (unless we align its values very precisely with our own).

But here's the rub - for me, anyway - might it be rational to take your own reasoning as suspect about such 'foggy' matters? That is, even if you are doing your best to reason rationally, might it be best to assume that your potential knowledge is reduced to very little and the remainder filled in by cognitive bias?

Okay, maybe this is a better approach---since questions like Singularity&FAI are potentially so important---can anyone point me to a resource that convinced them of its reality?

When confronted with highly speculative claims so beloved by philosophers, string theorists and certain AI apologists, my battle cry is "testable predictions!". If one argues in favor of a model that predicts a tiny probability of a really big harm, they better provide a testable justification of that model. In the case of Pascal's mugging, I have suggested a simple way to test if the model should be taken seriously. Such a test would have to be constructed specifically for each individual model, of course. If all you say is "I can't prove anything, but if I'm right, it'll be really bad", I yawn and move on.

Conversely, the inability of a putative Pascal's Mugger to make such predictions ought to apply a significant penalty to the plausibility of its claim. And simply increasing the threatened disutility won't necessarily help, since the more powerful the entity claims to be, the greater should be the implausibility of its inability to make testable predictions.

I can't rely on myself to falsify such stuff reliably.

You don't need to, as they would most likely be unable to simulate an amoeba, so you can call their bluff right there.

It occurs to me that I may have made my answer to your post a little long,but I figured that when I was unsure of how to judge something, more tests are better than fewer tests.

Starting with the basic tests from Wikipedia:

1: Does the model have organelles?

2: Does the model have cytoplasm?

3: Are the models organelles and cytoplasm enclosed by a cell membrane?

4: Does the model obtain its food through phagocytosis?

5: Is it a Heterotroph?

6: Does it have a single large tubular pseudopod at the anterior end, and several secondary ones branching to the sides?

It would probably be reasonable for a fake amoeba simulation to have all of these.

Then you start applying more tricky bits, like:

7: "Amoeba's are supposed to be able to recover from a forcible division if the nucleus is intact. If I remove some of the cytoplasm and keep the nucleus intact, does the simulation recover?"

8: "Cytoplasm is supposed to contain water. Does the bit of cytoplasm I extracted contain water?"

9: "Does the water divide into Hydrogen and Oxygen via electrolysis?"

Also, be sure to test things where the answer is expected to be no, as well, in case the simulation is just programmed to spit out "Yes." repeatedly.

10: "Does the Amoeba contain multiple nuclei?"

11: "Does the Amoeba have a human nervous system?"

12: "Is the Amoeba more than 99% Salt?"

Then ask 3 more questions which are LIKE question 7-12, but which I am not writing down, in case the person simply Googled for "Questions people have ever asked about Amoebas." and recorded appropriate answers.

Now make sure to ask questions which aren't Yes/No questions, because Yes/No questions don't actually accumulate bits of evidence all that quickly.

For instance

16: "How many Base pairs does the Ameoba have in it's genome?"

17: "How long is the Amoeba while moving?"

18: "How long is the Amoeba while not moving?"

19: "How fast can the Amoeba move?"

20: "Amoebas contain water. Water can put out forest fires. How many of your simulated Amoebas would I have to drop from 100 meters above a forest fire currently covering 100 acres of burning forest surrounded by 1000 acres of unburnt forest to put out the fire before the unburnt forest burns up?" (This seems like a bizarre question, but I am trying to be thorough. Also, someone's reaction to a question that they would never be expected to have an answer to under normal circumstances can itself be revealing.)

And again, ask a few more unlisted questions here, as well, including at least one other question just as odd as the forest fire question. (again, they might have prepared rote answers to all known Ameoba related items.) Also, you may want to ask some other later questions first if you suspect you might have a limited number of questions.

Now it is possible that the person will say "I'm only simulating an Amoeba, there isn't a simulated forest that I can run forest fire tests on." and "I'm only simulating an Amoeba, there isn't anything in the simulation which can be used to generate electricity to do electrolysis on water." and "I'm only simulating an Amoeba, there isn't anything in the simulation which can be used to forcibly divide it." And you would expect SOME of these. For instance, I would be surprised if anyone who was simulating an Amoeba ALSO simulated a forest for forest fire tests using the water inside that Amoeba. That might for instance indicate that they DID have unlimited computing power and simply simulated everything. Or that they were faking answers. It depends on the amount of time the answer takes and the type.

But you could say that to anything. "I'm only simulating an Amoeba, there isn't any light in the simulation to see it." "I'm only simulating an Amoeba, there are no objects in the simulation to test it's permeability to objects." "I'm only simulating an Amoeba, there is nothing in the simulation which can be used to judge how fast the Amoeba is."

If the mugger seems to be saying that or something similar to EVERYTHING, it's very probably a fake, particularly if it sounds almost like a rote answer with no delays.

Given an answer to all of those questions, and a list of objections if any, and timing knowledge about the questions, I think I would have a fairly good grasp of whether the model simulating an Amoeba was valid (or simulating anything really, I can change the specific questions given something else.) given about 25 questions or so. Of course there are edge cases. It is entirely possible to come up with a list of answers which I will see, and then come to the wrong conclusion about. but it should take enough effort that it would be easier to get the mugged amount of money from someone else rather than making a series of incredibly elaborate ruses just for a few dollars.