Let's build definitely-not-conscious AI

How does this sound like that?

• I think the term "conscious" is very overloaded and the source of endless confusion and should be tabood. I'll be answering as if the numbers are not "probability(-given-uncertainty) of conscious" but "expected(-given-uncertainty) amount of moral patienthood [LW(p) · GW(p)]", calibrated with 1 meaning "as much as a human" (it could go higher — some whales have more neurons/synapses than humans and so they might plausibly be more of a moral patient than humans, in the sense that in a trolley problem you should prefer to save 1000 such whales to 1001 humans).
• Besides the trivia I just mentioned about whales, I'm answering this mostly on intuition, without knowing off the top of my head (nor looking up) the amount of neurons/synapses. Not to imply that moral patienthood is directly linear to amount of neurons/synapses, but I expect that that amount probably matters to my notion of moral patienthood.
• I'll also assume that everyone has a "normal amount of realityfluid" flowing through them (rather than eg being simulated slower, or being fictional, or having "double-thick neurons made of gold" [LW · GW] in case that matters).

F, E, E, D, C, B, A, A, A, A, A, A, A, A

The As start at "Single-celled organisms, bacteria".

There's rather a jump in your list from sleeping humans to insects. How about, say, elephants? I'd give them a D.

ETA: For the second list: E or D for all the animals (I'm not sure where I'd switch, if at all), A for the rest. I'd go down to C for, say, earthworms, but what I would really mean is more of a D for there being anything like consciousness there, but it would only be a smidgeon.

All B (Yes, I know that variants of eliminativism are not popular here, but I was instructed to answer before reading further)

Discussions of consciousness without defining which meaning you're using will usually cause confusion and arguments, as Tamsin Leake points out.

Consciousness is used to label many aspects of human cognition. Which is complex. So consciousness almost means human-like, but without specifying on which dimensions you're defining the analogy.

I think there might be a low-cost route. As some motivation, consider how most jobs are definitely not "run around and find what needs doing and go do it" despite humans being so general. We prefer to toolify ourselves somewhat.

I think making the case properly is going to be some real effort though.

Yes excellent point! Someone said to me that they can add a for-loop to my precious constant-run-time system and it's not constant anymore. This is completely true.

But agents/employees suck in lots of ways compared to tools/services and I can see people just sticking with the tools if they're good tools.

If all the tools are sitting out there for anyone to use and compose as they please then the agentification is bound to happen. It's not always like this though. Eg almost nobody scripts with MS Word; banks make me come in person for a cashiers check.

I think I'll need to rewrite this post with clearer evidence & examples to avoid just going over the same old tool-vs-agent arguments again.

By "AI that does what we need AI to do" I meant "AI that can specifically do the things people ask of it" (as opposed to an AI that has already mastered every possible skill as preparation).

Perhaps I should have been more clear and modest in my description of the target... Like I think that N LLMs that each know 1 subject/skill is 95% as useful as 1 LLM that knows N subjects. And I think the N-LLMs setup is 20% as dangerous. But I come empty handed in terms of real evidence. Mixture of experts still being SOTA is some weak evidence.

I have nothing against tools, but for many desired outcomes, I don't want tools, I want someone who knows the tools and can do the work.

Same here but I end up going for the tool over the human more often than not because I can see & trust the tool. If I had a magic shapeshifting tool then I think I would have little need for the general agent.