Thanks! I’m pretty sure this isn’t the one I saw, but it works even better for my purposes.

Edit: I'm working on an AI timeline / risk scenario where BCIs and neuro-imitative AI play a big role. I've sent you the draft if you're interested.

I think using AI + BCI + human brains will be easier than straight AI for the same reason that it’s easier to finetune pretrained models for a specific task than it is to create a pretrained model. The brain must have pretty general information processing structure, and I expect it’s easier to learn the interface / input encoding for such structures than it is to build human level AI.

Part of that intuition comes from how adaptable the brain is to injury, new sensory modalities, controlling robotic limbs, etc. Another part of the intuition comes from how much success we’ve seen even with relatively unsophisticated efforts to manipulate brains, such as curing depression.

I’m actually woking on an AI progress timeline / alignment failure story where the big risk comes from BCI-enabled coordination tech (I've sent you the draft if you're interested). I.e., instead of developing superintelligence, the timeline develops models that can manipulate mood/behavior through a BCI, initially as a cure for depression, then gradually spreading through society as a general mood booster / productivity enhancer, and finally being used to enhance coordination (e.g., make everyone super dedicated to improving company profits without destructive internal politics). The end result is that coordination models are trained via reinforcement learning to maximize profits or other simple metrics and gradually remove non-optimal behaviors in pursuit of those metrics.

This timeline makes the case that AI doesn’t need to be superhuman to pose a risk. The behavior modifying models manipulate brains through BCIs with far fewer electrodes than the brain has neurons and are much less generally capable than human brains. We already have a proof of concept that a similar approach can cure depression, so I think more complex modifications like loyalty/motivation enhancement are possible in the not too distant future.

You may also find the section of my timeline addressing progress standard in AI interesting:

My rough mental model for AI capabilities is that they depend on three inputs:

1. Compute per dollar. This increases at a somewhat sub-exponential rate. The time between 10x increases is increasing. We were initially at ~10x increase every four years, but recently slowed to ~10x increase every 10-16 years (source).
2. Algorithmic progress in AI. Each year, the compute required to reach a given performance level drops by a constant factor, (so far, a factor of 2 every ~16 months) (source). I think improvements to training efficiency drive most of the current gains in AI capabilities, but they'll eventually begin falling off as we exhaust low hanging fruit.
3. The money people are willing to invest in AI. This increases as the return on investment in AI increases. There was a time when money invested in AI rose exponentially and very fast, but it’s pretty much flattened off since GPT-3. My guess is this quantity follows a sort of stutter-stop pattern where it spikes as people realize algorithmic/hardware improvements make higher investments in AI more worthwhile, then flattens once the new investments exhaust whatever new opportunities progress in hardware/algorithms allowed.

When you combine these somewhat sub-exponentially increasing inputs with the power-law scaling laws so far discovered (see here), you probably get something roughly linear, but with occasional jumps in capability as willingness to invest jumps.

I think there's a reasonable case that AI progress will continue at approximately the same trajectory as it has over the last ~50 years.

I don't have a specific mental image for what I mean when I say 'non-deterministic', I was placing a bet on the assumption that YimbyGeorge was hypothesizing that conscious was somehow fundamentally mysterious and therefore couldn't be 'merely' deterministic, based on pattern-matching this view rather than any specific mental image of what it would mean for consciousness to only be possible in non-deterministic systems.

Do you think it will ever be possible to simulate a human mind (or analagous conscious mind) on a deterministic computer?

Do you think it possible in principle that a 'non-deterministic' human mind can be simulated on a non-deterministic substrate analagous to our current flesh substrate, such as a quantum computer?

If yes to either, do you think that it is necessary to simulate the mind on the lowest level of physics (e.g. on a true simulated spacetime indistinguishable from the original) or are higer-level abstractions (like building a mathematical model of one neuron and then using this simple equation as a building block) permissible?

(Also, are you just asking about Robert Roger Penrose's view or is this also your view?)