I initially assumed something similar to what you just described. However, it's plausible to me that in practice the line between "program" and "data" might be blurry here.

Thanks for the link. The problem of how to have a cryptographic root of trust for an uploaded person and how to maintain an on going state of trusted operation is a tricky one that I'm aware people have discussed. Though it's mostly well over my cryptography pay grade. The main point I was trying to get at was not primarily about uploaded brains. I'm using them as an anchor at the extreme end of a distribution that I'm arguing we are already on. The problems of being able to trust its own cognition that an uploaded brain has we are already beginning to experience in the aspects of our cognition that we are outsourcing.

Human brains are not just general purpose CPUs much of our cognition is performed on the wetware equivalent of application-specific integrated circuits (ASICs). ASICs that were tuned for applications that are of waning relevance in the current environment. They were tuned for our environment of evolutionary adaptiveness but the modern world presents very different challenges. By analogy it's as if they were tuned for sha256 hashing but Ethereum changed the hash function so the returns have dropped. Not to mention that biology uses terrible, dirty hacky heuristics that would would make a grown engineer cry and statisticians yell WHY! at the sky in existential dread. These leave us wide open to all sorts of subtle exploits that can be utilised by those who have studied the systematic errors we make and if they don't share our interests this is a problem.

Note that I am regarding the specifics of an uploaded brain as personal data which should be subject to privacy protections (both at the technical and policy level) and not as code. This distinction may be less clear for more sophisticated mind upload methods which generate an abstract representation of your brain and run that. If, however, we take a conceptually simpler approach the data/code distinction is cleaner. let's say we have an 'image' of the brain which captures the 'coordinates' (quantum numbers) of all of the subatomic particles that make up your brain. We then run that 'image' in a physics simulation which can also emulate sensory inputs to place the uploadee in a virtual environment. The brain image is data, the physics and sensory emulation engine is code. I suspect a similar reasonable distinction will continue to continue to hold quite well for quite a while even once your 'brain' data starts being represented in a more complex data structure than and N dimensional matrix.

I actually think mind uploading is a much harder problem than many people seem to regard it as, indeed I think it is quite possibly harder than getting to AGI de novo in code. This is for reasons related to neurobiology, imaging technology and computational tractability of physics simulations and I can get into it at greater length if anyone is interested.

I might have misunderstood the part of the OP about 'freedom of compute'. I understood it as proposing a constitutional amendment making 'proprietary software' not a thing and mandating that literally all software be open source.

If that's not what it meant, what does it mean? Open source software is already a thing that exists, I'm not sure how else to interpret the proposed amendment.

It seems to me that what you're worried about is a tendency toward increased gating of resources that are not inherently monopolizable. Creating infrastructure that permits inherently hard-to-gate resources, like software or technological designs, to be gated more effectively creates unnecessary opportunities for rent-seeking.

On the other hand, the traditional argument in favor of allowing such gates to be erected is that it creates incentives to produce the goods behind the gates, and we tend to reap much greater rewards in the long run by allowing gates to be temporarily erected and then torn down than we would by prohibiting such gates from ever being erected at all.

The fear is that some savvy agents will erect an elaborate system of gates such that they create, perhaps not a monopoly, but a sufficient system of gates to exact increasing fractions of created wealth over time. I think this is potentially worth worrying about, but it's not clear to me why we'd particularly focus on software as the linchpin of this dynamic, as opposed to all the other forms of gateable wealth. I think this is my primary objection to your argument.

When working with open government software or at least open APIs civic hackers have been able to get improvements in things like government budgetary transparency, the ease with which you can file your tax forms, the ability to locate retailers with face masks in stock etc.

Note that at least budgetary transparency and location of retailers with face masks are questions of data access. Sure, software is required to access that data, but it's making the data available that's key here. Forking software is useless without access to the data that software is meant to deliver. It's also useless without access to sufficiently powerful computing power. For example, if I had the source code for GPT-3, it wouldn't do me any good unless I had a sufficiently powerful supercomputer to run that code on. Furthermore, the human knowledge required to implement and maintain the code base can't just be forked even if you have access to the source code.

Data, microchips, and expertise are where bits meet atoms. Source code is just a fraction of the total capital controlled by a given company.

That retaliation allowed the users to limit the damage because they could work around it. It however did nothing that was disadvantageous to the creator and thus there was no deterrent. The parties were not aligned to cooperate. 

Any overt attempts to make things difficult loses the project good will with the community on which they often somewhat depend. 

You don't need to make any overt attempt to make things difficult to use for complex programs to become difficult to use. If you don't invest effort into making them easy to use they usually become hard to use.

The incentive to make it difficult to use if you are providing support has some interesting nuances especially with open core models but it is somewhat cancelled out by the community incentive to make it easy for them to use. 

There's little community incentive to make software easy to use. Ubuntu's useability for the average person is better than that of more community-driven projects. 

Starting in 2008 (number from memory) Microsoft invested a lot in useability for Microsoft Office. As far as I know, there wasn't anything similar for LibreOffice. GIMP's usability is still similarly bad as it was a decade ago.

Even though Blender basically won, its usability is awful. 

One of the key points of inadequate equilibria is that there are plenty of problems in our society where it's hard to create an alignment between stakeholders to get the problem solved. If we would legislate that all software has to be free software then that would prevent some forms that are currently effectively used to get problems solved from getting solved. 

This sounds to me like a bunch of buzzwords thrown together. You have argued in your post that it might be useful to have free software but I have seen no argument why it's currently a bottleneck for raising the waterline. 

I'll own up to a downvote on the grounds that I think you added nothing to this conversation and were rude. In the proposed scoring system [LW · GW], I'd give you negative aim and negative truth-seeking. In addition, the post you linked isn't an answer, but a question, so you didn't even add information to the argument, so I'd give you negative correctness as well.

If you thought the answers in that thread backed you up:

It's a mixed bag. A lot of near term work is scientific, in that theories are proposed and experiments run to test them, but from what I can tell that work is also incredibly myopic and specific to the details of present day algorithms and whether any of it will generalize to systems further down the road is exceedingly unclear. 

...

A lot of the other work is pre-paradigmatic, as others have mentioned, but that doesn't make it pseudoscience. Falsifiability is the key to demarcation.

That summarizes a few answers.

I agree, I wouldn't consider AI alignment to be scientific either. How is it a "problem" though?

Who it is aligned with is absolutely critical. 

One formula for friendly AI is, that it should be the kind of moral agent a human being would become, if they could improve themselves in a superhuman fashion. (This may sound vague; but imagine that this is just the informal statement, of a precise recipe expressed in terms of computational neuroscience.) 

If one were using a particular human or class of humans as the "seed", they would then only need a slight preponderance of virtue over vice (benevolence over malevolence? reason over unreason?) to be suitable, since such a being, given the chance to self-improve, would want to increase its good and decrease its bad; and, in this scenario, the improved possible person is what the AI is supposed to align with, not the flawed actual person. 

closed sourcing AI research approaches the most wrong and suicidal decisions possible 

One of the risks of open sourcing, arises from the separation between general problem-solving algorithms, and the particular values or goals than govern them. If you share the code for the intelligent part of your AI, you are allowing that problem-solving power to be used for any goal at all. One might therefore wish to only share code for the ethical part of the AI, the part that actually makes it civilized.