Posts
Comments
He already addressed this.
If somehow international cooperation gives us a pause on going full AGI or at least no ASI - what then?
Just hope it never happens, like nuke wars?
The answer now is to set later generations up to be more able.
This could mean doing fundamental research (whether in AI alignment or international game theory or something else), it could mean building institutions to enable it, and it could mean making them actually smarter.
Genes might be the cheapest/easist way to affect marginal chances given the talent already involved in alignment and the amount of resources required to get involved politically or in building institutions
A few notes on massive cities:
Cities of 10Ms exist, there is always some difficulty in scaling, but scaling 1.5-2 OOMs doesn't seem like it would be impossible to figure out if particularly motivated.
China and other countries have built large cities and then failed to populate them
The max population you wrote (1.6B) is bigger than china, bigger than Africa, similar to both American Continents plus Europe .
Which is part of why no one really wants to build something so big, especially not at once.
Everything is opportunity cost, and the question of alternate routes matters alot in deciding to pursue something. Throwing everything and the kitchen sink at something costs a lot of resources.
Given that VR development is currently underway regardless, starting this resource intense project which may be made obsolete by the time it's done is an expected waste of resources. If VR hit a real wall that might change things (though see above).
If this giga-city would be expected to 1000x tech progress or something crazy then sure, waste some resources to make extra sure it happens sooner rather than later.
Tl;dr:
Probably wouldn't work, there's no demand, very expensive, VR is being developed and would actually be able to say what you're hoping but even better
Vr might be cheaper
Have you thought about how to get the data yourself?
Perhaps offering payment to people willing to get iq tested and give a genetic sample, and paying more for higher scores on the test?
I understand that money is an issue, but as long as you're raising this seems like an area you could plug infinite money into and get returns
This seems... evil or at the very least zero-sum thinking to me.
Would you want to stop the successful from paying for their children's education? Spending their time on raising their children? Do you want to take all children away from their parents to make sure they aren't put on different footing? Perhaps genetically enforce equality?
I would much rather governments try to preserve hereditary positive dynamics, while getting involved with negative ones.
We'll have won once all trees are positive and successful, and bad apples do not create generations of bad trees
There is something fundamentally compelling about the idea that every generation should start fresh, free from the accumulated advantages or disadvantages of their ancestors.
...
...
The death tax does not punish success—it prevents success from becoming hereditary. It ensures that the cycle of opportunity begins anew with each generation.Keeping humans around is the correct move for a powerful AGI, assuming it isn't being existentially threatened.
For a long while human inputs will be fairly different from silicon inputs, and humans can do work - intellectual or physical - and no real infrastructure is necessary for human upkeep or reproduction (compared to datacenters).
Creating new breeds of human with much higher IQs and creating (or having them create) neuralink-like tech to cheaply increase human capabilities will likely be a very good idea for AGIs.
Most people here seem worried about D tier ASIs, ASIs should see the benefits of E tier humans (250+ IQ and/or RAM added through neuralink-like tech) and even D tier humans (genesmith on editing, 1500+ IQs with cybernetics vastly improving cognition and capability)
'Sparing a little sunlight' for an alternative lifeform which creates a solid amount of redundancy as well as being more effecient for certain tasks and allowing for more diverse research, as well as having minimal up-front costs is overdetermined
The Fønix team is just heating water, which is great but actual distillation (with automated re-adding of specific minerals) is probably what you're actually going to want so as to avoid all contamination not just biological.
In this size of structure growing food isn't really worth it, storing food for 10 years is actually easier (according to claude). It does need to come stocked though
It's more that it stops being relevant to humans, as keeping humans in the loop slows down the exponential growth
I do think VR and neuralink-like tech will be a very big deal though, especially in regards to allowing people experiences that would otherwise be expensive in atoms
At what IQ do you think humans are able to "move up to higher levels of abstraction"?
(Of course this assumes AIs don't get the capability to do this themselves)
Re robotics advancing while AI intelligence stalls, robotics advancing should be enough to replace any people who can't take advantage of automation of their current jobs.
I don't think you're correct in general, but it seems that automation will clear out at least the less skilled jobs in short order (decades at most)
I very much hope the computers brought in were vetted and kept airgapped.
You keep systems separate, yes.
For some reason I assumed that write permissions were on user in the actual system/secure network and any data exporting would be into secured systems. If they created a massive security leak for other nations to exploit, that's a crux for me on whether this was reckless.
Added: what kind of idiot purposely puts data in the wrong system purposely? The DOGE guys doing this could somehow make sense, governmental workers??
No.
I'm not familiar with public documentation on this.
I know people who have gotten access to similarly important governmental systems at younger ages.
Don't worry about it too much.
If they abuse it, it'll cost their group lots of political goodwill. (Recursive remove for example)
Musk at least is looking to upgrade humans with Neuralink
If he can add working memory can be a multiplier for human capabilities, likely to scale with increased IQ.
Any reason the 4M$ isn't getting funded?
Any good, fairly up-to-date lists of the relevant papers to read to catch up with AI research (as far as a crash course will take a newcomer)?
Preferably one that will be updated
Reading novels with ancient powerful beings is probably the best direction you have for imagining how status games amongst creatures which are only loosely human look.
Resources being bounded, there will tend to always be larger numbers of smaller objects (given that those objects are stable).
There will be tiers of creatures. (In a society where this is all relevant)
While a romantic relationship skipping multiple tiers wouldn't make sense, a single tier might.
The rest of this is my imagination :)
Base humans will be F tier, the lowest category while being fully sentient. (I suppose dolphins and similar would get a special G tier).
Basic AGIs (capable of everything a standard human is, plus all the spikey capabilities) and enhanced humans E tier.
Most creatures will be here.
D tier:
Basic ASIs and super enhanced humans (gene modding for 180+ IQ plus SOTA cyborg implants) will be the next tier, there will be a bunch of these in absolute terms but relative to the earlier tier rarer.
C tier:
Then come Alien Intelligence, massive compute resources supporting ASIs trained on immense amounts of ground reality data, biological creatures that have been redesigned fundamentally to function at higher levels and optimally synergize with neural connections (whether with other carbon based or silicon based lifeforms)
B tier:
Planet sized clusters running ASIs will be a higher tier.
A, S tiers:
Then you might get entire stars, then galaxies.
There will be much less at each level.
Most tiers will have a -, neutral or +.
- : prototype, first or early version. Qualitatively smarter than the tier below, but non-optimized use of resources, often not the largest gap from the + of the earlier tier
Neutral: most low hanging optimizations and improvements and some harder ones at this tier are implemented
+: highly optimized by iteratively improved intelligences or groups of intelligences at this level, perhaps even by a tier above.
Writing tests, QA and Observability are probably going to stay for a while and work hand in hand with AI programming, as other forms of programming start to disappear. At least until AI programming becomes very reliable.
This should allow for working code to be produced way faster, likely giving more high-quality 'synthetic' data, but more importantly massively changing the economics of knowledge work
Is there a reason that random synthetic cells will not be mirror cells?
We have here some scientists making cells. Looks like a dangerous direction
Humans seem way more energy and resource efficient in general, paying for top talent is an exception not the rule- usually it's not worth paying for top talent.
Likely to see many areas where better economically to save on compute/energy by having human do some of the work.
Split information workers vs physical too, I expect them to have very different distributions of what the most useful configuration is.
This post ignores likely scientific advances in bioengineering and cyborg surgeries, I expect humans to be way more efficient for tons of jobs once the standard is 180 IQ with a massive working memory
I do things like this at times with my teams.
Important things:
-
Don't think you need to solve the actual problem for them
-
Do solve 'friction' for them as much as possible
-
Do feel free to look up other sources so you can offer more perspective and to take off the load of having to find relevant info
-
positive energy, attentive etc
-
if they're functioning well just watch and listen while being interested and unobtrusive, at most very minor inputs if you're pretty sure it'll be helpful
If stuck at a crossroads ask them how long they think each path will take/ how hard it'll be, and give them feedback if you think they're wrong. Help them start working on one, people can get stuck for longer than it would take to actually do one option.
- if lost, methodically go through the different areas where the issue could be and methodically go through all the directions they could take for each area and in general. You don't need to think these up, but keep track of them and help guide towards picking apart the problem and solution spaces. This takes some mental load off.
A message from Claude:
'''This has been a fascinating and clarifying discussion. A few key insights I'll take away:
The distinction between bounded and unbounded optimization is more fundamental than specific value differences between AIs. The real existential threat comes from unbounded optimizers. The immune system/cancer metaphor provides a useful framework - it's about maintaining a stable system that can identify and prevent destructive unbounded growth, not about enforcing a single value set. The timing challenge is critical but more specific than I initially thought - we don't necessarily need the "first" AGI to be perfect, but we need bounded optimizers to establish themselves before any unbounded ones emerge.
Some questions this raises for further exploration:
What makes a Schelling fence truly stable under recursive self-improvement? Could bounded optimizers coordinate even with different base values, united by shared meta-level constraints? Are there ways to detect early if an AI system will maintain bounds during capability gain?
The framing of "cancer prevention" versus "value enforcement" feels like an important shift in how we think about AI governance and safety. Instead of trying to perfectly specify values, perhaps we should focus more on creating robust self-limiting mechanisms that can persist through capability gains.'''
A few thoughts.
-
Have you checked what happens when you throw physic postdocs at the core issues - do they actually get traction or just stare at the sheer cliff for longer while thinking? Did anything come out of the Illiad meeting half a year later? Is there a reason that more standard STEMs aren't given an intro into some of the routes currently thought possibly workable, so they can feel some traction? I think either could be true- that intelligence and skills aren't actually useful right now, the problem is not tractable, or better onboarding could let the current talent pool get traction - and either way it might not be very cost effective to get physics postdocs involved.
-
Humans are generally better at doing things when they have more tools available. While the 'hard bits' might be intractable now, they could well be easier to deal with in a few years after other technical and conceptual advances in AI, and even other fields. (Something something about prompt engineering and Anthropic's mechanistic interpretability from inside the field and practical quantum computing outside).
This would mean squeezing every drop of usefulness out of AI at each level of capability, to improve general understanding and to leverage it into breakthroughs in other fields before capabilities increase further. In fact, it might be best to sabotage semiconductor/chip production once the models one gen before super-intelligence/extinction/ whatever, giving maximum time to leverage maximum capabilities and tackle alignment before the AIs get too smart.
- How close is mechanistic interpretability to the hard problems, and what makes it not good enough?
The point was more about creating your own data being easy, just generate code then check it by running it. Save this code, and later use it for training.
If we wanted to go the way of AlphaZero it doesn't seem crazy.
De-enforce commands, functions, programs which output errors, for a start.
I didn't think of the pm as being trained by these games, that's interesting. Maybe have two instances competing to get closer on some test cases the pm can prepare to go with the task, and have them competing on time, compute, memory, and accuracy. You can de-enforce the less accurate, and if fully accurate they can compete on time, memory, cpu.
I'm not sure "hard but possible" is the bar - you want lots of examples of what doesn't work along with what does, and you want it for easy problems and hard ones so the model learns everything
Product manager, non-technical counterpart to a team lead in a development team
I notice that I'm confused.
Google made an amazing AI for playing chess, by allowing it to make its own data.
Why hasn't the same thing happened for programming? Have it generate a bunch of pictures with functionality expectations (a PM basically), have it write and run code, then check the output against the requirements it created, then try again when it doesn't come out right.
This is even easier where the pm is unnecessary - leetcode, codewars, euler...
You could also pay PMs to work with the AI developers, instead of the code tutors xAI is hiring.
There seems to be a preference to having the LLMs memorize code instead of figuring things out itself.
If you run out of things like that you could have it run random programs in different languages, only learning from those that work.
I haven't used genesis, but that also seems like a mostly-built validator for programs that AIs can use to create and train on their own data.
With the amount of compute going into training, it should be easy to create huge amounts of data?
There's a certain breadth of taste in reading you can only aquire by reading (and enjoying!) low quality internet novels after you've already developed sophisticated taste.
So unbundle it?
There is a beautiful thing called unilateral action.
I believe most employers mostly don't care about conformity as such.
The inner circle stuff is only true of elite schools AFAIK. You can outcompete the rest of the universities
University education can be made free pretty cheaply.
The cost at scale is in the credentials- you need to make tests, test students, and check those tests.
The classes can be filmed once, and updated every few years if necessary. Each course can have a forum board for discussion and meeting up for studying in groups.
See course credentials for things like AWS.
This implies that we should stop life from developing independently, and that if contact is made with aliens then the human making contact and any environment that's been in chain of proximity should be spaced
Start small, once you have an attractive umbrella working for a few projects you can take in the rest of the US, the the world
In my work I aggregate multiple other systems' work as well as doing my own.
I think a similar approach may be useful. Create standardized outputs each project has to send to the overarching org, allow each to develop their own capabilities and to a degree how what is required to make those outputs meaningfully reflect on the capabilities and R&D of the project.
This will lay the ground to self-regulate, keeps most of the power with the org (assuming it is itself good at actual research and creation) conditional on the org playing nice and being upstanding with the contributing members, and without limiting any project before it is necessary.
DOGE.
This is an opportunity to work with the levers of real power. If there are 5 people here who work on this for two years, that's an in with Senators, Congressman, bureaucrats and possibly Musk.
Just showing up and making connections while doing hard work is the most efficient way to get power right now, in the time before AI gets dangerous and power will be very relevant.
I do not believe that this should be taken as an opportunity to evangelize. People, not ideology.
This seems like something worth funding if someone would like to but can't afford it.
The first issue seems minor - even if true, a 40 year old man could have a new arm by 60
What happened to regrowing limbs? From what little I understand, with pluripotent stem cells we could do a lot, except cancer.
Why don't we use stem cells instead of drilling for cavities? While there are a few types of tissue, tumors are fairly rare in teeth, likely due to minimal blood flow.
Why aren't research companies made in Africa/Middle East/China for human research- cut out most of the bureaucracy and find out fast if something works, if it does set up a company in a 1st world country to go through the steps?
Something like iterative/cliff, with fast and slow expressing time scales
Can you sort the poll options by popularity?
Iterative/Sudden
I can only describe the Product, not the tech. The idea would be to plug in a bigger working memory in the area of the brain currently holding working memory. This is the piece I think matters most
On reflection something like wolfram alpha should be enough for calculations, and a well indexed reservoir of knowledge with an LLM pulling up relevant links with summaries should be good enough for the rest
Inside the super organism you are correct, but the genome is influenced by outside forces as whole over the ages - and any place where this breaks down for long enough you eventually get two species instead of one.
Therefore outside groups can treat the species as a super organism in general, the individual members must be dealt with individually when there is previous loyalty to another member of the other species.
For example, an Englishman and his dog vs an eskimo and his dog. The two humans may be against each other, the dogs may be against each other, but the opposite human/dog interactions would be standard if they weren't already attached to other in-species members.
This gives the bones of a proper theoretical foundation on the moral duties between members of different species.
For example, this would back the intuition of eating dog to be worse than eating a bear or octupus, regardless of intelligence, and of killing rats out of hand
They'd not identical. First, they have a different status, much the same as citizens and aliens have different rights. Second, different species of animals have different relationships with humanity: Dogs are bred to be symbiotic companions Cats are parasites if allowed, pest control if tolerated Rats are disease vector scavengers Chickens are livestock - they lay infertile eggs for human consumption!
I'm not sure how well curated and indexed most information is.
Working memory allows for looking at the whole picture at once better with the full might of human intelligence (which is better at many things than LLMs), while removing frictions that come from delays and effort expended in search for data and making calculations.
Of course we have smart people together now, but getting multiple 7+SD people together would have many further benefits beyond having them work solo.
We probably have at least a generation (we're probably going to slow down before we hit SAGI due to the data wall, limited production of new compute, and regulation).
The focus should be on moving quickly to get a group ecliping current human capabilities ASAP, not on going much further
The idea of inertia is excellent, extending the idea of momentum far further (and naturally inspiring thoughts on mass, velocity, etc)
Devops Mentality is correct:
Friction is a big deal.
Being able to change code and deploy immediately (or in a few minutes) matters immensely.
This might just be feedback loops at an extreme, but I believe it's something more.
The benefit of going from 20wpm to 40wpm was not ×2 to my writing and experience, it was more like ×8. I fully intend to reach 60wpm.
It was closer to a ×2 to my developing experience, which is interesting as most of software development isn't the actual typing. Another anecdote is that the best programmers I know all have 60+wpm (there was one react front end guy who was good, but mostly at design).
Reducing friction is underrated, even if friction is only 10% of a job (typing during coding), improving that friction substantially matters more than 10%.
This may have to do with flow, concentration or enjoyment
On human-computer interfaces: Working memory, knowledge reservoirs and raw calculation power seem like the easiest pieces, while fundamentally making people better at critical thinking, philosophy or speeding up actual comprehension would be much for difficult.
The difference being upgrading the core vs plug-ins.
Curated reservoirs of practical and theoretical information, well indexed, would be very useful to super geniuses.
On human-human: You don't actually need to hook them up physically. Having multiple people working on different parts of a problem lets them all bounce ideas off each other.
Overall: The goal should be to create a number of these people, then let them plan out the next round if their intelligence doesn't do it.
If humanity can make 100 7+SD humans hooked up with large amounts of computing power, curated knowledge + tons of raw data, and massive working memories, they'll be able to figure out any further steps much better than we can.
The virus example doesn't seem right to me.
- The claim doesn't seem necessarily true. Why would a more transmissible virus be more deadly? (aside from general virulence I suppose..). It isn't hard to think of some very contagious viruses that don't seem all that deadly (herpes, cold, flu) and some much less transmissible viruses that are quite deadly (AIDS).
I suppose more deadly viruses generally require more transmissibility to survive than less deadly ones do, but A -> B does not mean B -> A.
- Viruses are generally horizontal, I don't see why one horizontal thing would a priori be more negative than another horizontal thing, just based on a vertical vs horizontal idea
Well now I need to read your simalcrum posts, this is brilliant