LessWrong 2.0 Reader

Another possible interpretation of the titular question: an amino acid sequence with a fixed stable functional configuration but one that it cannot naturally reach because some intermediate stage of the folding is forbidden. I suspect such a thing is possible, and one might even be able to synthesize the final structure (in pieces, perhaps?).

My first thought was knotted proteins, but somehow those actually exist in nature (how?!):  link.

I am sad they're not getting as much use. I have wondered if they would work well as part of the comment section UI, where if you're having a back-and-forth with someone, the site instead offers you "Would you like to have a dialogue instead?" with a single button.

If it's just questions about the notation, please also feel free to ask here whatever is unclear, I'm happy to clarify (I understand that I kept the wording of the questions quite brief).

We are clearly looking at things differently. That's fine. But if two people see things differently I don't think it's wise to map what they are saying into your ontology. 

I'm currently in the Catalyze Impact AI safety incubator program. I'm working on creating infrastructure for automating AI safety research. This startup is attempting to fill a gap in the alignment ecosystem and looking to build with the expectation of under 3 years left to automated AI R&D. This is my short timelines plan [LW · GW].

I'm looking to talk (for feedback) to anyone interested in the following:

• AI control
• Automating math to tackle problems as described in Davidad's Safeguarded AI programme.
• High-assurance safety cases [LW · GW]
• How to robustify society in a post-AGI world
• Leverage large amounts of inference-time compute to make progress on alignment research
• Short timelines
• Profitability while still reducing overall x-risk

If you're interested in chatting or giving feedback, please DM me!

One interesting example of humans managing to do this kind of compression in software: .kkrieger is a fully-functional first person shooter game with varied levels, detailed textures and lighting, multiple weapons and enemies and a full soundtrack.  Replicating it in a modern game engine would probably produce a program at least a gigabyte large, but because of some incredibly clever procedural generation, .kkrieger managed to do it in under 100kb.

Some tech stacks / tools / resources for research. I have used most of these and found them good for my work. 

• TODO: check out https://www.lesswrong.com/posts/6P8GYb4AjtPXx6LLB/tips-and-code-for-empirical-research-workflows#Part_2__Useful_Tools [LW · GW]

Finetuning open-source language models. 

• Docker images: Nvidia CUDA latest image as default, or framework-specific image (e.g Axolotl)
• Orchestrating cloud instances: Runpod
  • Connecting to cloud instances: Paramiko
  • Transferring data: SCP
• Launching finetuning jobs: Axolotl
  • Efficient tensor ops: FlashAttention, xFormers
  • Multi-GPU training: DeepSpeed
  • [Supports writing custom cuda kernels in Triton]
• Monitoring ongoing jobs: Weights and Biases
• Storing saved model checkpoints: Huggingface
• Serving the trained checkpoints: vLLM.
  • [TODO: look into llama-cpp-python and similar things for running on worse hardware] 

Finetuning OpenAI language models. 

• End-to-end experiment management: openai-finetuner

Evaluating language models. 

• Running standard benchmarks: Inspect
• Running custom evals: [janky framework which I might try to clean up and publish at some point] 

AI productivity tools.

• Programming: Cursor IDE
• Thinking / writing: Claude
  • Plausibly DeepSeek is now better
• More extensive SWE: Devin
• [TODO: look into agent workflows, OpenAI operator, etc] 

Basic SWE

• Managing virtual environments: PDM
• Dependency management: UV
• Versioning: Semantic release
• Linting: Ruff
• Testing: Pytest
• CI: Github Actions
• Repository structure: PDM
• Repository templating: PDM
• Building wheels for distribution: PDM
• [TODO: set up a cloud development workflow] 

Research communication. 

• Quick updates: Google Slides
• Extensive writing: Google Docs, Overleaf
  • Some friends have recommended Typst
• Making figures: Google Draw, Excalidraw

I think you are correct with respect to my estimate of $\alpha$ and the associated model I was using. Sorry about my error here. I think I was fundamentally confusing a few things in my head when writing out the comment.

I think your refactoring of my strategy is correct and I try to check it myself, though I don't feel as confident in this as I feel in my approach being incorrect.

Your estimate doesn't account for the conversion between algorithmic improvement and labor efficiency, but it is easy to add this in by just changing the historical algorithmic efficiency improvement of 3.5x/year to instead be the adjusted effective labor efficiency rate and then solving identically. I was previously thinking the relationship was that labor efficiency was around the same as algorithmic efficiency, but I now think this is more likely to be around $algo\_efficienc{y}^2$ based on Tom's comment [LW(p) · GW(p)].

Plugging this is, we'd get:

$$\frac{\lambda }{\beta }(1-p)=\frac{r}{q}(1-p)=\frac{\ln \left({3.5}^2\right)}{0.4\ln \left(4\right)+0.6\ln \left(1.6\right)}(1-0.4)=2\frac{\ln \left(3.5\right)}{\ln \left(2.3\right)}(1-0.4)=2\cdot 1.5\cdot 0.6=1.8$$

(In your comment you said $\frac{\ln \left(3.5\right)}{\ln \left(2.3\right)}=1.6$, but I think the arithmetic is a bit off here and the answer is closer to 1.5.)

Not necessarily.

I think any method that calculates the value/utility of your wealth as a timeless function of utility per amount will be pretty disconnected to how people behave in practice. It doesn't account for people making plans and having to scrap them because an accident cost them their savings, for instance.

(But then again, I'm not an economist, maybe there are timeless frameworks that account for that.)

To be clear, I am not claiming that this failure mode is very likely very hard to resolve. Just harder than "run it twice on the original question and a rephrasing/transformation of the question".