LessWrong 2.0 Reader

Things done:

1. Followed Jan's advice, and made sure that I would return just a plain string in GetTextResponse(model_id=model_id, request=request, txt=response, raw_responses=[], context=None) [important for later, I'm sure! But the failure is occurring before that point, as confirmed with print statements.]
2. tried without the global variables, just in case (global variables in python are always suspect, even though pretty standard to use in the specific case of instantiating an api client which is going to be used a bunch)

Traceback (most recent call last):
  File "/home/ub22/.pyenv/versions/3.11.2/envs/neurolib/bin/sad", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/ub22/projects/data/sad/sad/main.py", line 446, in main
    fire.Fire(valid_command_func_map)
  File "/home/ub22/.pyenv/versions/3.11.2/envs/neurolib/lib/python3.11/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ub22/.pyenv/versions/3.11.2/envs/neurolib/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
                                ^^^^^^^^^^^^^^^^^^^^
  File "/home/ub22/.pyenv/versions/3.11.2/envs/neurolib/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
exception calling callback for <Future at 0x7f427eab54d0 state=finished raised RuntimeError>
Traceback (most recent call last):
  File "/home/ub22/.pyenv/versions/3.11.2/lib/python3.11/concurrent/futures/_base.py", line 340, in _invoke_callbacks
    callback(self)
  File "/home/ub22/.pyenv/versions/3.11.2/envs/neurolib/lib/python3.11/site-packages/evalugator/api/api.py", line 113, in _log_response
    response_data = future.result().as_dict()
                    ^^^^^^^^^^^^^^^
  File "/home/ub22/.pyenv/versions/3.11.2/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/ub22/.pyenv/versions/3.11.2/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/home/ub22/.pyenv/versions/3.11.2/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ub22/projects/data/sad/providers/goodfire_provider.py", line 33, in execute
    response_text = client.chat.completions.create(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ub22/.pyenv/versions/3.11.2/envs/neurolib/lib/python3.11/site-packages/goodfire/api/chat/client.py", line 408, in create
    response = self._http.post(
               ^^^^^^^^^^^^^^^^
  File "/home/ub22/.pyenv/versions/3.11.2/envs/neurolib/lib/python3.11/site-packages/goodfire/api/utils.py", line 33, in post
    return run_async_safely(
           ^^^^^^^^^^^^^^^^^
  File "/home/ub22/.pyenv/versions/3.11.2/envs/neurolib/lib/python3.11/site-packages/goodfire/utils/asyncio.py", line 19, in run_async_safely
    loop = asyncio.get_event_loop()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ub22/.pyenv/versions/3.11.2/lib/python3.11/asyncio/events.py", line 677, in get_event_loop
    raise RuntimeError('There is no current event loop in thread %r.'
RuntimeError: There is no current event loop in thread 'ThreadPoolExecutor-1_0'.
    component = fn(*varargs, **kwargs)
exception calling callback for <Future at 0x7f427ec39ad0 state=finished raised RuntimeError>

That'd be a bug! Just to confirm, you were subscribed before I put this post up on Saturday morning, and don't have an email? Also reminder to check spam if you haven't.

"If you'd like to get notified of future events, you can subscribe to our meetup below to get an email whenever we add another one."

fwiw I'm subscribed and I haven't received any emails. I tried verifying my email and fiddling with notification settings and haven't been able to fix it

neural networks routinely generalize to goals that are totally different from what the trainers wanted

I think this is slightly a non sequitor. I take Tom to be saying "AIs will care about stuff that is natural to express in human concept-language" and your evidence to be primarily about "AIs will care about what we tell it to", though I could imagine there being some overflow evidence into Tom's proposition.

I do think the limited success of interpretability is an example of evidence against Tom's proposition. For example, I think there's lots of work where you try and replace an SAE feature or a neuron (R) with some other module that's trying to do our natural language explanation of what R was doing, and that doesn't work.

My take is it's fine/good, but the article is much more likely to be read (by me and many others) if the full content is crossposted (or even the opening bunch of paragraphs).

I'm glad you asked this. I think there are many good suggestions by others. A few more:

1: Have explicit, written plans for various scenarios. When it makes sense, have drills for them. Make your plans public or, at least, solicit external feedback on them.

Examples of such scenarios:

• Anthropic needs to pause down due to RSP commitments
• A model is caught executing a full-blown escape attempt
• Model weights are stolen
• A competing AI company makes credible claims about having AIs that imply decisive competitive advantage

2: Have a written list of assumptions you aim to maintain for each model's lifecycle. Make your lists public or, at least, solicit external feedback on them. Iterate on them regularly. Updates and violations are communicated at least internally.

These lists could vary based on ASL-levels etc., and could include things like:

• During the training of the model, we will continually evaluate the model on our evaluation suite for sabotage, sandbagging and scheming at [frequency].
• We solicit third-party evaluations on the model before internal deployment.
• Throughout the model's external deployment, we will have such-and-such monitoring schemes in place.

They could also have conditional statements (e.g. "if the model is [surprisingly capable] on [metric], we will do further investigation / take counter-measures ABC / ..."). C.f. safety cases. I intend this as less binding and formal than Anthropic's RSP.

3: Keep external actors up-to-speed. At present, I expect that in many cases there are months of delay between when the first employees discover something to when it is publicly known (e.g. research, but also with more informal observations about model capabilities and properties). But months of delay are relatively long during fast acceleration of AI R&D, and make the number of actors who can effectively contribute smaller. 

This effect strengthens over time, so practicing and planning ahead seems prudent. Some ideas in that direction:

• Provide regular updates about internal events and changes (via blog posts, streamed panel conversations, open Q&A sessions or similar)
• Interviews, incident reporting and hotlines with external parties (as recommended here: https://arxiv.org/pdf/2407.17347)
• Plan ahead for how to aggregate and communicate large amounts of output (once AI R&D has been considerably accelerated)

4: Invest in technical governance. As I understand it, there are various unsolved problems in technical governance (e.g. hardware-based verification methods for training runs), and progress in those would make international coordination easier. This seems like a particularly valuable R&D area to automate, which is something frontier AI companies like Anthropic are uniquely fit to advance. Consider working with technical governance experts on how to go about this.

I'm on both for the past year (vs just sertraline for prev decade). Love the extra energy and reduction in anxiety I've experienced from the buproprion.

On at least 5 different occasions I've tried weaning myself gradually off of sertraline. Every time I fall back into depression within a month or two. My friends and family then beg me to go back on it ASAP.

Why don’t you expect AGIs to be able to do that too?

I do, I just expect it to take a few iterations. I don't expect any kind of stable niche for humans after AGI appears.

I thought the argument about the kindly mask was assuming that the scenario of "I just took over the world" is sufficiently out-of-distribution that we might reasonably fear that the in-distribution track record of aligned behavior might not hold?

I wonder if you could set up a conditional donation? “I donate $X, minus if total donations exceed $3M"