Testing for parallel reasoning in LLMs 2024-05-19T15:28:03.771Z

Comment by meemi (meeri) on A quick experiment on LMs’ inductive biases in performing search · 2024-04-15T16:03:56.783Z · LW · GW

Great post! I'm glad you did this experiment.

I've worked on experiments where I test gpt-3.5-turbo-0125 performance in computing iterates of a given permutation function in one forward pass. Previously my prompts had some of the instructions for the task after specifying the function. After reading your post, I altered my prompts so that all the instructions were given before the problem instance. As with your experiments, this noticeably improved performance, replicating your result that performance is better if instructions are given before the instance of the problem.