Please do not use AI to write for you

My suggestion for a LLM policy for LW2 might be:

If AI-written or edited text is not being posted as AI-written/edited text samples, then it must be improved by a human before posting.

If someone is posting a GPT-4 sample as a response or example of "what would GPT-4 write here?", that is totally legitimate and doesn't need to be edited other than to put it in blockquotes etc; if it's an exercise in "and the punchline is, an AI wrote this!", well, that's fine too, and readers will upvote/downvote as they find the exercise of value. These are not the problem. The problem is when people slip in AI stuff purely as an (inferior) substitute for their own work.

I am also fine with use of AI in general to make us better writers and thinkers, and I am still excited about this. (We unfortunately have not seen much benefit for the highest-quality creative nonfiction/fiction or research, like we aspire to on LW2, but this is in considerable part due to technical choices & historical contingency, which I've discussed many times before, and I still believe in the fundamental possibilities there.) We definitely shouldn't be trying to ban AI use per se.

However, if someone is posting a GPT-4 (or Claude or Llama) sample which is just a response, then they had damn well better have checked it and made sure that the references existed and said what the sample says they said and that the sample makes sense and they fixed any issues in it. If they wrote something and had the LLM edit it, then they should have checked those edits and made sure the edits are in fact improvements, and improved the improvements, instead of letting their essay degrade into ChatGPTese. And so on.

Anything else pollutes the commons. Every comment here is a gift from the author, but it's also a gift from the readers, which they make in good faith under the belief that the author tried to make the comment worthwhile & put in enough effort that it would be worth potentially many people reading it. It should never take the author much less effort to write a comment than the readers will take to read it (as is the case with spamming sections with LLM junk that the 'author' didn't even read but merely skimmed and went 'lgtm', judging from cases that have been flagged here in the past). Because you know, bro, I am just as capable as you are of copying a comment into the neighboring ChatGPT or Claude tab and seeing what it says; I don't need you doing that manually on LW2 and it doesn't help me if I have to waste time reading it to realize that I was better off ignoring it because you are just going to paste in random average AI slop without any kind of improvement: filtering, critique, improvement, evaluation, commentary, fact-checking, editing, curation, comparison of LLMs...

Such comments are spam, plain and simple, indistinguishable from spammers karma-farming to flip an account: creating fake contributions to gain status in order to parasitize the community without giving anything in return. And should be treated as such: downvoted, and banned.

I have a counterpoint, which is that I often see low effort posts or comments by people (less often on LessWrong) where I think: "I wish this person had had a discussion with Claude about this before posting."

I don't like it when people post verbatim what the models output, because of all the reasons you mention, but I do think that having a debate about your ideas with a model can help clarify. You need to actually manage to have a debate though, not just get it to sycophantically agree with you.  Try tactics like starting out pretending to have the opposite point of view you actually have, then switching. Ask for pros and cons and for insightful critiques, and for it to avoid padding phrases, social niceties, and flattery.

Then rewrite, in your own words, your updated thought after this process, and it'll probably be at least a bit improved.

So, this isn't quite disagreeing with the point here exactly. I guess my summary is, 'Use LLMs thoughtfully and deliberately, not sloppily and carelessly.'

Funnily enough, I feel pretty similar about AI-generated images by now. I've always struck by how people stick huge useless images on top of their blogposts, and other people want to read such articles more?? But now with AI generated images this has multiplied to a point that's just frustrating—it means more time spent scrolling, and it evokes in me the feeling of someone trying to (and failing to) set an æsthetic atmosphere for a post and then convincing me through that atmosphere, instead of providing arguments or evidence.

I have seen this being done well, especially in some of Joseph Carlsmith's posts, in which he uses a lot of images of old paintings, real photographs etc. I always thought of myself as having no taste, but now seeing other people sticking the same-flavored AI images on top of their posts makes me reconsider. (And I notice that there's a clear difference in beauty between /r/museum and /r/Art.)

I'll probably get disagree points, but wanted to share my reaction: I honestly don't mind the AI's output. I read it all and think it's just an elaboration of what you said. The only problem I noticed is it is too long.

Then again, I'm not an amazing writer, and my critical skills aren't so great for critiquing style. I will admit I rarely use assistance, because I have a tight set of points I want to include, and explaining them all to the AI is almost the same as writing the post itself.

I added to your prompt the instructions

Be brief and write concise prose in the style of Paul Graham. Don't hedge or repeat yourself or go on tangents.

And the output is still bad, but now mostly for the flaw (also present in your output) that ChatGPT can't resist making the complaint about "human authenticity" and "transparency/trust" when that's not what you're talking about:

I've noticed a troubling trend on online forums: a surge in posts that clearly seem to be generated by AI. These posts are verbose, meandering, and devoid of real substance. They present every argument with an opposing one, wrapped in so much "on the one hand" and "on the other" that they end up saying nothing at all.

AI in its current form shouldn’t be used for this purpose. The problem isn’t just that these posts waste time, though they do. It’s that they dilute meaningful discourse. When an AI-generated post floods a forum, it clogs the flow of genuine ideas. The writing might seem polished, but it’s hollow, a shell of real thought. And that’s a disservice to any community that values actual discussion.

Forums are places for people to exchange ideas, not for machines to simulate them. Using AI to generate content in this way erodes the authenticity of conversation. It’s a misuse of technology that contributes to the noise rather than to meaningful exchange.

AI can be powerful, but that power should be wielded carefully. Right now, the tools aren’t good enough to replace real human input, and pretending they are does more harm than good.

EDIT: Below is what it looks like if I replace "in the style of Paul Graham" with "in the style of LessWrong":

AI-generated content in online forums is a growing problem. These tools are designed to mimic human language, but they often fail to deliver meaningful or original contributions. Instead, they produce verbose, meandering prose that tries to cover all bases but ultimately says nothing. This kind of content is worse than useless—it clogs up discussions with noise, making it harder for genuine insights to emerge.

AI, in its current state, lacks the ability to form and defend coherent, novel ideas. It doesn't understand context, nuance, or the importance of taking a clear stance. The result is writing that hedges endlessly, offering a shallow imitation of thoughtful discourse but devoid of real substance. When forums are flooded with such content, it dilutes the value of the platform. It wastes the time of readers who are searching for real engagement and stifles the development of ideas that can only come from genuine human interaction.

AI tools should not be used to generate content for forums, period. They don't contribute to the quality of discussion and instead, undermine it. The promise of AI lies in its ability to augment human capabilities, not replace them in areas where human judgment, creativity, and insight are paramount. Until AI can meaningfully contribute to a discussion, we should resist the temptation to use it in contexts where clarity, originality, and human understanding are crucial.

Yeah, I think people asking AIs to write essays for them is currently pretty doomed. The RLHF also makes the writing style of basically all AI systems into a weird kind of polite-markety-slop that tends to waffle in a way that I find deeply grating. 

That said, I sure use AI systems as part of a lot of my writing and thinking, and I also desperately wish I could somehow get them to help me with editing. Many times has it happened that I changed some small part of an article, and where I wished I could just ask my AI assistant to propagate the obvious consequences of that change throughout the rest of the article (for example, let's say I start my article with two examples, I remove the second one because I no longer like it, now I would like to rewrite the rest to no longer reference the second example). 

I think the real lesson to take away from this is that, at least right now, chat models are bad writers and bad writing assistants. I think things like base models or autocomplete models have more of a chance of producing writing that isn't terribly grating, and in almost no circumstance would I recommend that someone directly copies more than maybe 1 paragraph of LLM output into their essay without clearly attributing it to AI systems. 

On the other hand, I would personally be in favor of people embedding more references to LLM conversations into their essays. It's an easy way to be transparent about your sources, and I often find it interesting to understand why an LLM got something wrong. 

Right now, I would consider it to be a mod-warneable offense to publish an obviously LLM-written essay under your own name, without telling people an LLM wrote it, or most of it. And at least for now it's pretty easy to tell. 

Upvoted and agreed, but the ship has sailed.  We need to help people use LLM article generation well, to help them focus and express their thoughts, rather than just making longer versions of a brief thesis.  This is usally a multi-step process, generating and iterating on an outline, then individually generating and iterating on each section (of a paragraph or two).  The iteration - human control of the conciseness and correctness of each portion - is absolutely key.

As a low-effort very small change to your prompt to demonstrate the variance, I did a one-shot with GPT 4o that specified LessWrong and added an instruction for tone.

I have recently seen a substantial increase in the proportion of postings to the LessWrong online forum that were clearly generated by AI, although not marked as such. The writing is verbose and meandering, full of "on the one hand" and "on the other", never making an assertion without also suggesting its opposite, sounding oh-so-measured and even-handed that it ends up saying nothing at all. Please write a brief article for me, written in the first person, arguing that AI tools in the current state of the art should not be used for this purpose. Keep it factual and direct, without unnecessary elaboration or verbosity.

and got

I've noticed a growing trend on the LessWrong forum where AI-generated content is becoming more prevalent, and it's not marked as such. This raises significant concerns, especially when the writing is verbose, indecisive, and ultimately devoid of substance. The hallmark of these posts is a constant balancing act—presenting multiple sides of an argument without ever committing to a clear stance. This approach, while superficially measured, often results in content that says nothing meaningful.

AI tools, as they currently stand, are excellent at generating text that mimics human writing, but they lack the ability to produce genuinely insightful or novel ideas. Their output tends to be safe, repetitive, and lacking in depth because they are designed to avoid controversy and adhere to patterns seen in their training data. This often leads to the production of content that feels empty—words strung together in a way that looks polished but fails to engage or challenge the reader.

The problem with using AI for content creation in thoughtful forums like LessWrong is that it undermines the quality of discussion. The platform thrives on clear, well-reasoned arguments that push the boundaries of rational thought. When AI-generated content is introduced into this space without proper identification, it dilutes the discourse. Readers are left sifting through noise to find genuine contributions, which can erode the overall value of the community.

In its current state, AI should not be used for generating posts on platforms that prioritize intellectual rigor and meaningful debate. Until AI can contribute something more than surface-level analysis and pseudo-argumentation, its use in these contexts should be reconsidered. Identifying AI-generated content clearly when it is used is also essential to maintain the integrity of the forum and respect the expectations of its users.

This is much better (IMO) than your initial attempt, and I believe it could be much better with more significant effort into multi-shot prompt engineering and light edits afterward.

I'd rephrase your title as "If you're going to use AI to write for you, use it as a fairly limited assistant rather than an open-ended delegation of responsibility".  People need to put in the effort to clarify their own thoughts and put them clearly into text - AI is really helpful for grammar and the mechanics of writing, and not (yet) any good at conceptual presentation that's not basically a summary of something else.

I agree with your intended point, but disagree with it as stated. There are certainly ways to use AI to write for you well. Claude Opus is a much more natural writer than GPT-4o, especially with more than simple prompting. Even to the specific points of conciseness, filler phrases, equivocating, and inordinately high proportions of applause-light [LW · GW] text.

In the extreme, Janus [LW · GW] sometimes simulates conversational branches with a base LLM before having the conversation, and copies text over from the AI's completions during the conversation. I've certainly never been able to tell reliably when I'm speaking to human-Janus or AI-Janus.

Grok-2 and Llama-3-405b seem better at opinion/essay writing (with Llama able to decide to be brief when appropriate), Claude-3.5-Sonnet thinks most clearly (closely keeping track of the intended plausible position). GPT-4o doesn't seem exceptional on any relevant dimension. Gemini-1.5-Pro can occasionally notice some non-trivial implication all others would miss. Weaker chatbots are even more pointlessly verbose and non-committal, so I expect this problem will be gone in a year as strengths of the current frontier models become robust with additional scale.

I’ll admit to recently having written a post myself, asked an AI to improve my writing, made a few edits myself, posted it and then come back later and “thought omg how did I let some of these AI edits through”. Hopefully the post in question is best now.

You may be right, but this is missing a major point so I didn't continue. You didn't even try to prompt it to write in a style you like more.

As for letting it come up with the logic, of course you shouldn't do that; humans still reason better. At least very well educated humans do.

OTOH, you should ask it for help in thinking through domains you're not expert in; that will improve your opinions and writing.

Your experiment is contaminated: If a piece of training document said that AI texts are overly verbose, and then announced that the following is a piece of AI-written text, it'd be a natural guess that the document would continue with overly verbose text, and so that's what an autocomplete engine will generate.

Due to RLHF, AI is no longer cleanly modelled as an autocomplete engine, but the point stands. For science, you could try having AI assist in the writing of an article making the opposite claim :).

I have nothing against AI as a Jarvis/Friday-like assistant/advisor. A bad workman blames his tools (absit iniuria). Some us don't know how to use stuff properly. My reckoning suggests that I'm aware of only 5% of my smart phone's capabilities. Sometimes I get these random notifications full of interesting suggestions.