LLM-based Fact Checking for Popular Posts?

post by azergante · 2025-04-18T21:26:25.230Z · LW · GW · 1 comments

Contents

  1. Some pros and cons
    Cons:
    Pros:
  2. Some thoughts on implementation details
    Should fact checking be done on all posts or only some of them?
    Where should the LLM's report go?
None
1 comment

Hello,

I think having some LLM-based fact checking on LessWrong posts would be a valuable addition.

To some extent comments already serve this purpose, but LLMs can do this in a more automated and systematic way, side-stepping human blind spots and tendency to conserve energy.

1. Some pros and cons

Cons:

• time cost to implement & maintain the feature
• money cost to power the LLMs that do the fact-checking
• potential errors in the fact-checking

Pros:

• higher confidence that what is in the post is true
• incentive for the author of the post to put more effort in researching the material
• the post is read in a broader context, with opposite views (if any) readily available
• spark some interesting discussions in the comments based on the LLM's feedback
• save reader time as they have less fact checking to do

2. Some thoughts on implementation details

Should fact checking be done on all posts or only some of them?

I would like fact checking to be enabled on the most read posts. This includes: The Sequences, The Codex, and the Best of Less Wrong for each year. Enabling them for other popular posts also makes sense.

I am not sure about less popular posts as it might cost a lot of money. Another concern is that it could intimidate new authors, though this could be mitigated by allowing the author to run the fact-checking before publishing.

Another consideration is whether the fact-checking should happen automatically when the post reaches some degree of popularity, or should be performed on demand by (some) users.

Where should the LLM's report go?

I imagine it going in a comment, perhaps pinned or otherwise highlighted, but subject to user feedback.

---

What do you think are the pros & cons of LLM-based fact checking for LW posts?

If this is to be implemented, how should it be implemented?

1 comments

Comments sorted by top scores.