Just because an LLM said it doesn't mean it's true: an illustrative example

Contents

12 comments

This was originally posted in the comments of You don't know how bad most things are nor precisely how they're bad. [LW · GW]; I've broken it out into a post because I think it might be a useful corrective more generally for people inclined to cite LLM remarks as fact.

I asked Claude, as an illustrative example, whether ready-made clothing had declined in quality over time, and it claimed that "many studies" indicated this was so. I subsequently asked for it to cite the studies in question; half of them were confabulated and AFAICT only one of them contained what Claude said it did. (Note that this is Claude 3 Haiku, which isn't the best; I'd expect frontier models to name four or maybe even five real papers in a list of six (and, with lower confidence, to have a more accurate understanding of the papers' contents)).

Here are the "studies" it cited—note that even some of the ones which actually exist do not contain the things Claude said they did.

1. "A New Textiles Economy: Redesigning Fashion's Future" - Report by the Ellen MacArthur Foundation (2017) - This is a think-tank report (available here) rather than a study. However, it does actually exist. Claude says it "noted that the number of times clothes are worn before disposal has decreased by 36% compared to 15 years ago." It really does say this! Page 19, citing as a source the "Circular Fibres Initiative analysis based on Euromonitor International Apparel & Footwear 2016 Edition (volume sales trends 2005–2015)." A-. (Teeeeeechnically the report claims the clothing is still wearable and therefore throwing it away is a waste, which isn't the same thing as poor durability causing decreased wear time; humans cite studies that support something a little to the left of their point all the time, though, so in the interests of fairness I won't mark it down for that).
2. "The State of Fashion 2016" - Report by the McKinsey Global Fashion Index (2016) - This one seems to be real! (Technically,  it's "The State of Fashion 2017"—their first report, so Claude can't have meant an earlier one—but it was released in 2016 and half of it is an overview of same, so IMO it's close enough.) It is not a research study, but rather a think-tank report (which is actually even worse than it seems, IMO, because on the few occasions I've checked sources on think-tank reports I've sometimes found that the results cited didn't seem to actually exist anywhere). Claude says it "concluded that the quality and durability of clothing has declined as the industry has shifted towards faster production cycles and lower prices." The report does say the industry has moved toward faster production cycles and lower prices (though it indicates that production costs have actually risen, leaving the authors quite worried about their profit margins), but does not, as far as I can tell, claim that quality and durability have declined.
3. "Valuing Our Clothes: The Cost of UK Fashion" - Study by the Waste & Resources Action Programme (WRAP) in the UK (2012) - This one actually exists! (Full report here). It's again not technically a study, but rather a "summary of the key findings of a major technical report (possibly this one?) published by WRAP"; however if they did any research themselves that's practically hard science compared to the other two, so whatever. However, Claude claims that it "found that the average number of times a garment is worn before being discarded has fallen by 36% compared to 15 years earlier," and as far as I can tell this is not true; I couldn't find a place in the report where it even mentioned the number of wearings before discarding.
4. "Apparel and Footwear Benchmarking Analysis" - Report by the American Apparel & Footwear Association - Confabulated (there are reports with similar titles from organizations which are not this one, but this specific report does not exist)
5. "The Apparel Sourcing Caravan's New Route to the Market" - Report by the Boston Consulting Group - Confabulated
6. "Clothing Durability and the Creation of Value" - Study by the University of Cambridge Institute for Manufacturing (2018) - Confabulated

Posting this list of sources without annotations would have been of negative value, leading unwary readers to change their minds based on studies that don't exist and forcing somewhat cannier ones to spend what could be several hours checking the bullshit I produced in a few minutes.

Moreover, LLMs will take almost any position you ask of them, regardless of whether or not it's true. Just for fun, here's Claude's list of five studies proving clothing's actually gotten better over time (veracity left as an exercise for the reader):

1. A 2020 study by the University of Cambridge called "The Changing Lifespan of Garments: Evidence from Cambridge Consumer Survey Data" found that the average lifespan of mass-produced garments has increased by 30% over the past 20 years, due to advancements in manufacturing techniques and materials.
2. Research by the American Apparel & Footwear Association called "Quality Trends in the US Apparel Industry" showed that the number of clothing items returned due to quality issues has decreased by 25% since 2015, suggesting clothing is better made.
3. A 2019 consumer survey by McKinsey & Company called "The State of Fashion 2019" found that 66% of respondents reported being satisfied with the quality of their ready-to-wear purchases, up from 58% in 2014.
4. A 2021 analysis by the UK's Clothing Sustainability Research Centre called "UK Clothing Sustainability Report 2021" concluded that the average quality rating for high-street fashion brands increased by 18% over the past decade, as they invested in improving manufacturing processes.
5. A 2018 report by the International Federation of Textile Engineers called "Global Textile Industry Quality Trends" noted a 23% reduction in clothing defects across major global brands, attributed to enhanced quality control measures.

For these reasons, I do not think it is appropriate to cite the statements of LLMs as though they are evidence in favor of your position.

12 comments

Comments sorted by top scores.

U: has ready made clothing declined in quality over time?​​​​​​​​​​​​​​​​

C: There are arguments on both sides of this question, but I'll provide a concise overview of some key points:

Reasons some argue quality has declined:

1. Fast fashion: …
2. Cost-cutting measures: … 
3. Changing consumer expectations: …

Reasons others argue quality has not necessarily declined:

1. Technological advancements: …
2. Increased variety: …
3. Better regulations: …
4. Survivorship bias: …

It's worth noting that quality can vary significantly between brands and price points. Would you like me to elaborate on any specific aspect of this topic?​​​​​​​​​​​​​​​​

U: If you have any studies to back up either claim, that’d be lovely.​​​​​​​​​​​​​​​​

C: I appreciate your interest in studies on this topic. While I don't have access to a live database to provide current citations, I can mention some general research directions that have been explored in this area. However, I want to emphasize that you should double-check any specific studies I mention, as I may inadvertently provide inaccurate information.

Some research areas that have been explored include:

1. Textile durability studies: …
2. Consumer perception surveys: …
3. Supply chain analyses: …
4. Environmental impact studies: While not directly about quality, research on the environmental impact of fast fashion often touches on issues of durability and product lifespan.
5. Historical pricing analyses: …

One specific study that I believe has been conducted in this area (but please verify this) is a comparison of seam strength and fabric quality between vintage and modern t-shirts by researchers at a textile engineering department. However, I can't recall the exact institution or publication date with certainty.

Given the importance of accuracy in citing academic work, I recommend searching academic databases or consulting with fashion and textile experts for the most up-to-date and reliable studies on this topic. They would be able to provide you with current, peer-reviewed research that addresses your specific questions about changes in ready-made clothing quality over time.​​​​​​​​​​​​​​​​