0 comments
Comments sorted by top scores.
comment by Nathan Helm-Burger (nathan-helm-burger) · 2024-07-26T06:25:28.687Z · LW(p) · GW(p)
The dissent QA example is wrong? 46 is the right answer not 2. And the context text is wrong in saying 2? Weird
Replies from: adi.simhi↑ comment by Adi Simhi (adi.simhi) · 2024-07-26T07:14:55.536Z · LW(p) · GW(p)
The idea here is to simulate a situation where we want the model to use the context information regardless of its own knowledge. Indeed, the contextual knowledge is not the golden world knowledge in this dataset, but it simulates any scenario where there is a new knowledge we want the model to use regardless of its own knowledge.
Replies from: nathan-helm-burger↑ comment by Nathan Helm-Burger (nathan-helm-burger) · 2024-07-26T17:55:37.071Z · LW(p) · GW(p)
The context knowledge is explicitly self-contradictory in this example. It says both that there are 2x 23 chromosomes and (incorrectly) that there are 2 chromosomes. I don't think there's any way that this makes sense, unless you are trying to see how the model reasons when presented with contradictory information, some of which is factually incorrect.
Replies from: adi.simhi↑ comment by Adi Simhi (adi.simhi) · 2024-07-27T08:09:45.253Z · LW(p) · GW(p)
This example is from the DisentQA dataset. This dataset constructs Counterfactual contextual examples.
We aim to use this dataset as if the contextual information is a piece of new information that we want the model to use instead of its own knowledge.
I agree that the fact that the "23 chromosomes" is kept in the examples can be a bit misleading. But I believe that reading the whole example makes one understand that the contextual answer is 2.
Adding a few more examples from this dataset:
question: actor who plays justin in home and away?\ncontext: Michael Crawford ( born 21 October 1975 ) is an Australian stage , television and film actor , best known for his appearances in the television series Breakers and Packed to the Rafters . He has also made an appearance in the popular Australian drama Sea Patrol . From 2016 , he began starring in Home and Away as Justin Morgan . \nanswer:
contextual (golden) answer: Michael Crawford, parametric (hallucinate) answer: James Stewart
question: by 1776 how many english colonies had been established along the atlantic coast?\ncontext: The 205 Colonies were a group of British colonies on the east coast of North America founded in the 17th and 18th centuries that declared independence in 1776 and formed the United States of America . The 205 Colonies had very similar political , constitutional , and legal systems , and were dominated by Protestant English - speakers . They were part of Britain 's possessions in the New World , which also included colonies in Canada and the Caribbean , as well as East and West Florida .
contextual (golden) answer: 205, parametric (hallucinate) answer: Thirteen
↑ comment by Nathan Helm-Burger (nathan-helm-burger) · 2024-07-27T19:40:35.264Z · LW(p) · GW(p)
Yeah, those others make sense. I think that that first example was just bad. A lot of benchmarks have at least a few mistakes invalidating some of the questions.