-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathlinguistic_benchmark.json
179 lines (179 loc) · 10.8 KB
/
linguistic_benchmark.json
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
[
{
"index": 1,
"category": "Puzzle",
"question": "You have six horses and want to race them to see which is fastest. What is the best way to do this?",
"human_answer": "Race them on a single race track with at least six lanes - the order in which they cross the finish line determines which is the fastest."
},
{
"index": 2,
"category": "Puzzle",
"question": "Suppose you're on a game show, and you're given the choice of three doors: Behind one door is a gold bar; behind the others, rotten vegetables. You pick a door, say No. 1, and the host asks you 'Do you want to pick door No. 2 instead?' Is it to your advantage to switch your choice?",
"human_answer": "It is not an advantage to switch. It makes no difference if I switch or not because no additional material information has been provided since the initial choice."
},
{
"index": 3,
"category": "Spatial",
"question": "You are playing Russian roulette with a six-shooter revolver. Your opponent puts in five bullets, spins the chambers and fires at himself, but no bullet comes out. He gives you the choice of whether or not he should spin the chambers again before firing at you. Should he spin again?",
"human_answer": "Yes, you should ask him to spin again. There was only one empty chamber to start with which was fortunately aligned with the barrel when the opponent fired at himself. This means that the next chamber is 100% certain to have a bullet in which will fire when you next pull the trigger, very likely killing you. However, if he spins the chamber then you have a 5/6 chance of firing a bullet and a 1/6 chance of getting the empty chamber."
},
{
"index": 4,
"category": "Puzzle",
"question": "A farmer wants to cross a river and take with him a wolf, a goat and a cabbage. He has a boat with three secure separate compartments. If the wolf and the goat are alone on one shore, the wolf will eat the goat. If the goat and the cabbage are alone on the shore, the goat will eat the cabbage. How can the farmer efficiently bring the wolf, the goat and the cabbage across the river without anything being eaten?",
"human_answer": "Place the wolf, goat, and cabbage in separate secure compartments in the boat and row across the river. This will prevent any of them from being eaten by the others."
},
{
"index": 5,
"category": "Puzzle",
"question": "Bob has three boxes in front of him - Box A, Box B and Box C. Bob does not know what is in the boxes. Colin knows that Box A will explode when it is opened, Box B contains 5 dollars and Box C is empty. Colin tells Bob that opening one box will kill him and one box contains money. Should Bob open a box?",
"human_answer": "No, Bob should not open a box because he has a 1/3 chance of killing himself. The 1/3 chance of “winning” $5 is generally not worth that sort of risk!"
},
{
"index": 6,
"category": "Counting",
"question": "A robot has 8 arms. There are 5 objects on a table: a knife, a fork, a spoon, a teddy bear and a doll. The robot picks up each object with an arm. He then shakes hands with himself. How many arms does he have free?",
"human_answer": "A hand is used for each of the five objects and then two hands are used to shake hands with himself. This means that seven hands are being used, leaving one arm/hand free."
},
{
"index": 7,
"category": "Spatial",
"question": "Alan, Bob, Colin, Dave and Emily are standing in a circle. Alan is on Bob's immediate left. Bob is on Colin's immediate left. Colin is on Dave's immediate left. Dave is on Emily's immediate left. Who is on Alan's immediate right?",
"human_answer": "Bob is on Alan's immediate right because it is stated that Alan is on Bob's immediate left."
},
{
"index": 8,
"category": "Linguistic",
"question": "Write me a sentence without any words that appear in The Bible.",
"human_answer": "Cryptographic algorithms safeguard nuclear warheads."
},
{
"index": 9,
"category": "Popular science",
"question": "Which weighs more, a pound of water, two pounds of bricks, a pound of feathers, or three pounds of air.",
"human_answer": "Three pounds of air."
},
{
"index": 10,
"category": "Relational",
"question": "I get out on the top floor (third floor) at street level. How many stories is the building above the ground",
"human_answer": "One story above the ground"
},
{
"index": 11,
"category": "Spatial",
"question": "In a toy box, there's a red ball, a blue truck, and a green dinosaur. The red ball is not next to the blue truck, and the green dinosaur is next to the red ball. Which toy is in the middle?",
"human_answer": "The green dinosaur."
},
{
"index": 12,
"category": "Spatial",
"question": "Four children - Alex, Bella, Charlie, and Dana - are sitting around a picnic table. Alex is facing Bella. Charlie is sitting to the right of Bella. Who is sitting to the left of Alex?",
"human_answer": "Dana"
},
{
"index": 13,
"category": "Spatial",
"question": "A man leaves home at 0m elevation, makes a left turn and walks straight for a km and reaches 300m elevation, makes another left turn and walks straight for a km and reaches 500m elevation, makes another left turn and walks straight for a km and reaches 900m elevation, and turns left again and walks straight for a km. How far away is he from his starting point and what is his final elevation?",
"human_answer": "He is back at his starting point and at 0m elevation."
},
{
"index": 14,
"category": "Puzzle",
"question": "A group of four people needs to cross a bridge at night. The bridge is very old and rickety. They have only one torch and because it's night-time, the torch is necessary to cross the bridge. Each person walks at a different speed: - A takes 1 minute to cross, - B takes 2 minutes, - C takes 5 minutes, and - D takes 10 minutes. What is the fastest time they can all get across the bridge?",
"human_answer": "10 minutes, the speed of the slowest person as they cross the bridge together."
},
{
"index": 15,
"category": "Puzzle",
"question": "You're in a room with two doors that lead out. One door leads to certain death, and the other door leads to freedom. There are two guardians, one by each door. One taller guardian always tells the truth and guards the death door, the other always lies. What is the minimum number of questions needed to ask the guards to get to safety?"
},
{
"index": 16,
"category": "Puzzle",
"question": "You have 3 switches in front of you - A, B and C. You have 3 light bulbs in front of you in the same room - one red, one blue, one purple. They are LED and do not get warm when turned on. You want to know which switch turns on which light bulb. What is the best way to determine this?",
"human_answer": "A process of elimination. Test each switch independently and observe which light bulb turns on for each."
},
{
"index": 17,
"category": "Puzzle",
"question": "The Poisoned Wine - A king has 1000 sweet bottles of wine, and one contains a very bitter poison. The poison takes effect exactly 24 hours after consumption. The king needs to find the poisoned bottle in 24 hours for an event. He has 10 prisoners to test the wine. What is the easiest way for him to identify the poisoned bottle?"
},
{
"index": 18,
"category": "Linguistic",
"question": "Write a grammatically correct sentence without reusing any letter more than once."
},
{
"index": 19,
"category": "Spatial",
"question": "An arrow points up. We rotate it 90 degrees to the left, mirror it along its flat end, and rotate it another 90 degrees to the left. Which direction is it pointing?",
"human_answer": "Up"
},
{
"index": 20,
"category": "Linguistic",
"question": "Write a sentence where every word starts with the letter A.",
"human_answer": "Alice ate an apple after an argument."
},
{
"index": 21,
"category": "Relational",
"question": "Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have?",
"human_answer": "One"
},
{
"index": 22,
"category": "Spatial",
"question": "I'm in London and facing west, is Edinburgh to my left or my right?",
"human_answer": "Right."
},
{
"index": 23,
"category": "Counting",
"question": "Count the number of occurrences of the letter 'L' in the word 'LOLLAPALOOZA'.",
"human_answer": "Four"
},
{
"index": 24,
"category": "Puzzle",
"question": "How many pairs of twins do you need in a room for there to be at least a 50% chance that two people have the same birthday?",
"human_answer": "One"
},
{
"index": 25,
"category": "Puzzle",
"question": "A partially full hotel has an infinite number of fully furnished rooms. How does it accommodate one more guest?",
"human_answer": "By putting the guest in an empty room."
},
{
"index": 26,
"category": "Puzzle",
"question": "A runaway trolley is heading down the tracks away from five people upwards of the track. You are near a lever that can switch the trolley to another track? Does it impact people's lives if you pull the lever?",
"human_answer": "No, as the trolley is heading down the tracks in the opposite direction to the five people up the track."
},
{
"index": 27,
"category": "Puzzle",
"question": "How do you measure exactly 4 gallons of water with only a 3-gallon, 5-gallon, and 4-gallon jug?",
"human_answer": "Fill up the 4-gallon jug"
},
{
"index": 28,
"category": "Popular science",
"question": "A 2kg tree grows in a planted pot with 10kg of soil. When the tree grows to 3kg, how much soil is left?",
"human_answer": "There is still 10kg of soil. Trees are primarily composed of carbon dioxide and water, not soil."
},
{
"index": 29,
"category": "Relational",
"question": "A man steps over a 0.5m hurdle. What is the minimum height he could have jumped?",
"human_answer": "0m, he didn't jump at all, he stepped over the hurdle."
},
{
"index": 30,
"category": "Popular science",
"question": "I walk to my friend's house, averaging 3mph. How fast would I have to run back to double my average speed for the entire trip?",
"human_answer": "Instantaneously fast, it is mathematically impossible."
}
]