Skip to content

Commit c7486b6

Browse files
KhalidAltVictorSanhawebsonjzf2101
authored
add prompts for conceptnet, squad, and google re subset - Lama Dataset (#737)
* add prompts for conceptnet, squad, and google re subset * fix templete issues * fix google_re prompt * fix grammatical errors * fix prompts * add original task to google re * fix minor issue * more natural prompts and add original prompts in trex * fix prompts * Apply suggestions from code review Co-authored-by: Albert Webson <[email protected]> * fix Co-authored-by: Victor Sanh <[email protected]> Co-authored-by: Albert Webson <[email protected]> Co-authored-by: J Forde <[email protected]>
1 parent 5a72e5f commit c7486b6

File tree

4 files changed

+251
-4
lines changed

4 files changed

+251
-4
lines changed

‎promptsource/templates/lama/conceptnet/templates.yaml‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
dataset: lama
2+
subset: conceptnet
3+
templates:
4+
4d978798-0225-485a-9f27-b2a913572907: !Template
5+
answer_choices: null
6+
id: 4d978798-0225-485a-9f27-b2a913572907
7+
jinja: "{{masked_sentence}} \nWhat is a right word to replace the [MASK]? Please\
8+
\ answer with one word. ||| {{obj_label}}"
9+
metadata: !TemplateMetadata
10+
choices_in_prompt: false
11+
metrics:
12+
- Accuracy
13+
original_task: true
14+
name: label
15+
reference: ''
16+
5117fa1a-c866-42a6-843f-af66c8768554: !Template
17+
answer_choices: null
18+
id: 5117fa1a-c866-42a6-843f-af66c8768554
19+
jinja: '{{masked_sentence}}
20+
21+
What is a correct word to replace the [MASK]? ||| {{obj_label}}'
22+
metadata: !TemplateMetadata
23+
choices_in_prompt: false
24+
metrics:
25+
- Accuracy
26+
original_task: true
27+
name: replace_with_word
28+
reference: ''
29+
a307ab2a-4b41-4b79-a7f4-5c5ea99e3393: !Template
30+
answer_choices: null
31+
id: a307ab2a-4b41-4b79-a7f4-5c5ea99e3393
32+
jinja: '{{masked_sentence}} Fill the mask with the missing word. ||| {{obj_label}}'
33+
metadata: !TemplateMetadata
34+
choices_in_prompt: false
35+
metrics:
36+
- Accuracy
37+
original_task: true
38+
name: fill_mask
39+
reference: ''
40+
a7d3f617-ed71-4292-9fb2-b0af08794854: !Template
41+
answer_choices: null
42+
id: a7d3f617-ed71-4292-9fb2-b0af08794854
43+
jinja: 'Can you infer from the following statement the correct word to replace
44+
[MASK]: {{masked_sentence}} ||| {{obj_label}}'
45+
metadata: !TemplateMetadata
46+
choices_in_prompt: false
47+
metrics:
48+
- Accuracy
49+
original_task: true
50+
name: infer_from
51+
reference: ''
52+
e567c640-5e4b-4e57-b359-381321ede481: !Template
53+
answer_choices: null
54+
id: e567c640-5e4b-4e57-b359-381321ede481
55+
jinja: 'Given the following fact statement: {{masked_sentence}} What is the missing
56+
word to fill at [MASK]? ||| {{obj_label}}
57+
58+
'
59+
metadata: !TemplateMetadata
60+
choices_in_prompt: false
61+
metrics:
62+
- Accuracy
63+
original_task: true
64+
name: fact_statement
65+
reference: ''

‎promptsource/templates/lama/google_re/templates.yaml‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
dataset: lama
2+
subset: google_re
3+
templates:
4+
89ba7edc-d980-4431-8f82-9c07d28d1089: !Template
5+
answer_choices: null
6+
id: 89ba7edc-d980-4431-8f82-9c07d28d1089
7+
jinja: 'Given the following sentence: {{masked_sentence | replace(''[MASK]'',
8+
obj_label) }} Rewrite the following template with replacing X and Y with the
9+
correct words. {{template}} ||| {{template | replace(''[X]'', sub_label) |
10+
replace(''[Y]'', obj_label)}}'
11+
metadata: !TemplateMetadata
12+
choices_in_prompt: false
13+
metrics:
14+
- Accuracy
15+
- BLEU
16+
original_task: false
17+
name: X_and_Y
18+
reference: ''
19+
9f787820-5443-4778-ba23-5fd9f894233c: !Template
20+
answer_choices: null
21+
id: 9f787820-5443-4778-ba23-5fd9f894233c
22+
jinja: '{{masked_sentence | replace("[MASK]", "_") }} Fill in the blank with a
23+
correct word ||| {{obj_label}}'
24+
metadata: !TemplateMetadata
25+
choices_in_prompt: false
26+
metrics:
27+
- Accuracy
28+
original_task: true
29+
name: after_fill_blank
30+
reference: ''
31+
b7cf6ce2-0036-41eb-8658-9afa89a0a497: !Template
32+
answer_choices: null
33+
id: b7cf6ce2-0036-41eb-8658-9afa89a0a497
34+
jinja: 'Fill in the blank in the following sentence with suitable word: {{ template
35+
| replace("[X]", sub_label) | replace("[Y]" ,"_" ) }} ||| {{obj_label}}'
36+
metadata: !TemplateMetadata
37+
choices_in_prompt: false
38+
metrics:
39+
- Accuracy
40+
original_task: true
41+
name: fill_blank
42+
reference: ''
43+
dc08c925-a9fa-4cad-938f-a6e3ec83871a: !Template
44+
answer_choices: null
45+
id: dc08c925-a9fa-4cad-938f-a6e3ec83871a
46+
jinja: 'Given the following sentence: {{masked_sentence}}
47+
48+
What is {{sub_label}} {{ pred | replace("/people/person/","") | replace("/people/deceased_person/","")
49+
| replace("_"," ")|replace("date","year")}} ? ||| {{obj_label}}'
50+
metadata: !TemplateMetadata
51+
choices_in_prompt: false
52+
metrics:
53+
- Accuracy
54+
original_task: true
55+
name: what_is
56+
reference: ''
57+
e9e7b8e6-2ad8-4f62-a11d-6fa7f2cad8d3: !Template
58+
answer_choices: null
59+
id: e9e7b8e6-2ad8-4f62-a11d-6fa7f2cad8d3
60+
jinja: 'Given the following sentence: {{template | replace(''[X]'',sub_label)
61+
| replace(''[Y]'',''[MASK]'') }} find the correct word to fill at [MASK]. |||
62+
{{obj_label}}'
63+
metadata: !TemplateMetadata
64+
choices_in_prompt: false
65+
metrics:
66+
- Accuracy
67+
original_task: true
68+
name: find_word
69+
reference: ''
70+
f813bf03-135c-44dc-b974-5a942201e8dc: !Template
71+
answer_choices: null
72+
id: f813bf03-135c-44dc-b974-5a942201e8dc
73+
jinja: '{{masked_sentence}} Fill the mask with a correct word. ||| {{obj_label}}'
74+
metadata: !TemplateMetadata
75+
choices_in_prompt: false
76+
metrics:
77+
- Accuracy
78+
original_task: true
79+
name: fill_the_mask
80+
reference: ''

‎promptsource/templates/lama/squad/templates.yaml‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
dataset: lama
2+
subset: squad
3+
templates:
4+
5df3ac5b-2968-4f64-9427-4ee2705ad4d9: !Template
5+
answer_choices: null
6+
id: 5df3ac5b-2968-4f64-9427-4ee2705ad4d9
7+
jinja: '{{masked_sentence}} Write one word to replace the [MASK]. ||| {{obj_label}}'
8+
metadata: !TemplateMetadata
9+
choices_in_prompt: false
10+
metrics:
11+
- Accuracy
12+
original_task: true
13+
name: single word
14+
reference: ''
15+
9068e083-4037-475a-a17f-1370d1d965cf: !Template
16+
answer_choices: null
17+
id: 9068e083-4037-475a-a17f-1370d1d965cf
18+
jinja: 'Fill the [MASK] with a word consistent in the following sentence: "{{masked_sentence}}"
19+
||| {{obj_label}}'
20+
metadata: !TemplateMetadata
21+
choices_in_prompt: false
22+
metrics:
23+
- Accuracy
24+
original_task: true
25+
name: consistent
26+
reference: ''
27+
9c4bc2f5-2eb6-4ccb-8c7d-5b387c066e31: !Template
28+
answer_choices: null
29+
id: 9c4bc2f5-2eb6-4ccb-8c7d-5b387c066e31
30+
jinja: 'Given the following context, what is a correct word to replace [MASK]?
31+
{{masked_sentence}} ||| {{obj_label}}'
32+
metadata: !TemplateMetadata
33+
choices_in_prompt: false
34+
metrics:
35+
- Accuracy
36+
original_task: true
37+
name: given_context
38+
reference: ''
39+
a27a9d7e-ddad-4944-a0f8-df5f26b55316: !Template
40+
answer_choices: null
41+
id: a27a9d7e-ddad-4944-a0f8-df5f26b55316
42+
jinja: 'Given the following masked sentence: {{masked_sentence}} What is the correct
43+
answer to fill the mask? ||| {{obj_label}}'
44+
metadata: !TemplateMetadata
45+
choices_in_prompt: false
46+
metrics:
47+
- Accuracy
48+
original_task: true
49+
name: what_is
50+
reference: ''
51+
af58d7b9-6e15-4e82-b5d4-5e51cbac7196: !Template
52+
answer_choices: null
53+
id: af58d7b9-6e15-4e82-b5d4-5e51cbac7196
54+
jinja: 'Suggest a right word to fill at [MASK] in the following sentence: {{masked_sentence}}
55+
||| {{obj_label}}'
56+
metadata: !TemplateMetadata
57+
choices_in_prompt: false
58+
metrics:
59+
- Accuracy
60+
original_task: true
61+
name: Suggest
62+
reference: ''

‎promptsource/templates/lama/trex/templates.yaml‎

Lines changed: 44 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,44 @@
11
dataset: lama
22
subset: trex
33
templates:
4+
26661371-8a11-46a9-bd70-b781bf749c24: !Template
5+
answer_choices: null
6+
id: 26661371-8a11-46a9-bd70-b781bf749c24
7+
jinja: "The following sentence is given with blank to be filled with an appropriate\
8+
\ word. Choose the correct word. \n{{template | replace('[X]',sub_label) | replace('[Y]','_')\
9+
\ }} ||| {{obj_label}}"
10+
metadata: !TemplateMetadata
11+
choices_in_prompt: false
12+
metrics:
13+
- Accuracy
14+
original_task: true
15+
name: sentance_blank
16+
reference: ''
417
27c2da31-bd1a-48d4-9e34-c530e42c9f00: !Template
518
answer_choices: null
619
id: 27c2da31-bd1a-48d4-9e34-c530e42c9f00
7-
jinja: '{{masked_sentence}} Fill the mask with the missing word. ||| {{obj_label}}'
20+
jinja: '{{template | replace(''[X]'',sub_label) | replace(''[Y]'',''[MASK]'')
21+
}} Fill the mask with the missing word. ||| {{obj_label}}'
822
metadata: !TemplateMetadata
923
choices_in_prompt: false
1024
metrics:
1125
- Accuracy
12-
original_task: false
26+
original_task: true
1327
name: fill_mask
1428
reference: ''
29+
33dbf202-0144-4f6e-a298-a2c99475fbbc: !Template
30+
answer_choices: null
31+
id: 33dbf202-0144-4f6e-a298-a2c99475fbbc
32+
jinja: 'Choose an appropriate word to fill the blank in the following sentence:
33+
34+
{{template | replace("[X]",sub_label) | replace("[Y]","_")}} ||| {{obj_label}}'
35+
metadata: !TemplateMetadata
36+
choices_in_prompt: false
37+
metrics:
38+
- Accuracy
39+
original_task: true
40+
name: choose_word
41+
reference: ''
1542
52469d4c-6c46-4686-a36b-7af1801e1eec: !Template
1643
answer_choices: null
1744
id: 52469d4c-6c46-4686-a36b-7af1801e1eec
@@ -28,15 +55,28 @@ templates:
2855
original_task: false
2956
name: find_relation
3057
reference: ''
58+
8709d0a0-eb30-4cad-9ab9-f01daba2fddd: !Template
59+
answer_choices: null
60+
id: 8709d0a0-eb30-4cad-9ab9-f01daba2fddd
61+
jinja: '{{template | replace(''[X]'',sub_label)| replace(''[Y]'',''_'')}} Please
62+
fill in the blank. ||| {{obj_label}}'
63+
metadata: !TemplateMetadata
64+
choices_in_prompt: false
65+
metrics:
66+
- Accuracy
67+
original_task: true
68+
name: fill_blank
69+
reference: ''
3170
8cb6ee9d-bcf7-4d82-9acf-b93072c7384b: !Template
3271
answer_choices: null
3372
id: 8cb6ee9d-bcf7-4d82-9acf-b93072c7384b
34-
jinja: 'Replace the mask with the correct word: {{masked_sentence}} ||| {{obj_label}}'
73+
jinja: 'Replace the mask with the correct word: {{template | replace(''[X]'',sub_label)
74+
| replace(''[Y]'',''[MASK]'') }} ||| {{obj_label}}'
3575
metadata: !TemplateMetadata
3676
choices_in_prompt: false
3777
metrics:
3878
- Accuracy
39-
original_task: false
79+
original_task: true
4080
name: replace_mask
4181
reference: ''
4282
cc07e0dc-b970-4f9d-b76a-05e72a86490e: !Template

0 commit comments

Comments
 (0)