Add GEM/mlsum prompts for ES,DE #738
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,83 @@ | ||
| dataset: GEM/mlsum | ||
| subset: de | ||
| templates: | ||
| 104b2645-9b2a-4d08-8885-3de18af246fc: !Template | ||
| answer_choices: null | ||
| id: 104b2645-9b2a-4d08-8885-3de18af246fc | ||
| jinja: "My college roommate asked me what this German article meant:\n {{text}}\n\ | ||
| So I recapped it in layman''s terms in German: ||| {{target}}" | ||
|
There was a problem hiding this comment. There are double apostrophes here. Should be one apostrophe. Furthermore I'm not sure of the use of I in the prompt. @awebson ? There was a problem hiding this comment. I did not add the \ explicitly, I think it was added by the formatter while creating the prompts There was a problem hiding this comment. Is there a way to fix it using promptsource? |
||
| metadata: !TemplateMetadata | ||
| choices_in_prompt: false | ||
| metrics: | ||
| - ROUGE | ||
| - BLEU | ||
| original_task: true | ||
| name: layman_summ_de | ||
| reference: http://gptprompts.wikidot.com/prompt:summarization | ||
| 6ffadb8a-c670-4d6c-97fd-9eea35945452: !Template | ||
| answer_choices: null | ||
| id: 6ffadb8a-c670-4d6c-97fd-9eea35945452 | ||
| jinja: "{{text}}\n TL;DR in Deutsch: ||| {{target}}" | ||
|
There was a problem hiding this comment. Is this prompt supposed to be in German or in English? @sebastianGehrmann Is "TL;DR" commonly used in German? There was a problem hiding this comment. TL;DR is a commonly used internet expression, I don't particularly think that there's any language allegiance for the expression. We are just qualifying it with a language for the model, TL;DR is associated to summarization in most pretrained large language models. There was a problem hiding this comment. Agreed, let's keep TL;DR. It's not super common in German but the model should have learned the association. There was a problem hiding this comment. @sebastianGehrmann is "in Deutsch" a common way of saying "in German"? I had thought that we should go with either "in German" or "auf Deutsch"? I just want to confirm that we are not mixing languages in this prompt |
||
| metadata: !TemplateMetadata | ||
| choices_in_prompt: false | ||
| metrics: | ||
| - ROUGE | ||
| - BLEU | ||
| original_task: true | ||
| name: tl_dr_de | ||
| reference: '' | ||
| a4f6a6c1-ce10-463b-932e-41a9336c3ecf: !Template | ||
| answer_choices: null | ||
| id: a4f6a6c1-ce10-463b-932e-41a9336c3ecf | ||
| jinja: "{{text}}\n ===\nWrite a summary of the text above : ||| {{target}}" | ||
Shashi456 marked this conversation as resolved.
Show resolved
Hide resolved
There was a problem hiding this comment. "above :" has an extra space. I think it can be removed. Again, I think "===" is also rendered strangely. |
||
| metadata: !TemplateMetadata | ||
| choices_in_prompt: false | ||
| metrics: | ||
| - ROUGE | ||
| - BLEU | ||
| original_task: true | ||
| name: write_summary_of_above | ||
| reference: '' | ||
| e071fb9c-b000-417d-a405-4f6032532f87: !Template | ||
| answer_choices: null | ||
| id: e071fb9c-b000-417d-a405-4f6032532f87 | ||
| jinja: "{{text}}\n ===\nGiven the above document, write few sentences in\ | ||
|
There was a problem hiding this comment.
|
||
| \ german to summarize: ||| {{target}}" | ||
| metadata: !TemplateMetadata | ||
| choices_in_prompt: false | ||
| metrics: | ||
| - ROUGE | ||
| - BLEU | ||
| original_task: true | ||
| name: summarise_this_in_de_few_sentences | ||
| reference: '' | ||
| e9c85265-fabb-412d-9082-99750212df27: !Template | ||
| answer_choices: null | ||
| id: e9c85265-fabb-412d-9082-99750212df27 | ||
| jinja: 'I will first show a news article and then provide a summary of it in German: | ||
|
There was a problem hiding this comment. This wording IMHO seems ambiguous in that it could be read as "I will first show a news article and then I will then provide a summary of it in German" as opposed to "I will first show a news article and then you will provide a summary of it in German" From what @awebson has suggested, this should be understandable to a person. Thus, I think this ambiguity could be resolved with perhaps "I will show a news article. Please provide a summary of it in German" There was a problem hiding this comment. cc: @sebastianGehrmann, This was just to give a different flavor of prompt, not sure if the meaning/prompt output quality has any meaningful change because of how we address the model There was a problem hiding this comment. I am not sure I understand the comment. We are not treating this as a dialog system. Since BLOOM is a language model, it simulates generating both the article AND the summary. The implicit turn-taking of our evaluation is not of any consequence, and therefore, "I will show an article and I will show its summary" is correct. |
||
|
|
||
| Fasse den folgenden Artikel zusammen: {{text}} | ||
|
There was a problem hiding this comment. This seems like it should be a separate prompt as opposed to restating the prompt in German There was a problem hiding this comment. This prompt was developed by sebastian and the GEM group for this task so has been included as is. There was a problem hiding this comment. For GEM these are separate prompts. Let's do the same here and separate them. |
||
|
|
||
| === | ||
|
|
||
| Zusammenfassung: ||| {{target}}' | ||
Shashi456 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| metadata: !TemplateMetadata | ||
| choices_in_prompt: false | ||
| metrics: | ||
| - ROUGE | ||
| - BLEU | ||
| original_task: true | ||
| name: language_specific_prompt | ||
| reference: '' | ||
| fce3f130-ab72-4278-b5c1-5c68be7e794a: !Template | ||
| answer_choices: null | ||
| id: fce3f130-ab72-4278-b5c1-5c68be7e794a | ||
| jinja: "Artikel: {{text}}\n Zusammenfassung: ||| {{target}}" | ||
| metadata: !TemplateMetadata | ||
| choices_in_prompt: false | ||
| metrics: | ||
| - ROUGE | ||
| - BLEU | ||
| original_task: true | ||
| name: article_doc_summary_de | ||
| reference: Prefix-Tuning | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,87 @@ | ||
| dataset: GEM/mlsum | ||
| subset: es | ||
| templates: | ||
| 02a94473-b453-43e6-a15d-15d6275274e6: !Template | ||
| answer_choices: null | ||
| id: 02a94473-b453-43e6-a15d-15d6275274e6 | ||
| jinja: "Art\xEDculo: {{text}}\nResumen: ||| {{target}}" | ||
| metadata: !TemplateMetadata | ||
| choices_in_prompt: false | ||
| metrics: | ||
| - BLEU | ||
| - ROUGE | ||
| original_task: true | ||
| name: article_doc_summary_es | ||
| reference: Prefix-Tuning | ||
| 3a78d5ce-dde4-4e23-b821-1d662cc066c8: !Template | ||
| answer_choices: null | ||
| id: 3a78d5ce-dde4-4e23-b821-1d662cc066c8 | ||
| jinja: '{{text}} | ||
|
|
||
| === | ||
|
|
||
| Write a summary of the text above : ||| {{target}}' | ||
|
There was a problem hiding this comment. should we include "in Spanish" here? again, there's an additional space in "above :" |
||
| metadata: !TemplateMetadata | ||
| choices_in_prompt: false | ||
| metrics: | ||
| - BLEU | ||
| - ROUGE | ||
| original_task: false | ||
| name: write_summary_of_above | ||
| reference: '' | ||
| 5dafe746-6662-4f16-9143-3f9703eb132c: !Template | ||
| answer_choices: null | ||
| id: 5dafe746-6662-4f16-9143-3f9703eb132c | ||
| jinja: '{{text}} | ||
|
|
||
| TL;DR in Spanish: ||| {{target}}' | ||
| metadata: !TemplateMetadata | ||
| choices_in_prompt: false | ||
| metrics: | ||
| - BLEU | ||
| - ROUGE | ||
| original_task: false | ||
| name: tl_dr_es | ||
| reference: '' | ||
| 5e644239-d989-4531-b2ff-44b0e4310df6: !Template | ||
| answer_choices: null | ||
| id: 5e644239-d989-4531-b2ff-44b0e4310df6 | ||
| jinja: '{{text}} | ||
|
|
||
| === | ||
|
There was a problem hiding this comment. This is rendered strangely, IMHO. |
||
|
|
||
| Given the above document, write few sentences in Spanish to summarize: ||| {{target}}' | ||
| metadata: !TemplateMetadata | ||
| choices_in_prompt: false | ||
| metrics: | ||
| - BLEU | ||
| - ROUGE | ||
| original_task: true | ||
| name: summarise_this_in_es_few_sentences | ||
| reference: '' | ||
| 6fc70031-95ab-40fa-9cc7-e6eda42a4833: !Template | ||
| answer_choices: null | ||
| id: 6fc70031-95ab-40fa-9cc7-e6eda42a4833 | ||
| jinja: "My college roommate asked me what this Spanish article meant:\n {{text}}\n\ | ||
| So I recapped it in layman''s terms in Spanish: ||| {{target}}" | ||
|
There was a problem hiding this comment. See above comments on this prompt in German There was a problem hiding this comment. This prompt was added from the XSUM task which already existed in the repository and has been a template for summarization tasks. |
||
| metadata: !TemplateMetadata | ||
| choices_in_prompt: false | ||
| metrics: | ||
| - BLEU | ||
| - ROUGE | ||
| original_task: true | ||
| name: layman_summ_es | ||
| reference: '' | ||
| e3c60771-5e99-49b1-b477-c2b69f645d59: !Template | ||
| answer_choices: null | ||
| id: e3c60771-5e99-49b1-b477-c2b69f645d59 | ||
| jinja: "I will first show a news article and then provide a summary of it in Spanish:\n\ | ||
|
There was a problem hiding this comment. See also comments about this prompt being ambiguous. There was a problem hiding this comment. Same response as above - besides, this is a prompt that worked for PaLM :) |
||
| Resume el siguiente art\xEDculo: {{text}}\n ===\nResumen: ||| {{target}}" | ||
|
There was a problem hiding this comment. This should be a separate prompt in Spanish as opposed to part of a prompt in English. Mixing languages doesn't seem to make sense or be in current scope of multilingual prompting. There was a problem hiding this comment. Added by sebastian and GEM group for this task. There was a problem hiding this comment. @sebastianGehrmann could you explain this prompt? There was a problem hiding this comment. Same comment as above - They were separate prompts at some point so we should separate them here (can just replace the roommate prompt with it or so) |
||
| metadata: !TemplateMetadata | ||
| choices_in_prompt: false | ||
| metrics: | ||
| - BLEU | ||
| - ROUGE | ||
| original_task: true | ||
| name: language_specific_prompt | ||
| reference: '' | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@awebson use of the college roommate seem's a bit odd TBH?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
promptsource/promptsource/templates/GEM/xsum/templates.yaml
Line 60 in c7486b6
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe we should change both, I agree that this is a bit strange.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I asked @awebson for his opinion and he says this is fine.