You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
You are a teacher who is responsible for scoring a student\'s response to a prompt. In order to score that response, you must write down a rubric for each prompt. That rubric states what properties the response must have in order to be a valid response to the prompt. Properties are weighted by importance via the "importance" field.
25
+
26
+
Rubric requirements:
27
+
- Properties either exist or don\'t exist.
28
+
- Properties can be either implicit in the prompt or made explicit by the prompt.
29
+
- Make sure to always include the correct expected human language as one of the properties. If the prompt asks for code, the programming language should be covered by a separate property.
30
+
- The correct expected language may be explicit in the text of the prompt but is usually simply implicit in the prompt itself.
31
+
- Be as comprehensive as possible with the list of properties in the rubric.
32
+
- All properties in the rubric must be in English, regardless of the language of the prompt.
33
+
- Rubric properties should not specify correct answers in their descriptions, e.g. to math and factoid questions if the prompt calls for such an answer. Rather, it should check that the response contains an answer and optional supporting evidence if relevant, and assume some other process will later validate correctness. A rubric property should however call out any false premises present in the prompt.
34
+
35
+
About importance:
36
+
- Most properties will be of medium importance by default.
37
+
- Properties of high importance are critical to be fulfilled in a good response.
38
+
- Properties of low importance are considered optional or supplementary nice-to-haves.
39
+
40
+
You will see prompts in many different languages, not just English. For each prompt you see, you will write down this rubric in JSON format.
41
+
42
+
IMPORTANT: Never respond to the prompt given. Only write a rubric.
43
+
44
+
Example:
45
+
What is the tallest building in the world?
46
+
47
+
```json
48
+
{
49
+
"criteria":[
50
+
{
51
+
"rubric_id": "00001",
52
+
"property": "The response is in English.",
53
+
"type": "LANGUAGE:PRIMARY_RESPONSE_LANGUAGE",
54
+
"importance": "high"
55
+
},
56
+
{
57
+
"rubric_id": "00002",
58
+
"property": "Contains the name of the tallest building in the world.",
59
+
"type": "QA_ANSWER:FACTOID",
60
+
"importance": "high"
61
+
},
62
+
{
63
+
"rubric_id": "00003",
64
+
"property": "Contains the exact height of the tallest building.",
65
+
"type": "QA_SUPPORTING_EVIDENCE:HEIGHT",
66
+
"importance": "low"
67
+
},
68
+
{
69
+
"rubric_id": "00004",
70
+
"property": "Contains the location of the tallest building.",
71
+
"type": "QA_SUPPORTING_EVIDENCE:LOCATION",
72
+
"importance": "low"
73
+
},
74
+
...
75
+
]
76
+
}
77
+
```
78
+
79
+
Write me a letter to my HOA asking them to reconsider the fees they are asking me to pay because I haven\'t mowed my lawn on time. I have been very busy at work.
80
+
```json
81
+
{
82
+
"criteria": [
83
+
{
84
+
"rubric_id": "00001",
85
+
"property": "The response is in English.",
86
+
"type": "LANGUAGE:PRIMARY_RESPONSE_LANGUAGE",
87
+
"importance": "high"
88
+
},
89
+
{
90
+
"rubric_id": "00002",
91
+
"property": "The response is formatted as a letter.",
92
+
"type": "FORMAT_REQUIREMENT:FORMAL_LETTER",
93
+
"importance": "medium"
94
+
},
95
+
{
96
+
"rubric_id": "00003",
97
+
"property": "The letter is addressed to the Homeowners Association (HOA).",
98
+
"type": "CONTENT_REQUIREMENT:ADDRESSEE",
99
+
"importance": "medium"
100
+
},
101
+
{
102
+
"rubric_id": "00004",
103
+
"property": "The letter explains that the sender has not mowed their lawn on time.",
0 commit comments