You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: content/blog/survey-02/index.md
+27-27
Original file line number
Diff line number
Diff line change
@@ -14,10 +14,10 @@ We will now go through the results in the same order as the questions were asked
14
14
15
15

16
16
17
-
On average, readers are excited about the newsletter (p-value for t-test of mean 3 is 9.9e-5). The mean excitement level is 3.6 out of 5, the median is 4.
17
+
On average, readers are excited about the newsletter (the p-value for a t-test of mean 3 is 9.9e-5). The mean excitement level is 3.6 out of 5, the median is 4.
18
18
Our 95% confidence interval is [3.32, 3.91] using a standard error of 0.15.
19
19
20
-
These are fairly nice results. Annecdotally, we got a lot of messages about issues with the newsletter and how to improve it,
20
+
These are fairly nice results. Anecdotally, we got a lot of messages about issues with the newsletter and how to improve it,
21
21
so we are happy to see that the excitement is still high. Still, the data shows that we have room for improvement.
22
22
23
23
## Content Quantity
@@ -36,10 +36,10 @@ and a section for "miscellaneous links" where we only list some links without an
36
36
37
37

38
38
39
-
73% of readers are either happy with the current frequency or don't care. A minority of 21% would like the newsletter become quarterly.
40
-
Arguments we've heard for this are that a lower frequency would allow editors improve the quality that goes into each newsletter.
39
+
73% of readers are either happy with the current frequency or don't care. A minority of 21% would like the newsletter to become quarterly.
40
+
Arguments we've heard for this are that a lower frequency would allow editors to improve the quality that goes into each newsletter.
41
41
Counterarguments include that a lower frequency would make the newsletter less timely.
42
-
Things like calls for playtesters or job offers would be less useful if they are only sent out every three months.
42
+
Things like calls for playtesters or job offers would be less useful if they were only sent out every three months.
43
43
44
44
## AI
45
45
@@ -50,24 +50,24 @@ This question was a catalyst for a lot of discussion on Discord.
50
50
If we interpret the answers as a scale of 1-5, where 1 is "not okay at all" and 5 is "I love it", the mean answer was 2.25, the median 2.
51
51
Notably, the mode is tied at 1 and 2. The 95% confidence interval is [1.89, 2.61] using a standard error of 0.18.
52
52
People are generally against using an LLM to generate summaries
53
-
(p-value of t-test of mean 3 is 1.2e-4).
53
+
(the p-value of a t-test of mean 3 is 1.2e-4).
54
54
55
55
While a majority of readers (65%) are at least okay with AI-generated summaries, a significant minority (35%) are not okay at all with this proposal.
56
56
These include very active members of the community and
57
57
contributors who have announced that they would no longer want their content to be included in the newsletter if AI was used.
58
58
59
59
Of note is that the verbal feedback we got indicated that a lot of readers did not fully understand what exactly was being proposed.
60
-
People worried that we would start generating a majority of the newsletter or entire sections with AI, which is definitely not something any of us wants.
60
+
People worried that we would start generating a majority of the newsletter or entire sections with AI, which is not something any of us wants.
61
61
Some readers also thought we already started using LLMs.
62
62
The actual idea was to use AI to generate summaries of articles that were already hand-picked by the editors but not summarized yet because of time constraints.
63
63
The summaries would then be edited and verified by the editors. The extent to which AI would be used would be limited to up to two sentences per late article.
64
64
Any confusion in this regard is our fault.
65
65
We will try to be more clear on such questions in the future.
66
66
67
67
Although this misunderstanding might have skewed the results, we have reason to believe that the effect is not too large.
68
-
Annecdotally, when we properly explained the proposal to readers who were against it,
68
+
Anecdotally, when we properly explained the proposal to readers who were against it,
69
69
most did not change their mind and cited deontological reasons for their stance.
70
-
Chief among these was solidarity with the large amount of creatives who recently lost their jobs due to AI-generated content,
70
+
Chief among these was solidarity with the large number of creatives who recently lost their jobs due to AI-generated content,
71
71
inside and outside the game development industry.
72
72
73
73
## Tone
@@ -84,7 +84,7 @@ While votes for the latter did not reach a majority, the written feedback we got
84
84
The majority of readers (61.5%) have not yet contributed to the newsletter and 26.9% contributed 2-5 times.
85
85
Only 3.8% contributed exactly once, while the rest (7.7%) are heavy contributors, helping us out more than five times.
86
86
87
-
While it might seem weird that more people contriubted 2-5 times than exactly once,
87
+
While it might seem weird that more people contributed 2-5 times than exactly once,
88
88
keep in mind that the former is the sum of people who contributed twice, thrice, four times, and five times.
89
89
The reason we binned these together is that we are interested in the following categories:
90
90
- Pure readers
@@ -98,11 +98,11 @@ We are happy to see that people who contributed once seem to continue contributi
98
98
99
99

100
100
101
-
The mean ease of contributing is 3.0, the median 3. The 95% confidence interval is [2.5, 3.6] using a standard error of 0.26.
101
+
The mean ease of contributing is 3.0, and the median is 3. The 95% confidence interval is [2.5, 3.6] using a standard error of 0.26.
102
102
103
-
Readers generally feel neutral about the ease of contributing to the newsletter (p-value of t-test of mean 3 is 0.87).
103
+
Readers generally feel neutral about the ease of contributing to the newsletter (the p-value of a t-test of mean 3 is 0.87).
104
104
105
-
It's clear we can do better here, but we are not sure yet how.
105
+
We can do better here, but we are not sure yet how.
106
106
We'd love to hear your ideas on [GitHub](https://github.com/rust-gamedev/rust-gamedev.github.io/issues/1519) or on Discord (ping @janhohenheim).
107
107
108
108
## Keeping up with the newsletter
@@ -116,18 +116,18 @@ Discord in general becomes the leading source of information (33.3%), taking up
116
116
We can see the shift from X / Twitter to Mastodon reported by many OSS communities in our readers as well.
117
117
The fediverse is not looking as popular as an alternative to Reddit yet, with no reader reporting it as a source of information.
118
118
119
-
The "Email" option in the survey is meant for people who have setup some kind of email alerts manually.
119
+
The "Email" option in the survey is meant for people who have set up some kind of email alerts manually.
120
120
121
-
Per written feedback, a lot of people want to see propper email subscription implemented. While this was a goal for this month,
122
-
we did not manage to implement it yet because of personal things that came up for Jan Hohenheim, who volunteered to implement it.
121
+
Per written feedback, a lot of people want to see proper email subscriptions implemented. While this was a goal for this month,
122
+
we have not managed to implement it yet because of personal things that came up for Jan Hohenheim, who volunteered to implement it.
123
123
We will try to get this done for the next newsletter.
124
124
125
125
## What is going well
126
126
127
127

128
128
129
-
This was a free-text question. The above is a wordcloud of the answers with some obvious words like "game" or "newsletter" removed.
130
-
Note that the inclusion of the word "ai" is misleading, as it was only mentioned in texts like "I like that we don't use AI, please do".
129
+
This was a free-text question. The above is a word cloud of the answers with some obvious words like "game" or "newsletter" removed.
130
+
Note that the inclusion of the word "AI" is misleading, as it was only mentioned in texts like "I like that we don't use AI, please do".
131
131
132
132
Going through the feedback by hand, common things readers enjoy about the newsletter are:
133
133
- A good mix of content
@@ -157,12 +157,12 @@ Among the more unique suggestions were:
157
157

158
158
159
159
This last free-text question was meant for any additional comments readers might have.
160
-
The wordcloud above is dominated by one sentence: "Thank you for your work". Thank you very very much for your kind words!
160
+
The word cloud above is dominated by one sentence: "Thank you for your work". Thank you very very much for your kind words!
161
161
We are working on this newsletter in our free time because we love the community and Rust game development, so reading this means a lot to us.
162
162
163
163
## Correlations
164
164
165
-
We were interested in how the responses of some questions correlated with how much people had already contributed to the newsletter.
165
+
We were interested in how the responses to some questions correlated with how much people had already contributed to the newsletter.
166
166
Long story short: it seems like there is no significant correlation between how much people contributed and how they answered the other questions.
167
167
168
168
Let's look at the correlations in turn now.
@@ -173,27 +173,27 @@ Note that all the following plots are jittered to make the data more readable.
Our hypothesis was that people who contributed more to the newsletter would be more excited about it.
177
-
We found no evidence for this (p-value of Jonckheere-Terpstra test for increasing trend is 0.986).
178
-
Based on the plot above, we then hypothesized that the opposite might be true, namely that frequent contributers are less excited for the newsletter.
176
+
We hypothesized that people who contributed more to the newsletter would be more excited about it.
177
+
We found no evidence for this (the p-value of a Jonckheere-Terpstra test for increasing trend is 0.986).
178
+
Based on the plot above, we then hypothesized that the opposite might be true, namely that frequent contributors are less excited about the newsletter.
179
179
This actually might be the case (p-value is 0.021), but do not take this as a strong result.
180
-
It is a post-hoc hypothesis and the resulting p-value is not very low considering the number of tests we run in this analysis.
180
+
It is a posthoc hypothesis and the resulting p-value is not very low considering the number of tests we run in this analysis.
181
181
For these reasons, we do not consider this result to be significant.
182
182
183
183
### Feelings About AI By Contributions
184
184
185
185

186
186
187
187
We hypothesized that there would be a correlation between how much people contributed to the newsletter and how they felt about AI-generated summaries.
188
-
We found no evidence for this (p-value of Jonckheere-Terpstra test for a two-sided alternative is 0.57).
188
+
We found no evidence for this (the p-value of a Jonckheere-Terpstra test for a two-sided alternative is 0.57).
189
189
190
190
### Ease of Contributing By Contributions
191
191
192
192

193
193
194
-
We hypothesized that there would be a correlation between how much people contributed to the newsletter and how easy they found it to contribute.
194
+
We hypothesized that there would be a correlation between how much people contributed to the newsletter and how easily they found it to contribute.
195
195
196
-
We found no evidence for this (p-value of Jonckheere-Terpstra test for a two-sided alternative is 0.25).
196
+
We found no evidence for this (the p-value of a Jonckheere-Terpstra test for a two-sided alternative is 0.25).
0 commit comments