Aurornis
7 hours ago
The spread isn't as large as I thought from the headline. It's closer to a toss-up than I would have expected.
The length of the response is a huge factor:
> Students tended to prefer longer responses. The selected answer was 37% longer on average than the alternatives. The longest response won 47.7% of decisive writing comparisons. The shortest still won 25.0%.
So the score is partially a proxy for longest responses.
Makes me wonder how much the reviewers actually read the text. Were lazy evaluators picking the text that looked the longest or most structured without reading it all?
xtiansimon
an hour ago
Matches my experience. When I need to think or reason about something, between Clause and ChatGPT, the latter is my go to. With more to read, there’s just more to work with. For the same reason it’s bonkers for tech issues. How many times did I get a ChatGPT reply a mile long, do the first thing, and then read more after it didn’t work. Only then realize chatgpt didn’t offer the best answer first.
pasharayan
6 hours ago
Yeah it's hard to get good data on what's best. We did read the a good sample amount of essays. Lots of arguments about what made them better, but its uncontroversial that longer ones were more preferred.
Interestingly we trained a simple LORA layer on top of Inkling and it turns out a model can smell which model wrote a response 70% of the time. I wonder if the smell of gemini is just more preferred by students.
dr_dshiv
6 hours ago
Hmm, seems like which model teachers prefer would be the more important headline…
pasharayan
6 hours ago
You know.. might try to see what we can do here