Analyzing student votes across AI models for college essay help

40 pointsposted 10 hours ago
by pasharayan

74 Comments

Aurornis

7 hours ago

The spread isn't as large as I thought from the headline. It's closer to a toss-up than I would have expected.

The length of the response is a huge factor:

> Students tended to prefer longer responses. The selected answer was 37% longer on average than the alternatives. The longest response won 47.7% of decisive writing comparisons. The shortest still won 25.0%.

So the score is partially a proxy for longest responses.

Makes me wonder how much the reviewers actually read the text. Were lazy evaluators picking the text that looked the longest or most structured without reading it all?

xtiansimon

an hour ago

Matches my experience. When I need to think or reason about something, between Clause and ChatGPT, the latter is my go to. With more to read, there’s just more to work with. For the same reason it’s bonkers for tech issues. How many times did I get a ChatGPT reply a mile long, do the first thing, and then read more after it didn’t work. Only then realize chatgpt didn’t offer the best answer first.

pasharayan

6 hours ago

Yeah it's hard to get good data on what's best. We did read the a good sample amount of essays. Lots of arguments about what made them better, but its uncontroversial that longer ones were more preferred.

Interestingly we trained a simple LORA layer on top of Inkling and it turns out a model can smell which model wrote a response 70% of the time. I wonder if the smell of gemini is just more preferred by students.

dr_dshiv

6 hours ago

Hmm, seems like which model teachers prefer would be the more important headline…

pasharayan

6 hours ago

You know.. might try to see what we can do here

conception

24 minutes ago

Gemini is the only model that has made legitimately good jokes - subtle, clever. I was terrified.

demibabs

7 hours ago

Author, that doesn’t mean you need to get Gemini to write your article.

pasharayan

6 hours ago

TBH i did a bit of both - we used AI to get us insights in the db, we then wrote and rewrote the article to be readable. It's really hard because usually when i write reports they're so terse its not fun to read

avaer

7 hours ago

> How to use Gemini for a college essay

Given that this is increasingly the go-to for a college degree, college needs to rethink its cirricula and place in the world. Or at least get rid of the essay.

Lest it become a place where student and teacher ais go to play pay-to-win social deduction video games.

Aurornis

7 hours ago

My professor and teacher friends are painfully aware of how many students submit AI generated essays.

They're adding steps like having the students discuss and defend their essay, which immediately reveals the people who had AI write something and thought they could bluff. This triggers complaints about social anxiety and such, which are unfortunately becoming the go-to defense when unable to discuss the work.

They're also moving toward more in-person writing. Instead of long essays, shorter writing segments as part of the test. Submitting a written essay earns you feedback from the professor and a better understanding of the topic, but that's it.

pasharayan

6 hours ago

I think it's less about curricula and more about how professors should eval their students. In previous era's essays are a proxy of the students ability to reason, remember and argue certain points - but its clear they fall short now.

I suspect the best schools into the future will integrate lots of socratic defenses of theses, building real things in real time or solving problems with a professor in a case study manner. My last company was a good hint to the future.

brainwad

7 hours ago

The homework essay is dead. High schoolers in my home state were just relieved of any take-home assignments because it was pointless now, I guess universities won't be far behind.

xtiansimon

an hour ago

> “The best response must be complete without becoming shapeless, structured without sounding mechanical…”

Anyone else scratching their head at these descriptions? Uh, your writing is shapeless. You’re 5% too mechanical. Who talks like this? Fiction writers and editors??

pasharayan

an hour ago

TBH :') that one was an AI slip through. In other comments i've flagged the things that were hard rewritten - but even on me that's bad.

John7878781

7 hours ago

In my experience, Gemini excels at explaining new concepts clearly and intuitively. Much more so than other models.

zorked

7 hours ago

Gemini is the best general-purpose model. We hear a lot about the other ones here because we are focusing on coding.

inbx0

7 hours ago

So, given that developers these days mix and match models anyway with harnesses like Pi, has anyone tried letting Claude or GPT do the coding, and Gemini do the comments and documentation? Maybe even asking it to "translate" Claude's output messages to better human language for the developer or subagents?

SturgeonsLaw

7 hours ago

I use Qwen/Claude for coding and Gemini/Gemma to draft text intended for a human. To me that feels like each model playing to their strengths.

pkulak

7 hours ago

Seems reasonable. You could just switch the model after the code is written, when you need to ask questions. You’ll lose your token cache, but that’s it.

pasharayan

6 hours ago

I wonder how much is due to their investment in notebook LLM + gmail work

gherkinnn

8 hours ago

Anything is better than the current crop of Claudes, its prose has become painful.

StableAlkyne

7 hours ago

For real, the "Claudish" has become so painful to read that it just takes me out of whatever task I was working on.

When prompted to use simple English without jargon, it's still filled with load bearing honest caveats in every footgun seam it talks about — what I should have led with. <Insert whatever other Claude cliche you prefer>

And I'm not the only one to notice this. Next time it's up for renewal, my team is abandoning it for GH Copilot in order to use literally any other frontier model.

glimshe

4 hours ago

That claim about load bearing is doing a lot of work there!

I'm curious to know how "Claudish" emerged during the training process. Why each AI has a particular voice if much of the training material is the same across AIs?

pmontra

7 hours ago

Yes, Claude speaks Claudish but at the end of the day I care about the Ruby, Python, JS it writes. It still does a good job at it even if maybe I prefer the way DeepSeek talks. I did not use other models in an agentic harness.

gherkinnn

6 hours ago

The code is fine, yes. But now that I am dabbling in spec-driven development (unsure whether I like it) I have to read a lot of prose and I simply cannot get myself to read a page of claudisms. It is that painful.

The odd phrases are one thing, what numbs my brain is how everything is evenly bombastic and lacks any sense of rhythm. Technical documents should not read like catchphrases strung together.

levelZero

3 hours ago

don't just dabble, dive in! domain-expert, buck-stopping, orchestrating architects of intent under constraints are going to be the only survivors ;-)

mobius47

7 hours ago

good thing claudish isnt a programming language

kuboble

7 hours ago

Say a word.

user

7 hours ago

[deleted]

purplecats

8 hours ago

You make some good points, but the best one is the one you might not be aware you've made.

levelZero

3 hours ago

You may not be aware of the most important emergent property of this comment: it made me piss myself laughing. Strong work, architect of intent.

senderista

7 hours ago

It's so bad that my workflow has Sol rewrite all comments and error messages produced by Claude.

whateveracct

7 hours ago

yeah i think Claude might just be compounding its way to a local maximum rn

asdfologist

7 hours ago

Fair challenge, the ability to write good prose is load-bearing for essential tasks.

gexla

7 hours ago

GPT gets the most out there, but creates interesting nuance that turn an idea into a more creative exercise. Claude is like a more grounded GPT but may miss the nuance. Gemini is my last pick, but does better than the rest for making something clear and understandable. It gets to be exhausting parsing through Claude outputs and breaking it down into something more easily understandable. One way I use to improve this is to ask the LLM to pull upstream ideas from prior work on the topics. This makes me feel better about the possibility of hallucination, gives me alternate places to look, but the model still may pull things out of context or fall over on the interpretation.

Given all that, I can definitely see how Gemini would be preferred. And good for Google, because I would rather my offering be the top choice for the most people rather than better serving a small subset of users.

ShawWang

9 hours ago

Gemini is bad at coding, but good at writing. More human like

parasti

7 hours ago

Gemini is not bad at coding? Give it a /plan and it does a very decent job in a fraction of the time/tokens that Claude needs.

madhatter999

7 hours ago

Makes sense when you consider what type of data Google had available in abundance. Natural written language (e.g., Docs, Gmail, Books) and natural spoken language (e.g., YouTube).

One could argue that Google Colab would supply the training data for better coding performance. I would argue that Colab is mostly used for non-complex (e.g., small number of variables) and self-contained (i.e., runnable in one page) code that can’t train a model for multi-folder and multi-page projects that rely on global connections, which real-life coding would often require.

danpalmer

7 hours ago

madhatter999

7 hours ago

Given it's Google, I don't believe it.

danpalmer

7 hours ago

That's up to you, but occam's razor would suggest the most likely thing is that they are in fact just doing what they say they do in their T&Cs.

madhatter999

7 hours ago

If that's what Occam's razor would suggest, then Occam presumes a world where actors do what they say. We see more and more the opposite of that in Silicon Valley.

You may well be right about Google not including the said data but your support for this point is weak.

danpalmer

6 hours ago

Yes Occam does quite literally state that – the option with the fewest assumptions is most likely to be correct. Assuming that someone is lying is an extra assumption to make over what they are claiming to do.

Malicious acts from large corporations are certainly well known, but they're also grossly over-represented in the news and mindshare. Mostly things just work as they say they work.

I don't really know what proof you're looking for, it's well known that it's hard to prove a negative. I'm not suggesting taking Google (or anyone else) at face-value, but rather saying that when you think about T&Cs, laws, PR, etc, it's unlikely.

WithinReason

7 hours ago

They cached the entire internet an have Google Books though

blauditore

7 hours ago

Google also has millions of lines of code in several languages written over decades. Probably access to more non-public (quality) code than most other companies in the world.

hesprs

8 hours ago

True. Whenever I ask LLMs to translate English to other languages, Gemini is always the most natural one.

Honali

7 hours ago

I'd love to see the same experiment with responses normalized for length, and with actual essay quality scored separately from how helpful the model's feedback felt

pasharayan

6 hours ago

Will give this a shot - we're going to try to work with students and schools better on normalizing this stuff out

mogili

7 hours ago

I've found Gemini to be better at writing. Claude has gotten worse while it gets better at coding / agentic use cases.

kuuuzya

7 hours ago

IT students should use their own LLM.

pasharayan

6 hours ago

As in they should make their own foundation models for essays?

lwansbrough

7 hours ago

If you use Gemini after reading Claude's simlish, it will feel very refreshing.

jdthedisciple

7 hours ago

skeptical

at most, the result could be useful for fellow students

I have no doubt other cohorts would rate differently

it is known (on HN at least) e.g. that SWEs tend to prefer brevity, contrary to these students apparently

pasharayan

5 hours ago

Students do have different incentives in writing than SWE's. There's the habit of reaching word counts for assignments, some social signalling of how intellgent you are with essay length etc.

The data we have is just there to compare for the student base. I'd love to try it out on other cohorts - but the acquisition of such users and the product to make them happy is hard to achieve!

foolfoolz

7 hours ago

when i use gemini i ask it to stop asking me questions, to stop showing me media and just use text, and be concise in output. with these directions it’s pretty useful

explodes

7 hours ago

My settings are similar. I also added to be very concise without losing important information. It's output is so slim now that I can ask yes or no questions. Asking for a code snippet gives me just a snippet without an accompanying paragraph of explanation that I would just ignore anyway. Of course I can still ask follow up questions, but it gets right to the point. No filler is a dream.

mial

8 hours ago

From my own tests, the best model for writing complex, nuanced prose is Opus 4.6. All the next versions are impossible to coax into good writing. Gemini is OK, GPT ok but requires significant prompt tuning.

userbinator

7 hours ago

Having coworkers who use it and thus unfortunately needing to read its output, Claude's "English" is very obviously unnatural-sounding, and extremely distinctive in a bad and irritating way. It's almost like another dialect.

...and of course this article itself has a bit of AI-ish tone to it.

pasharayan

5 hours ago

You'll find some paragraphs are half written by Ai and some written totally by me. It's heavily edited from its original form (as it was a data report by codex on our db) - but some paragraphs which were good enough stayed.

E.g. Contrast my human written paragraph vs the AI written first paragraph

Human written:

"The three most popular AI models used by college students are ChatGPT, Gemini, and Claude. As of August 2026 the data on StudyArena shows they prefer Gemini. "

vs

AI Written "Most AI comparsions are written like wine reviews. Claude is subtle. ChatGPT is dependable.."

s18m

7 hours ago

its starting to feel that way with newer models. right after opus 4.6, I have started to notice the same

s18m

7 hours ago

I have the same workflow for about 3 months now (constant). same workflow since opus 4.5 and now with fable (variable). start with a plan about a problem (somewhat similar), write down the plan, use the plan to execute the work. Lately the plans are so verbose and confusing, Im starting to think of skipping the plan stage entirely or tell it to be super concise.

rienbdj

7 hours ago

Students? The couldn’t even write the essay!

pasharayan

6 hours ago

Most students can write an essay - but most are also smart enough to try to save time too haha

levelZero

6 hours ago

Claude prose praising Gemini ;-)

pasharayan

6 hours ago

Codex - at least initially haha! I wrote lots of insights and findings and pushed out a lot of false logic it picked up. Personally I hate claude writing cause it feels bureaucratic. But definitely wrote a ton more with my bare hands and brain.

levelZero

4 hours ago

My bad. If you are interested, it was "Gemini's lead matters..." which tingled.

dzhiurgis

6 hours ago

Been using Gemini for about a year for code (much cheaper and actually sticks to your prompt so savings are many multiples of; I work with Salesforce implementations and Google is a user for Salesforce so I think their code might have went into training data, coupled with their code guidelines it's been a bliss for me) and for other uses for maybe 9 months.

I do go back to ChatGPT for things that need to be calculated, but _Gemini simply hallucinates less_. IMO that's whats important for students.

I did get suckered into extending my Grok subscription which is really fun for images/videos. Grok in car is a must too (my jaw dropped when I asked what is pantsula music and it added multiple albums into playlist for me).

adventured

7 hours ago

I use Fable for writing code and some planning. I use GPT for planning/roadmaps and code review, it checks Fable. I use Gemini for stray conversations. Gemini has a better conversational style than GPT, less robotic; it hallucinates more unfortunately, it has plainly fallen quite a bit behind, but it's still highly useful to converse with, research with, etc. And it spares me wasting usage of GPT or Fable. I use Gemini for the 'everything else' category, basically. Gemini never goes anywhere near anything serious.

Fable & Claude Opus 4.x or 5 are terrible to talk to about anything. I gave up on that entirely and just use Anthropic's models for work.

Any less serious technical work I'll use GPT for, as the usage limits are quite fantastic.

nickhodge

7 hours ago

Did they assess if any wanted to use their own brain?

pasharayan

6 hours ago

Interestingly, the smartest students I've worked with do use AI - but more in the sense that AI is a good comparison tool and a good bottom ceiling of the quality of work they should bring in.

Lots of research shows being able to see different answers improves the quality of a student's output - especially as the good ones are able to pick out what's good from the other work and then add their own insights or takes on top.

arcticfox2029

7 hours ago

I think that recently gemini has responded quickly but has not been effective, unable to support gpt.