It's been a while since I've seen an actual data science post submitted to Hacker News: both because AI has superset a lot of DS tasks (e.g. vector embeddings), but also because not much new has happened in DS. Polars has been around for a bit and as noted it is much better than pandas, but otherwise the DS ecosystem has been somewhat stagnant.

I'd write more tutorials about how to use data science tooling but one consequence of AI is that all the old data sources I used to analyze such as social media and Reddit are now completely locked down (I am surprised NYC Taxi is still being updated, though). Therefore in the meantime, I'm working on making better data science tooling...although unclear to what end due to the data issue above.

Cute title; I thought I was about to read a contrarian ecologist saying "a species that is so very specialized in its diet and finicky in its reproductive behavior doesn't deserve, evolutionarily speaking, to survive". (Seriously, it's seriously freakin' difficult to get pandas to reproduce in captivity). And while my knee-jerk reaction would be "maybe, but we should still preserve them because we can", I was prepared to see if the author had a serious argument to present.

Instead it's a cute bait-and-switch title, and the article tells you upfront that it's actually about the Python `pandas` library. Which I think I've encountered maybe once in my entire career (I'm not in the data-science field), so I don't have much meaningful to say about the article itself. I just want to commend the author on fooling me with the title. This is the kind of "clickbait" I can respect and actually wish there was a little bit more of sometimes. A nice chuckle, then a real article.

> People typically start with Excel and graduate to Pandas somewhere in the GB range. Pandas serves them well into the 10s of GBs range, and then they start hitting memory issues, slow computation, or become frustrated with Pandas’ baroque API.

Assumes that a project moves beyond 10s of GBs. I guess 99.9% of projects that import pandas fall well below this threshold.

People are usually surprised to hear that polars can be slower for fairly pedestrian operations, especially with smaller datasets. For example take a 5000 x 3 dataframe of float64 and sort by one column and you'll find polars takes about 2.5x as long. If you set POLARS_MAX_THREADS to 1 then it's faster. Though this all depends on the machine. Polars tends to shine with larger datasets or where it can heavily take advantage of query planning.

Don't skip your profiling

In my opinion a better argument to stop using pandas is the very unintuitive API pandas have. Additionally, a slight change in the query can force you to restructure the whole query (change all lines), while in Polars (and tidyverse in R) it's just a simple one-line change.

Am I crazy or did the OP swap the contents of the posts around accidentally?

https://eddie.codes/posts/pandas-should-go-extinct/ <=> https://eddie.codes/posts/source-code-comments/

I'm in the middle of wrapping up the edits for Effective Pandas 3rd Edition. (I also wrote a Polars book and just wrapped up a weeklong training session on pandas this week.)

Pandas is not perfect, it has a bunch of warts. But it is good enough for most. (And many of those folks are using Excel or tableau or power bi... These were the types I was training this week).

If you have medium data, migrating from pyarrow backed pandas to duck or Polars is trivial.

Only in the last few years did I start using SQL properly. Before that my pipelines would live in python. Now I offload as much to the db as possible, and keep my python simple glue. I'm very happy with this compared to other methods in pandas or polars.

If I still need to do db-like things in python I think duckdb is better.

Sup Eddie, the actual motivating example is we finally nixed Pandas from Data Ingestion, now reading excel files takes 2 seconds instead of 2 minutes. However unfortunately your

> TODO: rewrite this entire service

remains.

> If you were interested in whether cash usage declined during the Pandemic, it sure did. However, correlation !== causality, so please don’t draw any meaningful conclusions from this.

I think we can be certain that reduced cash use didn't cause the pandemic, and a common cause for both is also hard to imagine. That only leaves the pandemic causing a reduction in cash use. Am I missing something?

Claude loved pandas, at least around 4.5 opus. I had to keep reminding it to use polars instead. I wonder if it’s changed since.

> an Intel i5-1135G7 with 8 cores

The i5-1135G7 has only 4 cores. Hyperthreading is enabled, so it reports 8 "virtual processors".

Besides the performance benefits, I use Polars at work because it’s just (subjectively) nicer to work with. The “pl.col” API lets you create arbitrary generated/virtual columns anywhere you want, declaratively. You can throw in these column expressions in wherever without actually computing their values and storing that in memory. Very powerful stuff.

The reality is most of the pandas audience don't care about performance. Whenever performance is in the question people have always used polars, duckdb, dask etc. These users are usually software engineers and not data analysts. Agreed, that there is a real gap in latency and performance though!

Makes sense, especially with AI coding tools the rewrite and familiarity arguments hold less water. Similar for the rustify everything crazy.

The problem is, orgs who see themselves as big data orgs want to act that way, even if they're medium data. "But we'll need it when we grow", "we need to know the state of the art tools"

Nice post but they quickly disregard Dask, don't explain why, don't test it, even exclude it from the benchmark they quote. I don't know if is the better answer, but it seems worth testing if you want a balance approachable + scalable. That's kind the thing Dask was meant to do.

I've been saying this since using Databricks at a company almost a decade ago. Most folks do not need big data tools, and it's just so entrenched because everyone wanted to be a "big data" company and pandas was how you handled big data.

Real link here: https://eddie.codes/posts/source-code-comments/

Something going wonky on their blog, where two posts got their links swapped.

Liked the title

Nitpick for the author: it looks like you've got (a,b] when you actually mean [a,b). Either that or change the ≥ to be >.

I wonder what they taste like.

I would love to see a speed comparison with R incorporated into this.

You could still squeeze out some performance with cudf.pandas

Pandas should go extinct? I’m confused Edit: Oh link was broken before

I use polars or duckdb now exclusively. Better syntax, better performance. But pandas is deeply entrenched - I try to get my team off it but it’s an uphill battle. It’s not going anywhere anytime soon.

I've plugged this several times here (just a fan) but chdb's DataStore is a "lazy" drop-in replacement for pandas dataframe. Pandas API with chdb performance. I don't use it myself (I do everything mostly in sql with either duckdb or chdb) but always found pandas more convinient than polars for quick and dirty data crunching (less typing and frankly more pythonic if you're used to slicing).

Agreed on all counts.

In many cases I’ve found directly using python primitives to be less confusing than pandas.

Similarly, in companies I’ve worked at, the datasets just aren’t that big. Especially if you’ve got access to modern hardware.

Flagged - clickbait title

Data scientist here. I still use pandas. It is a work horse. But today, the progress in data science is much more in integrating sources and outputs. I find Grist (via API) and n8n much more applicable and process oriented than core DS pandas - although all of these components fit together somewhere. Most of the time, there is very little binding glue needed, e.g. a bash script or standard python file. DS is still fun, but it is much more process oriented today!

I'd drop pandas if polars worked eamlessly with sklearn.

import fireducks.pandas as pd

Title is currently, err...: Useful Code Comments

Hoping OP can fix this on their end so the url has the expected content. Whoops!

Polars seems nice but in my experience using it, the "lazy" APIs would still immediately materialize a ton of stuff in memory and had very spotty support on what data formats and storage integrations were possible with scan_* functions (though that was half a year ago and the support is slowly improving). It's frustrating, I mean really frustrating, to think I could solve a lot of my "scan through heinous amounts of data without any memory hungry things like window aggregations without blowing out my memory" with Polars and then watch my scan_thisorthat() call result in instant memory usage ballooning.

DuckDB on the other hand is wonderful and truly doesn't use any more memory than it really needs to.

The article is harsh for no reason. Pandas served us, for a lot of time and was the de-facto library for Data Scientists. It evolves. Polars was a rewrite following bets practices learnt while working with Pandas.

I do not understand the aggressiveness towards previous evolutionary steps. Who knows, may they come up with newer methods. I still use it. I also use DuckDB and Polars. DuckDB is more useful to me these days from either. Still Pandas has a place in my arsenal. Mostly because I know it better and I like it.

There is also libraries for doing type checking.

so this story is something that's really important for everybody to know about and should not get downvoted...

Wait, wait, wait... you're telling me that polars, something made in this decade, is better at modern problems than something made two decades ago?!

I'm shocked! Shocked, I say!

Sometimes you just want something that works with all the things. That's pandas. But, like the bamboo eaters, it wont be long...

Strongly disagree with the author.

For dumb simple select group by OLAP queries on medium data ? Sure clickhouse local or duckdb works perfectly well.

But if you need to construct dataset ? Or process existing dataset, do heavy filtering, transformation, reshaping, splitting? The proper ETL work, then pandas is really the perfect use case.

And pandas can work with small memory footprint as well, its actually trivial to do that, plus there are libraries like Modin that are upgrades over pandas with pandas api

Pandas is the swiss knife tool of data science that lets you do anything with the data and it integrates well with ML libraries