Ask HN: Why Isn't Elixir More Popular?

52 pointsposted 14 hours ago
by wkyleg

Item id: 41792304

43 Comments

hakunin

8 hours ago

Here’s my anecdote.

I built 3 production projects in elixir around 2015-2018, and it was a blast to learn and work with. Those were interesting projects that were a great fit for the stack. One was March Madness bracket game which required a huge throughput on day one, and another was a football game audience play calling each play from their phone as the team is playing live at the stadium. This one needed a lot of timing coordination and poor connection handling. I even put out an open source fast/customizable leaderboard on top of ETS.

However, I didn’t feel the need for this stack on most projects, and if I’m totally honest, I never got good at the novel way of building applications in it.

While it was very enjoyable, everything felt a little awkward. Even 3 years in I constantly felt that I’m not doing it right, and I’m fighting the language to do data transformations in a purely functional style. I never got used to writing Ecto queries, and always had to look up their syntax. Plus, there didn’t seem to be a good architecture story, just isolated praise of OTP. And Phoenix further fueled the confusion, making it hard to understand whether I’m supposed to reason about my app like a Rails app (just build controllers, models, and views) with no regard for processes, or I should orchestrate some creative supervision trees, that I can’t even tell how they would be arranged in a typical web app. Going back to Ruby on Rails was kind of back to super productive comfort zone after a bit, and I just continued staying there.

I’m still looking back at Elixir with nostalgia, wondering if I’m going to have a chance to go back to it and really make it an extension of my arms/brain like Ruby had become. And whether I can do all that amazing supervision-based architecting I keep theorizing in my mind. I loved Dave Thomas’s vision (and agreed with his controversial takes) and really wished I could get as good as Sasa Juric at really deeply reasoning in this framework. Maybe one day.

josevalim

4 hours ago

> I’m still looking back at Elixir with nostalgia, wondering if I’m going to have a chance to go back to it and really make it an extension of my arms/brain like Ruby had become.

You should give it another try. :)

You jumped in quite early, when we were still collectively figuring out what it meant to build Phoenix applications, and many things were in flush back then. You probably went through Ecto 1 -> 2 -> 3 and Phoenix was migrating to contexts.

But I also have to say there was a lot of FOMO in relation to OTP back then: people felt they had to build these amazing supervision designs, otherwise they were not using the language correctly. But the truth is that they are building blocks for frameworks and certain libraries, in the same way threads/mutexes are for other languages. Of course, GenServers are higher-level, more expressive, and bring more properties, but the overall idea is that they are about infrastructure, and not design. Phoenix, Ecto, Broadway, etc should be spawning 99% of the processes that you need for you.

steve_adams_86

5 hours ago

I have to say this is a really good summary from my perspective too. Different exposure and projects, but the experience in terms of ergonomics and other developer experience was much the same.

I never felt truly proficient. I recall years ago interviewing for a role and straight up telling the interviewer I wasn’t very good with Elixir. That’s just how it made me feel. In retrospect I was fine, I built some interesting stuff. I probably could have promoted myself for that. But the language kind of left me feeling like I wasn’t getting it.

Even so, great fun to learn and apart from those aspects, I enjoyed building with it a lot.

ipnon

4 hours ago

I have this same feeling using Elixir, and I think it’s because it’s excellent at showing you exactly where you have undefined behavior in your application, much in the same way you spend most of your time writing Rust by wrangling with the compiler that keeps showing you all the holes in your logic. You can write a seemingly equivalent program in Python, and it runs just fine because it’s interpreted, so it seems like you hit the bullseye first try. But as we’re all aware, there are still countless bugs in your system, they just don’t surface until later when it’s Friday night and your website finally went viral. It’s the downside of the “fail fast, fail often” OTP paradigm: you develop the program by working through your faulty logic step by step as determined by the (many) errors raised by the BEAM. The upside being that unlike Python none of these errors are usually strong enough to be fatal to your application.

toldyouso2022

34 minutes ago

Used elixir a few years ago for tutorials and such, worked fine.

Tried to use it again in 2023, vscode extension would break. Tried on 2 windows machine and one linux machine, exetension always broke.

I then figured out why (I don't remember the exact reason, it was one of those "yeah I can see why it didn't work and it's my fault but it kinda isn't really" situations) but at that point I was done because I had better stuff to do.

Point being, if you want your stuff to be adopted make sure the tooling is noob friendly. The noobs of today can be the professionals of tomorrow.

Especially today when there is so much stuff to learn and elixir is so far from the usual and requires a certain time investment

yen223

12 hours ago

The simple reason is that Elixir is competing in the backend web-services space, which is a very crowded space with many equally-compelling alternatives.

As someone who likes working with off-meta languages, it's clear that the default state for any new language is to die an early death. The fact that Elixir hasn't died is testament to its strengths.

johnklos

13 hours ago

Got me. Running Akkoma on Elixir is a piece of cake, relatively speaking. Trying to run Mastodon on Ruby by simply using the latest version of the required Ruby components hasn't worked for me, ever.

The idea that we'd need to keep a snapshot of modules from a specific date in order to run certain software is ridiculous, and the idea of running it in a container like Docker is just giving in to bad practices. Too often it means things are too fragile to update when there are security issues.

I haven't seen this happen with Elixir. What do they do better / differently from Ruby that updating doesn't cause the house of cards to come crashing down?

wkyleg

13 hours ago

Yes! That's a good point

Docker is another technology that is sort of ridiculous from this perspective. Deploying an entire OS just to make dealing with dependencies easier

madamelic

11 hours ago

Since this thread is going to be bait to more experienced Elixir devs, can someone tell me books, blogs, etc that you like that discuss performance in Elixir?

I am working on an Elixir project that in no circumstance should be taxing but it's falling over miserably at like 100 - 200 events per minute. The detail is it is distributed-ish IoT. I didn't write it, the person(s) who did write it are gone, and no one else writes Elixir. I've gotten some good gains already but I'd like to squeeze the 80 - 90% of juice I think I can get before resorting to beefier hardware.

I've gotten into instrumenting and measuring it and I have some ideas but I'd love to hear others point me to other ideas. The real problem is that the hardware is miserably underpowered and it is real-time, by that I mean I can't defer, schedule for later, or de-prioritize anything.

---

To actually contribute, I really like Elixir. I am not yet sure why I would advocate for it over something more 'simple' like nodejs (My background is, accidentally, Javascript World) but it's certainly a very nice language to write in. It feels magical but not too magical where you get scared it's trapping you into its web.

Before anyone jumps too much on me for it, I gauge "simplicity" by how many people can I hire to write it. You can barely swing a cat without hitting 3 competent Javascript developers. I tried for many years to hire another golang dev so I could write it professionally, I only encountered a few despite having been in most interviews my employers would do. With that said, it may just be that the Venn diagram between "writes Javascript" and "writes golang" is small.

elcritch

7 hours ago

Interesting, Elixir should scale far more than that. Are you doing a lot of non-io processing or computations? I run Elixir on raspberry pi 4s doing IoT and they easily handle say generatings graphs with hundred of thousands data points.

One possibility is you're using a single process instead of parallelizing things. For example, you may want to use one process per event, etc. Though if the hardware is very underpowered and say single core, I could see it becoming problematic.

madamelic

6 hours ago

> Are you doing a lot of non-io processing or computations?

Unfortunately.

From metrics, computing AWS signatures takes up an absurdly large amount of CPU time. The actual processing of events is quite minimal and honestly well-architected, a lot of stuff is loaded into memory rather than read from disk. There's syncing that happens fairly frequently from the internet which refreshes the cache.

The big problem is each event computes a new signature to send back to the API. I do have to wonder if the AWS signature is 99% of the problem and once I take that burden off, the entire system will roar to life. That's what makes me so confused because I had heard Erlang / Elixir could do on the scale of significantly more per minute even with pretty puny hardware.

One thing I am working on is batching then I am considering dropping the AWS signatures in favor of short-lived tokens since either way, it's game over if someone gets onto the system anyway since they could exploit the privilege. The systems are air-gapped anyway so the risk is minimal in my opinion.

> One possibility is you're using a single process instead of parallelizing things. For example, you may want to use one process per event, etc.

This is done by pushing it to a task ie: `Task.Supervisor.async_nolink`? That's largely where I found my gains actually.

It took a dive into how things schedule, because a big issue that was happening was the queue would get massively backed up, and I realized that I needed to apparently toggle on a flag telling it to pack the scheduler more (`+scl true`). I also looked into the wake-up lengths of threads. I am starting to get my head around "dirty schedulers" but I am not entirely sure how to affect those or how I can besides it doing it forever me.

The other one just for posterity is that I believe events get unnecessarily queued because they don't / didn't have locks. So if event A gets queued then creates a timer to re-queue it in 5 minutes, event A (c|w)ould continue to get queued despite the fact the first event A hadn't been processed yet. So the queue would just continue to compound and starve itself.

elcritch

6 hours ago

> From metrics, computing AWS signatures takes up an absurdly large amount of CPU time. The actual processing of events is quite minimal and honestly well-architected, a lot of stuff is loaded into memory rather than read from disk. There's syncing that happens fairly frequently from the internet which refreshes the cache.

Oh, sounds nice! Caching in Elixir really is nice.

Okay, that makes sense. Elixir isn't fast at pure compute. It can actually be slower than Python or Ruby. However, the signatures likely are NIFs (native code). If the AWS signs are computed using NIFs then the CPUs are likely just can't keep up with them. Tokens would make sense in that scenario. But you should check the lib or code you're using for them.

> The big problem is each event computes a new signature to send back to the API. I do have to wonder if the AWS signature is 99% of the problem and once I take that burden off, the entire system will roar to life. That's what makes me so confused because I had heard Erlang / Elixir could do on the scale of significantly more per minute even with pretty puny hardware.

Yeah, crypto compute can be expensive especially on older / smaller cpus without builtin primitives. Usually I find Elixir performs better than equivalent NodeJS, Python, etc due to it's built in parallelism.

Also one thing to lookout for would be NIF C functions blocking the BEAM VM. The VM can now do "dirty nifs", but if they're not used and the code assumes the AWS signs will run fast, it could create knock on effects by blocking the Beam VM's schedulers. That's also not always easy to find with Beams built in tools.

On that note, make sure you've tried the `:observe` tooling. It's fantastic.

> One thing I am working on is batching then I am considering dropping the AWS signatures in favor of short-lived tokens since either way, it's game over if someone gets onto the system anyway since they could exploit the privilege. The systems are air-gapped anyway so the risk is minimal in my opinion.

Definitely, seems logical to me.

madamelic

6 hours ago

Thank you! You gave me a great term that I can jump off from (NIF).

I'll have to rig up Observer. I've been using recon because I was being lazy overall.

throwaway81523

10 hours ago

Disclaimer: I've written Erlang but not Elixir. From what I can tell the issues are similar though. I'd say look at Fred Hebert's Erlang blog (ferd.ca) and "Erlang in Anger" for some tuning hints. Erlang has been around a lot longer and its devs have gotten more into nitty gritty issues, it seems to me.

madamelic

10 hours ago

Thanks! That is a helpful tip in general I should remember: search for Erlang, not Elixir ;)

imjonse

6 hours ago

Probably unrelated to Elixir itself but caused by doing too much/redundant work on network/disk so it needs an algo change.

DoesntMatter22

6 hours ago

This fits my experience too. I've been able to crush elixir apps pretty easily, cowboy and phoenix.

I actually think Elixir really doesn't have great performance. TechEmpower, which is IMO the most real world standardized tests out there shows that Phoenix doesn't even complete. And Elixir+Plug+Ecto performs worse than Rails, which is an entire framework.

Everyone in Elixir land tells me "Oh those benchmarks don't matter". Yet they are heavily talked about, and referred to here and other places. They only don't matter if you perform terribly on them I suppose.

And they say "Oh we didn't care to put much effort into it", Yet Jose Valim himself tried to work on it and didn't fix it. He's written extensively about how this type of test doesn't really fit elixir, etc, but ultimately it's just doing DB queries, why does this not work?

I really think Elixir is mostly propaganda at this point. It's a huge mental paradigm shift and I have seen myself that it wasn't performant, and as you said you keep thinking "Oh I must be doing it wrong".

I just cannot fathom why anyone in a decent sized company would use with all the negatives it has going for it. YMMV

Proof below

https://www.techempower.com/benchmarks/#section=data-r22&hw=...

josevalim

4 hours ago

> And they say "Oh we didn't care to put much effort into it", Yet Jose Valim himself tried to work on it and didn't fix it.

This is inaccurate. I have started looking again into solutions only last week [1]. My suspicion was always the database pool size was too small but, when I tried to contribute 4+ years ago, fine tuning was hard because it took too long, so I didn't pursue it further [2].

My discontent with the benchmarks is that they are not measuring what people effectively run in production. Since you mentioned Rails, here is how a Rails application looks like:

https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...

But almost nobody runs a Rails application like that in production.

And then you look at the configuration of most stacks and they have been explicitly fine tuned to match exactly the concurrency traffic and patterns that the benchmark sends through. But, for most web applications, your web traffic is not homogeneous and you don't have a predetermined number of concurrent requests.

I still believe those benchmarks are not indicative of what you will actually see in production. Most companies who have gone from Rails to Phoenix, for example, report a 10-20x reduction in operation size and costs. But it is clear at this point people put way too much stock on these benchmarks. The irony of it all is that, if someone copies these setups into their actual applications, they will most likely perform worse. Oh well.

[1]: https://github.com/TechEmpower/FrameworkBenchmarks/pull/9302

[2]: https://github.com/TechEmpower/FrameworkBenchmarks/pull/5432 - here you can see me increasing the database pool to 40... but most benchmarks today run with 512-1024 connections (which, once again, is most likely not what you would do in prod). In any case, we need to bump our numbers accordingly.

pansa2

3 hours ago

Ultimately, Elixir is a functional programming language.

Has any functional language ever really become popular?

ipnon

4 hours ago

No manager is going to lose their job by hiring 10 Python and JavaScript engineers to accomplish what could be done with 1 Elixir engineer. And that’s how most time and effort is allocated in our tech labor market economy. It’s by working backwards from “I want to still have a job in tech next year.”

The number of people who are calculating which language to write in by thinking “how can I write a fault-tolerant distributed system with less time and energy so that I can quickly release performant products” is minuscule. The lack of popularity of Elixir is evidence of this I think.

In regards to your point on ZIRP: billions of dollars have been poured in to LLMs that are biased toward legacy languages like Python and JavaScript. Even the file structure of these languages is conducive to LLMs. A HTTP server can be and is frequently defined in a single file or function, where everything from socket creation to connection handling to request parsing to database queries to response can be composed in literally a few lines of code. This immense expressiveness is a testament to the power of HTTP. But as I’m sure we’re all aware there are limits to what can be accomplished with a single machine serving these stateless requests, and the limit can be reached very quickly. But LLMs gravitate towards producing these haiku-like incantations, it’s trivial for them.

Elixir’s power comes from having a well-defined API and explicit failure modes for each of these layers, each in their own expressive modules. This makes it difficult for LLMs to write Phoenix code when they’ve been optimized to output a 3-line FastAPI decorator-definition-query endpoint. Each of these layers in Phoenix is by itself quite simple, each layer has about 2-4 required functions to be implemented, and Phoenix can generate all of these for you. But ChatGPT doesn’t seem to be able to grok it all at once the way a good engineer can after readings the docs for an afternoon.

Will Elixir survive an era of programming where RTFM is a lost art? I suppose we will find out soon enough!

goralph

9 hours ago

In a commercial setting (i.e. not a side-project) the choice of programming language is also a business decision.

The main factors being:

- How large is the pool of available candidates for this language? A recruiting risk.

- How mature is this language? A business continuity risk.

viccis

5 hours ago

>How large is the pool of available candidates for this language? A recruiting risk.

I worked at a company with a massive Erlang codebase. Really nasty, not really following good OTP practices, etc. HUGE hiring problem and it took 6-12 months to spin up new devs to the point that they could really have ownership over things. And this is not a system that even remotely needed this.

Elixir might seem better until you write enough of it to realize that you do basically have to learn Erlang or else you'll always be at a disadvantage when it comes to really understanding it.

josevalim

4 hours ago

You don't have to learn the Erlang language (and this is commonly echoed by the community). At a certain scale, you have to learn the runtime, but this is the same for any technology. You have to learn JVM if you are doing Kotlin, there are plenty of discussions on V8 internals if you are running Node, etc.

ram_rar

4 hours ago

A limited job market, considerably smaller ecosystem (compared to java/python), and lack of support from major tech corporations (unlike Go or Rust) makes it challenging to adopt Elixir.

At my company , we inherited a legacy Elixir service, but we're actively moving away from it. We found it more practical to rewrite the service in Go (entire team is proficient in + LLMs make it easier to migrate) rather than invest time and resources into learning Elixir and its associated ecosystem. The return on investment for mastering Elixir didn't justify the effort required, given our specific circumstances and needs.

I do like to tinker with Elixir for hobby projects. But supporting a whole new language in a company is an uphill battle, that's incredibly hard to justify.

maz1b

6 hours ago

From what i've read, it is the "upgrade" to rails for people who want higher scale. I don't know if I agree, as Rails definitely does scale if you know what you're doing. Would love to hear more about others experience using Elixir for fast, low-latency high throughput scale.

jatins

3 hours ago

I can't say about mass popularity but for me personally the lack of static types has held me back from using it. Now that it's being gradually introduced I am looking forward to trying it

sergiotapia

13 hours ago

It's a wonderful language. Unfortunately the only places I've been able to use it and get paid were places where I ultimately had the decision making ability of the tech stack.

Why isn't it used? It's niche and betting on such a small community is risky for the majority of companies. Why use Elixir when you could hire 10 engineers to pump out javascript. That's the mentality of most.

Hiring for elixir was great, it self selected people who wrote code as their craft. You kind of have to be curious about code to even be aware of Elixir, know how to write elixir. These types of devs would pick it up really quickly because the language is just so damn ergonomic.

I'm using Elixir now, and I wake up so happy that I get paid to do this. I am really blessed.

sbaiabBzic

10 hours ago

> Hiring for elixir was great, it self selected people who wrote code as their craft

I worked at a company that hired like this. On the whole it was good, but it wasn’t a panacea. A surprising number ended up being academic types that were extremely smart but could never actually finish anything. Amazing guys to talk to at lunch though!

hyperbrainer

5 hours ago

The stereotype of Haskell programmers seems to extend to the entirety of FP, then.

ipnon

an hour ago

When you want to write a monad the first step is usually to take a coffee break to make sure everyone agrees on what a monad is.

wkyleg

13 hours ago

It's funny I think I remember explicitly reading this argument somewhere, that startups need to meet headcount as a metric for VC funding so they will end up picking "worse" tech for that reason.

Terr_

10 hours ago

I don't follow, wouldn't it be faster/cheaper to meet an arbitrary headcount goal using a more-common language or stack?

julienmarie

9 hours ago

I think @wkyleg meant that an Elixir dev can be as productive as multiple js devs because of the specificity of the language and the beam virtual machine. Complicated things are trivial in Elixir, dependency management is a breather, and all in all, a really simple and productive environment as the mental model is pretty simple once it clicks.

sgarland

11 hours ago

> it self selected people who wrote code as their craft

I’ve often thought that would be true for niche languages. I interviewed at a fintech whose backend is written in Haskell, and definitely got that vibe from the interviewers.

7bit

4 hours ago

I'm Not a Professional dev, but a sysadmin WHO Likes to code. I learned a bunch of languages over the year. C#, Java, Python, JS/TS, PowerShell, Rust. Not professionally, but enough to finish some Personal projects.

Elixir was so vastly different, had such a small community that I lost interest very, very fast. It just felt like a waste of time. Any of the other languages I learned I could and did apply at my job. Even just knowing the languages helps a great deal when evaluating the next tech stack to implement. And Elixir just doesn't. It's a niche language for niche applications and doesn't personally help me in the least.

I also just did not like the DX. I did fail to understand how to use the VS Code debugger to step through functions. Printing to console to inspect variables felt like being in the stone age again, no thank you.

I still believe Phoenix LiveView is a really, really good tech and currently wish that Svelte could implement something like that.

And that's my personal reason for not looking into it.

andrewstuart

3 hours ago

Straight up - you should not use it without much more compelling reasons than those you have outlined.

It's not OK as CTO to push your company to use for favorite pet slightly obscure technology.

The company needs ordinary technologies to get its job done - Java, C#, TypeScript, Golang, Python, something like that. Maybe even Ruby but frankly that causes hiring headaches too.

Unless the company is doing something so extraordinary that it NEEDS Elixir over the technologies above, and it probably doesn't need it.

I know companies that went all in on Elixir - they spend all their time trying to recruit Elixir developers from a tiny recruiting pool, or trying to find people willing to learn it and bring them up to speed. This is poor business practice - there was nothing they were doing that could not be done just as effectively with any of the other major languages.

Just use normal technologies, have all the people and documentation and training and other resources available and get on with the job of building the software that the business needs.

I guarantee you that the company that went with Elixir due to the enthusiasm of some current or past CTO will have to go through the painful process of getting rid of it.

Elixir is for personal projects. Corporate software development is mature enough now that ordinary technologies are extraordinarily powerful and there's almost no reason sufficiently compelling to not use them. Toys are for home, tools are for work.

rsrsrs86

9 hours ago

I regret using it. Recruiting risk. Smaller ecosystem. I should have gone for Python.

elixir_newbie

4 hours ago

Will you consider hiring me (remote, non-US) if I pickup the basics in 15 days? I'm open to technical rounds that don't assume elixir familiarity.

throwawaymaths

5 hours ago

Hired a python dev at an elixir gig because I was forced to by the CTO. Picked by the CTO "because we need someone that can do python". Dev was absolute trash. Never hired a shit elixir dev. Python is a huge recruiting risk.

ddyevf635372

7 hours ago

Because it is not strictly typed language.

KingMob

6 hours ago

You probably meant "static" or "strong", not "strictly".

Regardless, both are still not right, since Elixir is getting a gradually-typed set-theoretic type system. It's been in the works for a few years now.

kylecazar

7 hours ago

a lot of extremely popular languages aren't.