Ask HN: How can I experiment with LLMs with a old machine?

4 pointsposted 13 hours ago
by hedgehog0

Item id: 41611156

5 Comments

roosgit

10 hours ago

I've never used it, but I think Google Colab has a free plan.

As another option, you can rent a machine with a decent GPU on vast.ai. An Nvidia 3090 can be rented for about $0.20/hr.

tarun_anand

12 hours ago

I am on a 2019 mac and finding it difficult too.

Best bet would be to start with a small language model?

hedgehog0

12 hours ago

May I ask how small? From what I see on r/localllama, it seems that at least 3B and 7B ones are starting to be useful? Also I cannot use CUDA or modern AMD cards..

LargoLasskhyfv

12 hours ago

Make yourself comfortable with

https://blogs.oracle.com/database/post/freedom-to-build-anno...

https://gist.github.com/rssnyder/51e3cfedd730e7dd5f4a816143b...

https://www.reddit.com/r/oraclecloud/

or any other offer.

Deploy some minimal Linux on them, or use what's offered.

Plus optionally, if you don't want to instantly start coding from first principles/scratch, make use of established and excellent solutions, like

https://future.mozilla.org/builders/news_insights/introducin...

https://ai-guide.future.mozilla.org/content/running-llms-loc...

https://github.com/mozilla-Ocho/llamafile

https://justine.lol/matmul/

and parallelize them with

https://github.com/b4rtaz/distributed-llama

Obviously this needs some knowledge of the command line, so get a good terminal emulator like

https://iterm2.com/

Mend, bend, rend that stuff and see what works how and why, and what not.

Edit: Optionally, if you really want to go low-level, with some debugger like

https://justine.lol/blinkenlights/

for 'toy-installations' of smallest models.

'Toy' because that doesn't fully support the CPU-instructions which are used in production.

Could still help conceptually.

hedgehog0

9 hours ago

Thank you for the Mozilla AI guide, it's really useful! I knew some of Justine's work and they are really cool.