PSA: I spent 3 days trying to get a local AI model to run on my old laptop

I mean, everyone online says you can run these small language models on any hardware now, right? So I tried to set up Llama 2 7B on my 5 year old laptop with 8GB of RAM. The guides made it sound simple, just a few commands. But I kept hitting this memory error that would crash everything. I tried different versions of the model file, messed with Python settings, and even tried a different program to run it. I must have followed six different forum posts and watched a bunch of videos. It just would not work. I finally got it running after I found a super specific setting to limit how much memory it could use at once. The whole thing took me like 20 hours over three days. Has anyone else had this much trouble with a simple local setup?

3 comments

3 Comments

ward.jamie1mo ago

My 2018 MacBook Air with 8GB RAM had the exact same memory crash issue.

cole3561mo ago

Yeah the "memory crash issue" you mentioned is super common on those older 8GB Airs. I saw a whole thread about it where people said macOS just eats up all the RAM with basic tasks. It got so bad for some that the system would freeze up trying to open a few browser tabs. Apple really dropped the ball selling those with only 8GB for so long. Not surprised your 2018 model had the same problems.

the_alice1mo ago

Honestly I think people blow this out of proportion. Had a 2017 Air with 8GB for years and it ran fine for me. Sure you couldn't have 30 tabs open and edit video at the same time but basic browsing, emails, Spotify... never had it freeze up once. Sounds like some folks just loaded theirs down with background apps and blamed the RAM.