Show HN: I Scraped 2,200 Software Engineering Jobs from Career Pages Using LLMs

8 pointsposted a year ago
by kylem866

15 Comments

toomuchtodo

a year ago

You should connect with the person building https://hiring.cafe. They are scraping something like 1.6M jobs using ChatGPT, might be some collaboration opportunity or knowledge transfer.

https://news.ycombinator.com/item?id=42806956

Worst case, proven pattern to emulate. Wishing you success!

kylem866

a year ago

Thanks! I've actually already sent a message to the hiring cafe creator and didn't hear back. Might be worth another shot

spicy_ranch

a year ago

I really enjoy the simple, elegant design and look of this site. Well done!

I did notice that the mid-level jobs are returning mainly senior roles though.

kylem866

a year ago

Thanks! Yeah I have noticed accuracy problems with the seniority too. I'm using 4o-mini + structured output to extract the seniority. Currently the seniority output is defined as an array to handle edge cases where a job could technically be either mid level or senior. But, in reality the LLM is over eager at assigning multiple seniorities. It frequently gives a mid level seniority to jobs which literally have 'Senior' in the title. I'll work on it!

wbakst

a year ago

cool stuff! I wish there were a fuzzy search / filter bar to make it easier to search for more specific things.

I'm also curious, what are you using to structure the outputs?

kylem866

a year ago

Thank you! What more specific things would you like to search for?

I'm using 4o-mini + structured output mode

wbakst

a year ago

mostly just in the UI like a free form fuzzy search so I could look for more specific things rather than the drop down select

kylem866

a year ago

Right, that can definitely be done. I was just wondering what specific things you're hoping to find with a fuzzy search so I can make sure it's implemented well

wbakst

a year ago

oh things like languages, tech stack, more specifics around role, etc.

kylem866

a year ago

Update: just added support for tech stack filtering. Let me know what you think!

wbakst

a year ago

seems to work!

still want an open-ended fuzzy search bar though

kylem866

a year ago

On it!

wbakst

a year ago

yay! excited to play around with it