r/MachineLearningJobs • u/Greedy_Extent4028 • 2h ago
I built an AI/ML job scraper that tracks 150+ companies — here's what I learned
Hey everyone,
I got tired of checking 50+ career pages daily to find AI/ML job openings, so I built a scraper that tracks 150+ companies in one run.
The problem:
Every AI company posts jobs on their own career page. Some use Greenhouse, some use Lever, some have custom boards. If you're job hunting or doing market research, you're checking 50+ tabs daily.
What I built:
- Scrapes Greenhouse, Lever, RemoteOK, WeWorkRemotely, and YC in parallel
- Extracts skills, frameworks, salary, experience level, remote status
- Change detection tracks new/removed/updated jobs between runs
- Webhook alerts for Slack/Discord when matching jobs appear
- Optional AI enrichment via Groq (free tier)
Tech stack:
- TypeScript, runs on Apify platform
- Greenhouse/Lever APIs (no HTML scraping)
- Keyword-based classifier handles 60-80% of jobs without LLM
- State persisted in KV store for change detection
What I found interesting:
- 62% of AI jobs are remote
- Median salary: $185k
- PyTorch mentioned 2x more than TensorFlow
- Top hirers: OpenAI (45), Anthropic (32), Google (28)
Lessons learned:
Greenhouse and Lever have public APIs — most companies expose jobs via JSON endpoints. No scraping HTML needed.
Keyword classification is underrated — a simple keyword matcher handles most jobs. LLMs are only needed for ambiguous cases.
Change detection is the killer feature — users don't want to browse 1,000 jobs. They want to know "what's new since yesterday."
Pricing matters — at $1 per 1,000 jobs, the tool is accessible to individual job seekers, not just enterprises.
What's next:
- Adding more sources (Indeed, LinkedIn jobs)
- Better salary normalization
- Company analytics dashboard
Would love feedback on what other sources or features would be useful.
Live on Apify Store: https://apify.com/lissome_dancer/ai-ml-jobs-scraper