🚀

Proof of Skill

Subject Matter Expert - Web Scraping & Crawling (Web Data Platform)

📍 Hyderabad, Telangana, India

AT A GLANCE

Subject Matter Expert to own and extend a production web-crawling platform with LLM verification for healthcare data compliance

LEVEL

senior · 8+ yrs

KEY SKILLS

Web scrapingWeb crawlingPythonPlaywrightScrapyasynciohttpxBeautifulSoupSeleniumAWSDockerKubernetesSQLPydantic

WHAT YOU'LL DO

Replace scraped SERPs with paid search API behind existing search_domain() interface
Migrate fetch.py to async httpx with per-domain concurrency limits and politeness budgets
Implement proxy rotation and egress management to eliminate single-IP failure mode
Add typed outcomes, structured logs, metrics, drift and volume alerts for failure visibility
Reverse-engineer payer XHR/JSON endpoints to replace Playwright recipes
Honor robots.txt Disallow and Crawl-delay; build per-domain ToS and licensing register
Containerize and schedule pipeline; add CI with offline tests on every change
Add HAR replay and golden-file parser tests on real payer HTML with daily canary crawls

THE ROLE

Subject Matter Expert - Web Scraping & Crawling (Web Data Platform) at Proof of Skill. Hyderabad, Telangana, India. Sourced from the fund's portfolio job board — apply on the company's site.

THIS ROLE IS FOR

📊

Data Whisperer

Not sure if that's you? Find your archetype free →

APPLY FOR THIS ROLE

Apply on Proof of Skill →

APPLICATION HAPPENS ON PROOF OF SKILL’S SITE

Want work you can prove — not just apply to?

get your archetype + join the pool →

COMPANY SNAPSHOT

LocationHyderabad, Telangana, India

RemoteIn-office

See similar roles →

🚀

Proof of Skill