Senior Engineer (Python)

What You’ll Do

We are seeking a Software Engineer with expertise in web scraping, data processing, and search technologies to help build a large-scale data ingestion and classification system. You will be responsible for extracting data from diverse sources (web pages, APIs, PDFs), cleaning and normalizing it, and building search capabilities using ElasticSearch/OpenSearch. You will work with Python, Scrapy, Airflow, Kubernetes, AWS, and Spark to create scalable, high-performance data pipelines.

• Build and design large scale, distributed crawling bots (perhaps AI agents) and infrastructure that operate in an adversarial environment aiming at low operational overhead
• Develop and maintain data pipelines to extract data from large volumes of web pages, documents, PDFs (OCR), and APIs.
• Help unify heterogeneous documents into a coherent data schema across varied source formats
• Preprocess and normalize raw data for downstream classification, ML/NLP, and search indexing.
• Build APIs to expose structured, classified data