For an AI platform project in the construction and infrastructure consulting space (1M+ documents per client).
Brief requirements: • Production experience with large-scale batch pipelines (millions of documents / multi-TB): Spark, Ray, AWS Batch/Glue, Kubernetes • Search and vector databases at scale: OpenSearch/Elasticsearch, pgvector, Milvus, Qdrant, hybrid search, index design • Document parsing and OCR for scans and complex files: Docling, Unstructured, PaddleOCR, Textract, Azure Document Intelligence • RAG and retrieval evaluation: chunking, embeddings, rerankers, golden sets • Python, AWS (Azure/GCP also fine), cost control of GPU and LLM tokens
