Senior Data Engineer

Що робити

AI-Driven Data Scoring: Design and implement pipelines that utilize LLMs to analyze and score identity data in real-time. You will integrate AI models directly into the decision-making loop, balancing accuracy with latency and cost.
Own the Data Architecture: Architect scalable data solutions using GCP and Python. You will manage data storage and retrieval using BigQuery and Apache Iceberg to support querying of TBs of data.
Heavy Data Processing: Utilize Apache Spark for data transformations and batch processing when lightweight cloud functions are not enough.
Manage High-Load Orchestration: Maintain and optimize our system instances. It involves complex dataflows, custom Python nodes, and performance tuning for 10k+ execs/minute.
Release Lifecycle (CI/CD): Take ownership of the deployment process, ensuring that updates to pipelines and infrastructure are released safely with proper testing and rollback strategies.

4-5+ years of experience in Data Engineering or Backend Engineering with a strong data focus.
Production AI Integration: Experience integrating LLMs (OpenAI, Anthropic, Gemini) into production applications via API. You understand latency, token limits, and how to structure data for AI scoring.
Expertise in GCP: Understanding of Google Cloud Platform (Cloud Functions, IAM, Networking).
Strong Python: You write clean, efficient, and testable code. You are comfortable building custom logic where standard tools fall short.
Big Data Stack: Experience with BigQuery, Apache Spark, and modern table formats like Apache Iceberg.