Data Engineer - Quality Assurance

Що робити

Automate data quality and reconciliation checks across varied storage layers, including Snowflake, SQL, and RDF/SPARQL databases
Test and verify data lineage, governance, and visualization components using Snowflake, data catalogs (ie. DataHub), Thoughtspot, and other visualization tools
Integrate test suites into the core infrastructure orchestrated by Apache Airflow and utilizing Iceberg table formats, while monitoring data pipeline health, alerting, and observability metrics using Prometheus and Grafana Cloud
Establish AI Evaluation Loops (Evals) and Guardrails: Build rigorous verification protocols— including structural tests, checks, and watchdog agents—to validate AI-generated artifacts, catch false positives, and ensure all automated outputs are secure, reliable, and free from hallucinations.
Integrate automated testing workflows into CI/CD pipelines using GitHub Actions, ensuring continuous stability and quality gates across all deployment environments

Data engineering & data testing: dbt, data lakehouse concepts, Medallion architecture
Databases & storage testing: SQL, Snowflake, AWS S3, Iceberg
Integrating quality check into data pipelines: Apache Airflow
API testing & automation: REST/OpenAPI, GraphQL
Integrating test automation into CI/CD: GitHub Actions (or similar like ArgoCD/GoCD)

Graph databases: RDF / SPARQL
Data governance & analytics tools: DataHub, Thoughtspot
AI/ML testing & MLOps: AI evals, guardrails, RAG, vector databases, AI drift monitoring
Advanced / emerging data tech: StarRocks, DuckDB
Regulated environments: GxP, 21 CFR Part 11, HIPAA

Знайдено через trackr.help/jobs · Канал: @trackrhelp · Бот для персональних сповіщень: @trackrhelpBot