Lead Data Engineer
Bethesda, MD
On-siteLead & above
Job details
- Bethesda, MD
- On-site
- Full-time · Lead & above
- Platform & DevOps
- Posted 1 month ago
Sponsorship history
Past filings by Black Canyon Consulting in public government data. Not a promise of sponsorship.
- Filed 6 H-1B LCAs in FY2025
Full job description
Overview
We are seeking a Lead Data Engineer to design and implement the core analytical data platform for a federal permitting modernization initiative.
The Lead Data Engineer will establish a modern lakehouse environment using DuckDB, Apache Iceberg, Amazon S3, and AWS Glue Catalog and will lead the engineering patterns used to ingest, harmonize, and serve data across multiple source systems. The role will also help implement metadata-driven and agentic approaches to data engineering, including source analysis, mapping, pipeline generation, testing, reconciliation, and quality assurance.
This is a hands-on technical leadership role with responsibility for guiding and reviewing the work of other Data Engineers.
This position is remote.
Duties & Responsibilities
- Design and implement the core DuckDB / Apache Iceberg data platform on AWS.
- Define physical data organization, schema, partitioning, file layout, snapshot, compaction, and optimization strategies for Iceberg tables.
- Integrate DuckDB with Amazon S3, Apache Iceberg, and AWS Glue Catalog.
- Establish bronze, silver, and gold data-layer conventions and reusable engineering patterns.
- Lead and mentor Data Engineers, providing technical direction, code reviews, and engineering guidance.
- Build and maintain analytical datasets supporting application services, reporting, analytics, and data science.
- Develop reusable patterns for incremental loads, CDC, schema evolution, late-arriving data, backfills, and historical reconstruction.
- Work with the Data Architect to implement the common canonical data model and source-to-canonical mappings.
- Lead migration and harmonization of datasets from multiple source systems.
- Implement reconciliation processes that demonstrate completeness and accuracy between source and target systems.
- Establish automated data-quality monitoring, operational observability, and testing.
- Implement agentic and AI-enabled data engineering patterns supporting activities such as source analysis, mapping, pipeline generation, troubleshooting, reconciliation, and quality assurance.
- Support integration of structured data with associated document and geospatial datasets.
- Lead development of certified data products, analytical datasets, and business metrics.
- Optimize DuckDB queries and Iceberg datasets for performance and efficiency.
- Develop engineering documentation, operational procedures, and technical handoff materials.
