Data Engineer

Data Engineer

Overview

We're looking for a Data Engineer to design, build, and operate scalable data architectures and pipelines that transform diverse structured and unstructured sources into high-quality data repositories. You'll develop robust ETL/ELT processes in AWS, create parsers and extraction logic for complex formats (e.g., PDFs, contracts, procurement documents, budgetary reports), and deliver curated datasets that serve as reliable sources for analytics and AI/ML applications.

Key Responsibilities

• Design cloud data architectures for ingestion, storage, transformation, and consumption (batch and, where needed, near real-time).
• Build and maintain ETL/ELT pipelines that are reliable, testable, observable, and cost efficient.
• Ingest and integrate data from diverse sources including APIs, relational databases, file drops, event streams, SaaS platforms, and external data providers.
• Work extensively with structured and unstructured data, including normalization, enrichment, and metadata management.
• Develop data parsers and extraction logic for complex unstructured sources such as PDFs, contracts, procurement documents, and budgetary reports; implement validation and error handling for imperfect inputs.
• Implement and optimize data storage patterns (e.g., lake/lakehouse/warehouse), indexing/partitioning strategies, and query performance tuning.
• Build and manage data repositories designed to support AI (feature-ready datasets, document corpora, embeddings-ready stores, retrieval-oriented schemas, lineage and provenance).
• Apply data quality practices (automated checks, anomaly detection, reconciliation, SLAs) and implement governance-friendly patterns (cataloging, RBAC, encryption).
• Partner with stakeholders (product, analytics, data science, engineering) to translate requirements into scalable datasets and interfaces.
• Create and maintain documentation: data models, interfaces, lineage, runbooks, and operational playbooks.

Required Qualifications

• Bachelor's Degree A Bachelor's degree in a quantitative or business field (e.g., Statistics, Mathematics, Engineering, Computer Science). (Required)
• 8+ years of experience in data engineering (or 3-5 years with demonstrable senior-level impact), building production-grade pipelines and data systems.
• Strong proficiency in SQL and at least one general-purpose language (Python strongly preferred).
• Proven experience designing data architectures (e.g., data lake/lakehouse/warehouse patterns) and selecting fit-for-purpose storage/compute.
• Hands-on experience with AWS data engineering, including several of the following:
  • S3, IAM, KMS, VPC, CloudWatch
  • Glue, Athena, EMR, Lambda, Step Functions
  • Redshift (or alternative warehouse)
  • Kinesis/MSK (streaming) and/or EventBridge (eventing)
• Practical understanding of data reliability practices: testing, CI/CD, monitoring/alerting, backfills, and cost/performance optimization.
• Strong communication skills-able to explain technical tradeoffs to both technical and non-technical audiences.

Preferred Qualifications

• Experience supporting AI/ML data products, such as building curated corpora, document stores, vector/embedding pipelines, and retrieval-optimized datasets.
• Familiarity with search and indexing concepts (e.g., OpenSearch/Elasticsearch) and/or graph/metadata systems.
• Exposure to Infrastructure-as-Code (Terraform/CDK/CloudFormation) and containerization (Docker/Kubernetes).

Certifications (Relevant / Preferred)

Candidates may have one or more of the following (or equivalent):

• AWS Certified Data Engineer - Associate
• AWS Certified Solutions Architect - Associate or Professional
• AWS Certified Developer - Associate
• AWS Certified Database - Specialty
• Databricks Certified Data Engineer (Associate/Professional)