Data Engineer

We are looking for a Data Engineer who specializes in managing and optimizing data pipelines, with a specific focus on AI/ML data and applications. The ideal candidate will have experience in designing and implementing data infrastructure, ensuring high-quality labeled datasets, and collaborating with cross-functional teams to support our computer vision initiatives.

Responsibilities

• Manage data releases, tracking data provenance, evaluating data quality, providing data insight reporting.
• Contribute to scalable data and labeling pipelines for computer vision projects.
• Manage and optimize data storage solutions to ensure efficient data retrieval and processing.
• Monitor and troubleshoot data pipeline issues, ensuring timely resolution and minimal disruption to project timelines.
• Collaborate with machine learning engineers and other data/infrastructure engineers to understand data requirements and ensure the availability of high-quality labeled datasets.
• Stretch: Develop and maintain tools and processes for data labeling, including annotation workflows, quality control, and validation procedure.

Requirements

• Bachelor’s in Computer Science, Data Engineering, and 2+ years of experience in a related field.
• Strong Python and SQL; Good software development fundamentals a must.
• Experience as a Data Engineer, working with data pipelines.
• Proficiency with:
  • Data modeling
  • ETL processes and data pipeline operations
  • OLTP databases (e.g. Postgresql)
  • OLAP databases (e.g. ClickHouse)
• Excellent problem-solving skills and attention to detail.
• Strong communication skills and the ability to work effectively in a collaborative team environment.

Strong Plus

• Experience with evaluation of ML models.
• Experience developing evaluation frameworks.

Nice to Have

• Data pipelines for computer vision applications.
• Data labeling.
• Workflow orchestration tools, particularly Dagster.
• Distributed computing frameworks (Apache Spark, Dask).
• Containerization technologies (e.g. Docker).
• Job scheduling systems (e.g. Slurm).