Senior Machine Learning / Data Engineer - Evaluation
- 10 hours ago
- 3 min read
As a Senior Software Engineer – Evaluation, you will design and implement systems that measure and monitor the performance of our computer vision, automatic speech recognition (ASR), and small language model (SLM) systems. You will develop evaluation methodologies, benchmarking pipelines, and monitoring tools that ensure our AI systems perform reliably in real-world environments. You will help establish the evaluation standards and performance benchmarks that guide the development of all AI systems at the company.
You will work closely with machine learning and data engineering teams to evaluate model performance, identify failure modes, and guide improvements to data collection and model training, helping bring AI-powered aviation tools to market.
This role is ideal for someone who enjoys designing robust evaluation systems and uncovering insights from model performance data, and wants to have a direct impact on the quality and reliability of our AI systems.
LOCATION
Work arrangement: On-site
Primary location: Seattle, WA or Bozeman, MT
About VTI Aerospace
VTI Aerospace builds AI-powered perception and pilot assist technologies to unlock the future of aviation. With offices in Bozeman, MT and Seattle, WA, we are a team of engineers and technologists from Boeing, Airbus, Aurora, and beyond. We are passionate about pushing the boundaries of autonomy and aviation safety. Our company is dedicated to advancing the field of aviation with cutting-edge solutions
What You'll Do
Define and implement evaluation methodologies for computer vision, ASR, and language systems
Identify and track key performance indicators (KPIs) that measure system and model effectiveness
Perform dataset coverage analysis to understand strengths, gaps, and biases in training and evaluation data
Identify model deficiencies and collaborate with ML engineers to improve training data and model performance
Build scalable model evaluation pipelines in Python for automated benchmarking and regression testing
Design and maintain systems for monitoring model performance and drift in production environments
Architect data models and storage systems for evaluation results using relational and time-series databases
Build dashboards and reporting tools to visualize model performance and evaluation metrics
Collaborate with ML and data engineering teams to improve evaluation workflows and data pipelines
Contribute to ground-truth data pipelines and processes that support reliable evaluation
What We're Looking For
5+ years of experience developing production software systems in Python
Strong experience designing and implementing data processing or analysis pipelines
Experience building systems for machine learning evaluation, experimentation, or benchmarking
Experience working with large datasets and building scalable data workflows
Familiarity with statistical methods, experimental design, and model performance analysis
Experience building automated pipelines using tools such as Airflow or similar orchestration frameworks
Experience working with evaluation or experiment tracking tools such as MLflow or similar systems
Experience designing data models and working with relational databases and time-series data
Strong collaboration skills and ability to work closely with machine learning and data engineering teams
Bonus Points
Experience working with computer vision, ASR, or language models in production environments
Experience using PyTorch or other deep learning frameworks
Experience evaluating multimodal or large-scale AI systems
Experience building monitoring systems for ML models in production
Experience designing dataset analysis or data quality tooling
Experience building dashboards or monitoring tools using Grafana or similar platforms
Why You'll Love Working Here
Opportunity to work on meaningful problems in aviation and deliver products to industry at a rapid pace
Small team with significant impact on company direction
Collaborative and innovative environment
Competitive compensation and benefits
Opportunities for growth and leadership
VTI offers a fast-paced and dynamic work environment that focuses on technical excellence and collaboration. Additionally, we foster a culture of work-life balance to prevent burn-out.
Compensation packages are comprised of salary and incentive stock options. Salaries may range from $80,000 to $175,000 per year depending on role and individuals experience. VTI additionally offers 401K with a 4% match, an unlimited PTO policy, and health care reimbursement up to the IRS allowable limit through QSEHRA.
VTI is an equal opportunity employer and values diversity. We do not discriminate on the basis of race, religion, color, national origin, sex, gender, gender expression, sexual orientation, age, marital status, veteran status or disability status. All job offers are contingent upon the candidate passing background, and reference checks.
At this time, we are unable to provide visa sponsorship or consider candidates who require visa transfers. Applicants must be authorized to work in the United States without the need for visa sponsorship now or in the future.