Interested in this AI/ML Engineer role at innoVet Health, LLC?
Apply Now →Skills & Technologies
About This Role
Job Summary
InnoVet Health is seeking an AI Architect to lead technical architecture and solution design for national AI initiatives across federal healthcare, with a primary focus on the Department of Veterans Affairs. This is a full‑performance‑level role: candidates must arrive with the architectural judgment, security fluency, and self‑sufficiency to make defensible design decisions inside federal environments from day one.
You will own the technical architecture for prioritized AI use cases, taking them from concept through operational pilot in authorized government cloud environments. Work includes designing secure and compliant solution patterns, defining evaluation and monitoring strategy, establishing model lifecycle management, and working hands‑on with engineering, infrastructure, cloud, and security partners to validate that solutions are deployable, performant, and scalable. The role is approximately half hands‑on technical work and half architecture, documentation, and stakeholder coordination.
This role offers remote flexibility, competitive benefits, and the opportunity to shape the technical foundation for responsible AI in federal healthcare.
ResponsibilitiesArchitecture \& Solution Design
- Lead AI technical architecture and solution design for prioritized use cases, covering model and configuration approach, data flows, integration patterns, environment and infrastructure dependencies, and technical readiness for alpha testing.
- Design solutions that align with VA data architecture, cloud, security, privacy, interoperability, and enterprise technology standards, including deployment across Azure Government, AWS GovCloud, VAEC, and other approved VA hosting environments.
- Select platform and serving patterns appropriate to each use case, with clear rationale for tradeoffs across latency, cost, scalability, and operational complexity.
Security, Compliance \& Data Protection
- Design for control inheritance and minimize authorization burden, working with ISSOs, Privacy Officers, and cloud teams to keep solutions inside existing authorized boundaries.
- Ensure solutions meet NIST 800‑53, FedRAMP, and VA privacy and PHI handling requirements, and support security authorization (ATO) activities.
- Apply secure architecture patterns for PHI workloads, including network isolation, private endpoints, managed identities and service principals, and secrets management.
Evaluation, Monitoring \& Model Lifecycle
- Define the evaluation strategy for each use case, including latency and throughput targets, task‑appropriate accuracy metrics, and operating point and threshold selection under real‑world prevalence.
- Design structured evaluations for LLM‑based and agentic approaches, including groundedness, hallucination rate, robustness, and shadow‑mode validation prior to user exposure.
- Define bias and subgroup performance testing, explainability needs, and human‑in‑the‑loop safeguards proportionate to use case risk.
- Define model lifecycle management patterns including versioning, retraining triggers, drift monitoring, rollback, and model registry integration.
Deployment \& Hands‑On Implementation
- Work hands‑on with AI engineers, data scientists, data engineers, IT infrastructure partners, cloud teams, and security stakeholders on solution installation, configuration, and initial validation.
- Confirm compatibility, performance, scalability, and deployment feasibility across approved VA hosting environments.
- Provide architecture support for pilot planning and execution, including technical success criteria, monitoring requirements, pilot data collection approach, and deployment approach needed to move solutions from controlled testing into operational pilots.
Stakeholder Engagement \& Workflow Integration
- Partner with Product Owners, field users, value management leads, and scaling partners to assess whether AI solutions are technically viable for broader adoption.
- Ensure AI solutions integrate into existing clinical and operational workflows, minimizing burden and maximizing adoption.
- Support handoff to permanent solution owners or enterprise scaling teams, including documentation of architecture decisions and lessons learned.
Deliverables \& Federal Contract Execution
- Develop and maintain VA‑specific architecture artifacts, including current‑state and future‑state workflow models, system context diagrams, data flow diagrams, integration and solution architecture views, deployment diagrams, and pilot‑readiness documentation.
- Prepare formal federal deliverables including architecture decision records, technical memos, and pilot‑readiness documentation suitable for audit, external review, and transition into federal environments.
- Ensure user‑facing components of AI solutions meet Section 508 accessibility requirements.
QualificationsRequired
- Bachelor’s degree in Computer Science, Engineering, Data Science, or a related technical field; Master’s preferred.
- 5\+ years of hands‑on experience in solution architecture, ML engineering, or applied AI, including 3\+ years delivering systems in federal or otherwise regulated environments.
- Demonstrated experience architecting and deploying AI/ML systems in secure government cloud environments (Azure Government, AWS GovCloud, VAEC), including working knowledge of service and model availability gaps relative to commercial regions.
- Hands‑on Databricks platform architecture experience, including workspace and cluster design, cluster policies, Delta, job orchestration, and model serving patterns.
- Proficiency in Python, SQL, and PySpark.
- Experience with production AI/ML deployment patterns, including containerized deployment, CI/CD pipelines, real‑time endpoint versus batch serving tradeoffs, and model registry integration (MLflow or equivalent).
- Demonstrated experience designing evaluation and monitoring approaches for ML and LLM systems, including metric selection, threshold and operating point analysis, and drift detection.
- Working knowledge of federal security authorization, including the ATO process, NIST 800‑53, FedRAMP, and control inheritance.
- Ability to produce architecture documentation including system context, data flow, integration, and deployment diagrams using Visio, Lucidchart, or Draw.io.
- Ability to clearly communicate architecture decisions and tradeoffs to technical, clinical, and executive audiences.
- Ability to obtain and maintain VA suitability and a federal PIV badge.
- U.S. Citizen or Green Card holder.
- No 1099, corp‑to‑corp, or international outsourcing.
Preferred
- Direct experience working within the Department of Veterans Affairs on architecture, data science, or adjacent projects.
- Familiarity with Unity Catalog or equivalent data governance, lineage, and access control tooling.
- Familiarity with the VA data landscape, including CDW, Millennium and Cerner data, and VistA‑era source systems.
- Experience with Azure OpenAI, Hugging Face, and LangChain or similar orchestration frameworks.
- Experience with Terraform, Kubernetes, and Azure DevOps, GitHub, or GitLab.
- Familiarity with monitoring and operations tooling such as Azure Monitor, CloudWatch, Grafana, Splunk, Dynatrace, or Datadog.
- Familiarity with the NIST AI Risk Management Framework.
- Experience with point‑of‑care delivery patterns including FHIR APIs, CDS Hooks, and SMART on FHIR.
Job Type: Full\-time
Pay: From $150,000\.00 per year
Benefits:
- 401(k)
- 401(k) matching
- Dental insurance
- Health insurance
- Paid time off
- Referral program
- Vision insurance
Application Question(s):
- This position works with government contracts. Under Order 11935, either U.S. Citizenship or valid permanent residency is required. Answer 2 if you are a US citizen, 1 if you have a permanent resident card.
- Please provide the link to your LinkedIn account.
- Please provide the link to your GitHub account.
- How many years of experience do you have working on federal contracts?
Education:
- Bachelor's (Required)
Experience:
- Python: 5 years (Required)
- SQL: 3 years (Required)
- AI technical architecture: 5 years (Required)
- Model evaluation: 5 years (Required)
- Retrieval\-Augmented Generation: 5 years (Required)
- healthcare data : 5 years (Required)
Work Location: Remote
Role Details
About This Role
AI/ML Engineers build and deploy machine learning models in production. They work across the full ML lifecycle: data pipelines, model training, evaluation, and serving infrastructure. The role has evolved significantly over the past two years. Where ML Engineers once spent most of their time on model architecture, the job now tilts heavily toward inference optimization, cost management, and integrating LLM capabilities into existing systems. Companies want engineers who can ship production systems, and the experimenter-only role is fading fast.
Day-to-day, you're writing training pipelines, debugging data quality issues, setting up evaluation frameworks, and figuring out why your model performs differently in staging than it did on your dev set. The best ML engineers are obsessive about reproducibility and measurement. They instrument everything. They know that a model is only as good as the data feeding it and the infrastructure serving it.
Across the 4,317 AI roles we're tracking, AI/ML Engineer positions make up 70% of the market. At innoVet Health, LLC, this role fits into their broader AI and engineering organization.
Demand for AI/ML Engineers has been strong and consistent. Unlike some AI roles that spike with hype cycles, ML engineering is a foundational need. Every company deploying AI models needs people who can keep them running, and the gap between research prototypes and production systems keeps growing.
What the Work Looks Like
A typical week might include: debugging a data pipeline that's silently dropping 3% of training examples, running A/B tests on a new model version, writing documentation for a feature flag system that lets you roll back model deployments, and reviewing a junior engineer's PR for a new evaluation metric. Meetings tend to be cross-functional since ML touches product, engineering, and data teams.
Demand for AI/ML Engineers has been strong and consistent. Unlike some AI roles that spike with hype cycles, ML engineering is a foundational need. Every company deploying AI models needs people who can keep them running, and the gap between research prototypes and production systems keeps growing.
Skills Required
Python and PyTorch dominate the requirements. Most roles expect experience with cloud platforms (AWS, GCP, or Azure) and familiarity with ML frameworks like TensorFlow or JAX. RAG (Retrieval-Augmented Generation) has become a top-3 skill requirement as companies integrate LLMs into their products. Docker and Kubernetes show up in about a third of postings, reflecting the production focus of the role.
Beyond the core stack, employers increasingly want experience with experiment tracking tools (MLflow, Weights & Biases), feature stores, and vector databases. Fine-tuning experience is valuable but less common than you'd think from reading Twitter. Most production LLM work is RAG and prompt engineering, not fine-tuning. If you have both, you're in a strong position.
Companies that are serious about AI/ML hiring tend to post specific infrastructure details in the job description: the frameworks they use, their model serving stack, their data pipeline tools. Vague postings that just say 'ML experience required' without specifics are often companies that haven't figured out what they need yet.
Compensation Benchmarks
AI/ML Engineer roles pay a median of $214,900 based on 6,420 positions with disclosed compensation. Mid-level AI roles across all categories have a median of $194,400.
Across all AI roles, the market median is $215,000. Top-quartile compensation starts at $266,300. The 90th percentile reaches $320,790. For comparison, the highest-paying categories include AI Safety ($287,500) and Research Engineer ($272,100). By seniority level: Entry: $110,000; Mid: $194,400; Senior: $227,400; Director: $274,554; VP: $241,000.
innoVet Health, LLC AI Hiring
innoVet Health, LLC has 1 open AI role right now. They're hiring across AI/ML Engineer. Based in Remote, US.
Remote Work Context
Remote AI roles pay a median of $180,000 across 1,196 positions. About 15% of all AI roles offer remote work.
Career Path
Common paths into AI/ML Engineer roles include Data Scientist, Software Engineer, Research Engineer.
From here, career progression typically leads toward ML Architect, AI Engineering Manager, Principal ML Engineer.
The fastest path into ML engineering is through software engineering with a self-directed ML education. A CS degree helps, but production engineering skills matter more than academic credentials. Build something that works, deploy it, and measure it. That portfolio project is worth more than a Coursera certificate. For career growth, the fork comes around the senior level: go deep on technical complexity (staff/principal track) or move into managing ML teams.
What to Expect in Interviews
Expect system design questions around ML pipelines: how you'd build a training pipeline for a specific use case, handle data drift, or design A/B testing infrastructure for model deployments. Coding rounds typically involve Python, with emphasis on data manipulation (pandas, numpy) and algorithm implementation. Take-home assignments often ask you to build an end-to-end ML pipeline from raw data to deployed model.
When evaluating opportunities: Companies that are serious about AI/ML hiring tend to post specific infrastructure details in the job description: the frameworks they use, their model serving stack, their data pipeline tools. Vague postings that just say 'ML experience required' without specifics are often companies that haven't figured out what they need yet.
AI Hiring Overview
The AI job market has 4,317 open positions tracked in our dataset. By seniority: 138 entry-level, 2,071 mid-level, 1,655 senior, and 453 leadership roles (Director, VP, C-Level). Remote roles make up 15% of the market (635 positions). The remaining 3,657 roles require on-site or hybrid attendance.
The market median for AI roles is $215,000. Top-quartile compensation starts at $266,300. The 90th percentile reaches $320,790. Highest-paying categories: AI Safety ($287,500 median, 34 roles); Research Engineer ($272,100 median, 227 roles); AI Engineering Manager ($244,000 median, 23 roles).
Demand for AI/ML Engineers has been strong and consistent. Unlike some AI roles that spike with hype cycles, ML engineering is a foundational need. Every company deploying AI models needs people who can keep them running, and the gap between research prototypes and production systems keeps growing.
The AI Job Market Today
The AI job market spans 4,317 open positions across 15 role categories. The largest categories by volume: AI/ML Engineer (3,004), Data Scientist (345), AI Software Engineer (309). These three account for the majority of open positions, though smaller categories often have higher per-role compensation because of specialized skill requirements.
The seniority mix tells a story about where AI teams are in their maturity. Entry-level roles (138) are outnumbered by mid-level (2,071) and senior (1,655) positions, reflecting that most companies are past the 'build a team from scratch' phase and need experienced engineers who can ship production systems. Leadership roles (Director, VP, C-Level) total 453 positions, representing the bottleneck between technical execution and organizational strategy.
Remote work availability sits at 15% of all AI roles (635 positions), with 3,657 requiring on-site or hybrid attendance. The remote share has stabilized after the post-pandemic correction. Senior and specialized roles (Research Scientist, ML Architect) are more likely to be remote-eligible than entry-level positions, partly because experienced hires have more negotiating power and partly because these roles require less hands-on mentorship.
AI compensation is structured in clear tiers. The market median sits at $215,000. Top-quartile roles start at $266,300, and the 90th percentile reaches $320,790. These figures include base salary with disclosed compensation. Total compensation (including equity, bonuses, and sign-on) runs 20-40% higher at companies that offer those components.
Category matters for compensation. AI Safety roles lead at $287,500 median, while Prompt Engineer roles sit at $145,000. The spread between highest and lowest-paying categories reflects the premium on specialized technical skills versus broader analytical roles.
The most in-demand skills across all AI postings: Python (2,249 postings), Aws (1,224 postings), Azure (938 postings), Rag (915 postings), Gcp (660 postings), Pytorch (640 postings), Prompt Engineering (624 postings), Kubernetes (559 postings). Python dominates, appearing in the vast majority of role descriptions regardless of category. Cloud platform experience (AWS, GCP, Azure) is the second most common requirement. The newer entrants to the top skills list (RAG, vector databases, LLM APIs) reflect the shift from traditional ML toward generative AI applications.
Frequently Asked Questions
Get Weekly AI Career Intelligence
Salary data, skills demand, and market signals from 16,000+ AI job postings. Every Monday.