Interested in this AI Software Engineer role at CVS Health?
Apply Now →Skills & Technologies
About This Role
We’re building a world of health around every individual — shaping a more connected, convenient and compassionate health experience. At CVS Health®, you’ll be surrounded by passionate colleagues who care deeply, innovate with purpose, hold ourselves accountable and prioritize safety and quality in everything we do. Join us and be part of something bigger – helping to simplify health care one person, one family and one community at a time.
Overview
We are seeking a Principal Individual Contributor (IC) to lead production engineering, observability, and operational excellence for our AI Platform. This role sits at the intersection of ML systems, distributed infrastructure, and production reliability, ensuring that our AI services are scalable, observable, and resilient in real\-world environments.
As a senior technical leader, you will define and drive best\-in\-class production practices, build robust monitoring and alerting ecosystems, and partner across engineering, ML, and platform teams to ensure mission\-critical AI systems meet high availability, performance, and reliability standards.
Key Responsibilities
Production Reliability \& Operations Leadership* Own and evolve production operations strategy for AI/ML platforms and services
- Define SLOs, SLIs, and error budgets for AI systems (online \& batch/inference pipelines)
- Lead root cause analysis (RCA) and drive systemic improvements post\-incident
- Establish operational readiness standards for launching new AI capabilities
- Build frameworks for on\-call excellence, incident response, and escalation
Observability, Monitoring \& Alerting* Design and implement end\-to\-end observability systems across AI workloads:
+ Model performance monitoring
+ Data pipeline health
+ Infrastructure metrics
- Build and scale monitoring and alerting frameworks using modern tooling (e.g., Prometheus, Grafana, OpenTelemetry, Datadog, Azure Monitor, etc.)
- Define actionable, low\-noise alerts tied to business and system impact
- Develop dashboards and telemetry standards for real\-time visibility across services
- Drive adoption of golden signals (latency, errors, throughput, saturation) in AI systems
AI/ML Production Systems Excellence* Ensure reliable deployment and operation of:
+ Real\-time inference services
+ Model pipelines (training, validation, deployment)
+ Data ingestion and feature pipelines
- Implement model observability (drift detection, data skew, performance degradation)
- Partner with ML engineers to improve production readiness of models
- Establish lifecycle standards for models in production environments
Automation \& Platform Development* Build internal platforms and tooling for:
+ Automated incident detection and response
+ Self\-healing systems
+ Deployment validation and canarying
- Drive Infrastructure as Code (IaC) and policy automation
- Improve system resilience through chaos testing and fault injection
Technical Leadership \& Strategy* Act as a trusted technical advisor across platform, ML, and product teams
- Set direction for operational excellence in AI systems at org scale
- Mentor senior engineers and influence cross\-team architectural decisions
- Lead adoption of industry best practices in reliability engineering and observability
Required Qualifications* 10\+ years in software engineering, production engineering, or SRE roles
- Deep experience operating large\-scale distributed systems in production
- Proven track record building monitoring, observability, and alerting systems
- Strong expertise in incident management and production support models
- Experience working with cloud platforms (Azure, AWS, GCP)
Preferred Qualifications* Experience supporting AI/ML platforms or data\-intensive systems
- Familiarity with model lifecycle management and MLOps practices
- Knowledge of:
+ OpenTelemetry, Prometheus, Grafana, Datadog
+ Kubernetes and containerized workloads
+ Streaming systems (Kafka, Event Hub, etc.)
- Experience defining and implementing SLO\-driven engineering
- Background in high\-availability, low\-latency systems
Key Competencies* Systems thinking and ability to reason about complex, interdependent systems
- Strong bias for automation, scalability, and long\-term solutions
- Exceptional debugging and incident management skills
- Ability to influence without authority across multiple teams
- Passion for operational excellence and reliability
Pay Range
The typical pay range for this role is:
$144,200\.00 \- $288,400\.00
This pay range represents the base hourly rate or base annual full\-time salary for all positions in the job grade within which this position falls. The actual base salary offer will depend on a variety of factors including experience, education, geography and other relevant factors. This position is eligible for a CVS Health bonus, commission or short\-term incentive program in addition to the base pay range listed above. This position also includes an award target in the company’s equity award program.
Our people fuel our future. Our teams reflect the customers, patients, members and communities we serve and we are committed to fostering a workplace where every colleague feels valued and that they belong.
Great benefits for great people
We take pride in offering a comprehensive and competitive mix of pay and benefits that reflects our commitment to our colleagues and their families.
This full‑time position is eligible for a comprehensive benefits package designed to support the physical, emotional, and financial well‑being of colleagues and their families. The benefits for this position include medical, dental, and vision coverage, paid time off, retirement savings options, wellness programs, and other resources, based on eligibility.
Additional details about available benefits are provided during the application process and on Benefits Moments.
We anticipate the application window for this opening will close on: 06/04/2026
Qualified applicants with arrest or conviction records will be considered for employment in accordance with all federal, state and local laws.
Salary Context
This $144K-$288K range is above the median for AI Software Engineer roles in our dataset (median: $190K across 218 roles with salary data).
Role Details
About This Role
AI Software Engineers build the applications and systems that AI models run inside. They own the API layers, data pipelines, frontend integrations, and infrastructure that turn a model into a product users interact with. Every AI company needs engineers who can build the software around the AI.
The challenge is building reliable systems around inherently unreliable components. Models are probabilistic. They'll give different answers to the same question. They hallucinate. They're slow. They're expensive. Your job is to build an application layer that handles all of this gracefully while delivering a product that users trust and enjoy.
Across the 3,963 AI roles we're tracking, AI Software Engineer positions make up 7% of the market. At CVS Health, this role fits into their broader AI and engineering organization.
AI Software Engineer roles are among the most numerous in the AI job market. Every company deploying AI needs software engineers who understand AI integration patterns. The demand is broad, spanning startups to enterprises, across every industry adopting AI capabilities.
What the Work Looks Like
A typical week includes: building API endpoints that serve model inference with caching and fallback logic, designing the data pipeline that feeds context to a RAG system, implementing streaming responses in the frontend, debugging a race condition in the async inference pipeline, and optimizing database queries for the vector search layer. It's full-stack engineering with AI at the center.
AI Software Engineer roles are among the most numerous in the AI job market. Every company deploying AI needs software engineers who understand AI integration patterns. The demand is broad, spanning startups to enterprises, across every industry adopting AI capabilities.
Skills Required
Full-stack engineering skills with AI integration experience. Python and TypeScript are the most common requirements. You'll need to understand API design, database architecture, and how to build reliable systems around probabilistic outputs. Experience with streaming, async processing, and caching patterns is increasingly important as real-time AI applications proliferate.
Knowledge of vector databases, embedding APIs, and LLM integration patterns (function calling, structured outputs, retry logic) differentiates AI software engineers from general software engineers. Understanding cost optimization (caching strategies, model routing, batched inference) is valuable since inference costs can dominate application economics.
Strong postings describe the product you'll be building, the AI integration patterns you'll work with, and the scale requirements. Look for companies that have existing AI features and need engineers to improve and expand them, not companies that are 'planning to add AI' someday.
Compensation Benchmarks
AI Software Engineer roles pay a median of $234,000 based on 758 positions with disclosed compensation. Senior-level AI roles across all categories have a median of $227,400. This role's midpoint ($216K) sits 8% below the category median. Disclosed range: $144K to $288K.
Across all AI roles, the market median is $200,000. Top-quartile compensation starts at $253,000. The 90th percentile reaches $307,500. For comparison, the highest-paying categories include AI Engineering Manager ($290,000) and AI Safety ($274,200). By seniority level: Entry: $97,760; Mid: $163,400; Senior: $227,400; Director: $244,800; VP: $250,000.
CVS Health AI Hiring
CVS Health has 3 open AI roles right now. They're hiring across AI Product Manager, AI Software Engineer, Data Scientist. Positions span Scottsdale, AZ, US, TX, US, NY, US. Compensation range: $158K - $288K.
Location Context
Across all AI roles, 15% (593 positions) offer remote work, while 3,349 require on-site attendance. Top AI hiring metros: New York (2,585 roles, $210,300 median); San Francisco (2,103 roles, $253,000 median); Los Angeles (1,764 roles, $190,500 median).
Career Path
Common paths into AI Software Engineer roles include Software Engineer, Full-Stack Developer, Backend Engineer.
From here, career progression typically leads toward Staff Engineer, AI Architect, Engineering Manager.
If you're a software engineer, you're already 80% there. Learn the AI integration patterns: RAG, streaming inference, function calling, structured outputs. Build a project that demonstrates you can wrap an AI model in a production-quality application with proper error handling, caching, and user experience. That's the portfolio piece that gets you hired.
What to Expect in Interviews
Technical screens look like standard software engineering interviews with an AI twist. Expect system design questions about building reliable applications around probabilistic models: handling streaming responses, implementing retry logic for API failures, and designing caching strategies for LLM outputs. Coding rounds test standard algorithms plus practical integration patterns like async processing and rate limiting.
When evaluating opportunities: Strong postings describe the product you'll be building, the AI integration patterns you'll work with, and the scale requirements. Look for companies that have existing AI features and need engineers to improve and expand them, not companies that are 'planning to add AI' someday.
AI Hiring Overview
The AI job market has 3,963 open positions tracked in our dataset. By seniority: 116 entry-level, 1,875 mid-level, 1,532 senior, and 440 leadership roles (Director, VP, C-Level). Remote roles make up 15% of the market (593 positions). The remaining 3,349 roles require on-site or hybrid attendance.
The market median for AI roles is $200,000. Top-quartile compensation starts at $253,000. The 90th percentile reaches $307,500. Highest-paying categories: AI Engineering Manager ($290,000 median, 39 roles); AI Safety ($274,200 median, 52 roles); Research Engineer ($260,000 median, 421 roles).
AI Software Engineer roles are among the most numerous in the AI job market. Every company deploying AI needs software engineers who understand AI integration patterns. The demand is broad, spanning startups to enterprises, across every industry adopting AI capabilities.
The AI Job Market Today
The AI job market spans 3,963 open positions across 15 role categories. The largest categories by volume: AI/ML Engineer (2,783), Data Scientist (297), AI Software Engineer (279). These three account for the majority of open positions, though smaller categories often have higher per-role compensation because of specialized skill requirements.
The seniority mix tells a story about where AI teams are in their maturity. Entry-level roles (116) are outnumbered by mid-level (1,875) and senior (1,532) positions, reflecting that most companies are past the 'build a team from scratch' phase and need experienced engineers who can ship production systems. Leadership roles (Director, VP, C-Level) total 440 positions, representing the bottleneck between technical execution and organizational strategy.
Remote work availability sits at 15% of all AI roles (593 positions), with 3,349 requiring on-site or hybrid attendance. The remote share has stabilized after the post-pandemic correction. Senior and specialized roles (Research Scientist, ML Architect) are more likely to be remote-eligible than entry-level positions, partly because experienced hires have more negotiating power and partly because these roles require less hands-on mentorship.
AI compensation is structured in clear tiers. The market median sits at $200,000. Top-quartile roles start at $253,000, and the 90th percentile reaches $307,500. These figures include base salary with disclosed compensation. Total compensation (including equity, bonuses, and sign-on) runs 20-40% higher at companies that offer those components.
Category matters for compensation. AI Engineering Manager roles lead at $290,000 median, while Prompt Engineer roles sit at $140,000. The spread between highest and lowest-paying categories reflects the premium on specialized technical skills versus broader analytical roles.
The most in-demand skills across all AI postings: Python (2,043 postings), Aws (1,241 postings), Azure (934 postings), Rag (886 postings), Gcp (774 postings), Pytorch (614 postings), Prompt Engineering (614 postings), Claude (564 postings). Python dominates, appearing in the vast majority of role descriptions regardless of category. Cloud platform experience (AWS, GCP, Azure) is the second most common requirement. The newer entrants to the top skills list (RAG, vector databases, LLM APIs) reflect the shift from traditional ML toward generative AI applications.
Frequently Asked Questions
Get Weekly AI Career Intelligence
Salary data, skills demand, and market signals from 16,000+ AI job postings. Every Monday.