Interested in this AI Product Manager role at Keeper Security, Inc.?
Apply Now →Skills & Technologies
About This Role
Description
Keeper Security is hiring a Senior Software Development Engineer in Test (SDET), AI & Automation to join our AI team and help build a modern, automated approach to quality across AI-powered products and services. This is a 100% remote position, with an opportunity to work a hybrid schedule for candidates based in the El Dorado Hills, CA or Chicago, IL metro areas.
Keeper’s cybersecurity software is trusted by millions of people and thousands of organizations globally. Keeper is published in 23 languages and sold in over 150 countries. Join one of the fastest-growing cybersecurity companies and help establish scalable quality engineering practices for AI-driven systems operating in security-sensitive environments.
About Keeper
Keeper Security is one of the fastest-growing cybersecurity software companies that protects thousands of organizations and millions of people in over 150 countries. Keeper is a pioneer of zero-knowledge and zero-trust security built for any IT environment. Its core offering, KeeperPAM®, is an AI-enabled, cloud-native platform that protects all users, devices and infrastructure from cyber attacks. Recognized for its innovation in the Gartner Magic Quadrant for Privileged Access Management (PAM), Keeper secures passwords and passkeys, infrastructure secrets, remote connections and endpoints with role-based enforcement policies, least privilege and just-in-time access. Learn why Keeper is trusted by leading organizations to defend against modern adversaries at KeeperSecurity.com.
About the Job
As a Senior SDET, AI & Automation, you will serve as an embedded quality engineering partner within Keeper’s AI team. You will work alongside engineers and Product throughout design and development to define test strategy, establish acceptance criteria and build automated coverage as AI-powered capabilities are being developed.
This role requires experience testing both traditional deterministic software systems and non-deterministic AI/ML systems. Traditional application behavior can often be validated against consistent expected outputs, while AI models may produce different but acceptable results across models, prompts and inputs. You will help design testing approaches that account for this variability while still establishing measurable quality, reliability and safety standards. You will build automation frameworks, develop deterministic and probabilistic validation strategies, integrate testing into CI/CD workflows and help Keeper move toward continuous quality engineering rather than end-of-cycle QA.
Responsibilities
- Partner directly with AI engineers, Software Engineers and Product Managers during design and development to define test strategy, acceptance criteria and automation requirements
- Design and maintain automated testing frameworks for AI-powered applications, backend services, APIs and supporting software systems
- Develop testing strategies for both deterministic software behavior and non-deterministic AI/ML outputs
- Define measurable evaluation criteria, tolerances and quality thresholds for AI model and LLM-driven functionality
- Build automated regression, integration, API and end-to-end coverage alongside feature development
- Develop test harnesses and evaluation workflows for prompts, models, agents, tool-calling workflows and AI-integrated applications
- Validate AI behavior across different models, prompts, inputs and system configurations
- Develop automated test utilities using languages such as Python, Java, JavaScript/TypeScript, Rust or similar technologies
- Integrate automated validation and quality gates into CI/CD pipelines using GitHub Actions, Jenkins, GitLab CI or similar platforms
- Build and maintain test datasets, mocks, fixtures and synthetic data required for repeatable AI and software validation
- Analyze model and application failures to distinguish defects, expected model variability, data issues and test instability
- Evaluate AI outputs for reliability, consistency, safety and alignment with expected product behavior
- Work with engineering teams to improve application testability, observability and diagnostic capabilities
- Track test cases, execution results and coverage using TestRail or similar test management platforms
- Analyze logs, telemetry and system behavior to identify root causes and produce clear, reproducible defect reports
- Reduce flaky tests and improve the speed, stability and maintainability of automated test suites
- Contribute to quality engineering standards, AI testing methodologies and automation best practices across Keeper’s AI organization
- Use AI-assisted tools such as Claude, ChatGPT, GitHub Copilot or similar platforms to improve automation development, test generation, debugging, analysis and documentation
Requirements
- 5+ years of experience in SDET, software development, test automation or another code-intensive quality engineering role
- Hands-on experience testing AI/ML, LLM-powered or model-integrated software systems
- Strong understanding of the differences between deterministic software testing and non-deterministic AI/ML validation
- Experience defining evaluation criteria, expected behaviors, tolerances or scoring approaches for AI-driven functionality
- Strong programming skills in Python, Java, JavaScript/TypeScript, Rust or another general-purpose programming language
- Significant experience designing, developing and maintaining automated test frameworks
- Experience building automated functional, integration, API and end-to-end test coverage
- Experience testing REST APIs and distributed application workflows using tools such as Postman, REST Assured or similar platforms
- Experience integrating automated tests into CI/CD pipelines using GitHub Actions, Jenkins, GitLab CI or similar technologies
- Experience working within an Agile software development team with close collaboration between QA, Engineering and Product
- Strong understanding of shift-left testing, continuous quality and testability principles
- Experience with TestRail or similar test management platforms
- Strong debugging and troubleshooting skills using logs, APIs, application telemetry and development tools
- Experience with Git-based development workflows, code review and modern software engineering practices
- Familiarity with AI-assisted development and testing tools such as Claude, ChatGPT, GitHub Copilot or similar technologies
- Strong analytical and problem-solving skills with the ability to evaluate variable AI behavior without relying solely on exact-match outcomes
- Strong written and verbal communication skills and ability to collaborate closely with AI engineers, Software Engineers, Product Managers and QA professionals
- Bachelor’s degree in Computer Science, Engineering or a related technical field, or equivalent practical experience
- Due to this role’s involvement with FedRAMP compliance requirements, qualified candidates must be a U.S. Person, defined as a U.S. citizen or lawful permanent resident.
Preferred Qualifications
- Experience testing LLMs, agentic systems, retrieval-augmented generation, tool calling or prompt-driven workflows
- Experience building automated model evaluation, benchmarking or regression frameworks
- Familiarity with AI evaluation metrics and techniques for measuring quality across non-deterministic outputs
- Experience with Rust in production or test automation environments
- Experience with performance, reliability, security or adversarial testing of AI-enabled systems
- Experience with AI-assisted generation of automated tests or synthetic test data
- Familiarity with Docker, Kubernetes or cloud-based testing environments
- Experience testing cybersecurity, identity, PAM or other security-sensitive software
- Experience building shared automation frameworks or internal testing tools used across engineering teams
Benefits
- Medical, Dental & Vision (inclusive of domestic partnerships
- Employer Paid Life Insurance & Employee/Spouse/Child Supplemental life
- Voluntary Short/Long Term Disability Insurance
- 401K (Roth/Traditional)
- A generous PTO plan that celebrates your commitment and seniority (including paid Bereavement/Jury Duty, etc)
Keeper Security, Inc. is an equal opportunity employer and participant in the U.S. Federal E-Verify program. We celebrate diversity and are committed to creating an inclusive environment for all employees.
Classification: Exempt
Keeper Candidate Privacy Notice
This notice explains how Keeper Security processes your personal data during recruitment. Depending on the role and location, the Controller of personal data (the organization responsible for determining why and how personal data is processed) will be Keeper Security Inc. (US), Keeper Security EMEA Ltd. (Ireland), or Keeper Security APAC K.K (Japan).
- Data We Collect
Information You provide:
- Contact details, CV/resume, cover letter
- Employment history, qualifications, work eligibility
- Application responses and uploaded documents
Information We generate:
- Interview notes, assessments, communications
- Scheduling information
Information From Others:
- Recruiter/referral information who submit your profile
- References (with your consent, before final offer)
- Public professional profiles
- Background verification (post offer)
Voluntary Diversity and Equal Opportunity Information
- We may ask you to voluntarily provide diversity information including race/ethnicity, gender, disability status and veteran status (US). Providing this information is optional and Keeper collects this data in order to comply with EEOC and similar requirements
- How We Use Your Data
- Assess your application and suitability
- Manage interviews and recruitment workflow
- Consider you for other/future roles (we may seek your consent to keep your information on our systems beyond the retention period specified)
- Comply with employment law obligations
- Legal Basis
- Legitimate Interests (recruitment management, security and integrity of the hiring process)
- Contracting steps (for progressed candidates)
- Legal and regulatory compliance obligations; explicit consent where required
- Who We Share Information With
Internal:
- HR, hiring managers, interviewers*, IT support for system administration
- Note - diversity and equal opportunity data is not shared with hiring managers.
Third Parties:
Service providers who assist with:
- Applicant tracking, recruitment systems and assessment providers
- Background verification vendors (post offer)
- Recruitment agencies (where applicable)
- Tools to support communication, collaboration and to securely store your data
Keeper ensures that all our third parties are contractually bound to protect your personal data with adequate safeguards in place.
- International Transfers
Your data may be accessed by Keeper entities globally as needed for the purposes of hiring and decision making. We protect any such data transfer between Keeper entities using appropriate safeguards under applicable data protection laws.
- Security
We implement appropriate technical and organizational measures to protect your data, consistent with our industry leading security standards.
- Retention
We keep your data for 24 months from your last application activity, then delete or anonymize it.
Exceptions:
- You opt into our talent database for further retention by providing consent (extended retention)
- You're hired (transfers to employee records)
- Your Rights
You have the following rights and can contact us at the email below to exercise them:
- Access, correct, or delete your data, subject to applicable law and retention requirements
- Object to or restrict processing
- Withdraw consent (where applicable)
- Request data portability
- Lodge a complaint with your data protection authority
If you become an employee, your rights regarding your employee record are governed by our internal Employee Privacy Notice and certain data will be retained as required under relevant laws such as employment or tax law.
When you request access to your personal data, some information may be redacted if it includes the personal data of other individuals or information that we must protect in order to preserve their privacy rights.
- Automated Decisions
Keeper does not make hiring decisions using solely automated processing.
- Contact - Candidates can send privacy questions to: [email protected]
Role Details
About This Role
AI Product Managers define what AI features get built and why. They translate business problems into ML-solvable tasks, work with engineering to scope model requirements, and own the metrics that determine if an AI feature is working. The role requires a rare combination of technical fluency and product instinct.
Unlike traditional product management, AI PM work involves managing uncertainty at a fundamental level. Your model might work 90% of the time. What happens the other 10%? What's the user experience when the AI is wrong? How do you measure 'good enough' for a probabilistic system? These questions don't have easy answers, and the AI PM is the person responsible for finding them.
Across the 9,079 AI roles we're tracking, AI Product Manager positions make up 5% of the market. At Keeper Security, Inc., this role fits into their broader AI and engineering organization.
AI Product Manager roles are growing as companies realize that shipping AI features requires different product thinking than traditional software. The best candidates combine product management experience with enough technical depth to have productive conversations with ML engineers about model capabilities and limitations.
What the Work Looks Like
A typical week includes: reviewing model evaluation results with the ML team, defining success metrics for a new AI feature, conducting user research on how customers respond to AI-generated outputs, writing product requirements that include accuracy thresholds and fallback behaviors, and presenting the AI roadmap to leadership. You're the translator between technical capability and business value.
AI Product Manager roles are growing as companies realize that shipping AI features requires different product thinking than traditional software. The best candidates combine product management experience with enough technical depth to have productive conversations with ML engineers about model capabilities and limitations.
Skills Required
Technical fluency with ML concepts is essential, though you won't be writing models. Expect to understand training data, evaluation metrics, model limitations, and responsible AI practices. SQL and basic Python are increasingly expected. Experience with A/B testing, data analysis, and product analytics is baseline. Understanding LLM capabilities and limitations is now a core requirement.
The differentiator is AI-specific product thinking: knowing when to use ML vs. heuristics, understanding the cost of training data collection, designing graceful degradation for model failures, and building products that improve with usage data. Experience with AI safety, bias mitigation, and responsible AI deployment is increasingly important.
Strong postings describe specific AI products the PM will own, mention the ML team structure, and talk about measurement methodology. Look for companies that have already shipped AI features. Roles at companies that are 'exploring AI' often mean you'll spend a year defining the strategy before any building happens.
Compensation Benchmarks
AI Product Manager roles pay a median of $220,200 based on 947 positions with disclosed compensation. Senior-level AI roles across all categories have a median of $229,300.
Across all AI roles, the market median is $213,000. Top-quartile compensation starts at $263,911. The 90th percentile reaches $318,400. For comparison, the highest-paying categories include AI Safety ($256,000) and Research Engineer ($252,000). By seniority level: Entry: $111,000; Mid: $193,200; Senior: $229,300; Director: $272,300; VP: $235,830.
Keeper Security, Inc. AI Hiring
Keeper Security, Inc. has 2 open AI roles right now. They're hiring across AI Product Manager, AI Software Engineer. Based in Remote, US.
Remote Work Context
Remote AI roles pay a median of $188,000 across 2,316 positions. About 6% of all AI roles offer remote work.
Career Path
Common paths into AI Product Manager roles include Product Manager, Data Analyst, Technical Program Manager.
From here, career progression typically leads toward Director of AI Product, VP Product, Head of AI.
The most effective path is PM experience plus self-directed AI education. Take Andrew Ng's courses, build a small ML project, and learn enough Python to read model evaluation code. The goal isn't to become an ML engineer. It's to have credibility in technical conversations and to understand what's possible, what's hard, and what's a bad idea.
What to Expect in Interviews
AI interviews typically combine coding challenges (Python-focused), system design questions tailored to the role, and discussions about your experience with relevant tools and frameworks. Strong candidates demonstrate both technical depth and the ability to make pragmatic engineering tradeoffs. Prepare portfolio projects that demonstrate end-to-end capability rather than isolated skills.
When evaluating opportunities: Strong postings describe specific AI products the PM will own, mention the ML team structure, and talk about measurement methodology. Look for companies that have already shipped AI features. Roles at companies that are 'exploring AI' often mean you'll spend a year defining the strategy before any building happens.
AI Hiring Overview
The AI job market has 9,079 open positions tracked in our dataset. By seniority: 106 entry-level, 4,384 mid-level, 3,343 senior, and 1,246 leadership roles (Director, VP, C-Level). Remote roles make up 6% of the market (531 positions). The remaining 8,478 roles require on-site or hybrid attendance.
The market median for AI roles is $213,000. Top-quartile compensation starts at $263,911. The 90th percentile reaches $318,400. Highest-paying categories: AI Safety ($256,000 median, 66 roles); Research Engineer ($252,000 median, 457 roles); AI Engineering Manager ($251,622 median, 55 roles).
AI Product Manager roles are growing as companies realize that shipping AI features requires different product thinking than traditional software. The best candidates combine product management experience with enough technical depth to have productive conversations with ML engineers about model capabilities and limitations.
The AI Job Market Today
The AI job market spans 9,079 open positions across 16 role categories. The largest categories by volume: AI/ML Engineer (6,862), AI Product Manager (453), AI Software Engineer (451). These three account for the majority of open positions, though smaller categories often have higher per-role compensation because of specialized skill requirements.
The seniority mix tells a story about where AI teams are in their maturity. Entry-level roles (106) are outnumbered by mid-level (4,384) and senior (3,343) positions, reflecting that most companies are past the 'build a team from scratch' phase and need experienced engineers who can ship production systems. Leadership roles (Director, VP, C-Level) total 1,246 positions, representing the bottleneck between technical execution and organizational strategy.
Remote work availability sits at 6% of all AI roles (531 positions), with 8,478 requiring on-site or hybrid attendance. The remote share has stabilized after the post-pandemic correction. Senior and specialized roles (Research Scientist, ML Architect) are more likely to be remote-eligible than entry-level positions, partly because experienced hires have more negotiating power and partly because these roles require less hands-on mentorship.
AI compensation is structured in clear tiers. The market median sits at $213,000. Top-quartile roles start at $263,911, and the 90th percentile reaches $318,400. These figures include base salary with disclosed compensation. Total compensation (including equity, bonuses, and sign-on) runs 20-40% higher at companies that offer those components.
Category matters for compensation. AI Safety roles lead at $256,000 median, while Prompt Engineer roles sit at $141,440. The spread between highest and lowest-paying categories reflects the premium on specialized technical skills versus broader analytical roles.
The most in-demand skills across all AI postings: Python (1,649 postings), Aws (898 postings), Azure (723 postings), Rag (671 postings), Gcp (536 postings), Pytorch (434 postings), Claude (410 postings), Kubernetes (406 postings). Python dominates, appearing in the vast majority of role descriptions regardless of category. Cloud platform experience (AWS, GCP, Azure) is the second most common requirement. The newer entrants to the top skills list (RAG, vector databases, LLM APIs) reflect the shift from traditional ML toward generative AI applications.
Frequently Asked Questions
Get Weekly AI Career Intelligence
Salary data, skills demand, and market signals from 16,000+ AI job postings. Every Monday.