Interested in this AI/ML Engineer role at Quantum Sky?
Apply Now →Skills & Technologies
About This Role
Description:
Quantum Sky is searching for a Principal AI SOC Engineer to drive efficiency, velocity, and operational effectiveness across Security Operations Centers supporting federal missions. This is a hands-on engineering role for someone who has worked inside a federal watch floor, has built and integrated SOC tooling at enterprise scale, and knows how to turn analyst pain points into production-grade automated capability.
The ideal candidate is a builder and a leader: an engineer who writes production Python daily, is deeply fluent in Splunk, and has carried AI/ML and GenAI capabilities through a real software development lifecycle — versioned, tested, evaluated, monitored, and governed — inside a federal agency's accreditation and oversight regime. The role blends technical execution with engineering leadership, including backlog ownership in Jira, design documentation in Confluence, delivery oversight, code review, and mentoring. Responsibilities:AI Capability Engineering* Design and engineer AI-powered SOC capabilities that improve analyst efficiency, reduce alert fatigue, and accelerate detection and response.
- Apply AI/ML and GenAI techniques to concrete SOC problems: alert enrichment, triage and prioritization, entity and campaign correlation, investigation summarization, phishing and insider-threat triage, and automated response recommendation.
- Own the full AI SDLC for delivered capabilities — problem framing, data curation and labeling, model or prompt development, evaluation harness design, CI/CD integration, ATO-compatible deployment, monitoring for drift and regression, and rollback.
- Build evaluation and test methodology for AI-enabled workflows: golden datasets, regression suites, precision/recall and false-positive measurement, and human-in-the-loop review gates before any capability influences analyst action or containment.
- Implement guardrails, output validation, prompt and response logging, and decision traceability so AI-assisted findings are auditable and defensible to agency leadership, oversight bodies, and assessors.
- Engineer AI capabilities consistent with federal AI governance expectations — current OMB AI guidance, agency Chief AI Officer requirements, AI use case inventory reporting, and NIST AI RMF — including the additional practices that apply when a capability is designated high-impact.
- Ensure AI capabilities handle sensitive government data appropriately, including PII and law enforcement sensitive material, with data minimization, retention controls, and model-training exclusions.
SOC Engineering and Automation* Build and evolve detection and response pipelines across SIEM, SOAR, EDR, email security, identity, and cloud security platforms, with Splunk as the primary analytic platform.
- Engineer Splunk content and infrastructure: advanced SPL, data models and CIM normalization, correlation searches and notable event tuning in Enterprise Security, ingest and index architecture, and performance tuning for high-volume telemetry.
- Build and maintain SOC automation in Python — SOAR playbooks, custom microservices, API-driven integrations, and AI-driven decisioning across security and infrastructure controls.
- Engineer onboarding and normalization pipelines for telemetry from organizationally distinct components with heterogeneous tooling, ownership models, and data-sharing constraints.
- Support enterprise event logging maturity requirements, including log source coverage, retention tiering, and log integrity.
- Build and maintain integrations supporting federal reporting and directive compliance, including CISA sensor and CDM data flows, Binding Operational and Emergency Directive response, and incident notification timelines.
- Improve SOC velocity and throughput by automating repetitive analyst tasks and standardizing response patterns into reusable, tested components.
- Support design of scalable SOC architectures for high-volume telemetry and real-time workflows in a 24/7 operations environment.
- Maintain a tool-agnostic engineering mindset; integrate Elastic, Microsoft Sentinel and Defender, or cloud-native services where the mission calls for it.
Delivery and Technical Leadership* Own and groom the engineering backlog in Jira; prioritize, decompose, estimate, and ship production-ready increments on a predictable cadence against contract deliverables.
- Translate analyst user stories and operational requirements into concrete technical designs; document architecture decisions, runbooks, SOPs, and capability documentation in Confluence.
- Set engineering standards and patterns for AI-enabled SOC capabilities — code review expectations, testing requirements, repository structure, CI/CD pipelines, and secure development practices aligned to NIST SSDF.
- Partner with government stakeholders, analysts, and fellow engineers to deliver solutions, setting a high technical bar through hands-on contribution and shared ownership.
- Mentor engineers on Python, Splunk, and AI engineering practice.
Qualifications: Required:* 8–12 years of hands-on enterprise IT and cybersecurity engineering experience spanning security operations, cloud platforms, automation, and AI/ML.
- 3–5 years of direct SOC engineering experience designing, building, and optimizing SOC tooling, with at least 2 years supporting federal SOC environments (civilian agencies, DoD, Intelligence Community, or federal law enforcement).
- Expert-level Python: production services, automation frameworks, API integrations, unit and integration testing, packaging, and code review of others' work.
- Deep hands-on Splunk expertise: advanced SPL, data model and CIM work, Splunk Enterprise Security content development, ingest pipeline and architecture design, and Splunk SOAR playbook development.
- 3+ years applying AI/ML techniques to cybersecurity or operational systems, including AI-enabled workflows for alert enrichment, triage, detection engineering, or automated response.
- 2+ years working with LLMs or GenAI systems in production or near-production environments, including RAG pipelines and LLM-integrated automation for SOC use cases.
- Demonstrated AI SDLC ownership: taking an AI/ML or GenAI capability from requirement through evaluation, deployment, and sustained operation, with versioning, reproducibility, testing, and monitoring in place.
- Strong experience with SOC automation, orchestration, and playbook design, including API-driven integrations and detection, correlation, and response pipeline engineering.
- Proficiency with Git-based workflows, CI/CD pipelines, and containerized deployment.
- Working fluency with Jira and Confluence as engineering delivery and documentation tools — backlog ownership, sprint execution, and written design and decision records.
- Working knowledge of the federal cybersecurity compliance stack: FISMA, NIST 800-53, RMF and the ATO process, FedRAMP for cloud services, and CISA directive-driven operations.
- Demonstrated ability to set technical strategy, review designs, and foster best practices across a team of engineers.
Desired:* Prior engineering support to a large, federated federal civilian agency SOC or a federal law enforcement environment.
- Familiarity with current OMB AI governance requirements, NIST AI RMF, and federal AI use case inventory and impact-assessment processes.
- Experience with TIC 3.0, CDM, and CISA incident reporting workflows.
- Experience deploying AI/ML workloads in FedRAMP-authorized environments, and experience self-hosting open-weight models where data sensitivity precludes commercial API use.
- Familiarity with CJIS Security Policy and handling of law enforcement sensitive data.
- Infrastructure-as-code and configuration management experience (Terraform, Ansible, Kubernetes).
- Experience with Azure and Microsoft Sentinel, including Azure OpenAI or Azure ML for GenAI use cases.
- Certifications: Splunk Enterprise Certified Architect, Splunk SOAR Certified Automation Developer; CISSP; GCIA, GCDA, or GCFA; Microsoft Certified: Azure Security Engineer Associate.
Clearance* Must have a current TS/SCI
Location* Hybrid remote at HQ in Reston, VA
- ~2 day work from home flexibility.
About Quantum Sky: Compensation:* Compensation is unique to each candidate and relative to the skills and experience they bring to the position. The salary range for this position is typically between $200,000-$225,000. This does not guarantee a specific salary as compensation is based upon multiple factors such as education, experience, certifications, and other requirements, and may fall outside of the above-stated range.
Benefits:* Highlights of our benefits include Health/Dental/Vision, 401(k) match, Paid Time Off, STD/LTD/Life Insurance, Referral Bonuses, professional development reimbursement, and parental leave.
The world the mission operates in is going post-quantum, contested, and machine-speed. Quantum Sky engineers the advantage across cyber, networks, software, and quantum because the mission demands dominance, not parity. We don't follow the map. We draw it.
At Quantum Sky, we believe that success starts with our people. We foster a collaborative, innovative, and mission-driven environment where every team member plays a critical role in shaping the future of technology. Are you ready to join #TeamQuantumSky?
Quantum Sky Engineering LLC is an Equal Opportunity Employer; all qualified applicants will receive consideration for employment without regard to race, color, religion, sex, [sexual orientation, gender identity,] national origin, disability, status as a protected veteran, or any characteristic protected by applicable law.
Salary Context
This $200K-$225K range is above the median for AI/ML Engineer roles in our dataset (median: $176K across 1540 roles with salary data).
View full AI/ML Engineer salary data →Role Details
About This Role
AI/ML Engineers build and deploy machine learning models in production. They work across the full ML lifecycle: data pipelines, model training, evaluation, and serving infrastructure. The role has evolved significantly over the past two years. Where ML Engineers once spent most of their time on model architecture, the job now tilts heavily toward inference optimization, cost management, and integrating LLM capabilities into existing systems. Companies want engineers who can ship production systems, and the experimenter-only role is fading fast.
Day-to-day, you're writing training pipelines, debugging data quality issues, setting up evaluation frameworks, and figuring out why your model performs differently in staging than it did on your dev set. The best ML engineers are obsessive about reproducibility and measurement. They instrument everything. They know that a model is only as good as the data feeding it and the infrastructure serving it.
Across the 9,653 AI roles we're tracking, AI/ML Engineer positions make up 76% of the market. At Quantum Sky, this role fits into their broader AI and engineering organization.
Demand for AI/ML Engineers has been strong and consistent. Unlike some AI roles that spike with hype cycles, ML engineering is a foundational need. Every company deploying AI models needs people who can keep them running, and the gap between research prototypes and production systems keeps growing.
What the Work Looks Like
A typical week might include: debugging a data pipeline that's silently dropping 3% of training examples, running A/B tests on a new model version, writing documentation for a feature flag system that lets you roll back model deployments, and reviewing a junior engineer's PR for a new evaluation metric. Meetings tend to be cross-functional since ML touches product, engineering, and data teams.
Demand for AI/ML Engineers has been strong and consistent. Unlike some AI roles that spike with hype cycles, ML engineering is a foundational need. Every company deploying AI models needs people who can keep them running, and the gap between research prototypes and production systems keeps growing.
Skills Required
Python and PyTorch dominate the requirements. Most roles expect experience with cloud platforms (AWS, GCP, or Azure) and familiarity with ML frameworks like TensorFlow or JAX. RAG (Retrieval-Augmented Generation) has become a top-3 skill requirement as companies integrate LLMs into their products. Docker and Kubernetes show up in about a third of postings, reflecting the production focus of the role.
Beyond the core stack, employers increasingly want experience with experiment tracking tools (MLflow, Weights & Biases), feature stores, and vector databases. Fine-tuning experience is valuable but less common than you'd think from reading Twitter. Most production LLM work is RAG and prompt engineering, not fine-tuning. If you have both, you're in a strong position.
Companies that are serious about AI/ML hiring tend to post specific infrastructure details in the job description: the frameworks they use, their model serving stack, their data pipeline tools. Vague postings that just say 'ML experience required' without specifics are often companies that haven't figured out what they need yet.
Compensation Benchmarks
AI/ML Engineer roles pay a median of $212,000 based on 11,857 positions with disclosed compensation. Senior-level AI roles across all categories have a median of $228,700. Disclosed range: $200K to $225K.
Across all AI roles, the market median is $213,400. Top-quartile compensation starts at $264,230. The 90th percentile reaches $319,400. For comparison, the highest-paying categories include AI Safety ($253,100) and Research Engineer ($252,000). By seniority level: Entry: $110,408; Mid: $193,200; Senior: $228,700; Director: $272,275; VP: $235,000.
Quantum Sky AI Hiring
Quantum Sky has 3 open AI roles right now. They're hiring across AI/ML Engineer. Based in Reston, VA, US. Compensation range: $140K - $275K.
Location Context
Across all AI roles, 6% (568 positions) offer remote work, while 9,010 require on-site attendance. Top AI hiring metros: New York (2,878 roles, $221,000 median); San Francisco (2,413 roles, $264,500 median); Los Angeles (1,340 roles, $210,000 median).
Career Path
Common paths into AI/ML Engineer roles include Data Scientist, Software Engineer, Research Engineer.
From here, career progression typically leads toward ML Architect, AI Engineering Manager, Principal ML Engineer.
The fastest path into ML engineering is through software engineering with a self-directed ML education. A CS degree helps, but production engineering skills matter more than academic credentials. Build something that works, deploy it, and measure it. That portfolio project is worth more than a Coursera certificate. For career growth, the fork comes around the senior level: go deep on technical complexity (staff/principal track) or move into managing ML teams.
What to Expect in Interviews
Expect system design questions around ML pipelines: how you'd build a training pipeline for a specific use case, handle data drift, or design A/B testing infrastructure for model deployments. Coding rounds typically involve Python, with emphasis on data manipulation (pandas, numpy) and algorithm implementation. Take-home assignments often ask you to build an end-to-end ML pipeline from raw data to deployed model.
When evaluating opportunities: Companies that are serious about AI/ML hiring tend to post specific infrastructure details in the job description: the frameworks they use, their model serving stack, their data pipeline tools. Vague postings that just say 'ML experience required' without specifics are often companies that haven't figured out what they need yet.
AI Hiring Overview
The AI job market has 9,653 open positions tracked in our dataset. By seniority: 193 entry-level, 4,500 mid-level, 3,455 senior, and 1,505 leadership roles (Director, VP, C-Level). Remote roles make up 6% of the market (568 positions). The remaining 9,010 roles require on-site or hybrid attendance.
The market median for AI roles is $213,400. Top-quartile compensation starts at $264,230. The 90th percentile reaches $319,400. Highest-paying categories: AI Safety ($253,100 median, 57 roles); Research Engineer ($252,000 median, 419 roles); AI Engineering Manager ($251,622 median, 51 roles).
Demand for AI/ML Engineers has been strong and consistent. Unlike some AI roles that spike with hype cycles, ML engineering is a foundational need. Every company deploying AI models needs people who can keep them running, and the gap between research prototypes and production systems keeps growing.
The AI Job Market Today
The AI job market spans 9,653 open positions across 16 role categories. The largest categories by volume: AI/ML Engineer (7,301), Data Scientist (441), AI Software Engineer (438). These three account for the majority of open positions, though smaller categories often have higher per-role compensation because of specialized skill requirements.
The seniority mix tells a story about where AI teams are in their maturity. Entry-level roles (193) are outnumbered by mid-level (4,500) and senior (3,455) positions, reflecting that most companies are past the 'build a team from scratch' phase and need experienced engineers who can ship production systems. Leadership roles (Director, VP, C-Level) total 1,505 positions, representing the bottleneck between technical execution and organizational strategy.
Remote work availability sits at 6% of all AI roles (568 positions), with 9,010 requiring on-site or hybrid attendance. The remote share has stabilized after the post-pandemic correction. Senior and specialized roles (Research Scientist, ML Architect) are more likely to be remote-eligible than entry-level positions, partly because experienced hires have more negotiating power and partly because these roles require less hands-on mentorship.
AI compensation is structured in clear tiers. The market median sits at $213,400. Top-quartile roles start at $264,230, and the 90th percentile reaches $319,400. These figures include base salary with disclosed compensation. Total compensation (including equity, bonuses, and sign-on) runs 20-40% higher at companies that offer those components.
Category matters for compensation. AI Safety roles lead at $253,100 median, while Prompt Engineer roles sit at $141,440. The spread between highest and lowest-paying categories reflects the premium on specialized technical skills versus broader analytical roles.
The most in-demand skills across all AI postings: Python (1,605 postings), Aws (916 postings), Azure (724 postings), Rag (634 postings), Gcp (522 postings), Pytorch (495 postings), Kubernetes (384 postings), Claude (383 postings). Python dominates, appearing in the vast majority of role descriptions regardless of category. Cloud platform experience (AWS, GCP, Azure) is the second most common requirement. The newer entrants to the top skills list (RAG, vector databases, LLM APIs) reflect the shift from traditional ML toward generative AI applications.
Frequently Asked Questions
Get Weekly AI Career Intelligence
Salary data, skills demand, and market signals from 16,000+ AI job postings. Every Monday.