Interested in this Data Scientist role at Los Alamos National Laboratory?
Apply Now →Skills & Technologies
About This Role
What You Will Do
This position is open for external candidates only to apply.
The High Performance Computing (HPC) Division at Los Alamos National Laboratory provides scientific computing resources consisting of some of the largest HPC systems in the world as well as numerous large commodity clusters. Our HPC Systems Group (HPC-SYS) is seeking a Team Leader for the AIOps team to make better use of our vast amounts of HPC System and Application data, this team will add AI solutions to our HPC infrastructure. The new team will work alongside the other Teams in HPC-SYS, Monitoring, Web Services and Cybersecurity. The Monitoring Team is responsible for collecting all data from our Data Centers, everything from Facilities to Clusters, and implementing operational dashboard, alerts and reports using tools like Splunk. Our Web Servers team runs our admin and user facing web sites, including user Documentation, Ticketing systems and Gitlab. Our Cybersecurity Team monitors and implements cybersecurity policies on our HPC systems.
The AIOps team will have three major focus areas, LLMs, System Data Analysis and System Automation. We have a large set of HPC specific documentation for both users and admins that will be integrated into LLMs, this team will be responsible for designing, building, and running the LLMs, as well as adding agentic capabilities to increase the usefulness to the user. We have massive amounts of system data. This team will implement ML and Data Science techniques to perform deeper analysis of the data to improve performance analysis, event correlation and anomaly detection. Finally, the team will investigate AI driven workflows for task automation within the Data Center. You will work closely with members of the AIOps team and System Matter Experts (SMEs) in different HPC areas to design and develop these tools using our on-prem analysis and AI systems.
The successful candidates' scope will include monitoring and analyzing system performance to identify anomalies. You will analyze large volumes of data to identify patterns and trends using ML and Data Science techniques with the goal of developing automation scripts and workflows to implement proactive measures. You and the team will maintain the AIOps platforms and tools including the user-facing LLMs. The successful candidate will continue actively growing their technical skills and keeping up to date with the latest technologies in the field. In addition, the selected candidate will have the opportunity to develop technical products such as technical documentation, presentations, technical papers, and reports, to communicate findings internally and at conferences. This position is full-time and is located at Los Alamos National Laboratory in Los Alamos, New Mexico.
What You Need
Minimum Job Requirements:
- HPC Systems & Linux Administration: Strong understanding of compute nodes, schedulers, high-speed networks, parallel file systems, accelerators, containers, and common HPC workloads. Proficient with the Linux command line and hardware and software security practices.
- Technical Leadership & AIOps Strategy: Experience leading multidisciplinary teams of HPC engineers, data scientists, ML engineers, and developers. Able to define and execute a roadmap for applying AI and ML to improve reliability, efficiency, performance, and user support.
- Python & Software Development: Strong programming skills with Python and AI frameworks for data analysis, model development, automation, APIs, testing, and production-quality services, including Git-based version control.
- Machine Learning & Statistical Analysis: Strong statistical skills and working knowledge of ML algorithms for anomaly detection, classification, clustering, forecasting, time-series analysis, event correlation, and predictive maintenance.
- Monitoring & Observability: Extensive experience analyzing system logs, telemetry, metrics, and traces, along with alerting and observability platforms.
- Root Cause & Performance Analysis: Able to correlate failures and anomalous behavior across compute, storage, network, scheduler, facility, and application data. Experienced in identifying bottlenecks, resource contention, and inefficient workloads.
- Data Engineering: Able to design pipelines that collect, normalize, enrich, store, and retrieve large volumes of HPC telemetry and operational data.
- MCP & Systems Integration: Experience developing secure MCP or similar interfaces that let AI assistants query schedulers, monitoring systems, documentation, and ticketing platforms.
- LLM, RAG & AI Agent Architecture: Knowledge of retrieval-augmented generation, tool-using agents, workflow orchestration, and LLM evaluation, including accuracy, retrieval quality, hallucination risk, latency, and cost.
- Security, Responsible AI & Communication: Understanding of authentication and authorization, least-privilege access, secrets management, and prompt-injection defenses. Sound judgment on when AI should recommend, require human approval, or act autonomously. Able to explain technical findings and AI limitations to diverse stakeholders.
Education/Experience at HPC Engineer 3:
Position Requires a bachelor's in Computer Science or Computer Engineering or a related field and 6 years of relevant experience in high performance computing or scalable AI computing, or data center environments or equivalent combination of education and experience in related field.
Desired Qualifications:
- Production RAG Experience: Hands-on experience building and running RAG-based LLM systems using documentation, tickets, knowledge bases, and operational history.
- End-to-End AIOps Implementation: Experience implementing AIOps tools and workflows from ML analysis through system automation and configuration management.
- MLOps & LLMOps: Experience with model deployment, versioning, monitoring, drift detection, reproducibility, and lifecycle management.
- Monitoring Tool Experience: Experience integrating operational metrics into Splunk and working with Syslog, Telegraf, Prometheus, Grafana, or similar tools.
- HPC User Support: Familiarity with common user issues, including job failures, resource requests, environments, software dependencies, storage, and performance.
- Knowledge Management: Able to turn documentation, runbooks, incident reports, and expert knowledge into reliable content for retrieval and automated assistance.
- Professional Effectiveness: Effective at working with customers and vendors. Detail-oriented, organized, and able to multitask in a fast-paced environment, with a commitment to evaluating emerging AI technologies for measurable value.
- Active DOE Q Clearance
Work Location: This position will be located in Los Alamos, NM, with the potential for a hybrid work arrangement (60% onsite/40% offsite) from a location within 2 hours ground commute of this location. Reporting onsite will be required. Hybrid is at the discretion of management and can change at any time with appropriate notice.
Position commitment: Regular appointment employees are required to serve a period of continuous service in their current position in order to be eligible to apply for posted jobs throughout the Laboratory. If an employee has not served the time required, they may only apply for Laboratory jobs with the documented approval of their Division Leader. The position commitment for this position is 1 year.
Note to Applicants:
For consideration, applicants should submit a cover letter addressing how their knowledge, skills and abilities meet the minimum requirements along with a resume.
Due to federal restrictions contained in the current National Defense Authorization Act, citizens of the People's Republic of China-including the special administrative regions of Hong Kong and Macau-as well as citizens of the Islamic Republic of Iran, the Democratic People's Republic of Korea (North Korea), and the Russian Federation, who are not Lawful Permanent Residents ("green card" holders) are prohibited from accessing facilities that support the mission, functions, and operations of national security laboratories and nuclear weapons production facilities, which includes Los Alamos National Laboratory.
Where You Will Work
Located in beautiful northern New Mexico, Los Alamos National Laboratory (LANL) is a multidisciplinary research institution engaged in strategic science on behalf of national security. Our generous benefits package includes:
§ PPO or High Deductible medical insurance with the same large nationwide network
§ Dental and vision insurance
§ Free basic life and disability insurance
§ Paid childbirth and parental leave
§ Award-winning 401(k) (6% matching plus 3.5% annually)
§ Learning opportunities and tuition assistance
§ Flexible schedules and time off (PTO and holidays)
§ Onsite gyms and wellness programs
§ Extensive relocation packages (outside a 50 mile radius)
Additional Details
Directive 206.2 - Employment with Triad requires a favorable decision by NNSA indicating employee is suitable under NNSA Supplemental Directive 206.2. Please note that this requirement applies only to citizens of the United States. Foreign nationals are subject to a similar requirement under DOE Order 142.3A.
Clearance: Q (Position will be cleared to this level). Selected applicants will be subject to a background investigation conducted by or on behalf of the Federal Government, and must meet eligibility requirements* for access to classified matter. This position requires a Q clearance, and obtaining such clearance requires US Citizenship except in extremely rare circumstances. Dependent upon the position, additional authorization to access classified information may be required, which may or may not be available to dual citizens. Receipt of a Q clearance and additional access authorization ultimately is a decision of the Federal Government and not of Triad.
- Eligibility requirements: To obtain a clearance, an individual must be at least 18 years of age; U.S. citizenship is required except in very limited circumstances. See DOE Order 472.2 for additional information.
New-Employment Drug Test: The Laboratory requires successful applicants to complete a new-employment drug test and maintains a substance abuse policy that includes random drug testing. Although New Mexico and other states have legalized the use of marijuana, use and possession of marijuana remain illegal under federal law. A positive drug test for marijuana will result in termination of employment, even if the use was pre-offer.
Regular position: Term status Laboratory employees applying for regular-status positions are converted to regular status.
Internal Applicants: Regular appointment employees who have served the required period of continuous service in their current position are eligible to apply for posted jobs throughout the Laboratory. If an employee has not served the required period of continuous service, they may only apply for Laboratory jobs with the documented approval of their Division Leader. Please refer to Policy Policy P701 for applicant eligibility requirements.
Equal Opportunity: Los Alamos National Laboratory is an equal opportunity employer. All employment practices are based on qualification and merit, without regard to protected categories such as race, color, national origin, ancestry, religion, age, sex, gender identity, sexual orientation, marital status or spousal affiliation, physical or mental disability, medical conditions, pregnancy, status as a protected veteran, genetic information, or citizenship within the limits imposed by federal, state, and local laws and regulations. The Laboratory is also committed to making our workplace accessible to individuals with disabilities and will provide reasonable accommodations, upon request, for individuals to participate in the application and hiring process. To request such an accommodation, please send an email to [email protected] or call (505)-664-6947. Apply
Salary Context
This $128K-$215K range is above the median for Data Scientist roles in our dataset (median: $161K across 138 roles with salary data).
View full Data Scientist salary data →Role Details
About This Role
Data Scientists extract insights and build predictive models from data. In the AI era, many roles now include LLM-powered analytics, automated reporting, and integration with generative AI tools. The role has evolved from 'the person who runs SQL queries' to 'the person who builds AI-powered data products.'
Modern data science roles fall into two camps: analytics-focused (insights, dashboards, experimentation) and ML-focused (building predictive models, recommendation systems, NLP features). The best data scientists can operate in both modes. The AI shift means that even analytics-focused roles now involve building automated insight pipelines using LLMs, going well beyond one-off reports.
Across the 9,079 AI roles we're tracking, Data Scientist positions make up 4% of the market. At Los Alamos National Laboratory, this role fits into their broader AI and engineering organization.
Data Scientist roles remain in high demand, though the definition keeps shifting. Companies increasingly want candidates who can bridge traditional statistics with modern ML and LLM capabilities. The 'pure insights' data scientist role is consolidating into analytics engineering, while the 'build models' data scientist role is merging with ML engineering.
What the Work Looks Like
A typical week includes: analyzing experiment results for a product feature launch, building a predictive model for customer churn, creating an automated reporting pipeline using LLM-powered summarization, presenting insights to stakeholders, and cleaning data (always cleaning data). The ratio of analysis to engineering varies by company, but expect both.
Data Scientist roles remain in high demand, though the definition keeps shifting. Companies increasingly want candidates who can bridge traditional statistics with modern ML and LLM capabilities. The 'pure insights' data scientist role is consolidating into analytics engineering, while the 'build models' data scientist role is merging with ML engineering.
Skills Required
Python, SQL, and statistical modeling are the foundation. Increasingly, roles want experience with LLMs for data analysis, automated insight generation, and building AI-powered data products. Familiarity with cloud data platforms (Snowflake, BigQuery, Databricks) and ML frameworks (scikit-learn, PyTorch) covers most job requirements.
Experimentation design and causal inference are underrated skills that separate strong candidates. Companies care about whether their product changes cause improvements, and can distinguish causation from correlation. A/B testing methodology, Bayesian statistics, and the ability to communicate uncertainty to non-technical stakeholders are high-value skills.
Good postings specify the data stack, the types of problems you'll work on, and the team structure. Look for companies that differentiate between analytics and ML data science. Vague 'data scientist' postings that list every skill under the sun usually mean the company doesn't know what they need.
Compensation Benchmarks
Data Scientist roles pay a median of $190,000 based on 1,503 positions with disclosed compensation. Mid-level AI roles across all categories have a median of $193,200. This role's midpoint ($171K) sits 10% below the category median. Disclosed range: $128K to $215K.
Across all AI roles, the market median is $213,000. Top-quartile compensation starts at $263,911. The 90th percentile reaches $318,400. For comparison, the highest-paying categories include AI Safety ($256,000) and Research Engineer ($252,000). By seniority level: Entry: $111,000; Mid: $193,200; Senior: $229,300; Director: $272,300; VP: $235,830.
Los Alamos National Laboratory AI Hiring
Los Alamos National Laboratory has 3 open AI roles right now. They're hiring across Data Scientist, AI/ML Engineer. Based in Los Alamos, NM, US. Compensation range: $215K - $215K.
Location Context
Across all AI roles, 6% (531 positions) offer remote work, while 8,478 require on-site attendance. Top AI hiring metros: New York (3,078 roles, $221,000 median); San Francisco (2,646 roles, $261,000 median); Los Angeles (1,486 roles, $210,000 median).
Career Path
Common paths into Data Scientist roles include Data Analyst, Statistician, Quantitative Researcher.
From here, career progression typically leads toward Senior Data Scientist, ML Engineer, AI Product Manager.
Start with statistics and SQL. Build a real analysis project on public data that demonstrates insight generation alongside model building. The market values data scientists who can communicate findings clearly to business stakeholders. If you want to move toward ML engineering, invest in software engineering fundamentals and production deployment skills.
What to Expect in Interviews
Interviews combine statistics, coding, and business acumen. SQL is almost always tested, often with complex joins and window functions. Expect a case study round where you're given a business problem and asked to design an analysis plan. Coding rounds focus on pandas, statistical modeling, and visualization. The strongest differentiator is how well you communicate insights to non-technical stakeholders during presentation rounds.
When evaluating opportunities: Good postings specify the data stack, the types of problems you'll work on, and the team structure. Look for companies that differentiate between analytics and ML data science. Vague 'data scientist' postings that list every skill under the sun usually mean the company doesn't know what they need.
AI Hiring Overview
The AI job market has 9,079 open positions tracked in our dataset. By seniority: 106 entry-level, 4,384 mid-level, 3,343 senior, and 1,246 leadership roles (Director, VP, C-Level). Remote roles make up 6% of the market (531 positions). The remaining 8,478 roles require on-site or hybrid attendance.
The market median for AI roles is $213,000. Top-quartile compensation starts at $263,911. The 90th percentile reaches $318,400. Highest-paying categories: AI Safety ($256,000 median, 66 roles); Research Engineer ($252,000 median, 457 roles); AI Engineering Manager ($251,622 median, 55 roles).
Data Scientist roles remain in high demand, though the definition keeps shifting. Companies increasingly want candidates who can bridge traditional statistics with modern ML and LLM capabilities. The 'pure insights' data scientist role is consolidating into analytics engineering, while the 'build models' data scientist role is merging with ML engineering.
The AI Job Market Today
The AI job market spans 9,079 open positions across 16 role categories. The largest categories by volume: AI/ML Engineer (6,862), AI Product Manager (453), AI Software Engineer (451). These three account for the majority of open positions, though smaller categories often have higher per-role compensation because of specialized skill requirements.
The seniority mix tells a story about where AI teams are in their maturity. Entry-level roles (106) are outnumbered by mid-level (4,384) and senior (3,343) positions, reflecting that most companies are past the 'build a team from scratch' phase and need experienced engineers who can ship production systems. Leadership roles (Director, VP, C-Level) total 1,246 positions, representing the bottleneck between technical execution and organizational strategy.
Remote work availability sits at 6% of all AI roles (531 positions), with 8,478 requiring on-site or hybrid attendance. The remote share has stabilized after the post-pandemic correction. Senior and specialized roles (Research Scientist, ML Architect) are more likely to be remote-eligible than entry-level positions, partly because experienced hires have more negotiating power and partly because these roles require less hands-on mentorship.
AI compensation is structured in clear tiers. The market median sits at $213,000. Top-quartile roles start at $263,911, and the 90th percentile reaches $318,400. These figures include base salary with disclosed compensation. Total compensation (including equity, bonuses, and sign-on) runs 20-40% higher at companies that offer those components.
Category matters for compensation. AI Safety roles lead at $256,000 median, while Prompt Engineer roles sit at $141,440. The spread between highest and lowest-paying categories reflects the premium on specialized technical skills versus broader analytical roles.
The most in-demand skills across all AI postings: Python (1,649 postings), Aws (898 postings), Azure (723 postings), Rag (671 postings), Gcp (536 postings), Pytorch (434 postings), Claude (410 postings), Kubernetes (406 postings). Python dominates, appearing in the vast majority of role descriptions regardless of category. Cloud platform experience (AWS, GCP, Azure) is the second most common requirement. The newer entrants to the top skills list (RAG, vector databases, LLM APIs) reflect the shift from traditional ML toward generative AI applications.
Frequently Asked Questions
Get Weekly AI Career Intelligence
Salary data, skills demand, and market signals from 16,000+ AI job postings. Every Monday.