Interested in this Research Scientist role at Datadog?
Apply Now →Skills & Technologies
About This Role
Team description
At Datadog, AI agents are becoming first\-class consumers of observability, security, and software delivery data — from third\-party coding agents like Claude Code, Cursor, and Copilot, to our own Bits SRE, Bits Assistant, and Bits Dev Agent. The Agentic Interfaces team owns the platform that connects these agents to Datadog: the MCP Server, the tools and retrieval surfaces agents call into, and — critically — the evaluation systems that tell us whether an agent's experience on Datadog data is actually getting better over time.
This role is about that last piece. We're hiring a Staff Applied Scientist to define what "good" means for an Agentic interface at Datadog and to build the measurement systems that make it true. "Good" isn't one number — it spans answer quality, tool\-selection accuracy, retrieval relevance, latency, token cost, and end\-to\-end agent success on real customer workflows. You'll design the evals, build the datasets, define the metrics, and partner with the AI engineers on the team to land the platform that lets every product group at Datadog ship integrations that are demonstrably better release over release.
The space is full of open research questions. How do you evaluate an agent end\-to\-end when the trajectory is non\-deterministic? How do you score tool selection when the tool catalog has hundreds of entries and grows weekly? How do you build a measurement system that catches regressions across first\-party and third\-party agents at once, without each team writing their own harness? If those are the problems you want to spend your time on, come build this with us.
*Datadog values people from all walks of life. We understand not everyone will meet all the above qualifications on day one. That's okay. If you're passionate about technology and want to grow your skills, we encourage you to apply.*
What You'll Do:
- Own the evaluation strategy for Datadog's AI agent integrations. Define the metrics — offline and online, quality and cost, single\-turn and trajectory\-level — that the team and the broader organization optimize against.
- Build the eval datasets, golden traces, and regression harnesses that catch quality changes before they hit customers, and make those assets reusable by every team contributing tools to the platform.
- Drive measurable improvements to retrieval relevance, tool\-selection accuracy, and context efficiency, partnering closely with the AI engineers on the team who build the underlying platform.
- Run applied research on the open problems in agent–data interaction: tool selection under large catalogs, multi\-turn agent evaluation, grounding and hallucination control on live telemetry, cost/quality tradeoffs at scale.
- Partner with the Bits SRE, Bits Assistant, and Bits Dev Agent teams so first\-party agents benefit from the same measurement substrate as third\-party integrations, and so learnings move freely in both directions.
- Provide technical leadership across the Agentic Interfaces team and the broader organization through design reviews, working groups, and mentorship, and represent the team externally through talks, blog posts, and contributions to the open agent ecosystem.
Who You Are:
- You have a BS/MS/PhD in a scientific field, or equivalent experience.
- 10\+ years of relevant engineering or applied science experience, including time as a technical lead.
- Proven track record of leading ML or GenAI initiatives in a product\-driven environment, from research through production.
- Significant experience with evaluation, experimentation, or measurement of ML systems at scale.
- You bring a strong product mindset and are comfortable driving initiatives across cross\-functional teams.
- You thrive in ambiguity and can make sound technical calls when the path isn't yet defined.
Benefits and Growth:
- New hire stock equity (RSUs) and employee stock purchase plan (ESPP)
- Continuous professional development, product training, and career pathing
- An inclusive company culture, giving programs, and the ability to join our Community Guilds (Datadog employee resource groups)
- Competitive global benefits and global Spring Health benefits for employees and dependents age 6\+
\#LI\-Onsite
About Datadog:
Datadog is the leading observability and security platform for the AI era, providing businesses with unified visibility across the technology stack to manage complexity at scale. It brings applications, infrastructure, data, models, and security into one place, using AI to detect and resolve issues before they impact customers. Trusted globally by Fortune 500 companies and high\-growth AI leaders, Datadog enables businesses to move faster with clarity and confidence. Learn more about \#DatadogLife on Instagram, LinkedIn, and Datadog Learning Center.
Equal Opportunity at Datadog:
Datadog is proud to offer equal employment opportunity to everyone regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, citizenship, marital status, disability, gender identity, veteran status, and other characteristics protected by law. We also consider qualified applicants regardless of criminal histories, consistent with legal requirements. Here are our Candidate Legal Notices for your reference.
Datadog endeavors to make our Careers Page accessible to all users. If you would like to contact us regarding the accessibility of our website or need assistance completing the application process, please complete this form. This form is for accommodation requests only and cannot be used to inquire about the status of applications.
Privacy and AI Guidelines:
Any information you submit to Datadog as part of your application will be processed in accordance with Datadog's Applicant and Candidate Privacy Notice. For information on our AI policy, please visit Interviewing at Datadog AI Guidelines.
Salary Context
This $276K-$345K range is above the 75th percentile for Research Scientist roles in our dataset (median: $196K across 87 roles with salary data).
Role Details
About This Role
Research Scientists push the boundaries of what AI can do. They design experiments, develop novel architectures, publish papers, and translate research breakthroughs into production capabilities. This is where the fundamental advances happen, from attention mechanisms to diffusion models to reasoning chains.
The work is intellectually demanding and often ambiguous. You might spend months on an approach that doesn't pan out. The best research scientists combine deep mathematical intuition with engineering pragmatism. They know when to go deep on theory and when to run experiments. They read papers voraciously and can spot incremental contributions from genuine breakthroughs.
Across the 3,739 AI roles we're tracking, Research Scientist positions make up 3% of the market. At Datadog, this role fits into their broader AI and engineering organization.
Research Scientist roles are concentrated at major AI labs (OpenAI, Anthropic, Google DeepMind, Meta FAIR) and well-funded AI startups. The competition is intense. PhD is effectively required for most positions, and publication track record matters. Compensation is among the highest in AI, reflecting both the scarcity of talent and the strategic importance of research breakthroughs.
What the Work Looks Like
A typical week includes: reading and discussing recent papers with your team, designing and running experiments on multi-GPU clusters, analyzing results and iterating on hypotheses, writing up findings for internal review or publication, and collaborating with engineering teams to productionize promising results. The ratio of thinking to coding is higher than in engineering roles.
Research Scientist roles are concentrated at major AI labs (OpenAI, Anthropic, Google DeepMind, Meta FAIR) and well-funded AI startups. The competition is intense. PhD is effectively required for most positions, and publication track record matters. Compensation is among the highest in AI, reflecting both the scarcity of talent and the strategic importance of research breakthroughs.
Skills Required
PhD strongly preferred for most roles. Deep expertise in a specific area (NLP, computer vision, reinforcement learning, multimodal) is expected. PyTorch is the standard. Publication track record matters. Strong mathematical foundations in linear algebra, probability, optimization, and information theory are assumed.
Beyond the fundamentals, companies value experience with large-scale distributed training, novel architecture design, and the ability to bridge theory and practice. Understanding of current frontier topics (reasoning, multimodal, long-context, alignment) is essential. Code quality matters more than many researchers expect. Labs want researchers who can implement their ideas cleanly.
Strong research postings specify the research area, mention the team you'd join, and describe the problems they're working on. They often list recent publications from the team. Vague 'AI research' postings without specifics usually mean the company wants to sound impressive but doesn't have a real research agenda.
Compensation Benchmarks
Research Scientist roles pay a median of $223,400 based on 223 positions with disclosed compensation. Senior-level AI roles across all categories have a median of $227,400. This role's midpoint ($310K) sits 39% above the category median. Disclosed range: $276K to $345K.
Across all AI roles, the market median is $200,000. Top-quartile compensation starts at $253,000. The 90th percentile reaches $307,500. For comparison, the highest-paying categories include AI Engineering Manager ($293,500) and AI Safety ($274,200). By seniority level: Entry: $97,380; Mid: $160,000; Senior: $227,400; Director: $243,000; VP: $250,000.
Datadog AI Hiring
Datadog has 6 open AI roles right now. They're hiring across Research Scientist, AI Product Manager, AI/ML Engineer. Based in New York, NY, US. Compensation range: $195K - $484K.
Location Context
AI roles in New York pay a median of $210,000 across 2,448 tracked positions. That's 5% above the national median.
Career Path
Common paths into Research Scientist roles include PhD Student, Research Engineer, Postdoc.
From here, career progression typically leads toward Research Lead, Distinguished Scientist, VP of Research.
The PhD is the entry point for most paths. Choose your advisor and research area carefully since they'll define your first industry position. Publish consistently, contribute to open-source projects in your area, and build relationships at conferences. Industry research offers better compensation and compute resources than academia, but the pressure to show product impact is real.
What to Expect in Interviews
Research interviews are multi-stage: a research talk (present your best paper), technical deep-dives on your methodology, and often a 'research proposal' exercise where you design an experiment to test a hypothesis. Coding rounds test implementation ability alongside theoretical knowledge. Be prepared to implement a paper from scratch and discuss the design choices the authors made. Strong candidates can critique papers constructively and identify gaps in experimental methodology.
When evaluating opportunities: Strong research postings specify the research area, mention the team you'd join, and describe the problems they're working on. They often list recent publications from the team. Vague 'AI research' postings without specifics usually mean the company wants to sound impressive but doesn't have a real research agenda.
AI Hiring Overview
The AI job market has 3,739 open positions tracked in our dataset. By seniority: 115 entry-level, 1,764 mid-level, 1,444 senior, and 416 leadership roles (Director, VP, C-Level). Remote roles make up 16% of the market (597 positions). The remaining 3,119 roles require on-site or hybrid attendance.
The market median for AI roles is $200,000. Top-quartile compensation starts at $253,000. The 90th percentile reaches $307,500. Highest-paying categories: AI Engineering Manager ($293,500 median, 31 roles); AI Safety ($274,200 median, 51 roles); Research Engineer ($260,000 median, 401 roles).
Research Scientist roles are concentrated at major AI labs (OpenAI, Anthropic, Google DeepMind, Meta FAIR) and well-funded AI startups. The competition is intense. PhD is effectively required for most positions, and publication track record matters. Compensation is among the highest in AI, reflecting both the scarcity of talent and the strategic importance of research breakthroughs.
The AI Job Market Today
The AI job market spans 3,739 open positions across 15 role categories. The largest categories by volume: AI/ML Engineer (2,650), Data Scientist (271), AI Software Engineer (252). These three account for the majority of open positions, though smaller categories often have higher per-role compensation because of specialized skill requirements.
The seniority mix tells a story about where AI teams are in their maturity. Entry-level roles (115) are outnumbered by mid-level (1,764) and senior (1,444) positions, reflecting that most companies are past the 'build a team from scratch' phase and need experienced engineers who can ship production systems. Leadership roles (Director, VP, C-Level) total 416 positions, representing the bottleneck between technical execution and organizational strategy.
Remote work availability sits at 16% of all AI roles (597 positions), with 3,119 requiring on-site or hybrid attendance. The remote share has stabilized after the post-pandemic correction. Senior and specialized roles (Research Scientist, ML Architect) are more likely to be remote-eligible than entry-level positions, partly because experienced hires have more negotiating power and partly because these roles require less hands-on mentorship.
AI compensation is structured in clear tiers. The market median sits at $200,000. Top-quartile roles start at $253,000, and the 90th percentile reaches $307,500. These figures include base salary with disclosed compensation. Total compensation (including equity, bonuses, and sign-on) runs 20-40% higher at companies that offer those components.
Category matters for compensation. AI Engineering Manager roles lead at $293,500 median, while Prompt Engineer roles sit at $142,800. The spread between highest and lowest-paying categories reflects the premium on specialized technical skills versus broader analytical roles.
The most in-demand skills across all AI postings: Python (1,929 postings), Aws (1,185 postings), Azure (869 postings), Rag (866 postings), Gcp (726 postings), Prompt Engineering (578 postings), Pytorch (575 postings), Claude (547 postings). Python dominates, appearing in the vast majority of role descriptions regardless of category. Cloud platform experience (AWS, GCP, Azure) is the second most common requirement. The newer entrants to the top skills list (RAG, vector databases, LLM APIs) reflect the shift from traditional ML toward generative AI applications.
Frequently Asked Questions
Get Weekly AI Career Intelligence
Salary data, skills demand, and market signals from 16,000+ AI job postings. Every Monday.