Interested in this AI Agent Developer role at Oracle?
Apply Now →Skills & Technologies
About This Role
The Senior Principal AI Agent / ML Software Engineer is a Senior Staff\-level, hands\-on technical leadership role responsible for defining, building, and operating next\-generation AI systems on Oracle Cloud Infrastructure (OCI). This person will set architecture and engineering direction for production\-grade agentic AI platforms, autonomous workflows, scalable inference infrastructure, and enterprise AI applications used in large\-scale, business\-critical environments.
This role requires a proven engineer who can translate ambiguous product and platform goals into durable technical strategy, lead multi\-team execution without direct authority, and remain deeply hands\-on in design, code, reviews, operations, and incident follow\-up. The ideal candidate combines deep distributed systems experience with practical AI\-native engineering, including orchestration of LLMs, tools, APIs, memory, retrieval, evaluation, guardrails, and cloud services. The expectation is to ship, scale, and operate reliable, secure, observable, and cost\-aware AI platform systems while raising the technical bar for engineers across the organization.
Responsibilities
- Serve as a senior technical owner for OCI AI platform capabilities, including agent execution, inference systems, model serving, AI workflow orchestration, evaluation, and observability.
- Design, architect, and deliver scalable agentic AI systems capable of reasoning, planning, tool use, workflow execution, multi\-step task orchestration, and safe human\-in\-the\-loop escalation.
- Build production\-grade services for tool calling, agent memory, context management, Model Context Protocol (MCP) integration, vector retrieval, multi\-agent coordination, policy enforcement, and evaluation.
- Lead architecture across distributed services optimized for low latency, high throughput, GPU efficiency, reliability, cost, operability, and secure multi\-tenant operation.
- Define service boundaries, APIs, data models, state management, consistency tradeoffs, failure modes, SLIs/SLOs, rollout strategies, and operational readiness criteria for AI platform services.
- Drive technical strategy across infrastructure, platform, security, data, and application engineering teams, converting broad goals into executable multi\-quarter plans and measurable milestones.
- Integrate AI agents securely and reliably with enterprise APIs, cloud services, databases, identity systems, secrets management, and external systems.
- Establish AgentOps and LLMOps practices for tracing, monitoring, eval suites, regression testing, experimentation, safety guardrails, prompt/tool versioning, and production reliability.
- Evaluate and operationalize emerging technologies in generative AI, agentic workflows, inference optimization, long\-context systems, reasoning models, AI developer tooling, and agentic\-first development.
- Drive engineering excellence through code reviews, design reviews, test strategy, deployment automation, incident analysis, documentation, and AI\-assisted development practices using tools such as Codex, Claude Code, Cursor, Copilot, or similar systems.
- Mentor Staff and senior engineers, raise architectural standards, and influence engineering practices across OCI without requiring direct management authority.
- Own critical production outcomes, including reliability, performance, security posture, cost efficiency, and supportability for the systems delivered.
Required Qualifications
- Bachelor's, Master's, or Ph.D. in Computer Science, AI/ML, Engineering, or a related field, or equivalent practical experience.
- 12\+ years of professional software engineering experience, including significant ownership of production systems; or equivalent experience demonstrating Senior Staff / Principal\-level impact.
- Proven track record as a Staff, Senior Staff, Principal, or equivalent technical leader influencing architecture and execution across multiple teams.
- Deep experience designing, building, and operating high\-scale distributed systems, cloud services, infrastructure platforms, or AI/ML platform services.
- Hands\-on experience with production AI systems, agentic AI applications, autonomous workflows, tool\-using agents, multi\-step orchestration, or multi\-agent systems.
- Practical experience with orchestration frameworks such as LangGraph, LangChain, CrewAI, AutoGen, LlamaIndex, or similar ecosystems.
- Deep understanding of LLM application patterns, including prompt design, structured outputs, function/tool calling, context management, RAG, memory, tool safety, and evaluation.
- Strong programming skills in Python and ability to contribute high\-quality production code, reviews, tests, and debugging in complex distributed environments.
- Strong expertise with Kubernetes, Docker, cloud\-native infrastructure, service\-to\-service communication, scalability, fault tolerance, observability, and performance analysis.
- Experience defining SLIs/SLOs, production readiness criteria, incident response practices, monitoring, tracing, experiments, and reliability programs for AI or distributed systems.
- Strong understanding of AI safety, governance, security, and operational risks for autonomous or semi\-autonomous systems, including data handling, access control, auditability, and human accountability.
- Excellent written and verbal communication, with demonstrated ability to lead technical direction, resolve ambiguity, and influence senior stakeholders.
Preferred Qualifications
- Experience optimizing large\-scale GPU inference or training workloads for latency, throughput, utilization, availability, and cost.
- Experience building or operating model serving, inference gateways, agent runtimes, workflow engines, developer platforms, or internal AI productivity platforms.
- Experience integrating AI systems with enterprise APIs, databases, cloud services, vector databases, embeddings, retrieval systems, identity systems, and policy enforcement layers.
- Experience with LLM fine\-tuning, long\-context systems, reasoning models, model routing, caching, batching, quantization, or emerging generative AI research.
- Experience building evaluation frameworks for agentic systems, including offline evals, online experiments, golden tasks, adversarial testing, regression gates, and observability dashboards.
- Experience using AI\-assisted software development tools such as Codex, Claude Code, Cursor, Copilot, or similar systems in large\-scale engineering environments.
- Track record of defining architectural standards, platform capabilities, or engineering practices adopted across multiple teams or organizations.
- Experience in enterprise, cloud infrastructure, regulated, security\-sensitive, or mission\-critical environments.
Role Details
About This Role
AI Agent Developers build autonomous systems that can reason, plan, and take actions. They design multi-step workflows, tool-use frameworks, and orchestration layers that let LLMs interact with external systems. This is the frontier of applied AI engineering.
Agent development is where the most interesting (and hardest) problems in applied AI live right now. Making an LLM answer a question is straightforward. Making it reliably execute a 15-step workflow that involves calling APIs, reading databases, making decisions, and recovering from errors is an unsolved problem. You're building systems that have to work despite the fact that the underlying model is non-deterministic.
Across the 3,823 AI roles we're tracking, AI Agent Developer positions make up 1% of the market. At Oracle, this role fits into their broader AI and engineering organization.
AI Agent Developer is one of the newest and fastest-growing AI role categories. The market is early but accelerating as companies move beyond simple chatbots toward AI systems that can take real actions. Compensation is high because the skill set is rare and the business impact is potentially enormous.
What the Work Looks Like
A typical week includes: designing the action space and tool definitions for a new agent use case, debugging why the agent chose the wrong action sequence on a specific input, building evaluation frameworks that test agent reliability across hundreds of scenarios, optimizing the prompt chain for cost and latency, and implementing safety guardrails to prevent the agent from taking destructive actions. The work is equal parts engineering and empirical science.
AI Agent Developer is one of the newest and fastest-growing AI role categories. The market is early but accelerating as companies move beyond simple chatbots toward AI systems that can take real actions. Compensation is high because the skill set is rare and the business impact is potentially enormous.
Skills Required
Deep experience with LLM APIs and agent frameworks (LangChain, CrewAI, AutoGen). Strong understanding of prompt engineering, function calling, and error handling for non-deterministic systems. Python is standard. Experience with orchestration patterns, state management, and workflow engines adds significant value.
The best agent developers think like systems engineers. They design for failure modes, build observability into every step, and understand that agent reliability is the product. Expertise in evaluation methodology for non-deterministic systems is the differentiator. Can you measure whether your agent works 'well enough'? Can you find the edge cases where it breaks?
Look for roles that describe specific agent use cases, mention evaluation methodology, and talk about production deployment. Early-stage companies exploring agents can be exciting, but be prepared for ambiguity. The most valuable roles are at companies that have already shipped a v1 and need to make it reliable.
Compensation Benchmarks
AI Agent Developer roles pay a median of $245,040 based on 106 positions with disclosed compensation. Senior-level AI roles across all categories have a median of $227,400.
Across all AI roles, the market median is $200,100. Top-quartile compensation starts at $253,500. The 90th percentile reaches $307,500. For comparison, the highest-paying categories include AI Engineering Manager ($275,000) and AI Safety ($274,200). By seniority level: Entry: $97,880; Mid: $165,000; Senior: $227,400; Director: $247,800; VP: $250,000.
Oracle AI Hiring
Oracle has 22 open AI roles right now. They're hiring across AI/ML Engineer, AI Agent Developer, AI Software Engineer, MLOps Engineer. Positions span US, Seattle, WA, US.
Location Context
AI roles in Seattle pay a median of $227,400 across 1,084 tracked positions. That's 14% above the national median.
Career Path
Common paths into AI Agent Developer roles include Software Engineer, LLM Engineer, Prompt Engineer.
From here, career progression typically leads toward AI Architect, Principal Engineer, Head of AI Engineering.
Build agents. That's the portfolio. Take an open-source agent framework, build something that completes a non-trivial multi-step task, evaluate it rigorously, and document what you learned about reliability, cost, and failure modes. The field is new enough that practical experience counts for more than credentials.
What to Expect in Interviews
Interviews focus on systems thinking and reliability engineering. Expect questions about agent architecture: how you'd design a multi-step workflow with error recovery, how you'd evaluate agent performance, and how you'd prevent agents from taking destructive actions. Coding exercises often involve building a simple agent with tool use and evaluating its behavior across different scenarios. Discussion of safety and guardrails is increasingly common.
When evaluating opportunities: Look for roles that describe specific agent use cases, mention evaluation methodology, and talk about production deployment. Early-stage companies exploring agents can be exciting, but be prepared for ambiguity. The most valuable roles are at companies that have already shipped a v1 and need to make it reliable.
AI Hiring Overview
The AI job market has 3,823 open positions tracked in our dataset. By seniority: 112 entry-level, 1,798 mid-level, 1,516 senior, and 397 leadership roles (Director, VP, C-Level). Remote roles make up 15% of the market (590 positions). The remaining 3,217 roles require on-site or hybrid attendance.
The market median for AI roles is $200,100. Top-quartile compensation starts at $253,500. The 90th percentile reaches $307,500. Highest-paying categories: AI Engineering Manager ($275,000 median, 41 roles); AI Safety ($274,200 median, 55 roles); Research Engineer ($260,000 median, 434 roles).
AI Agent Developer is one of the newest and fastest-growing AI role categories. The market is early but accelerating as companies move beyond simple chatbots toward AI systems that can take real actions. Compensation is high because the skill set is rare and the business impact is potentially enormous.
The AI Job Market Today
The AI job market spans 3,823 open positions across 15 role categories. The largest categories by volume: AI/ML Engineer (2,629), Data Scientist (322), AI Software Engineer (279). These three account for the majority of open positions, though smaller categories often have higher per-role compensation because of specialized skill requirements.
The seniority mix tells a story about where AI teams are in their maturity. Entry-level roles (112) are outnumbered by mid-level (1,798) and senior (1,516) positions, reflecting that most companies are past the 'build a team from scratch' phase and need experienced engineers who can ship production systems. Leadership roles (Director, VP, C-Level) total 397 positions, representing the bottleneck between technical execution and organizational strategy.
Remote work availability sits at 15% of all AI roles (590 positions), with 3,217 requiring on-site or hybrid attendance. The remote share has stabilized after the post-pandemic correction. Senior and specialized roles (Research Scientist, ML Architect) are more likely to be remote-eligible than entry-level positions, partly because experienced hires have more negotiating power and partly because these roles require less hands-on mentorship.
AI compensation is structured in clear tiers. The market median sits at $200,100. Top-quartile roles start at $253,500, and the 90th percentile reaches $307,500. These figures include base salary with disclosed compensation. Total compensation (including equity, bonuses, and sign-on) runs 20-40% higher at companies that offer those components.
Category matters for compensation. AI Engineering Manager roles lead at $275,000 median, while Prompt Engineer roles sit at $140,000. The spread between highest and lowest-paying categories reflects the premium on specialized technical skills versus broader analytical roles.
The most in-demand skills across all AI postings: Python (1,979 postings), Aws (1,190 postings), Azure (899 postings), Rag (839 postings), Gcp (726 postings), Pytorch (595 postings), Prompt Engineering (595 postings), Claude (540 postings). Python dominates, appearing in the vast majority of role descriptions regardless of category. Cloud platform experience (AWS, GCP, Azure) is the second most common requirement. The newer entrants to the top skills list (RAG, vector databases, LLM APIs) reflect the shift from traditional ML toward generative AI applications.
Frequently Asked Questions
Get Weekly AI Career Intelligence
Salary data, skills demand, and market signals from 16,000+ AI job postings. Every Monday.