Sr. Staff AI Engineer, AI Agent Platform (Agent & Evaluation Harness)

New York, NY Senior AI Agent Developer

Interested in this AI Agent Developer role at GEICO Norwalk?

Apply Now →

Skills & Technologies

AzureBedrockClaudeLlamaOpenaiPythonRagRlhf

About This Role

AI job market dashboard showing open roles by category

Why Join GEICO? At GEICO, we offer a rewarding career where your ambitions are met with endless possibilities.

Every day we honor our iconic brand by offering quality coverage to millions of customers and being there when they need us most. We thrive on relentless innovation to exceed our customers' expectations while making a real impact on local communities nationwide.

Founded in 1936, GEICO is a member of the Berkshire Hathaway family of companies and one of the largest auto insurers in the United States. When you join our company, we want you to feel valued, supported, and proud to work here. That's why we offer the GEICO Pledge: Great Company, Great Culture, Great Rewards, and Great Careers.

Sr. Staff AI Engineer, AI Agent Platform (Agent & Evaluation Harness) Why Join GEICO? GEICO is transforming how AI is built and deployed across the enterprise. As one of the largest insurers in the United States, we are investing heavily in next-generation AI platforms that empower more than 30,000 associates and enhance experiences for millions of customers.

We are looking for a Sr. Staff AI Engineer to push the frontier of what GEICO's AI Agent Platform can do, with a focus on two core systems: the agent harness that determines how agents plan, reason, use tools, and manage context, and the eval harness that tells us, with evidence, whether they are getting better. The field is moving faster than any job description can capture. Techniques that define best practice today, such as context engineering, MCP-based tool integration, skills, sub-agent orchestration, and LLM-as-judge evaluation, barely existed a few years ago and will look different a year from now. We are looking for someone who thrives in that environment and can move quickly from a new idea to a working prototype to a data-backed conclusion about whether it delivers value.

The Opportunity The AI Agent Platform team builds use-case-agnostic infrastructure that many business workflows across GEICO build on, from claims and underwriting to internal associate tools. In this role, you'll own the intelligence layer of that platform: the agent design patterns, context strategies, and evaluation methodology that make agents on the platform reliably good at their jobs.

You'll work alongside backend platform engineers who own durable execution, scaling, and operations. Your focus is agent behavior and quality, and the science of measuring and improving it, grounded in a solid understanding of distributed systems so that what works in an experiment also works at enterprise scale. Success here comes from experimentation: forming hypotheses, designing experiments, reading traces, and iterating until performance meets the bar for real business workflows.

The model is only the starting point. Your job is to find out what actually works, prove it, and make it reusable. What You Will Do Agent Harness

  • Design the core agent loop, including planning, reasoning, tool selection and calling, memory, and error recovery, as reusable patterns that generalize across many business workflows.
  • Develop context engineering strategies: prompt architecture, context window management, summarization and compaction, memory, and retrieval (RAG) integration.
  • Define how agents connect to tools and enterprise systems through the Model Context Protocol (MCP), including tool design, server patterns, and the conventions that make tools reliable for agents to use.
  • Design how agents discover, load, and apply skills (packaged instructions, scripts, and resources for specific tasks) so capabilities built once can be reused across many workflows.
  • Explore sub-agent and multi-agent patterns, and decide with evidence what belongs in the platform.
  • Evaluate new frontier and open-weight models as they are released, and understand their tradeoffs in capability, latency, and cost for platform use.

Evaluation Harness

  • Design evaluation methodology: task-level metrics, rubrics, golden datasets, LLM-as-judge graders, and human review, with automated graders calibrated against human judgment.
  • Build the eval harness that lets any team define, run, and compare evaluations for their agents, spanning offline benchmarks, simulated users and environments, and online quality monitoring.
  • Make eval results trustworthy by accounting for variance, statistical significance, dataset contamination, and grader bias.
  • Diagnose failure modes by analyzing traces and eval results, and turn qualitative observations into measurable hypotheses.

Experiment and Iterate

  • Run fast, well-designed experiments on prompts, architectures, models, and tools, owning the loop from hypothesis to result to decision.
  • Iterate until agent quality meets the bar for production business workflows, and help define what that bar should be.
  • Track the research frontier and industry practice, bring the best ideas into GEICO quickly, and filter out the hype.
  • Share what you learn through write-ups, internal talks, reusable patterns, and playbooks that raise the level of GenAI practice across the company.

Lead

  • Set technical direction for agent design and evaluation across the AI Agent Platform, in partnership with senior AI and engineering leaders.
  • Mentor engineers and data scientists on GenAI engineering practice and experimental rigor.
  • Partner with platform engineers, product managers, and business teams to turn workflow needs into measurable quality targets and generalized platform capabilities.

How You Work

  • Intellectual curiosity. You want to understand why something works, not just whether it does.
  • Speed to learn. A new model, paper, protocol, or framework drops, and you're productive with it within days.
  • Adaptability. You change direction when the evidence says so, and hold strong opinions loosely.
  • Empirical rigor. You trust data over intuition, and your intuition is well calibrated because of it.
  • Bias to action. You'd rather build a prototype and measure it than debate it in a meeting.

Minimum Qualifications

  • 10+ years of experience in software engineering, ML engineering, or applied science, with strong Python proficiency.
  • Tech-lead experience: owned the design and delivery of significant systems involving multiple engineers or teams.
  • Significant hands-on experience building GenAI applications and agentic systems with LLMs such as GPT, Claude, Llama, or Qwen, taken beyond prototype into real use.
  • Demonstrated experience designing and running experiments to improve LLM or agent performance, including defining metrics, building evaluation sets, analyzing results, and iterating.
  • Deep practical understanding of prompt and context engineering, tool calling, RAG, and agent architectures, including hands-on experience with MCP.
  • Solid distributed systems fundamentals, including concurrency, fault tolerance, latency and throughput tradeoffs, and API design, with experience building or working closely with production services at scale.
  • A track record of quickly learning and applying new techniques in a fast-moving field.
  • Demonstrated technical leadership influencing direction across teams.

Preferred Qualifications

  • Experience building evaluation frameworks or benchmarks for LLMs or agents, including LLM-as-judge methods and human annotation pipelines, using tools such as Inspect AI, Braintrust, DeepEval, or RAG.
  • Experience building MCP servers and tools, or authoring skills using the Agent Skills standard (or similar packaged agent capabilities) used across teams.
  • Experience with agent frameworks such as LangGraph, Microsoft Agent Framework, OpenAI Agents SDK, Claude Agent SDK, or Google ADK, and eval and observability tools such as Langfuse, LangSmith, Braintrust, or Arize Phoenix.
  • Background in ML or applied science, such as statistics and experimental design, fine-tuning, reinforcement fine-tuning (e.g., RLHF, RLVR), or model evaluation.
  • Experience with cloud AI platforms such as Microsoft Foundry (including Azure OpenAI and Foundry Agent Service) or Amazon Bedrock (including AgentCore).
  • Publications, open-source contributions, or public writing on GenAI or agents.
  • Understanding of AI security (e.g., prompt injection, tool and skill supply chain risks), safety, and responsible AI practices.

Why This Role Is Unique

  • Define how agents behave, and how their quality is measured, across one of the largest insurers in the United States.
  • Work hands-on with the newest models and techniques as they're released, not years later.
  • See your experiments turn directly into scalable platform capabilities used by teams across the enterprise.
  • Build AI capabilities that impact 30,000+ associates and help shape AI experiences used by millions of customers.
  • Influence AI strategy and architecture across the enterprise, partnering with senior AI and engineering leaders on the next generation of AI at GEICO.

Annual Salary $115,000.00 - $260,000.00

The above annual salary range is a general guideline. Multiple factors are taken into consideration to arrive at the final hourly rate/ annual salary to be offered to the selected candidate. Factors include, but are not limited to, the scope and responsibilities of the role, the selected candidate’s work experience, education and training, the work location as well as market and business considerations.

GEICO will consider sponsoring a new qualified applicant for employment authorization for this position.

The GEICO Pledge Great Company: Protecting customers through life’s twists and turns with innovation and integrity.

Great Careers: Personalized development programs, mentorship, and certification assistance.

Great Culture: Inclusive and collaborative culture rooted in shared success.

Great Rewards: Competitive pay, benefits, and flexibility to support your well-being and future.

The equal employment opportunity policy of the GEICO Companies provides for a fair and equal employment opportunity for all associates and job applicants regardless of race, color, religious creed, national origin, ancestry, age, gender, pregnancy, sexual orientation, gender identity, marital status, familial status, disability or genetic information, in compliance with applicable federal, state and local law. GEICO hires and promotes individuals solely on the basis of their qualifications for the job to be filled.

GEICO reasonably accommodates qualified individuals with disabilities to enable them to receive equal employment opportunity and/or perform the essential functions of the job, unless the accommodation would impose an undue hardship to the Company. This applies to all applicants and associates. GEICO also provides a work environment in which each associate is able to be productive and work to the best of their ability. We do not condone or tolerate an atmosphere of intimidation or harassment. We expect and require the cooperation of all associates in maintaining an atmosphere free from discrimination and harassment with mutual respect by and for all associates and applicants.

Role Details

Company GEICO Norwalk
Title Sr. Staff AI Engineer, AI Agent Platform (Agent & Evaluation Harness)
Location New York, NY
Experience Senior
Salary Not disclosed
Remote No

About This Role

AI Agent Developers build autonomous systems that can reason, plan, and take actions. They design multi-step workflows, tool-use frameworks, and orchestration layers that let LLMs interact with external systems. This is the frontier of applied AI engineering.

Agent development is where the most interesting (and hardest) problems in applied AI live right now. Making an LLM answer a question is straightforward. Making it reliably execute a 15-step workflow that involves calling APIs, reading databases, making decisions, and recovering from errors is an unsolved problem. You're building systems that have to work despite the fact that the underlying model is non-deterministic.

Across the 10,419 AI roles we're tracking, AI Agent Developer positions make up 1% of the market. At GEICO Norwalk, this role fits into their broader AI and engineering organization.

AI Agent Developer is one of the newest and fastest-growing AI role categories. The market is early but accelerating as companies move beyond simple chatbots toward AI systems that can take real actions. Compensation is high because the skill set is rare and the business impact is potentially enormous.

What the Work Looks Like

A typical week includes: designing the action space and tool definitions for a new agent use case, debugging why the agent chose the wrong action sequence on a specific input, building evaluation frameworks that test agent reliability across hundreds of scenarios, optimizing the prompt chain for cost and latency, and implementing safety guardrails to prevent the agent from taking destructive actions. The work is equal parts engineering and empirical science.

AI Agent Developer is one of the newest and fastest-growing AI role categories. The market is early but accelerating as companies move beyond simple chatbots toward AI systems that can take real actions. Compensation is high because the skill set is rare and the business impact is potentially enormous.

Skills Required

Azure (10% of roles) Bedrock (2% of roles) Claude (6% of roles) Llama (1% of roles) Openai (5% of roles) Python (23% of roles) Rag (9% of roles) Rlhf (1% of roles)

Deep experience with LLM APIs and agent frameworks (LangChain, CrewAI, AutoGen). Strong understanding of prompt engineering, function calling, and error handling for non-deterministic systems. Python is standard. Experience with orchestration patterns, state management, and workflow engines adds significant value.

The best agent developers think like systems engineers. They design for failure modes, build observability into every step, and understand that agent reliability is the product. Expertise in evaluation methodology for non-deterministic systems is the differentiator. Can you measure whether your agent works 'well enough'? Can you find the edge cases where it breaks?

Look for roles that describe specific agent use cases, mention evaluation methodology, and talk about production deployment. Early-stage companies exploring agents can be exciting, but be prepared for ambiguity. The most valuable roles are at companies that have already shipped a v1 and need to make it reliable.

Compensation Benchmarks

AI Agent Developer roles pay a median of $215,000 based on 252 positions with disclosed compensation. Senior-level AI roles across all categories have a median of $230,000.

Across all AI roles, the market median is $214,834. Top-quartile compensation starts at $266,000. The 90th percentile reaches $321,730. For comparison, the highest-paying categories include AI Safety ($274,500) and Research Engineer ($252,000). By seniority level: Entry: $114,400; Mid: $193,200; Senior: $230,000; Director: $272,100; VP: $235,000.

GEICO Norwalk AI Hiring

GEICO Norwalk has 3 open AI roles right now. They're hiring across AI Agent Developer, AI Product Manager, AI/ML Engineer. Positions span New York, NY, Palo Alto, CA.

Location Context

AI roles in New York pay a median of $222,480 across 3,785 tracked positions. That's 4% above the national median.

Career Path

Common paths into AI Agent Developer roles include Software Engineer, LLM Engineer, Prompt Engineer.

From here, career progression typically leads toward AI Architect, Principal Engineer, Head of AI Engineering.

Build agents. That's the portfolio. Take an open-source agent framework, build something that completes a non-trivial multi-step task, evaluate it rigorously, and document what you learned about reliability, cost, and failure modes. The field is new enough that practical experience counts for more than credentials.

What to Expect in Interviews

Interviews focus on systems thinking and reliability engineering. Expect questions about agent architecture: how you'd design a multi-step workflow with error recovery, how you'd evaluate agent performance, and how you'd prevent agents from taking destructive actions. Coding exercises often involve building a simple agent with tool use and evaluating its behavior across different scenarios. Discussion of safety and guardrails is increasingly common.

When evaluating opportunities: Look for roles that describe specific agent use cases, mention evaluation methodology, and talk about production deployment. Early-stage companies exploring agents can be exciting, but be prepared for ambiguity. The most valuable roles are at companies that have already shipped a v1 and need to make it reliable.

AI Hiring Overview

The AI job market has 10,419 open positions tracked in our dataset. By seniority: 117 entry-level, 4,975 mid-level, 3,840 senior, and 1,487 leadership roles (Director, VP, C-Level). Remote roles make up 6% of the market (580 positions). The remaining 9,776 roles require on-site or hybrid attendance.

The market median for AI roles is $214,834. Top-quartile compensation starts at $266,000. The 90th percentile reaches $321,730. Highest-paying categories: AI Safety ($274,500 median, 90 roles); Research Engineer ($252,000 median, 580 roles); AI Engineering Manager ($250,811 median, 72 roles).

AI Agent Developer is one of the newest and fastest-growing AI role categories. The market is early but accelerating as companies move beyond simple chatbots toward AI systems that can take real actions. Compensation is high because the skill set is rare and the business impact is potentially enormous.

The AI Job Market Today

The AI job market spans 10,419 open positions across 16 role categories. The largest categories by volume: AI/ML Engineer (7,849), AI Product Manager (516), AI Software Engineer (513). These three account for the majority of open positions, though smaller categories often have higher per-role compensation because of specialized skill requirements.

The seniority mix tells a story about where AI teams are in their maturity. Entry-level roles (117) are outnumbered by mid-level (4,975) and senior (3,840) positions, reflecting that most companies are past the 'build a team from scratch' phase and need experienced engineers who can ship production systems. Leadership roles (Director, VP, C-Level) total 1,487 positions, representing the bottleneck between technical execution and organizational strategy.

Remote work availability sits at 6% of all AI roles (580 positions), with 9,776 requiring on-site or hybrid attendance. The remote share has stabilized after the post-pandemic correction. Senior and specialized roles (Research Scientist, ML Architect) are more likely to be remote-eligible than entry-level positions, partly because experienced hires have more negotiating power and partly because these roles require less hands-on mentorship.

AI compensation is structured in clear tiers. The market median sits at $214,834. Top-quartile roles start at $266,000, and the 90th percentile reaches $321,730. These figures include base salary with disclosed compensation. Total compensation (including equity, bonuses, and sign-on) runs 20-40% higher at companies that offer those components.

Category matters for compensation. AI Safety roles lead at $274,500 median, while Prompt Engineer roles sit at $140,000. The spread between highest and lowest-paying categories reflects the premium on specialized technical skills versus broader analytical roles.

The most in-demand skills across all AI postings: Python (2,350 postings), Aws (1,173 postings), Azure (1,030 postings), Rag (986 postings), Gcp (714 postings), Pytorch (649 postings), Kubernetes (583 postings), Claude (581 postings). Python dominates, appearing in the vast majority of role descriptions regardless of category. Cloud platform experience (AWS, GCP, Azure) is the second most common requirement. The newer entrants to the top skills list (RAG, vector databases, LLM APIs) reflect the shift from traditional ML toward generative AI applications.

Frequently Asked Questions

Based on 252 roles with disclosed compensation, the median salary for AI Agent Developer positions is $215,000. Actual compensation varies by seniority, location, and company stage.
Deep experience with LLM APIs and agent frameworks (LangChain, CrewAI, AutoGen). Strong understanding of prompt engineering, function calling, and error handling for non-deterministic systems. Python is standard. Experience with orchestration patterns, state management, and workflow engines adds significant value.
About 6% of the 10,419 AI roles we track offer remote work. Remote availability varies by company and seniority level, with senior and leadership roles more likely to offer location flexibility.
GEICO Norwalk is among the companies actively hiring for AI and ML talent. Check our company profiles for detailed breakdowns of open roles, salary ranges, and hiring trends.
Common next steps from AI Agent Developer positions include AI Architect, Principal Engineer, Head of AI Engineering. Progression depends on whether you lean toward technical depth, people management, or product strategy.

Get Weekly AI Career Intelligence

Salary data, skills demand, and market signals from 16,000+ AI job postings. Every Monday.