Interested in this AI/ML Engineer role at College Board?
Apply Now →Skills & Technologies
About This Role
Director, Applied AI Assessment Solutions
College Board – Assessment Design and Development | Learning & Assessment
Location: This is a fully remote role. Candidates who live near CB offices have the option of being fully remote or hybrid (Tuesday and Wednesday in office).
Type: This is a full-time position
About the Team
Assessment Design & Development (AD&D) is Learning & Assessment’s content engine—the department responsible for the item and assessment development that powers the SAT Suite. AD&D is expanding its mandate: beyond sustaining operational excellence for SAT, the department is building the shared infrastructure to scale content production across programs and designing the new item types and assessment formats the future of assessments demands.
Applied AI Assessment Solutions is AD&D’s team dedicated to this work: leading the shared strategy, standards, systems design, and prototype tooling that let every College Board program accelerate item development without compromising quality. The team operates as an internal, enterprise-wide consultancy: Directors, Applied AI Assessment Solutions, work across College Board’s assessment programs—including the SAT Suite, AP, ACCUPLACER, CLEP, and PAA—partnering closely with each program’s content teams to design solutions that help programs accelerate and scale item development.
About the Opportunity
As a Director, Applied AI Assessment Solutions, you translate deep item-development expertise into working AI solutions for the programs you support—applying hands-on fluency in item development and review, automated item generation (AIG), retrieval-augmented generation (RAG), prompt and context engineering, agentic workflow design, and systematic output evaluation to accelerate assessment content development across College Board’s programs. Working as part of an internal consultancy, you help identify and scope the engagements where your expertise will have the greatest impact, partnering with the Executive Director and program stakeholders to assess opportunity, negotiate scope, and define what a successful engagement looks like before work begins. During engagements, you partner with a program’s content team representatives to design and build point solutions that address specific production challenges. You also contribute to longer-range AI-enablement initiatives to integrate applied AI functionality into content authoring and item banking systems.
Across your engagements, you’ll play a crucial role in shaping how College Board’s programs leverage new tools to support their content teams, ethically and effectively integrating AI technology in assessment creation and review. With your team, you work to maximize ROI and efficiency of each tool, minimizing duplicative efforts across the enterprise, while meeting each program’s unique requirements.
In this role, you will:
Program AI Solution Design & Implementation (60%)
- Lead end-to-end delivery for assigned CB assessment program engagements: apply deep item development and AI expertise to define the problem and requirements, design and build the AI tools, prompts, and workflows, and deliver a working solution along with a training and adoption plan that partner teams can execute against.
- Apply expert judgment throughout the iterative building process—evaluating output at each iteration against the standards of an advanced domain subject matter expert, catching construct misalignment, flawed reasoning, bias, and factual errors early enough to correct course, and refining prompts, context, and workflows accordingly—and own final quality sign-off on tools before they are delivered to partner teams.
- Establish the quality standards and evaluation criteria used to assess AI-generated content systematically, translating domain expertise into defined rubrics, benchmarks, or scoring frameworks that others on the team can apply consistently.
- Identify and articulate requirements for systems-based solutions requiring input and resources from additional stakeholders.
- Diagnose gaps and document conditions for scaling enablement, ranging from culture and staffing, process improvements, and technological capabilities and use these findings to shape the team's engagement priorities and each program's roadmap for AI-enabled item development.
Agentic Workflow Contribution (20%)
- Work within and across teams to define agentic workflow standards and inputs, contributing assessment expertise and on-the-ground evidence from across your program engagements to inform the broader agentic-human hybrid architecture the Applied AI Assessment Solutions team leads with partners in Technology and Digital Product.
- Surface patterns, gaps, and opportunities from your program engagements that inform enterprise-wide strategy and roadmaps.
- Represent the needs and constraints of the programs you support in cross-program discussions about shared tooling and methodology.
Training, Adoption & Change Management (10%)
- Design and deliver trainings that build content teams’ fluency and comfort with AI-assisted and agentic workflows across the programs you support, gathering and responding to feedback to improve adoption.
- Develop process documentation so program teams can adopt new tools effectively and responsibly.
- Establish and document impact metrics to measure and report program progress toward targets related to item acceleration, production quality, or other defined goals.
- Uphold College Board’s standards of excellence in assessment design and development, ensuring AI-enablement centers human content and assessment expertise—and never compromises the quality of any program’s assessments.
Professional Growth & Field Currency (10%)
- Continue learning and improving to stay current in both your content domain and applied AI so your practice keeps pace with rapidly evolving fields.
- Experiment hands-on with emerging AI tools, models, and techniques to build informed judgment about their strengths, limitations, and fit for assessment content development.
- Regularly present what you learn with your team and internal communities of practice, keeping yours and your team’s standards and practices current with frontier capabilities.
- Attend and represent the organization at relevant external professional meetings and conferences.
About You
You have:
- 7+ years of experience in standardized assessment item development across a range of item types (multiple-choice, free response, and discrete and set-based items), with advanced, expert-level command of item-writing craft and construct alignment—including the ability to quickly identify flaws, gaps in reasoning, and errors in AI-generated content.
- An infectious enthusiasm for experimentation and new technology; you are an early adopter, naturally drawn to testing new tools, models, and approaches.
- Deep subject-matter expertise in assessment of one of the following content domains: Reading Comprehension, Writing, Math, or Science.
- 3+ years of hands-on experience applying AI-assisted or agentic techniques—including automated item generation (AIG), retrieval-augmented generation (RAG), prompt engineering, and context engineering—within professional or assessment-development workflows.
- Demonstrated experience bringing ideas for process or technology improvements to adoption, with measurable impact on the efficiency and/or quality of item or content production.
- Working knowledge of large language model capabilities and limitations—including common failure modes such as hallucination, context-window constraints, and training-data staleness—sufficient to diagnose why a pipeline is producing flawed output and reason about a fix.
- Experience designing or working within agentic and multi-agent workflows, such as generator/reviewer or generator/critic patterns, tool-calling, and chained or orchestrated prompts.
- Experience with vibe coding (or coding) and rapid prototyping, with demonstrated ability to independently build working scripts, tools, or interactive prototypes to test new ideas quickly.
- Experience designing new processes, educating others, and enabling successful adoption of new tools, processes, and systems.
- Ability to bridge technical and non-technical conversations; translating user needs into technical requirements and technical concepts into user-accessible strategies and practices.
- Proven ability to collaborate effectively with cross-functional teams to drive strategic initiatives and achieve organizational goals.
- A relationship-building approach that’s motivated by a strong, user-centric orientation.
- Excellent organizational and planning skills, with the ability to manage multiple tasks and maintain close attention to detail.
- Demonstrated ability to implement processes and procedures, manage project plans and schedules, and maintain tracking documentation.
- Experience with the item- and test-development life cycle, digital item authoring and review tools, and content management / item banking systems.
- Comfort working with ambiguity in a fast-paced environment where priorities may adjust as new information or challenges arise.
- An ability to travel 6-8 times per year.
- A master’s degree or equivalent education and experience is strongly preferred.
All roles at College Board require:
- A passion for expanding educational and career opportunities and mission-driven work.
- Curiosity and enthusiasm for emerging technologies, with a willingness to experiment with and adopt new AI-driven solutions and comfort with learning and applying new digital tools independently and proactively.
- Clear and concise communication skills, written and verbal.
- A learner’s mindset and a commitment to growth: welcoming diverse perspectives, giving and receiving timely, respectful feedback, and continuously improving through iterative learning and user input.
- A drive for impact and excellence: solving complex problems, making data-informed decisions, prioritizing what matters most, and continuously improving through learning, user input, and external benchmarking.
- A collaborative and empathetic approach: working across differences, fostering trust, and contributing to a culture of shared success.
- Authorization to work in the United States.
About Our Process
- Application review will begin immediately and will continue until the position is filled. This role is expected to accept applications for a minimum of 5 business days.
While the hiring process may vary, it generally includes: resume and application submission, recruiter phone/video screen, hiring manager interview, performance exercise such as item review and analysis, a panel interview, a conversation with leadership and reference checks. *
What We Offer
At College Board, we offer more than a paycheck- we provide a meaningful career, a supportive team, and a comprehensive package designed to help you thrive. We’re a self-sustaining nonprofit that believes in fair and competitive compensation grounded in your qualifications, experience, impact, and the market.
A Thoughtful Approach to Compensation
- The hiring range for this role is $88,000 - $135,000.
- Your exact salary will depend on your location, experience, and how your background compares to others in similar roles at the College Board.
- We aim to make our best offer upfront, rooted in fairness, transparency, and market data.
- We adjust salaries by location to ensure fairness, no matter where you live.
#LI-Remote
#LI-MW1
Salary Context
This $88K-$135K range is in the lower quartile for AI/ML Engineer roles in our dataset (median: $175K across 1356 roles with salary data).
View full AI/ML Engineer salary data →Role Details
About This Role
AI/ML Engineers build and deploy machine learning models in production. They work across the full ML lifecycle: data pipelines, model training, evaluation, and serving infrastructure. The role has evolved significantly over the past two years. Where ML Engineers once spent most of their time on model architecture, the job now tilts heavily toward inference optimization, cost management, and integrating LLM capabilities into existing systems. Companies want engineers who can ship production systems, and the experimenter-only role is fading fast.
Day-to-day, you're writing training pipelines, debugging data quality issues, setting up evaluation frameworks, and figuring out why your model performs differently in staging than it did on your dev set. The best ML engineers are obsessive about reproducibility and measurement. They instrument everything. They know that a model is only as good as the data feeding it and the infrastructure serving it.
Across the 8,943 AI roles we're tracking, AI/ML Engineer positions make up 75% of the market. At College Board, this role fits into their broader AI and engineering organization.
Demand for AI/ML Engineers has been strong and consistent. Unlike some AI roles that spike with hype cycles, ML engineering is a foundational need. Every company deploying AI models needs people who can keep them running, and the gap between research prototypes and production systems keeps growing.
What the Work Looks Like
A typical week might include: debugging a data pipeline that's silently dropping 3% of training examples, running A/B tests on a new model version, writing documentation for a feature flag system that lets you roll back model deployments, and reviewing a junior engineer's PR for a new evaluation metric. Meetings tend to be cross-functional since ML touches product, engineering, and data teams.
Demand for AI/ML Engineers has been strong and consistent. Unlike some AI roles that spike with hype cycles, ML engineering is a foundational need. Every company deploying AI models needs people who can keep them running, and the gap between research prototypes and production systems keeps growing.
Skills Required
Python and PyTorch dominate the requirements. Most roles expect experience with cloud platforms (AWS, GCP, or Azure) and familiarity with ML frameworks like TensorFlow or JAX. RAG (Retrieval-Augmented Generation) has become a top-3 skill requirement as companies integrate LLMs into their products. Docker and Kubernetes show up in about a third of postings, reflecting the production focus of the role.
Beyond the core stack, employers increasingly want experience with experiment tracking tools (MLflow, Weights & Biases), feature stores, and vector databases. Fine-tuning experience is valuable but less common than you'd think from reading Twitter. Most production LLM work is RAG and prompt engineering, not fine-tuning. If you have both, you're in a strong position.
Companies that are serious about AI/ML hiring tend to post specific infrastructure details in the job description: the frameworks they use, their model serving stack, their data pipeline tools. Vague postings that just say 'ML experience required' without specifics are often companies that haven't figured out what they need yet.
Compensation Benchmarks
AI/ML Engineer roles pay a median of $212,000 based on 11,866 positions with disclosed compensation. Director-level AI roles across all categories have a median of $272,250. This role's midpoint ($111K) sits 47% below the category median. Disclosed range: $88K to $135K.
Across all AI roles, the market median is $213,380. Top-quartile compensation starts at $264,148. The 90th percentile reaches $319,310. For comparison, the highest-paying categories include AI Safety ($253,100) and Research Engineer ($252,000). By seniority level: Entry: $110,408; Mid: $193,200; Senior: $228,700; Director: $272,250; VP: $235,000.
College Board AI Hiring
College Board has 1 open AI role right now. They're hiring across AI/ML Engineer. Based in Remote, US. Compensation range: $135K - $135K.
Remote Work Context
Remote AI roles pay a median of $185,906 across 2,124 positions. About 6% of all AI roles offer remote work.
Career Path
Common paths into AI/ML Engineer roles include Data Scientist, Software Engineer, Research Engineer.
From here, career progression typically leads toward ML Architect, AI Engineering Manager, Principal ML Engineer.
The fastest path into ML engineering is through software engineering with a self-directed ML education. A CS degree helps, but production engineering skills matter more than academic credentials. Build something that works, deploy it, and measure it. That portfolio project is worth more than a Coursera certificate. For career growth, the fork comes around the senior level: go deep on technical complexity (staff/principal track) or move into managing ML teams.
What to Expect in Interviews
Expect system design questions around ML pipelines: how you'd build a training pipeline for a specific use case, handle data drift, or design A/B testing infrastructure for model deployments. Coding rounds typically involve Python, with emphasis on data manipulation (pandas, numpy) and algorithm implementation. Take-home assignments often ask you to build an end-to-end ML pipeline from raw data to deployed model.
When evaluating opportunities: Companies that are serious about AI/ML hiring tend to post specific infrastructure details in the job description: the frameworks they use, their model serving stack, their data pipeline tools. Vague postings that just say 'ML experience required' without specifics are often companies that haven't figured out what they need yet.
AI Hiring Overview
The AI job market has 8,943 open positions tracked in our dataset. By seniority: 175 entry-level, 4,132 mid-level, 3,268 senior, and 1,368 leadership roles (Director, VP, C-Level). Remote roles make up 6% of the market (521 positions). The remaining 8,351 roles require on-site or hybrid attendance.
The market median for AI roles is $213,380. Top-quartile compensation starts at $264,148. The 90th percentile reaches $319,310. Highest-paying categories: AI Safety ($253,100 median, 57 roles); Research Engineer ($252,000 median, 419 roles); AI Engineering Manager ($251,622 median, 51 roles).
Demand for AI/ML Engineers has been strong and consistent. Unlike some AI roles that spike with hype cycles, ML engineering is a foundational need. Every company deploying AI models needs people who can keep them running, and the gap between research prototypes and production systems keeps growing.
The AI Job Market Today
The AI job market spans 8,943 open positions across 16 role categories. The largest categories by volume: AI/ML Engineer (6,739), AI Product Manager (403), AI Software Engineer (402). These three account for the majority of open positions, though smaller categories often have higher per-role compensation because of specialized skill requirements.
The seniority mix tells a story about where AI teams are in their maturity. Entry-level roles (175) are outnumbered by mid-level (4,132) and senior (3,268) positions, reflecting that most companies are past the 'build a team from scratch' phase and need experienced engineers who can ship production systems. Leadership roles (Director, VP, C-Level) total 1,368 positions, representing the bottleneck between technical execution and organizational strategy.
Remote work availability sits at 6% of all AI roles (521 positions), with 8,351 requiring on-site or hybrid attendance. The remote share has stabilized after the post-pandemic correction. Senior and specialized roles (Research Scientist, ML Architect) are more likely to be remote-eligible than entry-level positions, partly because experienced hires have more negotiating power and partly because these roles require less hands-on mentorship.
AI compensation is structured in clear tiers. The market median sits at $213,380. Top-quartile roles start at $264,148, and the 90th percentile reaches $319,310. These figures include base salary with disclosed compensation. Total compensation (including equity, bonuses, and sign-on) runs 20-40% higher at companies that offer those components.
Category matters for compensation. AI Safety roles lead at $253,100 median, while Prompt Engineer roles sit at $141,440. The spread between highest and lowest-paying categories reflects the premium on specialized technical skills versus broader analytical roles.
The most in-demand skills across all AI postings: Python (1,422 postings), Aws (808 postings), Azure (638 postings), Rag (555 postings), Gcp (457 postings), Pytorch (433 postings), Claude (341 postings), Kubernetes (337 postings). Python dominates, appearing in the vast majority of role descriptions regardless of category. Cloud platform experience (AWS, GCP, Azure) is the second most common requirement. The newer entrants to the top skills list (RAG, vector databases, LLM APIs) reflect the shift from traditional ML toward generative AI applications.
Frequently Asked Questions
Get Weekly AI Career Intelligence
Salary data, skills demand, and market signals from 16,000+ AI job postings. Every Monday.