Untitled

Properties

title: AI Evaluation Engineer
family: eval
maturity: growing
owns: Builds the evaluation systems that decide whether a model or agent should ship: eval harnesses, datasets, LLM-based graders, and metrics tightly coupled to development so AI behavior is measured and trusted before deployment.
skills: [eval harness design (Inspect, Promptfoo, Braintrust, LangSmith)][LLM-as-judge and grader calibration against human assessments][dataset curation and golden-set design][statistical metric design and eval analytics][production Python][CI integration of eval quality gates]
pay: low: 130000
high: 250000
currency: USD
basis: base
seniority: senior-ic
distinctFrom: Unlike an AgentOps engineer (who runs agents in production observability) or a context engineer (who builds the retrieval and memory environment), the eval engineer builds the measurement systems, harnesses, and graders that judge whether models and agents are good enough to ship.
source: Distyl AI 'AI Evaluation Engineer' posting (SF/NY), $130K-$250K
notes: Posted range $130K-$250K (basis not explicitly stated; read as base for an SF/NY IC engineer). Total comp for the discipline runs $230K-$650K+ (JobsByCulture 2026 career guide), with frontier labs at the top (Anthropic Research Engineer, Model Evaluations $320K-$485K; Magic MTS Evals $200K-$550K; Microsoft AI Evaluations Engineering). A distinct, fast-forming discipline.

title: AI Evaluation Engineer
family: eval
maturity: growing
owns: Builds the evaluation systems that decide whether a model or agent should ship: eval harnesses, datasets, LLM-based graders, and metrics tightly coupled to development so AI behavior is measured and trusted before deployment.
skills: [eval harness design (Inspect, Promptfoo, Braintrust, LangSmith)][LLM-as-judge and grader calibration against human assessments][dataset curation and golden-set design][statistical metric design and eval analytics][production Python][CI integration of eval quality gates]
pay: low: 130000
high: 250000
currency: USD
basis: base
seniority: senior-ic
distinctFrom: Unlike an AgentOps engineer (who runs agents in production observability) or a context engineer (who builds the retrieval and memory environment), the eval engineer builds the measurement systems, harnesses, and graders that judge whether models and agents are good enough to ship.
source: Distyl AI 'AI Evaluation Engineer' posting (SF/NY), $130K-$250K
notes: Posted range $130K-$250K (basis not explicitly stated; read as base for an SF/NY IC engineer). Total comp for the discipline runs $230K-$650K+ (JobsByCulture 2026 career guide), with frontier labs at the top (Anthropic Research Engineer, Model Evaluations $320K-$485K; Magic MTS Evals $200K-$550K; Microsoft AI Evaluations Engineering). A distinct, fast-forming discipline.