90% of US working hours are spent on tasks not covered well by AI benchmarks

Measuring model capabilities by job occupation is essential – for enterprises deciding which models fit their needs, for policymakers forecasting the economic impact of AI, and for Americans thinking about their careers. Unfortunately, this is a hard problem to solve.

We could just try averaging model performance on all benchmark prompts relevant to a given job. But as our Benchmark Map suggests, benchmarks cover some economic tasks much more intensively than others. A simple average would give too much weight to frequently benchmarked tasks.

O*NET offers a way to potentially reduce this bias. If we average model performance on occupation-relevant benchmark prompts weighted by O*NET classification, so that each O*NET task is equally weighted in our occupation average, we get a step closer to a number that represents real economic performance.

As a first step, we try this approach on Mercor’s APEX benchmarks for four occupations: investment banking, law, accounting, and software engineering. We compare O*NET-weighted scores to simple averages of model performance on tasks related to each occupation. The resulting weighted and unweighted scores are generally quite similar:

In this case, at least, O*NET weighting changes very little. Model performance on APEX prompts is similar across the O*NET tasks included in the weighted scores above, which limits the value of reweighting. However, less than ⅓ of the O*NET tasks relevant to the 4 occupations above have enough APEX coverage to be included in our weighted scores. We are hopeful that applying O*NET weighting to a much larger set of benchmarks will result in high-quality occupation-specific model scores, and are working toward that goal.

However, even if we incorporate tasks from dozens of benchmarks, several obstacles remain:

1. 01

   How do we navigate low statistical power?

   The sheer breadth of work done in the US economy means that for any O*NET occupation and task, we have relatively few benchmark items. This means that our estimates are inherently underpowered, and just one or two low-quality items can severely bias our scores.

2. 02

   How do we account for the varying difficulty and fidelity of tasks?

   Some benchmark items are highly saturated, whereas others are still unsolved. This is partly a function of time (as older benchmarks were designed for less performant models and simpler environments) and partly a function of quality (some benchmark items are simplistically graded, have minor mistakes, or can be reward hacked). We do not yet have an explicit way of weighting by quality.

3. 03

   Can evals that primarily measure AI automation generalize to augmentation?

   In the real world, AI models are still primarily used by humans, rather than automating entire job functions end-to-end. Therefore, measuring the real-world occupation capabilities of a model should ideally include measuring how much the model augments human productivity. We have completed some early work in this direction, but finding a methodology that scales remains elusive.

4. 04

   Can every O*NET occupation and task be measured with benchmarks?

   Knowledge-based work is well-suited to measurement through benchmarks because it is largely done on digital machines, like laptops and phones. Dingel and Neiman estimate that only 37% of US jobs could feasibly be done entirely from home, suggesting that ~63% of jobs involve some tasks with a physical-world component which would be hard to incorporate into current benchmarks. For now, this may not be a meaningful oversight – with the rapid development of robotics capabilities, we anticipate a significant need for physical task benchmarks.

We are actively working on these problems; if you are too, please reach out!

We are indebted to Wang et al. for the O*NET tagging methodology, and thank Nathan Lambert for feedback on a draft of this work.