Reliance on Metrics is a Fundamental Challenge for AI

Research Paper

Jimmie Munyi


August 7, 2022

Link to the paper - arxiv


  • We cannot measure the things that matter the most
  • Metrics, can and will, be gamed
  • Metrics overemphasize short-term concerns
  • Many metrics gather data of what we do in highly addictive environments

Framework for a healthier use of metrics

  • Using a slate of metrics to get a fuller picture.
  • Combine with qualitative accounts
  • Involve a range of stakeholders, including those who will be most impacted