Reliance on Metrics is a Fundamental Challenge for AI

Research Paper

Link to the paper - arxiv

Problems

• We cannot measure the things that matter the most
• Metrics, can and will, be gamed
• Metrics overemphasize short-term concerns
• Many metrics gather data of what we do in highly addictive environments

Framework for a healthier use of metrics

• Using a slate of metrics to get a fuller picture.
• Combine with qualitative accounts
• Involve a range of stakeholders, including those who will be most impacted