Receiver Operating Characteristic (ROC) Curve

Authors

Affiliations

Nate Yomogida, B.S., SPT

Doctor of Physical Therapy

B.S. in Kinesiology

Chloë Kerstein, B.S., SPT

Doctor of Physical Therapy

B.A. in Neuroscience

Resources

• Ch42¹

The ROC curve is a visualization of the trade-off between high sensitivity and high specificity¹.

An ROC Curve¹

Reading an ROC Curve

Bottom left:

• An extreme where the test never returns a positive diagnosis.
• At this extreme, every patient is incorrectly diagnosed as healthy (sensitivity = 0%)¹
• Every control is correctly diagnosed as healthy (specificity = 100%)¹

Upper right extreme

• The test always returns a diagnosis that the person tested has the disease¹.
• Every true positive is correctly diagnosed
  • Sensitivity (true positive rate) = 100%¹
• Every true negative is incorrectly diagnosed as a positive
  • specificity (True negative rate) = 0%¹
  • False positive rate ($1-\mathbf{\text{specificity}}$) = 100%¹

What is the best cut-off?

Choosing the ideal cut-off point is dependent on the consequences of a false negative or false positive.

• If both consequences are equally bad: The best threshold is the one that corresponds to the point on the ROC curve that is closest to the upper-left corner of the graph.
• Generally, the consequences are not comparable.
• It will be difficult to decide what cut-off makes the most sense. That decision must be made by someone who understands the disease and the test. It is not a decision to be made by a computer program.

Bayesian Logic

See Bayesian logic p448¹

Area under the Curve (AUC)

See Receiver operating characteristic curve: overview and practical use for clinicians²

• ideal ROC curve thus has an AUC = 1.0.²
• Therefore, for any diagnostic technique to be meaningful, the AUC must be greater than 0.5
• in general, it must be greater than 0.8 to be considered acceptable

Confidence Interval

For any test to be statistically significant, the lower 95% CI value of the AUC must be > 0.5

Interpretation of the Area Under the Curve²

AUC	Interpretation
$0.9\ge \text{AUC}$	Excellent
$0.8\ge \text{AUC}\ge 0.9$	Good
$0.7\ge \text{AUC}\ge 0.8$	Fair
$0.6\ge \text{AUC}\ge 0.7$	Poor
$0.5\ge \text{AUC}\ge 0.6$	Fail
$0.0\ge \text{AUC}\le 0.5$	Worse than flipping a coin

Sample size

See²

Common Mistakes

Automating the decision about which point on an ROC curve to use as a cut-off

The ROC curve plots the trade-offs between sensitivity and specificity. Which combination is the best to define a critical value of a lab test? It depends on the consequences of making a false positive or a false negative diagnosis. That decision needs to be made in a clinical (or in some situations, scientific) context and should not be automated.

Thinking that a single value can quantify the accuracy of a test

There are many ways to quantify accuracy

References

1.

Motulsky H. Intuitive Biostatistics: A Nonmathematical Guide to Statistical Thinking. 4th ed. Oxford University Press; 2018.

2.

Nahm FS. Receiver operating characteristic curve: Overview and practical use for clinicians. Korean Journal of Anesthesiology. 2022;75(1):25-36. doi:10.4097/kja.21209