This page supports the program from the
Curve Fitting for Receiver Operator Characteristics (ROCs) Program Page
. The algorithm is adapted from that in a paper by Moses et.al. (see references)
The Need for Curve Fitting the Receiver Operator Characteristic (ROC) Curve
The underlying assumptions of the ROC curve is that the Outcome is binary, but the Test is a continuous measurement. In
establishing the ROC from a set of reference data, it is assumed that the sample size provided is large enough, and the
granularity of the values in the data set is fine enough, to produce a valid curve.
The modelling data set unfortunately often failed these assumptions. A major problem is that test measurements are often
rounded to clinically meaningful intervals, so that the granularity of the data is coarse, and changes occurred in a
series of steps rather than in a smooth curve.
Furthermore, with rounding of values, often there are clusters with the same test value in one or both predictor groups, making interpretation difficult
- A large cluster will cause a large change in True and False Positive Rates, when the change in cut off value is small
- A cluster will result in an uneven relationship between false and true positive rates, as the values of one may change while the other remains the same.
An example of this is shown in the plot to the left. If the cut off for decision making is 155 (the black line), data points
can be clearly separated into above or below that value. However, if the cut off value is set at 154, there are 3 data points
in the Outcome Positive group and 2 in the Outcome Negative group that have this exact value. Whether these are considered above
the cut off, below the cut off, or excluded from consideration, will have a large impact on subsequent results produced.
The resulting ROC plot would be as the black line in the plot to the right.
Curve fitting would result in a ROC curve that has the closest fit with the data, yet allows a continuous interpretation along
the whole range of the Test values, as shown in the red line in the plot to the right.
References
Hanley JA, McNeil BJ (1982) The meaning and use of the Area Under a Receiver
Operating Characteristic (ROC) curve. Radiology 143:29-36
Simel D.L., Samsa G.P., Matchar D.B. (1991) Likelihood ratios with confidence:
sample size estimation for diagnostic test studies. J. Clin. Epidemiology
vol 44 No. 8 pp 763-770
Moses LE, Shapiro D, and Littenberg B(1993) Combining independent studies of a
diagnostic test into a Summary ROC Curve: Data analysis approaches and some
additional considerations. Statistics in Medicine 12:p1293-1316.
Deeks J.J, and Morris J.M. (1996) Evaluating diagnostic tests. In Bailliere's
Clinical Obstetrics and Gynaecology Vol.10 No. 4, December 1996
ISBN 0-7020-2260-8 p. 613-631.
Fagan T.J. (1975) Nomogram for Bayer's Theorem. New England J. Med. 293:257
Sackett D, Haynes R, Guyatt G, Tugwell P. (1991) Clinical Epidemiology: A Basic
Science for Clinical Medicine. Second edition. ISBN 0-316-76599-6.
Test Value | FPR | TPR |
162.5 | 0.96 | 1 |
162 | 0.92 | 1 |
161.5 | 0.84 | 1 |
160 | 0.84 | 0.96 |
159.5 | 0.76 | 0.96 |
159 | 0.68 | 0.96 |
158.5 | 0.48 | 0.92 |
157.5 | 0.4 | 0.88 |
157 | 0.36 | 0.88 |
156.5 | 0.36 | 0.8 |
156 | 0.24 | 0.68 |
155.5 | 0.2 | 0.6 |
154 | 0.12 | 0.48 |
153.5 | 0.04 | 0.4 |
153 | 0.04 | 0.36 |
152.5 | 0 | 0.28 |
151.5 | 0 | 0.24 |
151 | 0 | 0.2 |
150.5 | 0 | 0.16 |
150 | 0 | 0.12 |
149 | 0 | 0.08 |
This example uses the default input data from the program in the
Curve Fitting for Receiver Operator Characteristics (ROCs) Program Page
. This set of the data
is produced from the first table of results in the
Receiver Operator Characteristics (ROC) Program Page
. The original data are computer generated and do not represent any real information.
The data is a study of using maternal height (cms) as a test to predict the outcome of Caesarean Section in childbirth, the shorter the height, the more likely Caesarean Section. The 3 columns are the test value (maternal height in cms), False Positive Rate (FPR), and True Positive Rate (TPR). The data is as shown in the table to the right.
The Algorithm
Polynomial curve fitting, as explained in Curve Fitting Explained Page
, is used. As both False and True positive rates are probabilities, their logit transforme were used as the dependent variable. The formulae are:
- p represents FPR or TPR, and y its logit transform.
- x represents the predictor (height in cms in this case)
- The formulae are:
y = log((p / (1-p)))
p = 1 / (1 + exp(-y))
- Polynomial curve fitting is y = Intercept + c1(x) + c2(x^{2}) + c3(x^{3})
- Before processing, FPR and TPR values less than 0.00001 are defaulted to 0.00001, and greater than 0.99999 to 0.99999 to avoid the division by zero error
Three options for selection data
Option A includes all the data, as entered.
Option B includes only those test values from the two outcome groups overlapped, excluding values where TPR or FPR are 0 or 1. The excluded data are usually irrelevant to the ROC, as their interpretation is obvious, and their inclusion may bias the fitted curve to hug the borders excessively. For these reasons, option B is recommended
Option C includes only data that are in the upper left quadrant of the ROC plot, where FPR<=0.5 and TPR>=0.5. The results of the option provides greater precision to those values that are of interest to the clinicians, as values where the predictor has a high probability of being wrong (FPR>0.5 or TPR<0.5) are excluded.
Three options for power
Polynomial Regression allows any value for power, but only 3 are provided for curve fitting the ROC. In most cases, the results from different options are very similar, as most test had a linear relationship with the outcome, and the curve only needs to be bent slightly to cater for slight difference in the probability distribution of test values.
The use of higher power also risk overfitting, a greater precision in fitting the data presented, but producing results that are less reproducible.
Users should thereforre examine the rsults of using different power, and choose the lowest level that produces a satisfactory result
Results and their Interpretations
The program in the Curve Fitting for Receiver Operator Characteristics (ROCs) Program Page
produces one plot and two tables. Using the default example input data,
the plots produced from different options are shown to the left. The green curve uses algorithm C, including only data where TPR>=0.5 and FPR<=0.5. All fitted linearly (power=1)
The plot to the right and the two tables on this page show the relationship between original
data (black circular dots) and the curve fitted ROC curve using algorithm B, including data that
have values >0 and <1. The curve is fitted linearly (power=1), and the short straight line joins the
coordinate of the original values to the values occupied on the fitted curve for each data point.
Data | Curve Fitted |
162.5 | 0.96 | 1 | 0.95 | 0.99 |
162 | 0.92 | 1 | 0.94 | 0.99 |
161.5 | 0.84 | 1 | 0.91 | 0.99 |
160 | 0.84 | 0.96 | 0.80 | 0.97 |
159.5 | 0.76 | 0.96 | 0.75 | 0.96 |
159 | 0.68 | 0.96 | 0.68 | 0.94 |
158.5 | 0.48 | 0.92 | 0.61 | 0.93 |
157.5 | 0.4 | 0.88 | 0.45 | 0.88 |
157 | 0.36 | 0.88 | 0.37 | 0.84 |
156.5 | 0.36 | 0.8 | 0.30 | 0.80 |
156 | 0.24 | 0.68 | 0.24 | 0.74 |
155.5 | 0.2 | 0.6 | 0.18 | 0.68 |
154 | 0.12 | 0.48 | 0.08 | 0.47 |
153.5 | 0.04 | 0.4 | 0.06 | 0.40 |
153 | 0.04 | 0.36 | 0.04 | 0.33 |
152.5 | 0 | 0.28 | 0.03 | 0.27 |
151.5 | 0 | 0.24 | 0.02 | 0.17 |
151 | 0 | 0.2 | 0.01 | 0.13 |
150.5 | 0 | 0.16 | 0.01 | 0.10 |
150 | 0 | 0.12 | 0.01 | 0.08 |
149 | 0 | 0.08 | 0.00 | 0.04 |
Please note that the program produces coefficients to 6 decimal point precision and results to 4 decimal point precision. Tables are rounded to 2 decimal points for presentation on this page.
Table of Transformed Data is the table on the left. The first 3 columns displays the original
data as they were entered, and the last two column the curve fitted FPR and TPR. This table is for
the user to review, to consider whether the algorithm chosen is appropriate, whether the
results reflect the inputted data, and whether the parameters of the curve fitted ROC can be
generalized for future clinical use.
It can be seen that, above a height of 160cms, all mothers delivered vaginally, and below 153cms all
mothers required a Caesarean Section The data overlapped between these two values were used for curve
fitting (in this example).
It can also be seen that, where the data overlapped, the curve fitted is
closed enough to the original for practical use.
Test Value | FPR | TPR | TNR | YI | TPR/TNR | TNR/TPR | LR+ | LR- |
149.0 | 0.00 | 0.04 | 1.00 | 0.04 | 0.04 | 22.80 | 12.89 | 0.96 |
149.3 | 0.00 | 0.05 | 1.00 | 0.05 | 0.05 | 19.57 | 12.63 | 0.95 |
149.5 | 0.00 | 0.06 | 1.00 | 0.05 | 0.06 | 16.81 | 12.35 | 0.95 |
149.8 | 0.01 | 0.07 | 0.99 | 0.06 | 0.07 | 14.46 | 12.07 | 0.94 |
150.1 | 0.01 | 0.08 | 0.99 | 0.07 | 0.08 | 12.46 | 11.77 | 0.93 |
150.4 | 0.01 | 0.09 | 0.99 | 0.08 | 0.09 | 10.75 | 11.46 | 0.92 |
150.6 | 0.01 | 0.11 | 0.99 | 0.10 | 0.11 | 9.29 | 11.14 | 0.90 |
150.9 | 0.01 | 0.12 | 0.99 | 0.11 | 0.12 | 8.05 | 10.81 | 0.89 |
151.2 | 0.01 | 0.14 | 0.99 | 0.13 | 0.14 | 6.99 | 10.46 | 0.87 |
151.4 | 0.02 | 0.16 | 0.98 | 0.15 | 0.16 | 6.09 | 10.09 | 0.85 |
151.7 | 0.02 | 0.18 | 0.98 | 0.17 | 0.19 | 5.32 | 9.71 | 0.83 |
152.0 | 0.02 | 0.21 | 0.98 | 0.19 | 0.21 | 4.66 | 9.31 | 0.81 |
152.2 | 0.03 | 0.24 | 0.97 | 0.21 | 0.24 | 4.10 | 8.90 | 0.78 |
152.5 | 0.03 | 0.27 | 0.97 | 0.24 | 0.28 | 3.62 | 8.47 | 0.76 |
152.8 | 0.04 | 0.30 | 0.96 | 0.26 | 0.31 | 3.21 | 8.03 | 0.73 |
153.1 | 0.04 | 0.33 | 0.96 | 0.29 | 0.35 | 2.86 | 7.58 | 0.70 |
153.3 | 0.05 | 0.37 | 0.95 | 0.32 | 0.39 | 2.55 | 7.13 | 0.66 |
153.6 | 0.06 | 0.41 | 0.94 | 0.35 | 0.44 | 2.29 | 6.67 | 0.63 |
153.9 | 0.07 | 0.45 | 0.93 | 0.38 | 0.48 | 2.07 | 6.21 | 0.59 |
154.1 | 0.08 | 0.49 | 0.92 | 0.40 | 0.53 | 1.88 | 5.76 | 0.56 |
154.4 | 0.10 | 0.53 | 0.90 | 0.43 | 0.59 | 1.71 | 5.32 | 0.52 |
154.7 | 0.12 | 0.57 | 0.88 | 0.45 | 0.64 | 1.56 | 4.90 | 0.49 |
154.9 | 0.13 | 0.61 | 0.87 | 0.47 | 0.70 | 1.43 | 4.49 | 0.46 |
155.2 | 0.16 | 0.64 | 0.84 | 0.49 | 0.76 | 1.31 | 4.11 | 0.42 |
155.5 | 0.18 | 0.68 | 0.82 | 0.50 | 0.83 | 1.21 | 3.76 | 0.39 |
155.8 | 0.21 | 0.71 | 0.79 | 0.51 | 0.90 | 1.11 | 3.43 | 0.36 |
156.0 | 0.24 | 0.74 | 0.76 | 0.51 | 0.98 | 1.02 | 3.13 | 0.34 |
156.3 | 0.27 | 0.77 | 0.73 | 0.50 | 1.06 | 0.94 | 2.86 | 0.31 |
156.6 | 0.31 | 0.80 | 0.69 | 0.49 | 1.16 | 0.87 | 2.61 | 0.29 |
156.8 | 0.34 | 0.83 | 0.66 | 0.48 | 1.26 | 0.79 | 2.39 | 0.27 |
157.1 | 0.39 | 0.85 | 0.61 | 0.46 | 1.38 | 0.73 | 2.20 | 0.25 |
157.4 | 0.43 | 0.87 | 0.57 | 0.44 | 1.51 | 0.66 | 2.03 | 0.23 |
157.6 | 0.47 | 0.88 | 0.53 | 0.41 | 1.67 | 0.60 | 1.88 | 0.22 |
157.9 | 0.51 | 0.90 | 0.49 | 0.39 | 1.85 | 0.54 | 1.75 | 0.21 |
158.2 | 0.56 | 0.91 | 0.44 | 0.36 | 2.06 | 0.49 | 1.64 | 0.20 |
158.5 | 0.60 | 0.92 | 0.40 | 0.33 | 2.31 | 0.43 | 1.54 | 0.19 |
158.7 | 0.64 | 0.94 | 0.36 | 0.30 | 2.60 | 0.38 | 1.46 | 0.18 |
159.0 | 0.68 | 0.94 | 0.32 | 0.27 | 2.94 | 0.34 | 1.39 | 0.17 |
159.3 | 0.72 | 0.95 | 0.28 | 0.24 | 3.35 | 0.30 | 1.33 | 0.17 |
159.5 | 0.75 | 0.96 | 0.25 | 0.21 | 3.83 | 0.26 | 1.28 | 0.16 |
159.8 | 0.78 | 0.96 | 0.22 | 0.18 | 4.40 | 0.23 | 1.24 | 0.16 |
160.1 | 0.81 | 0.97 | 0.19 | 0.16 | 5.08 | 0.20 | 1.20 | 0.16 |
160.3 | 0.83 | 0.97 | 0.17 | 0.14 | 5.89 | 0.17 | 1.17 | 0.16 |
160.6 | 0.86 | 0.98 | 0.14 | 0.12 | 6.85 | 0.15 | 1.14 | 0.16 |
160.9 | 0.88 | 0.98 | 0.12 | 0.10 | 7.99 | 0.13 | 1.12 | 0.15 |
161.2 | 0.89 | 0.98 | 0.11 | 0.09 | 9.35 | 0.11 | 1.10 | 0.15 |
161.4 | 0.91 | 0.99 | 0.09 | 0.08 | 10.96 | 0.09 | 1.08 | 0.15 |
161.7 | 0.92 | 0.99 | 0.08 | 0.06 | 12.88 | 0.08 | 1.07 | 0.15 |
162.0 | 0.93 | 0.99 | 0.07 | 0.06 | 15.17 | 0.07 | 1.06 | 0.15 |
162.2 | 0.94 | 0.99 | 0.06 | 0.05 | 17.89 | 0.06 | 1.05 | 0.16 |
Table of Curve Fitted Values
The table to the right provides the results of curve fitting between the range of input data, divided into 50 intervals. Once the user is satisfied with the results, this result can be copied and used as reference for decision making.
- Column 1 is the calculated intervals of test values (in this case maternal height). Column 2 and 3 are the False (FPR) and True (TPR) Positives
- Column 4 is the True Negative Rate TNR = 1-FPR. This is used in some of the subsequent calculations
- Column 5 is the Youden Index YI = TPR+TNR-1, which represents the overall accuracy of the values. The Test value associated
with the maximum YI is sometimes used as the cut off value for clinical decision making, and in this example, the closest
is a maternal height of 156cms (156.02), with Youden Index = 0.51
- Column 6 and 7 is the ratio of TPR/TNR and its inverse TNR/TPR. These are commonly used for the following purposes
- An alternative to the Youden Index for selecting the best cut off value is Q*, suggested by Moses et.al (see references).
This is a point where TPR/TNR = 1, and represents the intersection between the ROC and a line joining the top left and
bottom right corner of the plot. In this example, the closest is also a maternal height of 156cms, with TPR/TNR=0.98
- If the test is being used for screening purposes, to create an early warning system, then a High TPR is preferred over a
high TNR. A TPR/TNR ratio of between 2 and 3 is sometimes used for this purposes. In our example, the closest is a maternal
height of 159cms, where TPR/TNR=2.94. This can be set as an alert trigger, possibly making a rule that all
mothers admitted to the labour ward shorter than 159cms should be reviewed by an experienced obstetrician in early labour.
- If the test is to make decisions, particularly if the decision is expensive or has a high riek, then a High TNR is preferred over a
high TPR. A TNR/TPR ratio of between 2 and 3 is sometimes used for this purposes. In our example, the closest is a maternal
height of 153cms (153.1), where TPR/TNR=2.86. This can be set as an decision trigger, possibly making a rule that all
mothers admitted to the labour ward shorter than 153cms should be considered for Caesarean Section without a trial of labour.
- Column 8 and 9 are Likelihood Ratios for Test Positive and Test negative. Discussions on Likelihood Ratio are provided in
the Prediction Statistics Explanation Page
and will not be repeated here.
Disclaimer and Comments
The reasons for developing the curve fitting program for ROC are presented in the introduction. The program has gone through a number of changes.
- The initial development was to repeat the method described in the paper by Moses et.al., regressing logit(TPR) against logit(FPR), but using data from a single study rather than multiple sets of FPR and TNR from a meta-analysis. Although this seemed to work, there was no provision to include the test values as a predictor.
- A second attempt (2017) was to add a regression of logit(FPR) against the test value, so that, from the test value, one can estimate the FPR, and from the FPR one can estimate the TPR. Options to select which part of the data to analysed was also introduced. This also seemed to work, but a regression upon a regression seems to be adding approximation to approximation, giving rise to anxiety over reproducibility.
- The current version (Feb 2020) is a major departure, seperately regressing the logits of FPR and TPR agains the test values, and providing a polynomial curve fit instead of assuming a linear relationship between the test and the logits.
Please be aware that the curve fitting program used formula found in the paper by Moses et.al. (see reference), to draw a line through all the data points in a ROC curve. The only assumptions are that the predictor (test measurement) is a continuous measurement, and that FPR are TPR are probabilities with a logistic distribution.
The algorithm has not been rigorously tested or reviewed, and users should not assume that the method is a generally accepted statistical procedure. Currently (February 2020) I am unable to find any reference on this subject, and I have not received any feedback or criticism, so I am uncertain whether there are any pitfalls in this approach that I have overlooked. My own approach is to use this as a tool, and to separately study the validity of the results produced.
Users inexperienced in statistics should seek professional advice before using this algorithm. From the experienced users, feedback, advice, and criticisms would be most welcomed.
The following is a program written by me in R, which produces the same results as that written in php in
Curve Fitting for Receiver Operator Characteristics (ROCs) Program Page
.
This R program was used to check and validate the results produced in the php program, and can be copied and pasted into RStudio, and run directly.
The program is presented here so that those interested can cross check the calculations, or to develop the idea further.
No restrictions are placed on the copy and use of these codes, although an acknowledgement would be appreciated.
Feedback, advice, or criticism are all very welcomed.
# Curvefit ROC
# This program uses all of the data (Option A in the php program)
# Power of curve fit is set before running, by changing the value of pw
myDat = ("
Test FPR TPR
162.5 0.96 1.00
162.0 0.92 1.00
161.5 0.84 1.00
160.0 0.84 0.96
159.5 0.76 0.96
159.0 0.68 0.96
158.5 0.48 0.92
157.5 0.40 0.88
157.0 0.36 0.88
156.5 0.36 0.80
156.0 0.24 0.68
155.5 0.20 0.60
154.0 0.12 0.48
153.5 0.04 0.40
153.0 0.04 0.36
152.5 0.00 0.28
151.5 0.00 0.24
151.0 0.00 0.20
150.5 0.00 0.16
150.0 0.00 0.12
149.0 0.00 0.08
")
# create dataframe from input data. df is the dataframe
df <- read.table(textConnection(myDat),header=TRUE)
summary(df)
# create logits of FPR and TPR using function MakeLogit
MakeLogit <- function(vec)
{
vec[vec>0.99999] <- 0.99999
vec[vec<0.00001] <- 0.00001
vecRes <- log(vec / (1 - vec))
vecRes
}
logitFPR <- MakeLogit(df$FPR)
logitTPR <- MakeLogit(df$TPR)
# curvefit logitFPR and logitTPR against Test.
# pw = power of polynomial curve fitting
pw = 2
resFPR<-lm(formula = logitFPR ~ poly(df$Test, pw, raw=TRUE))
resTPR<-lm(formula = logitTPR ~ poly(df$Test, pw, raw=TRUE))
# Display results of curve fitting
summary(resFPR)
summary(resTPR)
# Extract coefficients vector for future calculations
coefFPR <- coef(summary(resFPR))[1:(pw+1)]
coefTPR <- coef(summary(resTPR))[1:(pw+1)]
# Calculate curve fitted values for FPR and TPR
# using function CalCurveFitValues
# and append them to data frame
CalCurveFitValues <- function(coefVec, datVec)
{
f = length(coefVec)
n = length(datVec)
vecResult <- vector()
for ( i in 1:n)
{
x = datVec[i]
y = coefVec[1] + coefVec[2] * x
for (j in seq(3, f, by=1)){ y = y + coefVec[j] * x^(j-1) }
vecResult[i] = 1 / (1 + exp(-(y)))
}
vecResult
}
df$cfFPR <- CalCurveFitValues(coefFPR, df$Test)
df$cfTPR <- CalCurveFitValues(coefTPR, df$Test)
# Display original and curve fitted values
df
# Create dataframe table of calculations for range of predictor
# divided into 50 intervals
minv = min(df$Test)
maxv = max(df$Test)
Test <- seq(from = minv, to = maxv, length.out = 50)
dfTable <- data.frame(Test)
# Calculate and append all columns into dfTable,
# rounded to 4 decimal places
# False Positive Rate FPR
dfTable$FPR <- round(CalCurveFitValues(coefFPR, dfTable$Test),4)
# True Positive Rate TPR
dfTable$TPR <- round(CalCurveFitValues(coefTPR, dfTable$Test),4)
# False Negative Rate FNR
dfTable$FNR <- round(1-dfTable$TPR,4)
# True Negative Rate TNR
dfTable$TNR <- round(1-dfTable$FPR,4)
# Youden Index YI
dfTable$YI <- round(dfTable$TPR + dfTable$TNR - 1,4)
# TPR / TNR PNRatio
dfTable$PNRatio <- round(dfTable$TPR / dfTable$TNR,4)
# TNR / TPR NPRatio
dfTable$NPRatio <- round(dfTable$TNR / dfTable$TPR,4)
# Likelihood Ratio Test Positive LRPos
dfTable$LRPos <- round(dfTable$TPR / dfTable$FPR,4)
# Likelihood Ratio Test Negative LRNeg
dfTable$LRNeg <- round(dfTable$FNR / dfTable$TNR,4)
# Display table
dfTable