How badly the predictions miss, on the scale the model was fitted on. Lower is better, and zero is a perfect fit.
Usage
calc_deviance(
observed,
predicted,
family = c("binomial", "poisson", "gaussian", "laplace"),
weights = NULL,
mean = TRUE
)Details
Deviance answers a question AUC cannot: AUC only cares about ranking, and will not notice predictions that are ordered correctly but wrong. Deviance penalises confident mistakes hardest, which is usually the failure that matters.
It is most useful as a relative measure. Compare a model's deviance to the null deviance – what you would get predicting the overall mean for every observation – and the ratio is the proportion of deviance explained, the quantity boosted regression tree work reports as a matter of course.
References
Elith, J., Leathwick, J. R., & Hastie, T. (2008). A working guide to boosted regression trees. Journal of Animal Ecology, 77(4), 802-813. doi:10.1111/j.1365-2656.2008.01390.x
See also
threshold_metrics() for discrimination,
calibration_estimates() for whether the probabilities are honest.
Other evaluation plots:
calibration_estimates(),
held_out(),
permutation_importance(),
plotCalibration(),
plotImportance(),
plotROC(),
plotThreshold(),
spatial_sorting_bias(),
threshold_metrics()
Examples
set.seed(1)
dat <- data.frame(x = runif(300, 1, 10))
dat$y <- rbinom(300, 1, plogis(-3 + 0.6 * dat$x))
fit <- glm(y ~ x, data = dat, family = binomial)
fitted.deviance <- calc_deviance(dat$y, fitted(fit), "binomial")
null.deviance <- calc_deviance(dat$y, rep(mean(dat$y), nrow(dat)),
"binomial")
# Proportion of deviance explained
1 - fitted.deviance / null.deviance
#> [1] 0.2456978