Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

Losses and GLMs

Most smooth losses in regression depend on the coefficients only through the linear predictor η=Xβ\eta = X\beta:

β↦ℓ(Xβ),ℓ(η)=∑i=1nwi ℓi(ηi).\beta \mapsto \ell(X\beta), \qquad \ell(\eta) = \sum_{i=1}^n w_i\, \ell_i(\eta_i).

regreg builds such losses from two pieces: a saturated loss ℓ\ell, a smooth function of the nn-vector η\eta that knows about the response (and case weights wiw_i), and a design matrix XX. The class rr.glm composes the two, and its class-method constructors build the common models:

constructorsaturated loss ℓi(ηi)\ell_i(\eta_i) (negative log-likelihood up to constants)
rr.glm.gaussian(X, y)12(yi−ηi)2\frac{1}{2}(y_i - \eta_i)^2
rr.glm.logistic(X, y, trials=None)nilog⁡(1+eηi)−yiηin_i \log(1 + e^{\eta_i}) - y_i\eta_i
rr.glm.probit(X, y)−yilog⁡Φ(ηi)−(ni−yi)log⁡(1−Φ(ηi))-y_i \log\Phi(\eta_i) - (n_i - y_i)\log(1 - \Phi(\eta_i))
rr.glm.cloglog(X, y)binomial with πi=1−exp⁡(−eηi)\pi_i = 1 - \exp(-e^{\eta_i})
rr.glm.poisson(X, y)eηi−yiηie^{\eta_i} - y_i\eta_i
rr.glm.huber(X, y, eps)hϵ(ηi−yi)h_\epsilon(\eta_i - y_i), hϵ(r)=r2/(2ϵ)h_\epsilon(r) = r^2/(2\epsilon) for ∣r∣≤ϵ\lvert r\rvert\le\epsilon, ∣r∣−ϵ/2\lvert r\rvert - \epsilon/2 otherwise
rr.glm.huber_svm(X, y, eps)a Huberized hinge loss in (2yi−1)ηi(2y_i - 1)\eta_i
rr.glm.cox(X, times, status)Cox negative log partial likelihood (not separable in ii)

Each constructor also takes case_weights, coef (a nonnegative multiplier of the whole loss), saturated_offset, offset and quadratic. Because they all expose the same smooth_objective method, any of them can be paired with any penalty in rr.simple_problem.

Value and gradient checks

For each binary-response and regression loss we compare the value with the formula above and the gradient with central finite differences. The gradient of β↦ℓ(Xβ)\beta \mapsto \ell(X\beta) is XT∇ℓ(Xβ)X^T \nabla\ell(X\beta), computed by the glm object from the saturated loss’s gradient.

gaussian   value   528.1484  rel. grad error 1.8e-10  (value matches formula)
logistic   value   227.0863  rel. grad error 1.9e-10  (value matches formula)
probit     value   256.0399  rel. grad error 1.3e-10  (value matches formula)
cloglog    value   271.9730  rel. grad error 2.0e-10  (value matches formula)
poisson    value   387.1479  rel. grad error 4.3e-11  (value matches formula)
huber      value   384.3447  rel. grad error 1.7e-10
huber_svm  value   244.1541  rel. grad error 6.3e-11

The glm object is literally the composition of a saturated loss with XX; the saturated losses are available on their own (e.g. rr.logistic_loglike, rr.poisson_loglike) and rr.glm(X, Y, saturated_loss) puts them together. The design can also be any regreg affine transform (for instance rr.normalize(X), which centers and scales columns without forming a new matrix).

logistic_loglike

Case weights and offsets

case_weights multiplies each observation’s contribution, ℓ(η)=∑iwiℓi(ηi)\ell(\eta) = \sum_i w_i \ell_i(\eta_i). There are two kinds of offset, distinguished by the space they live in:

Note the sign convention. A classical GLM offset enters as η=Xβ+o\eta = X\beta + o (e.g. o=log⁡(exposure)o = \log(\text{exposure}) in a Poisson rate model), so it is passed as saturated_offset=-o.

case weights and offsets behave as described

Penalized logistic regression vs scikit-learn

The ℓ1\ell_1-penalized logistic regression

minimizeβ  ∑i[log⁡(1+exiTβ)−yixiTβ]+λ∥β∥1\text{minimize}_\beta \; \sum_i \left[\log(1 + e^{x_i^T\beta}) - y_i x_i^T\beta\right] + \lambda\|\beta\|_1

is rr.simple_problem(rr.glm.logistic(X, y), rr.l1norm(p, lagrange=lam)). scikit-learn’s LogisticRegression with an ℓ1\ell_1 penalty minimizes ∥β∥1+C∑i(log-loss)i\|\beta\|_1 + C \sum_i (\text{log-loss})_i, which is the same problem divided by λ\lambda, so C=1/λC = 1/\lambda. (In scikit-learn ≥1.8\ge 1.8 the ℓ1\ell_1 penalty is requested with l1_ratio=1; older versions use penalty='l1'.) We use the saga solver with no intercept.

[ 0.9206 -0.6414  0.3233  0.0168  0.      0.0717]
max |regreg - sklearn| = 2.7977620220553945e-13

The same KKT check as for the LASSO applies, with the logistic score in place of the least squares gradient.

active-set KKT error 2.1316282072803006e-14
max |score| off the active set 1.3120709038067286 <= lam = 8.0

Unpenalized Poisson regression vs IRLS

With no penalty FISTA is just accelerated gradient descent on the smooth loss; rr.simple_problem.smooth(loss) builds that problem. Here we fit a Poisson rate model with exposures tit_i, i.e. log⁡E[yi]=log⁡ti+xiTβ\log E[y_i] = \log t_i + x_i^T\beta, and compare with Newton’s method (equivalently IRLS) written directly in numpy:

β←β+(XTWX)−1XT(y−μ),μ=t⊙eXβ,  W=diag(μ).\beta \leftarrow \beta + (X^T W X)^{-1} X^T (y - \mu), \qquad \mu = t \odot e^{X\beta},\; W = \text{diag}(\mu).
[ 0.4276  -0.52836  0.31295 -0.10574]
max |FISTA - Newton| = 5.551115123125783e-17

glm objects also provide hessian(beta), here XTWXX^T W X, which is what second-order methods need:

The Cox model

rr.glm.cox(X, times, status) is the negative log partial likelihood of the proportional hazards model, with status equal to 1 for an observed failure and 0 for a censored time. Its value, gradient and Hessian are computed by the coxdev package. Tied failure times are handled by Breslow’s approximation (the default) or Efron’s, selected by tie_breaking='breslow' or 'efron'. With Breslow’s method

ℓ(η)=−∑i:δi=1[ηi−log⁡∑j:Tj≥Tieηj],\ell(\eta) = -\sum_{i: \delta_i = 1}\left[\eta_i - \log \sum_{j: T_j \ge T_i} e^{\eta_j}\right],

and Efron’s method replaces the risk-set sum for a time with dd tied failures DD by ∑j∈Reηj−kd∑j∈Deηj\sum_{j \in R} e^{\eta_j} - \frac{k}{d}\sum_{j \in D} e^{\eta_j} for k=0,…,d−1k = 0, \dots, d - 1. We round the times to create plenty of ties and check both against direct implementations.

number of tied times: 262  failures: 185
breslow  value 954.013219  direct 954.013219  rel. grad error 6.8e-09
efron    value 946.636767  direct 946.636767  rel. grad error 1.9e-09

A lasso-penalized Cox model is again a simple_problem. We check it against the KKT conditions, and check the unpenalized fit against Newton’s method using the Hessian from coxdev.

lambda = 30.89; lasso Cox: [ 0.7162 -0.6397  0.295   0.      0.     -0.    ]
active-set KKT error 2.1316282072803006e-14
unpenalized: max |FISTA - Newton| = 6.661338147750939e-16

Multinomial regression

For a response with JJ categories the coefficients form a p×Jp \times J matrix BB and the linear predictor is the n×Jn \times J matrix XBXB. rr.mglm.multinomial(X, Y) takes YY as an n×Jn \times J matrix of counts (one-hot rows for a single draw per case) and uses the overparametrized loss

ℓ(XB)=−∑i[∑jyij(XB)ij−nilog⁡∑je(XB)ij]\ell(XB) = -\sum_i \left[\sum_j y_{ij} (XB)_{ij} - n_i \log \sum_j e^{(XB)_{ij}}\right]

(baseline=True instead fixes the first category’s coefficients at zero). Without a penalty BB is identified only up to adding a common vector to every column, but an ℓ1\ell_1 penalty on all entries picks a unique solution, and this is the same problem scikit-learn solves for multinomial ℓ1\ell_1-penalized logistic regression, again with C=1/λC = 1/\lambda.

coefficient shape (4, 3)
[[-2.0108 -0.      0.2877]
 [ 0.1134 -0.     -0.0749]
 [-0.979  -0.      0.5412]
 [-0.      1.0582 -0.    ]]
max |regreg - sklearn| = 3.6984304507825527e-13