Most smooth losses in regression depend on the coefficients only through the linear predictor :
regreg builds such losses from two pieces: a saturated loss ,
a smooth function of the -vector that knows about the response (and
case weights ), and a design matrix . The class rr.glm composes the
two, and its class-method constructors build the common models:
| constructor | saturated loss (negative log-likelihood up to constants) |
|---|---|
rr.glm.gaussian(X, y) | |
rr.glm.logistic(X, y, trials=None) | |
rr.glm.probit(X, y) | |
rr.glm.cloglog(X, y) | binomial with |
rr.glm.poisson(X, y) | |
rr.glm.huber(X, y, eps) | , for , otherwise |
rr.glm.huber_svm(X, y, eps) | a Huberized hinge loss in |
rr.glm.cox(X, times, status) | Cox negative log partial likelihood (not separable in ) |
Each constructor also takes case_weights, coef (a nonnegative multiplier
of the whole loss), saturated_offset, offset and quadratic. Because they
all expose the same smooth_objective method, any of them can be paired with
any penalty in rr.simple_problem.
import numpy as np
from scipy.stats import norm
import regreg.api as rr
rng = np.random.default_rng(0)
n, p = 300, 6
X = rng.standard_normal((n, p))
beta_true = np.array([1., -0.8, 0.5, 0, 0, 0])
eta_true = X @ beta_trueValue and gradient checks¶
For each binary-response and regression loss we compare the value with the
formula above and the gradient with central finite differences. The gradient
of is , computed by the
glm object from the saturated loss’s gradient.
y_bin = rng.binomial(1, 1 / (1 + np.exp(-eta_true)))
y_cont = eta_true + rng.standard_normal(n)
y_count = rng.poisson(np.exp(0.5 * eta_true))
losses = {
'gaussian': (rr.glm.gaussian(X, y_cont),
lambda e: 0.5 * np.sum((y_cont - e)**2)),
'logistic': (rr.glm.logistic(X, y_bin),
lambda e: np.sum(np.logaddexp(0, e) - y_bin * e)),
'probit': (rr.glm.probit(X, y_bin),
lambda e: -np.sum(y_bin * norm.logcdf(e) + (1 - y_bin) * norm.logsf(e))),
'cloglog': (rr.glm.cloglog(X, y_bin),
lambda e: -np.sum(y_bin * np.log(-np.expm1(-np.exp(e)))
- (1 - y_bin) * np.exp(e))),
'poisson': (rr.glm.poisson(X, y_count),
lambda e: np.sum(np.exp(e) - y_count * e)),
'huber': (rr.glm.huber(X, y_cont, 0.5), None),
'huber_svm': (rr.glm.huber_svm(X, y_bin, 0.5), None),
}
def fd_grad(f, b, h=1e-6):
E = np.eye(len(b))
return np.array([(f(b + h * E[j]) - f(b - h * E[j])) / (2 * h) for j in range(len(b))])
b = 0.3 * rng.standard_normal(p)
for name, (loss, formula) in losses.items():
value, grad = loss.smooth_objective(b, 'both')
fd = fd_grad(lambda v: loss.smooth_objective(v, 'func'), b)
grad_err = np.max(np.abs(grad - fd)) / max(1, np.max(np.abs(grad)))
msg = f'{name:10s} value {value:10.4f} rel. grad error {grad_err:.1e}'
if formula is not None:
np.testing.assert_allclose(value, formula(X @ b), rtol=1e-10)
msg += ' (value matches formula)'
print(msg)
assert grad_err < 1e-6gaussian value 528.1484 rel. grad error 1.8e-10 (value matches formula)
logistic value 227.0863 rel. grad error 1.9e-10 (value matches formula)
probit value 256.0399 rel. grad error 1.3e-10 (value matches formula)
cloglog value 271.9730 rel. grad error 2.0e-10 (value matches formula)
poisson value 387.1479 rel. grad error 4.3e-11 (value matches formula)
huber value 384.3447 rel. grad error 1.7e-10
huber_svm value 244.1541 rel. grad error 6.3e-11
The glm object is literally the composition of a saturated loss with ; the
saturated losses are available on their own (e.g. rr.logistic_loglike,
rr.poisson_loglike) and rr.glm(X, Y, saturated_loss) puts them together.
The design can also be any regreg affine transform (for instance
rr.normalize(X), which centers and scales columns without forming a new
matrix).
sat = rr.logistic_loglike(y_bin.shape, y_bin)
by_hand = rr.glm(X, y_bin, sat)
np.testing.assert_allclose(by_hand.smooth_objective(b, 'func'),
losses['logistic'][0].smooth_objective(b, 'func'))
print(type(losses['logistic'][0].saturated_loss).__name__)logistic_loglike
Case weights and offsets¶
case_weights multiplies each observation’s contribution, .
There are two kinds of offset, distinguished by the space they live in:
saturated_offsetis subtracted from the linear predictor: the loss is .offsetis subtracted from the coefficients: the loss is .
Note the sign convention. A classical GLM offset enters as
(e.g. in a Poisson rate
model), so it is passed as saturated_offset=-o.
w = rng.uniform(0.5, 2, n)
o = 0.2 * rng.standard_normal(n)
alpha = 0.1 * rng.standard_normal(p)
e = X @ b
def logistic_nll(eta, weights=1):
return np.sum(weights * (np.logaddexp(0, eta) - y_bin * eta))
checks = [
(rr.glm.logistic(X, y_bin, case_weights=w), logistic_nll(e, w)),
(rr.glm.logistic(X, y_bin, saturated_offset=o), logistic_nll(e - o)),
(rr.glm.logistic(X, y_bin, offset=alpha), logistic_nll(X @ (b - alpha))),
]
for loss, expected in checks:
np.testing.assert_allclose(loss.smooth_objective(b, 'func'), expected, rtol=1e-12)
print('case weights and offsets behave as described')case weights and offsets behave as described
Penalized logistic regression vs scikit-learn¶
The -penalized logistic regression
is rr.simple_problem(rr.glm.logistic(X, y), rr.l1norm(p, lagrange=lam)).
scikit-learn’s LogisticRegression with an penalty minimizes
, which is the same problem divided by
, so . (In scikit-learn the penalty is
requested with l1_ratio=1; older versions use penalty='l1'.) We use the
saga solver with no intercept.
import warnings
from sklearn.linear_model import LogisticRegression
def sk_l1_logistic(X, y, C, **kw):
import sklearn
major, minor = map(int, sklearn.__version__.split('.')[:2])
pen = dict(l1_ratio=1) if (major, minor) >= (1, 8) else dict(penalty='l1')
return LogisticRegression(C=C, solver='saga', fit_intercept=False,
tol=1e-12, max_iter=100000, **pen, **kw).fit(X, y)
lam = 8.
logistic_loss = rr.glm.logistic(X, y_bin)
problem = rr.simple_problem(logistic_loss, rr.l1norm(p, lagrange=lam))
beta_logit = problem.solve(tol=1e-12, min_its=200)
sk = sk_l1_logistic(X, y_bin, C=1 / lam)
print(np.round(beta_logit, 4))
print('max |regreg - sklearn| =', np.max(np.abs(sk.coef_.ravel() - beta_logit)))
np.testing.assert_allclose(sk.coef_.ravel(), beta_logit, atol=1e-6)[ 0.9206 -0.6414 0.3233 0.0168 0. 0.0717]
max |regreg - sklearn| = 2.7977620220553945e-13
The same KKT check as for the LASSO applies, with the logistic score in place of the least squares gradient.
g = logistic_loss.smooth_objective(beta_logit, 'grad')
active = beta_logit != 0
print('active-set KKT error', np.max(np.abs(g[active] + lam * np.sign(beta_logit[active]))))
print('max |score| off the active set', np.max(np.abs(g[~active])), '<= lam =', lam)
assert np.max(np.abs(g[active] + lam * np.sign(beta_logit[active]))) < 1e-5
assert np.all(np.abs(g[~active]) <= lam)active-set KKT error 2.1316282072803006e-14
max |score| off the active set 1.3120709038067286 <= lam = 8.0
Unpenalized Poisson regression vs IRLS¶
With no penalty FISTA is just accelerated gradient descent on the smooth
loss; rr.simple_problem.smooth(loss) builds that problem. Here we fit a
Poisson rate model with exposures , i.e. ,
and compare with Newton’s method (equivalently IRLS) written directly in numpy:
Xp = 0.5 * X[:, :4]
exposure = rng.uniform(1, 5, n)
counts = rng.poisson(exposure * np.exp(Xp @ np.array([0.5, -0.5, 0.25, 0.])))
pois = rr.glm.poisson(Xp, counts, saturated_offset=-np.log(exposure))
beta_pois = rr.simple_problem.smooth(pois).solve(tol=1e-14, min_its=200)
beta_newton = np.zeros(Xp.shape[1])
for _ in range(10):
mu = exposure * np.exp(Xp @ beta_newton)
beta_newton += np.linalg.solve(Xp.T @ (mu[:, None] * Xp), Xp.T @ (counts - mu))
print(np.round(beta_pois, 5))
print('max |FISTA - Newton| =', np.max(np.abs(beta_pois - beta_newton)))
np.testing.assert_allclose(beta_pois, beta_newton, atol=1e-6)[ 0.4276 -0.52836 0.31295 -0.10574]
max |FISTA - Newton| = 5.551115123125783e-17
glm objects also provide hessian(beta), here , which is what
second-order methods need:
mu = exposure * np.exp(Xp @ beta_pois)
np.testing.assert_allclose(pois.hessian(beta_pois), Xp.T @ (mu[:, None] * Xp), rtol=1e-8)The Cox model¶
rr.glm.cox(X, times, status) is the negative log partial likelihood of the
proportional hazards model, with status equal to 1 for an observed failure
and 0 for a censored time. Its value, gradient and Hessian are computed by the
coxdev package. Tied failure
times are handled by Breslow’s approximation (the default) or Efron’s,
selected by tie_breaking='breslow' or 'efron'. With Breslow’s method
and Efron’s method replaces the risk-set sum for a time with tied failures by for . We round the times to create plenty of ties and check both against direct implementations.
T = rng.exponential(np.exp(-eta_true))
cens = rng.exponential(2, n)
times = np.round(np.minimum(T, cens), 1) + 0.1
status = (T <= cens).astype(int)
print('number of tied times:', n - len(np.unique(times)), ' failures:', status.sum())
def breslow_nll(eta):
return -sum(eta[i] - np.log(np.exp(eta[times >= times[i]]).sum())
for i in range(n) if status[i])
def efron_nll(eta):
val = 0.
for t in np.unique(times[status == 1]):
D = (times == t) & (status == 1)
d = D.sum()
risk, tied = np.exp(eta[times >= t]).sum(), np.exp(eta[D]).sum()
val -= eta[D].sum() - sum(np.log(risk - k / d * tied) for k in range(d))
return val
for tb, ref in [('breslow', breslow_nll), ('efron', efron_nll)]:
cox = rr.glm.cox(X, times, status, tie_breaking=tb)
v, g = cox.smooth_objective(b, 'both')
fd = fd_grad(lambda u: cox.smooth_objective(u, 'func'), b)
grad_err = np.max(np.abs(g - fd)) / np.max(np.abs(g))
print(f'{tb:8s} value {v:.6f} direct {ref(X @ b):.6f} rel. grad error {grad_err:.1e}')
np.testing.assert_allclose(v, ref(X @ b), rtol=1e-10)
assert grad_err < 1e-6number of tied times: 262 failures: 185
breslow value 954.013219 direct 954.013219 rel. grad error 6.8e-09
efron value 946.636767 direct 946.636767 rel. grad error 1.9e-09
A lasso-penalized Cox model is again a simple_problem. We check it against
the KKT conditions, and check the unpenalized fit against Newton’s method using
the Hessian from coxdev.
cox = rr.glm.cox(X, times, status, tie_breaking='efron')
# a fraction of the smallest lambda giving the zero solution, ||grad at 0||_inf
lam_cox = 0.3 * np.max(np.abs(cox.smooth_objective(np.zeros(p), 'grad')))
beta_cox = rr.simple_problem(cox, rr.l1norm(p, lagrange=lam_cox)).solve(tol=1e-14, min_its=500)
print(f'lambda = {lam_cox:.2f}; lasso Cox:', np.round(beta_cox, 4))
g = cox.smooth_objective(beta_cox, 'grad')
active = beta_cox != 0
print('active-set KKT error', np.max(np.abs(g[active] + lam_cox * np.sign(beta_cox[active]))))
assert np.max(np.abs(g[active] + lam_cox * np.sign(beta_cox[active]))) < 1e-6
assert np.all(np.abs(g[~active]) <= lam_cox)
beta_mle = rr.simple_problem.smooth(cox).solve(tol=1e-14, min_its=200)
beta_nr = np.zeros(p)
for _ in range(10):
beta_nr -= np.linalg.solve(cox.hessian(beta_nr), cox.smooth_objective(beta_nr, 'grad'))
print('unpenalized: max |FISTA - Newton| =', np.max(np.abs(beta_mle - beta_nr)))
np.testing.assert_allclose(beta_mle, beta_nr, atol=1e-6)lambda = 30.89; lasso Cox: [ 0.7162 -0.6397 0.295 0. 0. -0. ]
active-set KKT error 2.1316282072803006e-14
unpenalized: max |FISTA - Newton| = 6.661338147750939e-16
Multinomial regression¶
For a response with categories the coefficients form a
matrix and the linear predictor is the matrix .
rr.mglm.multinomial(X, Y) takes as an matrix of counts (one-hot
rows for a single draw per case) and uses the overparametrized loss
(baseline=True instead fixes the first category’s coefficients at zero).
Without a penalty is identified only up to adding a common vector to every
column, but an penalty on all entries picks a unique solution, and this
is the same problem scikit-learn solves for multinomial -penalized
logistic regression, again with .
J = 3
Xm = X[:, :4]
B_true = rng.standard_normal((4, J))
prob = np.exp(Xm @ B_true)
prob /= prob.sum(1, keepdims=True)
labels = np.array([rng.choice(J, p=pr) for pr in prob])
Y = np.eye(J)[labels]
# zero counts trigger a harmless log(0) warning when the saturated
# log-likelihood is computed
with np.errstate(divide='ignore', invalid='ignore'):
multi = rr.mglm.multinomial(Xm, Y)
print('coefficient shape', multi.shape)
lam_m = 5.
B_hat = rr.simple_problem(multi, rr.l1norm(multi.shape, lagrange=lam_m)).solve(tol=1e-12, min_its=200)
print(np.round(B_hat, 4))
sk_m = sk_l1_logistic(Xm, labels, C=1 / lam_m)
print('max |regreg - sklearn| =', np.max(np.abs(sk_m.coef_.T - B_hat)))
np.testing.assert_allclose(sk_m.coef_.T, B_hat, atol=1e-6)coefficient shape (4, 3)
[[-2.0108 -0. 0.2877]
[ 0.1134 -0. -0.0749]
[-0.979 -0. 0.5412]
[-0. 1.0582 -0. ]]
max |regreg - sklearn| = 3.6984304507825527e-13