Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

scikit-learn interface

regreg.sklearn.api wraps penalized GLM-type problems as scikit-learn estimators, so that GridSearchCV, cross_val_score, clone, pipelines and so on can be used to tune and assess them. Each estimator pairs a loss with an arbitrary regreg atom as the regularizer.

familyestimators
Gaussian (least squares)gaussian, gaussian_lagrange
Huberhuber, huber_lagrange
smoothed hingehuber_svm, huber_svm_lagrange
logisticlogistic, logistic_lagrange, logistic_classifier, logistic_classifier_lagrange
probitprobit, probit_lagrange, probit_classifier, probit_classifier_lagrange
complementary log-logcloglog, cloglog_lagrange, cloglog_classifier, cloglog_classifier_lagrange
multinomialmultinomial, multinomial_lagrange, multinomial_classifier, multinomial_classifier_lagrange
Coxcox, cox_lagrange

There is no Poisson estimator (although regreg.paths has Poisson paths). The plain and _lagrange estimators are regressors: predict returns the linear predictor Xβ^X\hat\beta. The _classifier estimators return labels from predict (and probabilities from predict_proba) and are scored by accuracy.

The constructor

An estimator is built from an atom constructor and a dictionary of its arguments. For the plain estimators the penalty is atom_constructor(**atom_params), so the atom can be in bound or Lagrange form. The objective is

minimizeβ  ℓ(Xβ;y)+P(β)\text{minimize}_\beta \; \ell(X\beta; y) + \mathcal{P}(\beta)

where ℓ\ell is the negative log-likelihood (for the Gaussian, 12∥y−Xβ∥22\frac12\|y-X\beta\|_2^2; no division by nn) and P\mathcal{P} is the atom. There is no intercept: add a column of ones to X if you want one (and, if it should be unpenalized, use an atom that leaves it unpenalized, e.g. rr.weighted_l1norm with weight 0).

solve_args is passed to the solver. Its defaults are loose (coefficients are typically accurate to about 10-2), which is fine for model selection; pass e.g. solve_args={'tol': 1e-12, 'min_its': 200} when the coefficients themselves matter.

[ 1.732 -1.238  0.703  0.327 -0.     0.   ] 4.000000000000001

Since atom_params is an ordinary hyperparameter, it is what GridSearchCV searches over:

best bound: 5

Lagrange form: penalties relative to λmax⁡\lambda_{\max}

A Lagrange parameter is only meaningful relative to the scale of the data, and the scale differs between training folds. The _lagrange estimators therefore build the atom as

atom_constructor(score, **atom_params)

where score is the gradient of the loss at the null solution (β=0\beta=0, or the fit with only the unpenalized variables when unpenalized=True). For the lasso, λmax⁡=∥score∥∞\lambda_{\max} = \|\text{score}\|_\infty is the smallest λ\lambda giving β^=0\hat\beta=0, so the natural tuning parameter is a fraction of it, recomputed on every training fold:

As a check, scikit-learn’s Lasso minimizes 12n∥y−Xβ∥22+a∥β∥1\frac{1}{2n}\|y-X\beta\|_2^2 + a\|\beta\|_1, so it should agree with a=λ/na = \lambda / n:

max |regreg - sklearn| = 7.1e-15

Now tune the fraction by 5-fold CV. The default score is the negated deviance (larger is better, as GridSearchCV maximizes); score_method selects others:

frac = 0.01   CV mean deviance = 1.618
frac = 0.02   CV mean deviance = 1.553
frac = 0.05   CV mean deviance = 1.441
frac = 0.10   CV mean deviance = 1.522
frac = 0.20   CV mean deviance = 2.097
frac = 0.40   CV mean deviance = 3.683
selected: {'atom_params': {'frac': 0.05}}
nonzero coefficients of the refit: [ 0  1  2  3  4 16 18]

Elastic net and extra quadratic terms

For the Lagrange estimators, enet_alpha ∈[0,1]\in[0,1] splits the penalty between the atom and a ridge term: the atom’s Lagrange parameter λ\lambda becomes (1−enet_alpha)λ(1-\texttt{enet\_alpha})\lambda and enet_alpha λ2∥β∥22\frac{\texttt{enet\_alpha}\,\lambda}{2}\|\beta\|_2^2 is added. (Note that enet_alpha=0 is the pure lasso, the opposite convention from alpha in regreg.paths.) Every estimator also accepts quadratic, an rr.identity_quadratic added to the objective; in Lagrange form its gradient at the null solution is included in score.

max |enet - direct| = 6.7e-16

Case weights, offsets and sample_weight

scikit-learn passes only X and y through its splitters, so case weights and offsets are carried as extra columns of y. With case_weights=True the last column of y holds the weights; with offset=True the last column holds the offset; with both, the weights are second to last. The offset is subtracted from the linear predictor (the opposite of R’s convention). Because the extra columns travel with y, cross-validation subsets them along with the response.

score(X, y, sample_weight=w) uses w as case weights for scoring (multiplied into the weight column if there is one).

-201.77995047159453 -201.77995047159453
max |weighted - direct| = 0.0e+00

Classification: comparison with LogisticRegression

The _classifier estimators predict labels, so cross_val_score uses stratified folds and the accuracy score. Here an ℓ1\ell_1-penalized logistic classifier with λ=5\lambda=5 is compared with scikit-learn’s LogisticRegression with l1_ratio=1 and C=1/λC = 1/\lambda (scikit-learn minimizes C∑iloglossi+∥β∥1C\sum_i \text{logloss}_i + \|\beta\|_1) on the same folds.

regreg  accuracy by fold: [0.7   0.767 0.8   0.667 0.6  ]
sklearn accuracy by fold: [0.7   0.767 0.8   0.667 0.6  ]
max |coef difference| = 4.1e-13
predicted probabilities: [0.556 0.157 0.21  0.647]

solve_args matters for comparisons like this one: with the solver’s default tolerance the coefficients agree with scikit-learn only to about 10-2.

The same accuracy can be had from the regression estimator logistic with score_method='accuracy', whose predict returns the linear predictor instead of labels.

A group lasso through the Gaussian estimator

Any atom works. Here the 20 features form five groups of four and rr.group_lasso is the penalty; its Lagrange parameter is tuned by CV.

best lagrange: 10
group norms: [2.619 0.    0.    0.    0.091]

The first group, which holds all the signal, has by far the largest norm. As a check, the estimator’s coefficients solve the KKT conditions of the group lasso: for active groups −XgT(y−Xβ^)=−λwgβ^g/∥β^g∥2-X_g^T(y - X\hat\beta) = -\lambda w_g \hat\beta_g / \|\hat\beta_g\|_2, and for inactive groups ∥XgT(y−Xβ^)∥2≤λwg\|X_g^T(y-X\hat\beta)\|_2 \le \lambda w_g, with default weights wg=∣g∣w_g=\sqrt{|g|}.

group 0 active,   KKT residual 2.3e-14
group 1 inactive, ||grad|| / (lambda w) = 0.957
group 2 inactive, ||grad|| / (lambda w) = 0.626
group 3 inactive, ||grad|| / (lambda w) = 0.574
group 4 active,   KKT residual 4.4e-15

clone, get_params and set_params

The estimators are ordinary scikit-learn estimators: all constructor arguments are hyperparameters, so clone copies an unfitted estimator and set_params changes any of them (this is what GridSearchCV does internally).

['atom_constructor', 'atom_params', 'case_weights', 'coef', 'enet_alpha', 'initial', 'offset', 'quadratic', 'score_method', 'solve_args', 'unpenalized']
sklearn_gaussian_lagrange(atom_constructor=<function lasso_fraction at 0x7f8463e81800>,
                          atom_params={'frac': 0.05}, enet_alpha=0.25,
                          score_method='R2')