Smooth functions

Smooth functions#

regreg.smooth#

Smooth atoms: smooth functions together with their gradients.

This module defines the base class smooth_atom for smooth (differentiable) objectives, the composition affine_smooth of a smooth atom with an affine map, and a few simple smooth atoms (zero, sum).

class regreg.smooth.smooth_atom(shape, coef=1, offset=None, quadratic=None, initial=None)#

Bases: smooth

A class for representing a smooth function and its gradient.

Parameters:

shape (int or tuple) – Shape of the argument to smooth_objective.
coef (float, optional) – Scalar multiple to be applied (must be nonnegative).
offset (ndarray, optional) – Vector to be subtracted from the argument before evaluating smooth_objective.
quadratic (regreg.identity_quadratic.identity_quadratic, optional) – Quadratic term to be added to the overall objective.
initial (ndarray, optional) – Initial value for the coefficients.

smooth_objective(arg, mode='both', check_feasibility=False)#

Evaluate the smooth objective, its gradient, or both.

Abstract method: subclasses must implement it.

Parameters:

arg (ndarray) – The current parameter values.
mode ({'both', 'func', 'grad'}, optional) – Whether to return the objective value (‘func’), its gradient (‘grad’) or both (‘both’).
check_feasibility (bool, optional) – If True, return np.inf when the point is not feasible, i.e. when arg is not in the domain.

Returns:

f (float) – Objective value at arg; returned when mode is ‘func’ or ‘both’.
g (ndarray) – Gradient at arg; returned when mode is ‘grad’ or ‘both’.

classmethod affine(linear_operator, offset, coef=1, diag=False, quadratic=None, **kws)#

Compose the atom with an affine map.

The returned smooth function is \(\beta \mapsto f(X\beta + b)\) where \(X\) is linear_operator, \(b\) is offset and \(f\) is an instance of cls.

Parameters:

linear_operator (ndarray or regreg.affine.affine_transform) – Linear part of the affine map.
offset (ndarray) – Offset added to the output of linear_operator.
coef (float, optional) – Scalar multiple to be applied (must be nonnegative).
diag (bool, optional) – Whether linear_operator acts diagonally, i.e. is a rescaling. Passed to regreg.affine.linear_transform.
quadratic (regreg.identity_quadratic.identity_quadratic, optional) – Quadratic term to be added to the objective.
**kws – Further keyword arguments passed to the constructor of cls. They must be arguments of cls.__init__ other than shape, coef, quadratic, initial and offset.

Returns:

atom (affine_smooth) – The composition of the atom with the affine map.

classmethod linear(linear_operator, coef=1, diag=False, offset=None, quadratic=None, **kws)#

Compose the atom with a linear map.

The returned smooth function is \(\beta \mapsto f(X\beta)\) where \(X\) is linear_operator and \(f\) is an instance of cls.

Parameters:

linear_operator (ndarray or regreg.affine.affine_transform) – The linear map.
coef (float, optional) – Scalar multiple to be applied (must be nonnegative).
diag (bool, optional) – Whether linear_operator acts diagonally, i.e. is a rescaling.
offset (ndarray, optional) – Offset passed to the constructor of cls, i.e. subtracted from the output of linear_operator before evaluating the atom.
quadratic (regreg.identity_quadratic.identity_quadratic, optional) – Quadratic term to be added to the objective.
**kws – Further keyword arguments passed to the constructor of cls. They must be arguments of cls.__init__ other than shape, coef, quadratic, initial and offset.

Returns:

atom (affine_smooth) – The composition of the atom with the linear map.

classmethod shift(offset, coef=1, quadratic=None, **kws)#

Create an atom whose argument is shifted by offset.

The returned smooth function is \(\beta \mapsto f(\beta - \alpha)\) where \(\alpha\) is offset and \(f\) is an instance of cls.

Parameters:

offset (ndarray) – Vector subtracted from the argument; also determines the shape.
coef (float, optional) – Scalar multiple to be applied (must be nonnegative).
quadratic (regreg.identity_quadratic.identity_quadratic, optional) – Quadratic term to be added to the objective.
**kws – Further keyword arguments passed to the constructor of cls. They must be arguments of cls.__init__ other than shape, coef, quadratic, initial and offset.

Returns:

atom (cls) – The shifted atom.

scale(obj, copy=False)#

Multiply obj by the atom’s coef.

Parameters:

obj (float or ndarray) – Quantity to be scaled.
copy (bool, optional) – If True and coef is 1, return a copy of obj rather than obj itself.

Returns:

scaled (float or ndarray) – obj * self.coef.

get_conjugate()#

Return the convex conjugate of the atom.

To be implemented by subclasses.

property conjugate#: The convex conjugate of the atom.

regreg.smooth.combine_case_weights(case_weights, loss_case_weights, shape)#

Combine case weights passed to a loss with the loss’s own case weights.

Parameters:

case_weights (ndarray or None) – Case weights passed in by the caller (e.g. a glm). Treated as all ones if None. Never modified.
loss_case_weights (ndarray or None) – Case weights stored on the loss itself. Ignored if None.
shape (tuple) – Shape of the weights when case_weights is None.

Returns:

cw (ndarray) – A new float array, the product of the two sets of weights.

regreg.smooth.acceptable_init_args(cls, proposed_keywords)#

Check that keywords are valid extra arguments to cls.__init__.

Parameters:

cls (type) – Class whose constructor will receive the keywords.
proposed_keywords (dict) – Proposed keyword arguments.

Returns:

acceptable (bool) – True if every key of proposed_keywords is an argument of cls.__init__ and is not one of self, shape, coef, quadratic, initial or offset.

class regreg.smooth.affine_smooth(smooth_atom, atransform, store_grad=True, diag=False)#

Bases: smooth_atom

Composition of a smooth objective with an affine transform.

Represents \(\beta \mapsto f(X\beta + b)\) where \(f\) is a smooth atom and \(\beta \mapsto X\beta + b\) an affine transform.

smooth_objective(arg, mode='both', check_feasibility=False)#

Evaluate the composed objective, its gradient, or both.

The atom is evaluated at self.affine_transform.affine_map(arg) and its gradient is pulled back by the adjoint of the transform.

Parameters:

arg (ndarray) – The current parameter values.
mode ({'both', 'func', 'grad'}, optional) – Whether to return the objective value (‘func’), its gradient (‘grad’) or both (‘both’).
check_feasibility (bool, optional) – If True, return np.inf when the point is not feasible, i.e. when arg is not in the domain.

Returns:

f (float) – Objective value at arg; returned when mode is ‘func’ or ‘both’.
g (ndarray) – Gradient at arg; returned when mode is ‘grad’ or ‘both’.

property dual#

The (affine_transform, conjugate) pair for the dual problem.

None if the conjugate of the underlying atom is not available.

latexify(var=None, idx='')#

Return a LaTeX representation of the composed objective.

Parameters:

var (str, optional) – LaTeX name of the argument.
idx (str, optional) – Index appended as a subscript.

Returns:

latex (str)

class regreg.smooth.zero(shape, coef=1, offset=None, quadratic=None, initial=None)#

Bases: smooth_atom

The zero function.

smooth_objective(x, mode='both', check_feasibility=False)#

Evaluate the zero function, its (zero) gradient, or both.

Parameters:

x (ndarray) – The current parameter values.
mode ({'both', 'func', 'grad'}, optional) – Whether to return the objective value (‘func’), its gradient (‘grad’) or both (‘both’).
check_feasibility (bool, optional) – If True, return np.inf when the point is not feasible, i.e. when x is not in the domain.

Returns:

f (float) – Objective value at x; returned when mode is ‘func’ or ‘both’.
g (ndarray) – Gradient at x; returned when mode is ‘grad’ or ‘both’.

class regreg.smooth.sum(atoms, weights=None)#

Bases: smooth_atom

A weighted sum of smooth atoms.

smooth_objective(x, mode='both', check_feasibility=False)#

Evaluate the weighted sum of the atoms, its gradient, or both.

Parameters:

x (ndarray) – The current parameter values.
mode ({'both', 'func', 'grad'}, optional) – Whether to return the objective value (‘func’), its gradient (‘grad’) or both (‘both’).
check_feasibility (bool, optional) – If True, return np.inf when the point is not feasible, i.e. when x is not in the domain.

Returns:

f (float) – Objective value at x; returned when mode is ‘func’ or ‘both’.
g (ndarray) – Gradient at x; returned when mode is ‘grad’ or ‘both’.

regreg.smooth.glm#

Generalized linear model losses.

These are losses of the form \(\beta \mapsto \ell(X\beta)\) for a saturated loss \(\ell\). The glm class composes a saturated loss (e.g. gaussian_loglike, poisson_loglike, huber_loss, or the binary and Cox losses imported from regreg.smooth.binary and regreg.smooth.cox) with a design matrix, and has constructors for the common models.

class regreg.smooth.glm.glm(X, Y, loss, quadratic=None, initial=None, offset=None, case_weights=None)#

Bases: smooth_atom

A general linear model, usually a negative log-likelihood.

The loss is \(\beta \mapsto \ell(X\beta)\) for a response \(Y\) whose mean is modelled through \(X\beta\), where \(\ell\) is a saturated loss.

Usual examples are Gaussian (least squares regression), logistic regression and Poisson log-linear regression. The Huber loss is also implemented as an example.

smooth_objective(beta, mode='func', check_feasibility=False)#

Evaluate the loss \(\ell(X\beta)\), its gradient, or both.

Parameters:

beta (ndarray) – The current parameter values.
mode ({'both', 'func', 'grad'}, optional) – Whether to return the objective value (‘func’), its gradient (‘grad’) or both (‘both’).
check_feasibility (bool, optional) – If True, return np.inf when the point is not feasible, i.e. when beta is not in the domain.

Returns:

f (float) – Objective value at beta; returned when mode is ‘func’ or ‘both’.
g (ndarray) – Gradient at beta; returned when mode is ‘grad’ or ‘both’.

get_data()#: Return the data (X, Y).

set_data(data)#

Set the data.

Parameters:: data (tuple) – The pair (X, Y) of design and response.

property data#: Data for the general linear model.

linear_predictor(beta)#

Compute the linear predictor \(X\beta\).

Parameters:: beta (ndarray) – Parameters.
Returns:: linpred (ndarray) – The linear predictor.

objective(beta)#

Compute the loss \(\ell(X\beta)\).

Parameters:: beta (ndarray) – Parameters.
Returns:: objective (float) – Value of the loss at \(\beta\).

gradient(beta)#

Compute the gradient of the loss \(\nabla \ell(X\beta)\).

Parameters:: beta (ndarray) – Parameters.
Returns:: grad (ndarray) – Gradient of the loss at \(\beta\).

hessian(beta)#

Compute the Hessian of the loss \(\nabla^2 \ell(X\beta)\).

Requires the design to be an ndarray or a scipy.sparse matrix, and the saturated loss to have a hessian or hessian_mult method.

Parameters:: beta (ndarray) – Parameters.
Returns:: hess (ndarray) – Hessian of the loss at \(\beta\), i.e. \(X^T W X\).

latexify(var=None, idx='')#

Return a LaTeX representation of the loss.

Parameters:

var (str, optional) – LaTeX name of the argument.
idx (str, optional) – Index appended as a subscript.

Returns:

latex (str)

subsample(idx)#

Create a loss using a subsample of the data.

Makes a copy of the loss and multiplies its case weights by the indicator of idx.

Parameters:: idx (index) – Indices of np.arange(n) to keep.
Returns:: subsample_loss (glm) – Copy of the loss with zero case weight for all cases not in idx.

classmethod gaussian(X, response, case_weights=None, coef=1.0, saturated_offset=None, offset=None, quadratic=None, initial=None)#

Create a loss for a Gaussian regression model.

Parameters:

X (ndarray or regreg.affine.affine_transform) – Design matrix.
response (ndarray) – Response vector.
case_weights (ndarray, optional) – Non-negative case weights.
coef (float, optional) – Nonnegative scaling to be put in front of the loss.
saturated_offset (ndarray, optional) – Offset to be subtracted in the saturated (natural) parameter space before evaluating the loss.
offset (ndarray, optional) – Offset to be subtracted in parameter space before evaluating the loss.
quadratic (regreg.identity_quadratic.identity_quadratic, optional) – Quadratic term to be added to the objective.
initial (ndarray, optional) – Initial guess at the coefficients.

Returns:

glm_obj (regreg.smooth.glm.glm) – General linear model loss.

classmethod logistic(X, successes, trials=None, case_weights=None, coef=1.0, offset=None, saturated_offset=None, quadratic=None, initial=None)#

Create a loss for a logistic regression model.

Parameters:

X (ndarray or regreg.affine.affine_transform) – Design matrix.
successes (ndarray) – Number of successes for each case (non-negative).
trials (ndarray, optional) – Number of trials for each case. If None, the response must be binary and np.ones_like(successes) is used.
case_weights (ndarray, optional) – Non-negative case weights.
coef (float, optional) – Nonnegative scaling to be put in front of the loss.
offset (ndarray, optional) – Offset to be subtracted in parameter space before evaluating the loss.
saturated_offset (ndarray, optional) – Offset to be subtracted in the saturated (natural) parameter space before evaluating the loss.
quadratic (regreg.identity_quadratic.identity_quadratic, optional) – Quadratic term to be added to the objective.
initial (ndarray, optional) – Initial guess at the coefficients.

Returns:

glm_obj (regreg.smooth.glm.glm) – General linear model loss.

classmethod probit(X, successes, trials=None, case_weights=None, coef=1.0, offset=None, saturated_offset=None, quadratic=None, initial=None)#

Create a loss for a probit regression model.

Parameters:

X (ndarray or regreg.affine.affine_transform) – Design matrix.
successes (ndarray) – Number of successes for each case (non-negative).
trials (ndarray, optional) – Number of trials for each case. If None, the response must be binary and np.ones_like(successes) is used.
case_weights (ndarray, optional) – Non-negative case weights.
coef (float, optional) – Nonnegative scaling to be put in front of the loss.
offset (ndarray, optional) – Offset to be subtracted in parameter space before evaluating the loss.
saturated_offset (ndarray, optional) – Offset to be subtracted in the saturated (natural) parameter space before evaluating the loss.
quadratic (regreg.identity_quadratic.identity_quadratic, optional) – Quadratic term to be added to the objective.
initial (ndarray, optional) – Initial guess at the coefficients.

Returns:

glm_obj (regreg.smooth.glm.glm) – General linear model loss.

classmethod cloglog(X, successes, trials=None, case_weights=None, coef=1.0, offset=None, saturated_offset=None, quadratic=None, initial=None)#

Create a loss for a complementary log-log regression model.

Parameters:

X (ndarray or regreg.affine.affine_transform) – Design matrix.
successes (ndarray) – Number of successes for each case (non-negative).
trials (ndarray, optional) – Number of trials for each case. If None, the response must be binary and np.ones_like(successes) is used.
case_weights (ndarray, optional) – Non-negative case weights.
coef (float, optional) – Nonnegative scaling to be put in front of the loss.
offset (ndarray, optional) – Offset to be subtracted in parameter space before evaluating the loss.
saturated_offset (ndarray, optional) – Offset to be subtracted in the saturated (natural) parameter space before evaluating the loss.
quadratic (regreg.identity_quadratic.identity_quadratic, optional) – Quadratic term to be added to the objective.
initial (ndarray, optional) – Initial guess at the coefficients.

Returns:

glm_obj (regreg.smooth.glm.glm) – General linear model loss.

classmethod poisson(X, counts, case_weights=None, coef=1.0, saturated_offset=None, offset=None, quadratic=None, initial=None)#

Create a loss for a Poisson regression model.

Parameters:

X (ndarray or regreg.affine.affine_transform) – Design matrix.
counts (ndarray) – Response vector of non-negative integer counts.
case_weights (ndarray, optional) – Non-negative case weights.
coef (float, optional) – Nonnegative scaling to be put in front of the loss.
saturated_offset (ndarray, optional) – Offset to be subtracted in the saturated (natural) parameter space before evaluating the loss.
offset (ndarray, optional) – Offset to be subtracted in parameter space before evaluating the loss.
quadratic (regreg.identity_quadratic.identity_quadratic, optional) – Quadratic term to be added to the objective.
initial (ndarray, optional) – Initial guess at the coefficients.

Returns:

glm_obj (regreg.smooth.glm.glm) – General linear model loss.

classmethod huber(X, response, smoothing_parameter, case_weights=None, coef=1.0, saturated_offset=None, offset=None, quadratic=None, initial=None)#

Create a loss for a regression model using the Huber loss.

Parameters:

X (ndarray or regreg.affine.affine_transform) – Design matrix.
response (ndarray) – Response vector.
smoothing_parameter (float) – Smoothing parameter for the Huber loss.
case_weights (ndarray, optional) – Non-negative case weights.
coef (float, optional) – Nonnegative scaling to be put in front of the loss.
saturated_offset (ndarray, optional) – Offset to be subtracted in the saturated (natural) parameter space before evaluating the loss.
offset (ndarray, optional) – Offset to be subtracted in parameter space before evaluating the loss.
quadratic (regreg.identity_quadratic.identity_quadratic, optional) – Quadratic term to be added to the objective.
initial (ndarray, optional) – Initial guess at the coefficients.

Returns:

glm_obj (regreg.smooth.glm.glm) – General linear model loss.

classmethod huber_svm(X, successes, smoothing_parameter, case_weights=None, coef=1.0, offset=None, saturated_offset=None, quadratic=None, initial=None)#

Create a loss for a binary model using the Huber SVM loss.

Parameters:

X (ndarray or regreg.affine.affine_transform) – Design matrix.
successes (ndarray) – Binary (0/1) response vector.
smoothing_parameter (float) – Smoothing parameter for the Huberized hinge loss.
case_weights (ndarray, optional) – Non-negative case weights.
coef (float, optional) – Nonnegative scaling to be put in front of the loss.
offset (ndarray, optional) – Offset to be subtracted in parameter space before evaluating the loss.
saturated_offset (ndarray, optional) – Offset to be subtracted in the saturated (natural) parameter space before evaluating the loss.
quadratic (regreg.identity_quadratic.identity_quadratic, optional) – Quadratic term to be added to the objective.
initial (ndarray, optional) – Initial guess at the coefficients.

Returns:

glm_obj (regreg.smooth.glm.glm) – General linear model loss.

classmethod cox(X, event_times, censoring, case_weights=None, coef=1.0, saturated_offset=None, offset=None, quadratic=None, initial=None, tie_breaking='breslow')#

Create a loss for a Cox proportional hazards regression model.

Parameters:

X (ndarray or regreg.affine.affine_transform) – Design matrix.
event_times (ndarray) – Observed times for the Cox proportional hazards model.
censoring (ndarray) – Censoring indicator for the Cox proportional hazards model: 1 indicates the observation is a failure, 0 a censored observation.
case_weights (ndarray, optional) – Non-negative case weights.
coef (float, optional) – Nonnegative scaling to be put in front of the loss.
saturated_offset (ndarray, optional) – Offset to be subtracted in the saturated (natural) parameter space before evaluating the loss.
offset (ndarray, optional) – Offset to be subtracted in parameter space before evaluating the loss.
quadratic (regreg.identity_quadratic.identity_quadratic, optional) – Quadratic term to be added to the objective.
initial (ndarray, optional) – Initial guess at the coefficients.
tie_breaking ({'breslow', 'efron'}, optional) – Method for handling tied event times.

Returns:

glm_obj (regreg.smooth.glm.glm) – General linear model loss.

class regreg.smooth.glm.gaussian_loglike(shape, response, coef=1.0, offset=None, quadratic=None, initial=None, case_weights=None)#

Bases: smooth_atom

The Gaussian loss for observations \(y\).

\[\mu \mapsto \frac{1}{2} \sum_i w_i (y_i - \mu_i)^2\]

where \(w_i\) are the case weights.

Parameters:

shape (int or tuple) – Shape of the natural parameter.
response (ndarray) – Response vector \(y\).
coef (float, optional) – Nonnegative scaling to be put in front of the loss.
offset (ndarray, optional) – Offset to be subtracted from the natural parameter before evaluating the loss.
quadratic (regreg.identity_quadratic.identity_quadratic, optional) – Quadratic term to be added to the objective.
initial (ndarray, optional) – Initial guess at the coefficients.
case_weights (ndarray, optional) – Non-negative case weights.

smooth_objective(natural_param, mode='both', check_feasibility=False, case_weights=None)#

Evaluate the smooth objective, its gradient, or both.

Parameters:

natural_param (ndarray) – The current parameter values.
mode ({'both', 'func', 'grad'}, optional) – Whether to return the objective value (‘func’), its gradient (‘grad’) or both (‘both’).
check_feasibility (bool, optional) – If True, return np.inf when the point is not feasible, i.e. when natural_param is not in the domain.
case_weights (ndarray, optional) – Non-negative case weights, multiplied with any case weights stored on the loss.

Returns:

f (float) – Objective value at natural_param; returned when mode is ‘func’ or ‘both’.
g (ndarray) – Gradient at natural_param; returned when mode is ‘grad’ or ‘both’.

hessian(natural_param, case_weights=None)#

Compute the diagonal of the Hessian of the loss.

Parameters:

natural_param (ndarray) – Parameters at which the Hessian is evaluated.
case_weights (ndarray, optional) – Non-negative case weights.

Returns:

hess (ndarray) – 1D array holding the diagonal of the Hessian evaluated at natural_param.

get_data()#: Return the response.

set_data(data)#: Set the response.

property data#: Return the response.

subsample(case_idx)#

Create a saturated loss using a subsample of the data.

Makes a copy of the loss and multiplies its case weights by the indicator of case_idx.

Parameters:: case_idx (index) – Indices of np.arange(n) to keep.
Returns:: subsample_loss (regreg.smooth.smooth_atom) – Copy of the loss with zero case weight for all cases not in case_idx.

mean_function(eta)#: Mean function (inverse link), the identity.

class regreg.smooth.glm.poisson_loglike(shape, counts, coef=1.0, offset=None, quadratic=None, initial=None, case_weights=None)#

Bases: smooth_atom

The Poisson negative log-likelihood with log link.

\[\eta \mapsto \sum_i w_i \left(e^{\eta_i} - y_i \eta_i\right)\]

where \(y_i\) are the counts and \(w_i\) the case weights. A constant term is added through quadratic.

Parameters:

shape (int or tuple) – Shape of the natural parameter.
counts (ndarray) – Non-negative integer counts.
coef (float, optional) – Nonnegative scaling to be put in front of the loss.
offset (ndarray, optional) – Offset to be subtracted from the natural parameter before evaluating the loss.
quadratic (regreg.identity_quadratic.identity_quadratic, optional) – Quadratic term to be added to the objective.
initial (ndarray, optional) – Initial guess at the coefficients.
case_weights (ndarray, optional) – Non-negative case weights.

smooth_objective(natural_param, mode='both', check_feasibility=False, case_weights=None)#

Evaluate the smooth objective, its gradient, or both.

Parameters:

natural_param (ndarray) – The current parameter values.
mode ({'both', 'func', 'grad'}, optional) – Whether to return the objective value (‘func’), its gradient (‘grad’) or both (‘both’).
check_feasibility (bool, optional) – If True, return np.inf when the point is not feasible, i.e. when natural_param is not in the domain.
case_weights (ndarray, optional) – Non-negative case weights, multiplied with any case weights stored on the loss.

Returns:

f (float) – Objective value at natural_param; returned when mode is ‘func’ or ‘both’.
g (ndarray) – Gradient at natural_param; returned when mode is ‘grad’ or ‘both’.

hessian(natural_param, case_weights=None)#

Compute the diagonal of the Hessian of the loss.

Parameters:

natural_param (ndarray) – Parameters at which the Hessian is evaluated.
case_weights (ndarray, optional) – Non-negative case weights.

Returns:

hess (ndarray) – 1D array holding the diagonal of the Hessian evaluated at natural_param.

get_data()#: Return the counts.

set_data(data)#: Set the counts.

property data#: Return the counts.

subsample(case_idx)#

Create a saturated loss using a subsample of the data.

Makes a copy of the loss and multiplies its case weights by the indicator of case_idx.

Parameters:: case_idx (index) – Indices of np.arange(n) to keep.
Returns:: subsample_loss (regreg.smooth.smooth_atom) – Copy of the loss with zero case weight for all cases not in case_idx.

mean_function(eta)#: Mean function (inverse link), \(\eta \mapsto e^{\eta}\).

class regreg.smooth.glm.huber_loss(shape, response, smoothing_parameter, coef=1.0, offset=None, quadratic=None, initial=None, case_weights=None)#

Bases: smooth_atom

The Huber loss of the residuals.

\[\begin{split}\mu \mapsto \sum_i w_i \, h_{\epsilon}(\mu_i - y_i), \qquad h_{\epsilon}(r) = \begin{cases} r^2 / (2 \epsilon) & |r| \leq \epsilon \\ |r| - \epsilon / 2 & |r| > \epsilon \end{cases}\end{split}\]

where \(\epsilon\) is smoothing_parameter and \(w_i\) the case weights.

Parameters:

shape (int or tuple) – Shape of the natural parameter.
response (ndarray) – Response vector \(y\).
smoothing_parameter (float) – Smoothing parameter for the Huber loss.
coef (float, optional) – Nonnegative scaling to be put in front of the loss.
offset (ndarray, optional) – Offset to be subtracted from the natural parameter before evaluating the loss.
quadratic (regreg.identity_quadratic.identity_quadratic, optional) – Quadratic term to be added to the objective.
initial (ndarray, optional) – Initial guess at the coefficients.
case_weights (ndarray, optional) – Non-negative case weights.

smooth_objective(param, mode='both', check_feasibility=False, case_weights=None)#

Evaluate the smooth objective, its gradient, or both.

Parameters:

param (ndarray) – The current parameter values.
mode ({'both', 'func', 'grad'}, optional) – Whether to return the objective value (‘func’), its gradient (‘grad’) or both (‘both’).
check_feasibility (bool, optional) – If True, return np.inf when the point is not feasible, i.e. when param is not in the domain.
case_weights (ndarray, optional) – Non-negative case weights, multiplied with any case weights stored on the loss.

Returns:

f (float) – Objective value at param; returned when mode is ‘func’ or ‘both’.
g (ndarray) – Gradient at param; returned when mode is ‘grad’ or ‘both’.

hessian(param, case_weights=None)#

Hessian of the loss (not implemented).

The Huber loss is not twice differentiable, so this always raises NotImplementedError.

Parameters:

param (ndarray) – Parameters at which the Hessian would be evaluated.
case_weights (ndarray, optional) – Non-negative case weights.

Raises:

NotImplementedError –

get_data()#: Return the response.

set_data(data)#: Set the response.

property data#: Return the response.

subsample(case_idx)#

Create a saturated loss using a subsample of the data.

Makes a copy of the loss and multiplies its case weights by the indicator of case_idx.

Parameters:: case_idx (index) – Indices of np.arange(n) to keep.
Returns:: subsample_loss (regreg.smooth.smooth_atom) – Copy of the loss with zero case weight for all cases not in case_idx.

class regreg.smooth.glm.stacked_loglike(losses, coef=1.0, offset=None, quadratic=None, initial=None, case_weights=None)#

Bases: smooth_atom

Stack K = len(losses) saturated losses, summed over the losses.

The natural parameter is the concatenation of the natural parameters of the individual losses. Roughly speaking, this is a model of K independent measurements per individual.

Parameters:

losses (sequence) – Saturated losses to be stacked.
coef (float, optional) – Nonnegative scaling to be put in front of the loss.
offset (ndarray, optional) – Offset to be subtracted from the natural parameter before evaluating the loss.
quadratic (regreg.identity_quadratic.identity_quadratic, optional) – Quadratic term to be added to the objective.
initial (ndarray, optional) – Initial guess at the coefficients.
case_weights (ndarray, optional) – Non-negative case weights.

smooth_objective(natural_param, mode='both', check_feasibility=False, case_weights=None)#

Evaluate the smooth objective, its gradient, or both.

Parameters:

natural_param (ndarray) – The current parameter values.
mode ({'both', 'func', 'grad'}, optional) – Whether to return the objective value (‘func’), its gradient (‘grad’) or both (‘both’).
check_feasibility (bool, optional) – If True, return np.inf when the point is not feasible, i.e. when natural_param is not in the domain.
case_weights (ndarray, optional) – Non-negative case weights, of the same shape as natural_param. These are combined with this loss’s own case weights and passed on to the stacked losses.

Returns:

f (float) – Objective value at natural_param; returned when mode is ‘func’ or ‘both’.
g (ndarray) – Gradient at natural_param; returned when mode is ‘grad’ or ‘both’.

get_data()#: Return the stacked responses.

set_data(data)#: Set the stacked responses.

property data#: Return the stacked responses.

subsample(case_idx)#

Create a saturated loss using a subsample of the data.

Makes a copy of the loss and multiplies its case weights by the indicator of case_idx.

Parameters:: case_idx (index) – Indices of np.arange(n) to keep.
Returns:: subsample_loss (regreg.smooth.smooth_atom) – Copy of the loss with zero case weight for all cases not in case_idx.

classmethod gaussian(responses, case_weights=None, coef=1.0, offset=None, quadratic=None, initial=None)#

Create a stacked loss of Gaussian losses.

Parameters:

responses (sequence of ndarray) – Response vectors, one per stacked loss.
case_weights (ndarray, optional) – Non-negative case weights.
coef (float, optional) – Nonnegative scaling to be put in front of the loss.
offset (ndarray, optional) – Offset to be subtracted in parameter space before evaluating the loss.
quadratic (regreg.identity_quadratic.identity_quadratic, optional) – Quadratic term to be added to the objective.
initial (ndarray, optional) – Initial guess at the coefficients.

Returns:

stacked_loss (stacked_loglike) – Stacked saturated loss.

classmethod logistic(successes, trials=None, case_weights=None, coef=1.0, offset=None, quadratic=None, initial=None)#

Create a stacked loss of logistic losses.

Parameters:

successes (sequence of ndarray) – Numbers of successes, one array per stacked loss.
trials (sequence of ndarray, optional) – Numbers of trials, one array per stacked loss.
case_weights (ndarray, optional) – Non-negative case weights.
coef (float, optional) – Nonnegative scaling to be put in front of the loss.
offset (ndarray, optional) – Offset to be subtracted in parameter space before evaluating the loss.
quadratic (regreg.identity_quadratic.identity_quadratic, optional) – Quadratic term to be added to the objective.
initial (ndarray, optional) – Initial guess at the coefficients.

Returns:

stacked_loss (stacked_loglike) – Stacked saturated loss.

classmethod probit(successes, trials=None, case_weights=None, coef=1.0, offset=None, quadratic=None, initial=None)#

Create a stacked loss of probit losses.

Parameters:

successes (sequence of ndarray) – Numbers of successes, one array per stacked loss.
trials (sequence of ndarray, optional) – Numbers of trials, one array per stacked loss.
case_weights (ndarray, optional) – Non-negative case weights.
coef (float, optional) – Nonnegative scaling to be put in front of the loss.
offset (ndarray, optional) – Offset to be subtracted in parameter space before evaluating the loss.
quadratic (regreg.identity_quadratic.identity_quadratic, optional) – Quadratic term to be added to the objective.
initial (ndarray, optional) – Initial guess at the coefficients.

Returns:

stacked_loss (stacked_loglike) – Stacked saturated loss.

classmethod cloglog(successes, trials=None, case_weights=None, coef=1.0, offset=None, quadratic=None, initial=None)#

Create a stacked loss of complementary log-log losses.

Parameters:

successes (sequence of ndarray) – Numbers of successes, one array per stacked loss.
trials (sequence of ndarray, optional) – Numbers of trials, one array per stacked loss.
case_weights (ndarray, optional) – Non-negative case weights.
coef (float, optional) – Nonnegative scaling to be put in front of the loss.
offset (ndarray, optional) – Offset to be subtracted in parameter space before evaluating the loss.
quadratic (regreg.identity_quadratic.identity_quadratic, optional) – Quadratic term to be added to the objective.
initial (ndarray, optional) – Initial guess at the coefficients.

Returns:

stacked_loss (stacked_loglike) – Stacked saturated loss.

classmethod poisson(counts, case_weights=None, coef=1.0, offset=None, quadratic=None, initial=None)#

Create a stacked loss of Poisson losses.

Parameters:

counts (sequence of ndarray) – Non-negative integer counts, one array per stacked loss.
case_weights (ndarray, optional) – Non-negative case weights.
coef (float, optional) – Nonnegative scaling to be put in front of the loss.
offset (ndarray, optional) – Offset to be subtracted in parameter space before evaluating the loss.
quadratic (regreg.identity_quadratic.identity_quadratic, optional) – Quadratic term to be added to the objective.
initial (ndarray, optional) – Initial guess at the coefficients.

Returns:

stacked_loss (stacked_loglike) – Stacked saturated loss.

classmethod huber(responses, smoothing_parameter, case_weights=None, coef=1.0, offset=None, quadratic=None, initial=None)#

Create a stacked loss of Huber losses.

Parameters:

responses (sequence of ndarray) – Response vectors, one per stacked loss.
smoothing_parameter (float) – Smoothing parameter for the Huber loss.
case_weights (ndarray, optional) – Non-negative case weights.
coef (float, optional) – Nonnegative scaling to be put in front of the loss.
offset (ndarray, optional) – Offset to be subtracted in parameter space before evaluating the loss.
quadratic (regreg.identity_quadratic.identity_quadratic, optional) – Quadratic term to be added to the objective.
initial (ndarray, optional) – Initial guess at the coefficients.

Returns:

stacked_loss (stacked_loglike) – Stacked saturated loss.

classmethod huber_svm(successes, smoothing_parameter, case_weights=None, coef=1.0, offset=None, quadratic=None, initial=None)#

Create a stacked loss of Huber SVM losses.

Parameters:

successes (sequence of ndarray) – Binary (0/1) responses, one array per stacked loss.
smoothing_parameter (float) – Smoothing parameter for the Huberized hinge loss.
case_weights (ndarray, optional) – Non-negative case weights.
coef (float, optional) – Nonnegative scaling to be put in front of the loss.
offset (ndarray, optional) – Offset to be subtracted in parameter space before evaluating the loss.
quadratic (regreg.identity_quadratic.identity_quadratic, optional) – Quadratic term to be added to the objective.
initial (ndarray, optional) – Initial guess at the coefficients.

Returns:

stacked_loss (stacked_loglike) – Stacked saturated loss.

classmethod cox(event_times, censoring, coef=1.0, offset=None, quadratic=None, initial=None, case_weights=None)#

Create a stacked loss of Cox proportional hazards losses.

Parameters:

event_times (sequence of ndarray) – Observed times, one array per stacked loss.
censoring (sequence of ndarray) – Censoring indicators (1 for a failure, 0 for censored), one array per stacked loss.
coef (float, optional) – Nonnegative scaling to be put in front of the loss.
offset (ndarray, optional) – Offset to be subtracted in parameter space before evaluating the loss.
quadratic (regreg.identity_quadratic.identity_quadratic, optional) – Quadratic term to be added to the objective.
initial (ndarray, optional) – Initial guess at the coefficients.
case_weights (ndarray, optional) – Non-negative case weights.

Returns:

stacked_loss (stacked_loglike) – Stacked saturated loss.

regreg.smooth.glm.logistic_loss(X, Y, trials=None, coef=1.0)#

Construct a logistic loss for successes Y and design X.

This function is deprecated, use regreg.smooth.glm.glm.logistic instead.

Parameters:

X (ndarray or regreg.affine.affine_transform) – Design matrix.
Y (ndarray) – Number of successes.
trials (ndarray, optional) – Number of trials. If None, Y must be binary.
coef (float, optional) – Nonnegative scaling to be put in front of the loss.

Returns:

glm_obj (regreg.smooth.glm.glm) – General linear model loss.

regreg.smooth.mglm#

Multiresponse generalized linear models.

Losses of the form \(B \mapsto \ell(XB)\) where the coefficients \(B\) form a matrix with one column per response (or category), e.g. multinomial regression.

class regreg.smooth.mglm.mglm(X, loss, quadratic=None, initial=None, offset=None, case_weights=None)#

Bases: glm

A multiresponse general linear model.

Usually a negative log-likelihood, the loss is \(B \mapsto \ell(XB)\) for a response \(Y\) whose mean is modelled through \(XB\), where \(B\) has shape (p, q).

Usual examples are multivariate Gaussian (least squares regression) and multinomial regression.

subsample(idx)#

Create a loss using a subsample of the data.

Makes a copy of the loss and multiplies its case weights by the indicator of idx.

Parameters:: idx (index) – Indices of np.arange(n) to keep.
Returns:: subsample_loss (mglm) – Copy of the loss with zero case weight for all cases not in idx.

hessian(beta)#

Compute the Hessian of the loss \(\nabla^2 \ell(XB)\).

Not implemented for multiresponse models.

Parameters:: beta (ndarray) – Parameters.
Raises:: NotImplementedError –

latexify(var=None, idx='')#

Return a LaTeX representation of the loss.

Parameters:

var (str, optional) – LaTeX name of the argument.
idx (str, optional) – Index appended as a subscript.

Returns:

latex (str)

classmethod multinomial(X, successes, case_weights=None, coef=1.0, saturated_offset=None, offset=None, quadratic=None, initial=None, baseline=False)#

Create a loss for a multinomial regression model.

Parameters:

X (ndarray or regreg.affine.affine_transform) – Design matrix.
successes (ndarray) – Responses of shape (n, J): non-negative counts for each of the J categories.
case_weights (ndarray, optional) – Non-negative case weights.
coef (float, optional) – Nonnegative scaling to be put in front of the loss.
saturated_offset (ndarray, optional) – Offset to be subtracted in the saturated (natural) parameter space before evaluating the loss.
offset (ndarray, optional) – Offset to be subtracted in parameter space before evaluating the loss.
quadratic (regreg.identity_quadratic.identity_quadratic, optional) – Quadratic term to be added to the objective.
initial (ndarray, optional) – Initial guess at the coefficients.
baseline (bool, optional) – If True, use the baseline-category parametrization (multinomial_baseline_loglike) with coefficients for the first J - 1 categories only; otherwise use the overparametrized multinomial_loglike.

Returns:

mglm_obj (regreg.smooth.mglm.mglm) – Multiresponse general linear model loss.

classmethod stacked(X, losses, case_weights=None, coef=1.0, offset=None, saturated_offset=None, quadratic=None, initial=None)#

Create a multiresponse model from a stack of saturated losses.

Parameters:

X (ndarray or regreg.affine.affine_transform) – Design matrix.
losses (sequence) – Sequence of saturated losses with common shape.
case_weights (ndarray, optional) – Non-negative case weights.
coef (float, optional) – Nonnegative scaling to be put in front of the loss.
offset (ndarray, optional) – Offset to be subtracted in parameter space before evaluating the loss.
saturated_offset (ndarray, optional) – Offset to be subtracted in the saturated (natural) parameter space before evaluating the loss.
quadratic (regreg.identity_quadratic.identity_quadratic, optional) – Quadratic term to be added to the objective.
initial (ndarray, optional) – Initial guess at the coefficients.

Returns:

mglm_obj (regreg.smooth.mglm.mglm) – Multiresponse general linear model loss.

class regreg.smooth.mglm.stacked_common_loglike(losses, coef=1.0, offset=None, quadratic=None, initial=None, case_weights=None)#

Bases: smooth_atom

Stack K = len(losses) saturated losses with common shapes.

The natural parameter has one column per loss, and the objective is the sum of the losses evaluated on the corresponding columns. Roughly speaking, this is a model of K independent measurements per individual.

Parameters:

losses (sequence) – Saturated losses, all with the same shape.
coef (float, optional) – Nonnegative scaling to be put in front of the loss.
offset (ndarray, optional) – Offset to be subtracted from the natural parameter before evaluating the loss.
quadratic (regreg.identity_quadratic.identity_quadratic, optional) – Quadratic term to be added to the objective.
initial (ndarray, optional) – Initial guess at the coefficients.
case_weights (ndarray, optional) – Non-negative case weights.

smooth_objective(natural_param, mode='both', check_feasibility=False, case_weights=None)#

Evaluate the smooth objective, its gradient, or both.

Parameters:

natural_param (ndarray) – The current parameter values.
mode ({'both', 'func', 'grad'}, optional) – Whether to return the objective value (‘func’), its gradient (‘grad’) or both (‘both’).
check_feasibility (bool, optional) – If True, return np.inf when the point is not feasible, i.e. when natural_param is not in the domain.
case_weights (ndarray, optional) – Non-negative case weights, multiplied with any case weights stored on the loss.

Returns:

f (float) – Objective value at natural_param; returned when mode is ‘func’ or ‘both’.
g (ndarray) – Gradient at natural_param; returned when mode is ‘grad’ or ‘both’.

get_data()#: Return the stacked responses.

set_data(data)#: Set the stacked responses.

property data#: Return the stacked responses.

subsample(case_idx)#

Create a saturated loss using a subsample of the data.

Makes a copy of the loss and multiplies its case weights by the indicator of case_idx.

Parameters:: case_idx (index) – Indices of np.arange(n) to keep.
Returns:: subsample_loss (regreg.smooth.smooth_atom) – Copy of the loss with zero case weight for all cases not in case_idx.

classmethod gaussian(responses, case_weights=None, coef=1.0, offset=None, quadratic=None, initial=None)#

Create a stacked loss of Gaussian losses.

Parameters:

responses (sequence of ndarray) – Response vectors, one per stacked loss.
case_weights (ndarray, optional) – Non-negative case weights.
coef (float, optional) – Nonnegative scaling to be put in front of the loss.
offset (ndarray, optional) – Offset to be subtracted in parameter space before evaluating the loss.
quadratic (regreg.identity_quadratic.identity_quadratic, optional) – Quadratic term to be added to the objective.
initial (ndarray, optional) – Initial guess at the coefficients.

Returns:

stacked_loss (stacked_common_loglike) – Stacked saturated loss.

classmethod logistic(successes, trials=None, case_weights=None, coef=1.0, offset=None, quadratic=None, initial=None)#

Create a stacked loss of logistic losses.

Parameters:

successes (sequence of ndarray) – Numbers of successes, one array per stacked loss.
trials (sequence of ndarray, optional) – Numbers of trials, one array per stacked loss.
case_weights (ndarray, optional) – Non-negative case weights.
coef (float, optional) – Nonnegative scaling to be put in front of the loss.
offset (ndarray, optional) – Offset to be subtracted in parameter space before evaluating the loss.
quadratic (regreg.identity_quadratic.identity_quadratic, optional) – Quadratic term to be added to the objective.
initial (ndarray, optional) – Initial guess at the coefficients.

Returns:

stacked_loss (stacked_common_loglike) – Stacked saturated loss.

classmethod poisson(counts, case_weights=None, coef=1.0, offset=None, quadratic=None, initial=None)#

Create a stacked loss of Poisson losses.

Parameters:

counts (sequence of ndarray) – Non-negative integer counts, one array per stacked loss.
case_weights (ndarray, optional) – Non-negative case weights.
coef (float, optional) – Nonnegative scaling to be put in front of the loss.
offset (ndarray, optional) – Offset to be subtracted in parameter space before evaluating the loss.
quadratic (regreg.identity_quadratic.identity_quadratic, optional) – Quadratic term to be added to the objective.
initial (ndarray, optional) – Initial guess at the coefficients.

Returns:

stacked_loss (stacked_common_loglike) – Stacked saturated loss.

classmethod huber(responses, smoothing_parameter, case_weights=None, coef=1.0, offset=None, quadratic=None, initial=None)#

Create a stacked loss of Huber losses.

Parameters:

responses (sequence of ndarray) – Response vectors, one per stacked loss.
smoothing_parameter (float) – Smoothing parameter for the Huber loss.
case_weights (ndarray, optional) – Non-negative case weights.
coef (float, optional) – Nonnegative scaling to be put in front of the loss.
offset (ndarray, optional) – Offset to be subtracted in parameter space before evaluating the loss.
quadratic (regreg.identity_quadratic.identity_quadratic, optional) – Quadratic term to be added to the objective.
initial (ndarray, optional) – Initial guess at the coefficients.

Returns:

stacked_loss (stacked_common_loglike) – Stacked saturated loss.

classmethod huber_svm(successes, smoothing_parameter, case_weights=None, coef=1.0, offset=None, quadratic=None, initial=None)#

Create a stacked loss of Huber SVM losses.

Parameters:

successes (sequence of ndarray) – Binary (0/1) responses, one array per stacked loss.
smoothing_parameter (float) – Smoothing parameter for the Huberized hinge loss.
case_weights (ndarray, optional) – Non-negative case weights.
coef (float, optional) – Nonnegative scaling to be put in front of the loss.
offset (ndarray, optional) – Offset to be subtracted in parameter space before evaluating the loss.
quadratic (regreg.identity_quadratic.identity_quadratic, optional) – Quadratic term to be added to the objective.
initial (ndarray, optional) – Initial guess at the coefficients.

Returns:

stacked_loss (stacked_common_loglike) – Stacked saturated loss.

classmethod cox(event_times, censoring, coef=1.0, offset=None, quadratic=None, initial=None, case_weights=None)#

Create a stacked loss of Cox proportional hazards losses.

Parameters:

event_times (sequence of ndarray) – Observed times, one array per stacked loss.
censoring (sequence of ndarray) – Censoring indicators (1 for a failure, 0 for censored), one array per stacked loss.
coef (float, optional) – Nonnegative scaling to be put in front of the loss.
offset (ndarray, optional) – Offset to be subtracted in parameter space before evaluating the loss.
quadratic (regreg.identity_quadratic.identity_quadratic, optional) – Quadratic term to be added to the objective.
initial (ndarray, optional) – Initial guess at the coefficients.
case_weights (ndarray, optional) – Non-negative case weights.

Returns:

stacked_loss (stacked_common_loglike) – Stacked saturated loss.

class regreg.smooth.mglm.multinomial_loglike(shape, counts, coef=1.0, offset=None, initial=None, quadratic=None, case_weights=None)#

Bases: smooth_atom

An overparametrized multinomial loss for nominal responses.

See e.g. Agresti, Categorical Data Analysis, p. 267. With natural parameter \(\eta\) of shape (n, J), the loss is

\[\eta \mapsto -\sum_i w_i \left[\sum_j y_{ij} \eta_{ij} - n_i \log \sum_j e^{\eta_{ij}}\right]\]

where \(y_{ij}\) are the counts, \(n_i = \sum_j y_{ij}\) and \(w_i\) the case weights. The counts need not be integers. The log-likelihood of the saturated model is stored as a constant term in quadratic.

Parameters:

shape (tuple) – Shape (n, J) of the natural parameter.
counts (ndarray) – Non-negative counts of shape (n, J).
coef (float, optional) – Nonnegative scaling to be put in front of the loss.
offset (ndarray, optional) – Offset to be subtracted from the natural parameter before evaluating the loss.
initial (ndarray, optional) – Initial guess at the coefficients.
quadratic (regreg.identity_quadratic.identity_quadratic, optional) – Quadratic term to be added to the objective.
case_weights (ndarray, optional) – Non-negative case weights.

get_data()#: Return the data (counts, trials).

set_data(data)#

Set the counts.

Parameters:: data (ndarray or tuple) – The (n, J) array of counts, or a pair (counts, trials) (the trials are recomputed from the counts).

property data#: Return the data (counts, trials).

smooth_objective(natural_param, mode='both', check_feasibility=False, case_weights=None)#

Evaluate the smooth objective, its gradient, or both.

Parameters:

natural_param (ndarray) – The current parameter values.
mode ({'both', 'func', 'grad'}, optional) – Whether to return the objective value (‘func’), its gradient (‘grad’) or both (‘both’).
check_feasibility (bool, optional) – If True, return np.inf when the point is not feasible, i.e. when natural_param is not in the domain.
case_weights (ndarray, optional) – Non-negative case weights. If None, unit weights are used (case weights stored on the loss are not applied).

Returns:

f (float) – Objective value at natural_param; returned when mode is ‘func’ or ‘both’.
g (ndarray) – Gradient at natural_param; returned when mode is ‘grad’ or ‘both’.

subsample(case_idx)#

Create a saturated loss using a subsample of the data.

Makes a copy of the loss and multiplies its case weights by the indicator of case_idx.

Parameters:: case_idx (index) – Indices of np.arange(n) to keep.
Returns:: subsample_loss (regreg.smooth.smooth_atom) – Copy of the loss with zero case weight for all cases not in case_idx.

class regreg.smooth.mglm.multinomial_baseline_loglike(shape, counts, coef=1.0, offset=None, initial=None, quadratic=None, case_weights=None)#

Bases: smooth_atom

Baseline-category logistic loss for nominal responses.

See e.g. Agresti, Categorical Data Analysis, p. 267. The last of the J categories is the baseline, so the natural parameter \(\eta\) has shape (n, J - 1) and the loss is

\[\eta \mapsto -\sum_i w_i \left[\sum_{j < J} y_{ij} \eta_{ij} - n_i \log \left(1 + \sum_{j < J} e^{\eta_{ij}}\right)\right]\]

where \(y_{ij}\) are the (integer) counts, \(n_i = \sum_j y_{ij}\) and \(w_i\) the case weights. The log-likelihood of the saturated model is stored as a constant term in quadratic.

Parameters:

shape (tuple) – Shape (n, J - 1) of the natural parameter.
counts (ndarray) – Non-negative integer counts of shape (n, J).
coef (float, optional) – Nonnegative scaling to be put in front of the loss.
offset (ndarray, optional) – Offset to be subtracted from the natural parameter before evaluating the loss.
initial (ndarray, optional) – Initial guess at the coefficients.
quadratic (regreg.identity_quadratic.identity_quadratic, optional) – Quadratic term to be added to the objective.
case_weights (ndarray, optional) – Non-negative case weights.

get_data()#: Return the data (counts, trials).

set_data(data)#

Set the counts.

Parameters:: data (ndarray or tuple) – The (n, J) array of counts, or a pair (counts, trials) (the trials are recomputed from the counts).

property data#: Return the data (counts, trials).

smooth_objective(natural_parameter, mode='both', check_feasibility=False, case_weights=None)#

Evaluate the smooth objective, its gradient, or both.

Parameters:

natural_parameter (ndarray) – The current parameter values.
mode ({'both', 'func', 'grad'}, optional) – Whether to return the objective value (‘func’), its gradient (‘grad’) or both (‘both’).
check_feasibility (bool, optional) – If True, return np.inf when the point is not feasible, i.e. when natural_parameter is not in the domain.
case_weights (ndarray, optional) – Non-negative case weights. If None, unit weights are used (case weights stored on the loss are not applied).

Returns:

f (float) – Objective value at natural_parameter; returned when mode is ‘func’ or ‘both’.
g (ndarray) – Gradient at natural_parameter; returned when mode is ‘grad’ or ‘both’.

subsample(case_idx)#

Create a saturated loss using a subsample of the data.

Makes a copy of the loss and multiplies its case weights by the indicator of case_idx.

Parameters:: case_idx (index) – Indices of np.arange(n) to keep.
Returns:: subsample_loss (regreg.smooth.smooth_atom) – Copy of the loss with zero case weight for all cases not in case_idx.

regreg.smooth.losses#

Commonly used losses that are not in regreg.smooth.glm.

The only current example is the Huberized SVM.

class regreg.smooth.losses.huberized_svm(X, labels, smoothing_parameter, coef=1.0, offset=None, quadratic=None, initial=None)#

Bases: smooth_atom

Huberized SVM loss for a design matrix and binary labels.

The labels are encoded as \(y_i \in \{-1, 1\}\) and the loss is a smoothed version of the hinge loss

\[\beta \mapsto \sum_i \left(1 - y_i x_i^T \beta\right)_+,\]

obtained by (Nesterov) smoothing of the positive part: the quadratic \(\frac{\epsilon}{2} \|u\|^2_2\) is subtracted inside the conjugate representation, where \(\epsilon\) is smoothing_parameter.

Parameters:

X (ndarray or regreg.affine.affine_transform) – Design matrix.
labels (ndarray) – Binary labels (any two distinct values; the larger is coded as 1).
smoothing_parameter (float) – Smoothing parameter for the Huberized hinge loss.
coef (float, optional) – Nonnegative scaling to be put in front of the loss.
offset (ndarray, optional) – Offset to be subtracted in parameter space before evaluating the loss.
quadratic (regreg.identity_quadratic.identity_quadratic, optional) – Quadratic term to be added to the objective.
initial (ndarray, optional) – Initial guess at the coefficients.

smooth_objective(beta, mode='both', check_feasibility=False)#

Evaluate the smooth objective, its gradient, or both.

Parameters:

beta (ndarray) – The current parameter values.
mode ({'both', 'func', 'grad'}, optional) – Whether to return the objective value (‘func’), its gradient (‘grad’) or both (‘both’).
check_feasibility (bool, optional) – If True, return np.inf when the point is not feasible, i.e. when beta is not in the domain.

Returns:

f (float) – Objective value at beta; returned when mode is ‘func’ or ‘both’.
g (ndarray) – Gradient at beta; returned when mode is ‘grad’ or ‘both’.

regreg.smooth.quadratic#

Quadratic losses, such as squared error and the signal approximator.

class regreg.smooth.quadratic.quadratic_loss(shape, Q=None, Qdiag=False, coef=1.0, offset=None, quadratic=None, initial=None)#

Bases: smooth_atom

Half of a (possibly weighted) squared \(\ell_2\) norm.

Represents

\[\beta \mapsto \frac{C}{2} \beta^T Q \beta\]

with \(Q = I\) if Q is None.

Parameters:

shape (int or tuple) – Shape of the argument to smooth_objective.
Q (ndarray or regreg.affine.affine_transform, optional) – Positive definite matrix, defaults to the identity. If Qdiag then Q is one-dimensional and holds the diagonal.
Qdiag (bool, optional) – Is the quadratic form diagonal?
coef (float, optional) – Scalar multiple \(C\) to be applied (must be nonnegative).
offset (ndarray, optional) – Vector to be subtracted from the argument before evaluating smooth_objective.
quadratic (regreg.identity_quadratic.identity_quadratic, optional) – Quadratic term to be added to the overall objective.
initial (ndarray, optional) – Initial value for the coefficients.

static fromarray(Q, offset=None, quadratic=None, initial=None)#

Create a quadratic loss from a (dense) matrix.

Parameters:

Q (ndarray) – Positive definite matrix of shape (p, p).
offset (ndarray, optional) – Vector to be subtracted from the argument before evaluating the loss.
quadratic (regreg.identity_quadratic.identity_quadratic, optional) – Quadratic term to be added to the objective.
initial (ndarray, optional) – Initial value for the coefficients.

Returns:

loss (quadratic_loss)

static squared_transform(transform, offset=None, quadratic=None, initial=None)#

Create the quadratic loss of a linear transform.

The loss is \(\beta \mapsto \frac{1}{2}\|X\beta\|^2_2\), with \(Q = X^T X\) represented as a composition of transforms.

Parameters:

transform (ndarray or regreg.affine.affine_transform) – The linear transform \(X\).
offset (ndarray, optional) – Vector to be subtracted from the argument before evaluating the loss.
quadratic (regreg.identity_quadratic.identity_quadratic, optional) – Quadratic term to be added to the objective.
initial (ndarray, optional) – Initial value for the coefficients.

Returns:

loss (quadratic_loss)

static diagonal(D, offset=None, quadratic=None, initial=None)#

Create a quadratic loss with diagonal quadratic form.

Parameters:

D (ndarray) – Diagonal of the (positive definite) quadratic form.
offset (ndarray, optional) – Vector to be subtracted from the argument before evaluating the loss.
quadratic (regreg.identity_quadratic.identity_quadratic, optional) – Quadratic term to be added to the objective.
initial (ndarray, optional) – Initial value for the coefficients.

Returns:

loss (quadratic_loss)

smooth_objective(x, mode='both', check_feasibility=False)#

Evaluate the smooth objective, its gradient, or both.

Parameters:

x (ndarray) – The current parameter values.
mode ({'both', 'func', 'grad'}, optional) – Whether to return the objective value (‘func’), its gradient (‘grad’) or both (‘both’).
check_feasibility (bool, optional) – If True, return np.inf when the point is not feasible, i.e. when x is not in the domain.

Returns:

f (float) – Objective value at x; returned when mode is ‘func’ or ‘both’.
g (ndarray) – Gradient at x; returned when mode is ‘grad’ or ‘both’.

get_conjugate(factor=False)#

Return the convex conjugate of the loss.

Parameters:: factor (bool, optional) – For a non-diagonal Q, whether to compute a Cholesky factorization to represent the conjugate. Requires Q to be an ndarray.
Returns:: conjugate (regreg.smooth.smooth_atom)
Raises:: ValueError – If Q is not diagonal and factor is False, or if factor is True and Q is not an ndarray.

class regreg.smooth.quadratic.cholesky(Q, cholesky=None, banded=False)#

Bases: object

Linear transform given by multiplication by \(Q^{-1}\).

Given \(Q > 0\), represents multiplication by \(Q^{-1}\) by first computing the Cholesky decomposition of \(Q\). The transform is symmetric, so adjoint_map equals linear_map.

Parameters:

Q (ndarray) – Positive definite matrix (in banded storage if banded).
cholesky (optional) – A precomputed Cholesky factorization, as returned by scipy.linalg.cho_factor (or scipy.linalg.cholesky_banded if banded).
banded (bool, optional) – Whether Q is stored in banded form.

linear_map(x)#: Return \(Q^{-1}x\).

affine_map(x)#: Return \(Q^{-1}x\) (the transform has no offset).

adjoint_map(x)#: Return \(Q^{-1}x\) (the transform is symmetric).

regreg.smooth.quadratic.squared_error(X, Y, coef=1)#

Least squares loss with design \(X\).

\[\beta \mapsto \frac{C}{2} \|X\beta - Y\|^2_2\]

Parameters:

X (ndarray or regreg.affine.affine_transform) – Design matrix.
Y (ndarray) – Response vector.
coef (float, optional) – Scalar multiple \(C\) (must be nonnegative).

Returns:

loss (regreg.smooth.affine_smooth)

regreg.smooth.quadratic.signal_approximator(signal, coef=1)#

Least squares loss with identity design.

\[\beta \mapsto \frac{C}{2} \|\beta - Y\|^2_2\]

Parameters:

signal (ndarray) – The signal \(Y\) to be approximated.
coef (float, optional) – Scalar multiple \(C\) (must be nonnegative).

Returns:

loss (quadratic_loss)

regreg.identity_quadratic#

A class representing simple quadratic functions.

This module contains a single class that is meant to represent a quadratic of the form

\[\frac{L}{2} \|x-\mu\|^2_2 + \langle \eta, x \rangle + \gamma\]

with \((L, \mu, \eta, \gamma)\) = (coef, center, linear_term, constant_term).

class regreg.identity_quadratic.identity_quadratic(coef, center, linear_term, constant_term=0)#

Bases: object

A quadratic function with identity Hessian (up to scaling).

This object represents the function

\[x \mapsto \frac{L}{2} \|x-\mu\|^2_2 + \langle \eta, x \rangle + \gamma\]

used in the proximal methods of all atoms.

Parameters:

coef (float) – The coefficient \(L\). If None, it is set to 0.
center (ndarray or float) – The center \(\mu\). May be None.
linear_term (ndarray or float) – The linear term \(\eta\). May be None.
constant_term (float, optional) – The constant \(\gamma\). If None, it is set to 0.

property iszero#: True if every term of the quadratic is zero (or None).

noneify()#

Replace zero center and linear term with None.

A coefficient or constant term of None is replaced with 0.

zeroify()#: Replace any attribute that is None with 0, in place.

recenter(offset)#

Shift the center of the quadratic by offset.

Parameters:

offset (ndarray or None) – Offset to subtract from the center. If None or all zeros, no shift is made.

Returns:

offset (ndarray or None) – The offset, or None if it was None or all zeros.
quadratic (identity_quadratic) – The collapsed (recentered) quadratic.

objective(x, mode='both')#

Evaluate the quadratic and/or its gradient.

Parameters:

x (ndarray) – Point at which to evaluate.
mode (str, optional) – One of ['func', 'grad', 'both'].

Returns:

value (float or ndarray or tuple) – The value ('func'), the gradient ('grad') or a tuple of both ('both').

get_shapes()#

Determine shapes of the linear term and center, checking they agree.

Returns:

linear_term_shape (tuple) – Shape of the linear term.
center_shape (tuple) – Shape of the center.

collapsed()#

Return an equivalent identity quadratic with center of 0.

The center is absorbed into the linear and constant terms. It would be nice to have a center of None, but there are some places we are still multiplying by -1.

latexify(var='\\beta', idx='')#

Return a LaTeX representation of the quadratic.

Parameters:

var (str, optional) – LaTeX name for the argument of the quadratic.
idx (str, optional) – Subscript to attach to the parameters.

Returns:

latex (str) – LaTeX string.

property true_center#: Minimizer of the quadratic, or None if coef is not positive.

property conjugate#

The convex conjugate of the quadratic.

If coef is nonzero this is another identity_quadratic; otherwise it is a zero_constraint atom.

Smooth functions

Contents

Smooth functions#

regreg.smooth#

regreg.smooth.glm#

regreg.smooth.mglm#

regreg.smooth.losses#

regreg.smooth.quadratic#

regreg.identity_quadratic#