Parallelism, uniqueness, and large-sample asymptotics for the Dantzig selector

Abstract

The Dantzig selector (Candès & Tao, 2007) is a popular -regularization method for variable selection and estimation in linear regression. We present a very weak geometric condition on the observed predictors which is related to parallelism and, when satisfied, ensures the uniqueness of Dantzig selector estimators. The condition holds with probability 1, if the predictors are drawn from a continuous distribution. We discuss the necessity of this condition for uniqueness and also provide a closely related condition which ensures the uniqueness of lasso estimators (Tibshirani, 1996). Large sample asymptotics for the Dantzig selector, that is, almost sure convergence and the asymptotic distribution, follow directly from our uniqueness results and a continuity argument. The limiting distribution of the Dantzig selector is generally non-normal. Though our asymptotic results require that the number of predictors is fixed (similar to Knight & Fu, 2000), our uniqueness results are valid for an arbitrary number of predictors and observations. The Canadian Journal of Statistics 41: 23–35; 2013 © 2012 Statistical Society of Canada

Le sélecteur de Dantzig [Candès et Tao, 2007] est une méthode de régularisation pour le sélection de variables et l'estimation en régression linéaire. Nous présentons une condition géométrique très faible sur les prédicteurs observés qui est reliée au parallélisme, et, lorsque vérifiée, elle nous assure l'unicité des estimateurs du sélecteur de Dantzig. Si les prédicteurs proviennent d'une distribution continue, cette condition est vérifiée avec probabilité 1. Nous discutons de la nécessité de cette condition pour l'unicité et nous donnons aussi une condition étroitement reliée qui nous assure de l'unicité des estimateurs lasso [Tibshirani, 1996]. Les propriétés asymptotiques pour le sélecteur de Dantzig, c'est-à-dire la convergence presque sure et la distribution asymptotique, découlent directement de nos résultats d'unicité et d'un argument de continuité. La distribution limite du sélecteur de Dantzig n'est pas habituellement normale. Quoique nos résultats asymptotiques demandent que le nombre de prédicteurs soit fixe (similaire à [Knight et Fu, 2000]), nos résultats d'unicité sont valides pour un nombre arbitraire de prédicteurs et d'observations. La revue canadienne de statistique 41: 23–35; 2013 © 2012 Société statistique du Canada

1. INTRODUCTION

Regularized regression methods for variable selection and estimation have become an important tool for statisticians and have been the subject of intense statistical research during the past 15 years (Bickel & Li, 2006; Fan & Lv, 2010; Tibshirani, 2011). These methods provide a tractable approach to the analysis of high-dimensional datasets and are especially useful when the underlying signal is sparse. In this paper, we address some gaps in the literature, which pertain to uniqueness and large sample asymptotic theory for the Dantzig selector (Candès & Tao, 2007), a popular -regularized regression method that is closely related to lasso (Tibshirani, 1996).

First, we develop an intuitive geometric condition related to parallelism which ensures that the Dantzig selector has a unique solution and demonstrate that this condition holds in an overwhelming majority of instances (with probability 1, if the predictors follow an absolutely continuous distribution with respect to the Lebesgue measure). We also give a related necessary condition for the uniqueness of Dantzig selector solutions. These results originally appeared in the first author's Ph.D. thesis (Dicker, 2010) and, to our knowledge, are the first uniqueness results about the Dantzig selector to be found in the literature. In fact, our uniqueness condition for the Dantzig selector is easily translated into a similar prevalent condition which implies that lasso has a unique solution.

Aside from their independent interest, the uniqueness results presented here pave the way for a simple derivation of the almost sure limit and the asymptotic distribution of Dantzig selector estimators, when the number of predictors, p, is fixed (on the other hand, we emphasize that our uniqueness results are valid for arbitrary p). These asymptotic results are analogous to those found in Knight & Fu (2000) for the lasso and further highlight similarities between the two methods, which have been discussed by multiple authors (Meinshausen, Rocha, & Yu, 2007; James, Radchenko, & Lv, 2009). In fact, in comparison with Knight & Fu's (2000) results, uniqueness appears to be the major hurdle to obtaining large sample asymptotics for the Dantzig selector. The Dantzig selector is a convex—but not strictly convex—optimization problem. Thus, unique solutions are not guaranteed in general. However, once uniqueness is understood, asymptotic results for the Dantzig selector follow directly from continuity arguments. More specifically, we show that under the given uniqueness conditions the Dantzig selector may be viewed as a well-defined continuous mapping; asymptotic results then follow from the continuous mapping theorem. By contrast, for the lasso, uniqueness is assured in classical fixed p asymptotic analyses because the associated optimization problem is strictly convex (provided the predictors are non-degenerate). The foregoing discussion highlights the potential usefulness of uniqueness results for the Dantzig selector. More broadly, understanding uniqueness makes certain powerful tools—like the continuous mapping theorem—readily available for further analysis of the Dantzig selector.

Though much of the recent interest in regularized regression methods is spurred by applications that may perhaps be best approximated by an asymptotic regime where , we believe that it remains important to understand classical large sample asymptotics, where p is fixed and , in order to obtain a more complete understanding of these procedures. This paper helps shed light on this issue. Moreover, we believe that our uniqueness results, which are valid for all p, may be useful for formulating and deriving asymptotic results for regularized regression methods in settings where ; however, this is a topic for future research and is beyond the scope of this paper (though it is briefly addressed again in our concluding Section 5).

The rest of this paper proceeds as follows. In Section 2 we introduce the notation and definitions. In Section 3 we discuss uniqueness. Propositions 1 and 2 are the main results in Section 3 and summarize important uniqueness properties of the Dantzig selector and lasso vis-à-vis parallelism. In Section 4, we show that the Dantzig selector may be viewed as a continuous mapping from the space of predictors and associated outcomes to the space of parameter estimates (Proposition 3). Corollaries 1 and 2 give the almost-sure limit of Dantzig selector estimators and their asymptotic distribution, respectively. Section 5 contains a brief concluding discussion. The proofs may be found in the Appendix at the end of the paper.

2. NOTATION AND DEFINITIONS

Consider the linear model

• (1)

where and are observed outcomes and predictors, respectively, are unobserved iid integrable random variables with mean , and is an unknown parameter to be estimated. To simplify the notation, let denote the n-dimensional vector of outcomes and denote the matrix of predictors. Also let . Then (1) may be re-expressed as

• 

It will be useful to have a concise method for referring to sub-vectors and sub-matrices of various vectors and matrices. For a vector and a subset , let . Furthermore, for matrices let denote the matrix obtained from X by extracting columns corresponding to elements of A. If is a matrix, and has cardinality , let denote the matrix obtained from C by extracting rows corresponding to elements of A and columns corresponding to elements of B. For , let denote the j-th column of X. Finally, let denote the null-space of the matrix C and let denote the dimension of the vector space V.

The main object of study in this paper is the Dantzig selector—a linear programming problem for obtaining estimates of , which is defined as follows:

• (2)

where is a tuning parameter, denotes the -norm and denotes the -norm. Solutions to (2), denoted , will be referred to as Dantzig selector estimators.

We also introduce the lasso optimization problem and estimator at this time:

• (3)

where is the squared -norm. Though the lasso is not our primary concern in this paper, we will sometimes find it instructive to compare aspects of the Dantzig selector and lasso side-by-side. For instance, as discussed in the Introduction, notice that if X has rank p, then lasso is a strictly convex optimization problem, which ensures that is unique. On the other hand, the Dantzig selector (2) is a linear programming problem and the uniqueness properties are less clear, even when X has rank p.

In order to provide some additional context for the present study, we point out that one of the key features of both the Dantzig selector and lasso is that they perform simultaneous variable selection and estimation. By this we mean that and are often non-empty (contrast this with the ordinary least squares estimator for ). This implies that and often have reduced dimension (i.e., only a few non-zero entries) and can greatly enhance interpretability, along with estimation accuracy (Tibshirani, 1996; Candès & Tao, 2007; Bickel, Ritov, & Tsybakov, 2009).

3. PARALLELISM AND UNIQUENESS

Parallelism plays a large role in the discussion of the uniqueness of Dantzig selector solutions. Roughly speaking, the Dantzig selector has a unique solution if the feasible set,

• 

is not parallel to the -ball. Below, we describe parallelism as a geometric concept which is relevant to the Dantzig selector and then give a more formal definition.

First note that the feasible set F is polyhedral (it is the intersection of finitely many hyperplanes). Solutions of the Dantzig selector are points of minimal -norm. Let be the closed unit -ball centered at the origin. Geometrically, we can find solutions to the Dantzig selector by “growing” , , until it intersects F; the points of intersection are Dantzig selector solutions. More precisely, let . The collection of all Dantzig selector solutions is . When , the 1-dimensional faces of have slope 1 or ; the Dantzig selector has multiple solutions only if a 1-dimensional face of F has slope 1 or , that is, only if F is parallel to the -ball, .

As indicated by the situation when , if the Dantzig selector has multiple solutions, then F is parallel to (Figure 1). When , the notion of parallelism which is correct for our purposes is less straightforward. Geometric intuition suggests that parallelism is invariant under translation and scalar multiplication, in the sense that F is parallel to if and only if is parallel to for and . In particular, multiplying X by a (non-zero) scalar and adding vectors to does not affect parallelism. This leads to a definition of parallelism between F and which depends only on the matrix . In fact, in our view, the primitive concept is parallelism between a symmetric matrix C and the -ball.

Figure 1. An instance of the Dantzig selector with multiple solutions. The region F is the feasible set for the Dantzig selector and . The bold line represents the intersection of with F and is the solution set for this instance of the Dantzig selector.

Definition 1.

• (a)
  Let C be a symmetric matrix. The matrix C is parallel to the -ball if and only if the condition [Par] (found below) holds. [Par] There exist subsets and a vector such that , , and .
• (b)
  The feasible set for the Dantzig selector, F, is parallel to the -ball if and only if is parallel to the -ball.

Remarks

• (i)
  Parallelism, as defined here, is related to degenerate sub-matrices of C, which, in the context of the Dantzig selector, correspond to the nontrivial faces of F. In [Par], the requirement that is related to the fact that the faces of the -ball, , have normal vectors , where for some .
• (ii)
  When , it is easy to see that F is parallel to the -ball if and only if one of the columns of is a scalar multiple of some point in . This occurs if and only if a one-dimensional face of F has slope 1 or , as depicted in Figure 1.

As discussed above, parallelism is invariant under translation and scalar multiplication. On the other hand, translation and scalar multiplication of the feasible set F gives rise to various instances of the Dantzig selector, some with a unique solution and some, perhaps, with multiple solutions. This suggests that any sufficient condition for the existence of multiple Dantzig selector solutions must, unlike parallelism, involve and λ. To illustrate this concept, suppose that is invertible and is parallel to the -ball. Figure 2a depicts , which is equal to the feasible set for the Dantzig selector when and and is parallel to the -ball. Figures 2b,c depict and , potential feasible sets for the Dantzig selector that are both obtained from by scalar multiplication and translation. The feasible sets and are both parallel to the -ball, and correspond to feasible sets for the Dantzig selector with the predictor matrix X and different values for , λ (not given here). The instance of the Dantzig selector with feasible set has multiple solutions, while the Dantzig selector with feasible set has a unique solution.

Figure 2. (a) is parallel to the -ball, as evidenced by the bold face D. (b) is obtained from by scalar multiplication and translation; the Dantzig selector problem with feasible set has multiple solutions, indicated by the bold line segment labelled . (c) is obtained from by scalar multiplication and translation; the point labelled is the unique solution to the Dantzig selector problem with feasible set .

The following condition combines parallelism with additional constraints and is a sufficient condition for the existence of multiple Dantzig selector solutions.

• [Mult]
  There exist subsets and vectors , , such that

  • 1.
    , , and .
  • 2.
    for all .
  • 3.
    .
  • 4.
    for all and for all .

Note that Condition 1 in [Mult] implies that F is parallel to the -ball. Conditions 2–4 in [Mult] constrain the location of F in relative to the origin. Proposition 1 below characterizes the uniqueness properties of the Dantzig selector in terms of [Par] and [Mult]. A related necessary condition for the existence of multiple lasso solutions is given in Proposition 1(c). Proposition 1 is proved in the Appendix at the end of this paper.

Proposition. Proposition 1.

• (a)
  If [Mult] holds, then the Dantzig selector has multiple solutions.
• (b)
  If F is not parallel to the -ball, then the Dantzig selector has a unique solution.
• (c)
  Suppose that and that the lasso has multiple solutions (i.e., contains more than a single element). Then there exists a subset and a vector such that , , and .

Remarks

• (i)
  Proposition 1 is valid for any n and p.
• (ii)
  Proposition 1(c) may be rephrased as follows. If the lasso has multiple solutions, then is parallel to the -ball and, moreover, one may take in the definition of parallelism.
• (iii)
  If , then the lasso has multiple solutions whenever is singular.
• (iv)
  The condition in Proposition 1(c) implies that is parallel to the -ball. It follows that if is not parallel to the -ball, then both the Dantzig selector and lasso have unique solutions. The relationship between uniqueness for the Dantzig selector and uniqueness for lasso is discussed by Meinshausen, Rocha, & Yu (2007), who give a concrete -dimensional example (with pictures) where lasso has a unique solution, but the Dantzig selector does not.
• (v)
  A condition similar to [Mult] which ensures the existence of multiple lasso solutions may be developed. This is not pursued further here.

The next proposition suggests that the Dantzig selector and lasso have a unique solution in an overwhelming majority of instances.

Proposition. Proposition 2. Suppose that are iid and drawn from a continuous distribution with respect to the Lebesgue measure on . Then is parallel to the -ball with probability 0. Consequently, the Dantzig selector and lasso have a unique solution with probability 1.

Remarks

• (i)
  Proposition 2 is proved in the Appendix (a proof also appears in Dicker, 2010). To provide some intuition, note that the parallelism condition requires to both (i) contain a specific point in its range (that is, an element of ) and (ii) to have a degenerate range (in the sense that ). Proposition 2 implies that this occurs with probability 0, under the specified conditions.

4. LARGE SAMPLE ASYMPTOTICS FOR THE DANTZIG SELECTOR

Throughout the rest of this article, assume that p and are fixed. In this section, we formulate the Dantzig selector as a well-defined mapping from sample covariance matrices, , marginal covariances, , and tuning parameters, , to estimators, . To do this, we restrict our attention to symmetric matrices that are not parallel to the -ball—Proposition 2 suggests that this restriction is fairly weak. Then, we show that the Dantzig selector mapping is continuous. With this machinery in place, large sample asymptotics for the Dantzig selector follow easily.

Let denote the collection of positive semidefinite matrices that are not parallel to the -ball and let , where is the collection of all invertible matrices with real entries. Define the Dantzig selector mapping by , where solves the optimization problem

• (4)

It follows directly from Proposition 1(b) that G is well-defined. Furthermore, notice that . Note that the domain of G may be extended to a subset of , provided one imposes conditions to ensure that the feasible set in the optimization problem (4) is non-empty. More specifically, define . Then (4) defines for .

Proposition. Proposition 3. The mapping G is continuous on .

Remarks

• (i)
  A proof of Proposition 3 is found in the Appendix. A similar proof shows that G is also continuous on . In other words, assuming that the appropriate (anti-) parallelism conditions hold, if there is non-trivial regularization in the limit (i.e., ), then the Dantzig selector is continuous, regardless of whether or not the predictors and the limiting sample covariance matrix are singular.

Corollary 1. Suppose that and that . Then , almost surely, where solves

• 

Remarks

• (i)
  The corollary follows directly from Proposition 3, which implies that , almost surely.
• (ii)
  Corollary 1 implies that under the given conditions, the Dantzig selector is consistent for if and only if . Furthermore, it gives the almost sure limit of in cases where the Dantzig selector is not consistent (that is, when ).

Corollary 2. Suppose that . Also assume that , that , and that . Let and let denote the complement of in . Then , where denotes convergence in distribution, solves the optimization problem

• (5)

and .

Corollary 2 is proved in the Appendix.

Remarks 

• (i)
  The second moment condition on and the condition ensure that is asymptotically normal.
• (ii)
  If , then has the same asymptotic distribution as the ordinary least squares estimator. If , then the limiting distribution of the Dantzig selector is not normal.
• (iii)
  Corollary 2 should be compared with Theorem 2 of Knight & Fu (2000), which describes the limiting distribution of . Though the limiting distribution of lasso is determined by an unconstrained optimization problem, the term in the limiting optimization problem for the Dantzig selector (5) also appears in the limiting optimization problem for lasso.

5. DISCUSSION

The results in this paper address fairly long-standing open questions about uniqueness for the Dantzig selector and lasso. To summarize, we prove that the Dantzig selector and lasso estimators are unique in almost all instances. Though these results may appear to be somewhat esoteric, Proposition 2 and its corollaries demonstrate their potential usefulness. Indeed, we have shown that once uniqueness is understood, it is straightforward to obtain the almost sure limit and limiting distribution of Dantzig selector estimators. Taking a broader view, the results presented here may help clear the path for a more operator theoretic approach to studying the Dantzig selector, lasso, and other regularized regression procedures. Such an approach may offer additional insights into the properties of these methods in a variety of settings. For instance, one could potentially obtain a better understanding of the Dantzig selector in an asymptotic regime where by defining the Dantzig selector operator on an appropriate infinite dimensional space (analogous to the operator G defined in Section 4 above) and studying its continuity properties in this more abstract setting. Future research in this direction is needed.

Acknowledgements

We thank an associate editor and the reviewers for their detailed and very helpful comments. This research was supported by grants from the U.S. National Cancer Institute.

APPENDIX

Proof of Proposition 1. The following two lemmas establish the Karush–Kuhn–Tucker (KKT) conditions for the Dantzig selector and lasso optimization problems. The lemmas appear in various forms in several references (Efron, Hastie, & Tibshirani, 2007; Asif, 2008; Dicker, 2010; Asif & Romberg, 2010), and the proofs are omitted.

Lemma A1. The vector is a solution to the Dantzig selector (2) if and only if there is such that

• (6)

• (7)

• (8)

• (9)

Lemma A2. The vector is a solution to the lasso optimization problem (3) if and only if

• 

To prove Proposition 1 (a), we assume that [Mult] holds and show that the Dantzig selector has multiple solutions. Let , , and be as in [Mult] and take so that and , where is the complement of A in . Then it is clear from Lemma A1 that is a solution to the Dantzig selector. Furthermore, using Lemma A1, it is easy to check that is a solution to the Dantzig selector for sufficiently small (take ).

Now suppose that are distinct solutions to the Dantzig selector and let be vectors such that and , satisfy (6)–(9). Without loss of generality, assume that and , where we define or 0, according to or , for . Let , . Then (7) and (8) imply that and , . Additionally, (9) implies that . Hence, . It follows that is parallel to the -ball. This proves Proposition 1 (b).

Finally, to prove Proposition 1(c), suppose

• 

are distinct and suppose without loss of generality that . Let and . Notice that for we have

• (10)

Since (10) must hold for all and since , we must have and . It follows that

• (11)

and .

Now, let , where is the Moore–Penrose pseudoinverse of . Then Lemma A2 implies that

• 

and . Proposition 1(c) follows from these observations plus (11).

Proof of Proposition 2. To prove Proposition 2, we make use of the following lemma.

Lemma A3. Suppose that and that the rows of X are iid and drawn from a distribution which is continuous with respect to the Lebesgue measure on . Let W be an matrix of rank . Then has rank with probability 1.

Proof of Lemma A3. Let X and W be as in the statement of the lemma. Without loss of generality, suppose that . When , the result is true. For , let . To facilitate a proof by induction, assume that has rank with probability 1. In the event that has rank , the rank of is less than p if and only if

• (12)

where and . Since W has full rank, it follows that , with probability 1. Thus, conditioning on and using the fact that the conditional distribution of is continuous, it follows that (12) holds with probability 0. We conclude that has rank p with probability 1.

Getting back to the proof of Proposition 2, suppose that the rows of X are iid and drawn from a distribution which is continuous with respect to the Lebesgue measure on . Then X has rank with probability 1. Let and decompose so that , , and , , and J are disjoint. If , then has a non-trivial null space. Suppose for the moment that . When X has full rank, the dimension of the null space of is non-zero if and only if

• 

Furthermore, if X has full rank, then has full rank. Conditioning on and appealing to Lemma A3, it follows that the rank of is with probability 1. Thus the null-space of is non-trivial with positive probability if and only if .

Now suppose that . There are two cases: and . In each case, the probability that there exists such that is 0. We prove this for the case ; the case follows similarly. Assume that . Choose such that and let . Suppose that for some and . Then, assuming that X is full rank,

• 

and

• 

Thus, we have

• 

where Lemma A3 guarantees that is invertible with probability 1. Since, conditional on , the rows of are independent and have continuous distributions with respect to Lebesgue measure on , it follows that

• 

with probability 0. Thus, as claimed, the probability that there exists such that is 0.

The results from the last two paragraphs imply that

• 

It follows that is parallel to the -ball with probability 0, as was to be shown.

Proof of Proposition 3. For , let , , and and assume that , , and . Let and let . We show that .

Since , there exists a subsequence and a vector such that . To prove the proposition, it suffices to show that . By continuity of the -norm, we must have

• 

Also, by the optimality properties of , we must have

• (13)

for any sequence , with and

• (14)

We consider two cases: and . First suppose and define . Then (14) holds and . From (13), it follows that and the optimality of implies that . Now suppose that and define . Then (14) holds and, as in the previous case, we conclude that . Thus, in either case, , as was to be shown.

Proof of Corollary 2. The conditions and ensure that , by the Lindeberg–Feller central limit theorem. By the Skorokhod representation theorem, we may assume without loss of generality that almost surely.

Now let and notice that the Dantzig selector (2) is equivalent to the optimization problem

• (15)

In particular, solves (15). We show that , the solution to (5), almost surely. This suffices to prove the corollary.

Since , almost surely, and , it follows that there is an almost surely finite random variable M such that whenever is feasible for the optimization problem (15). Let and notice that if and is feasible for (15), then . It follows that

• 

whenever . Taking , Proposition 3 implies that almost surely and it is straightforward to check that .