## Introduction

Linkage analysis of a quantitative trait involves detecting regions of the genome that influence the trait. These regions of the genome are referred as quantitative trait loci (QTL). The QTL mapping approach is a statistical technique to find a region *x* (QTL) by studying the trait-dependent segregation of alleles at *x* from founders to nonfounders in pedigrees. Under no linkage between the trait and *x*, we observe Mendelian segregation at *x*. Thus, under no linkage, the amount of genetic sharing at *x* between family members is determined purely as a function of family relationship and has nothing to do with the trait values. But if the location *x* is linked to the trait, there is expected to be excess genetic sharing among people with concordant phenotypes and less sharing among people with discordant phenotypes in a pedigree. The genetic sharing among family members referred to above is called identity by descent (IBD), which is defined as the sharing of the same copy of a founder allele. People who have similar phenotypes tend to have a higher than expected levels of IBD sharing near the genes that influence the trait.

A number of different statistical models (Pratt et al., 2000; Feingold, 2001, 2002) have been developed to assess linkage between specific regions of the genome and a quantitative phenotype. The power and robustness of these QTL mapping methods vary based on their assumptions on the underlying trait model (Feingold, 2001). The particularly popular approach that has been extensively used for many QTL studies is the variance component (VC) approach (Goldgar, 1990; Schork, 1993; Amos, 1994; Fulker et al., 1995; Almasy & Blangero, 1998; Pratt et al., 2000). The approach (Almasy & Blangero, 1998) assumes that the trait values for the family members follow a multivariate normal distribution where the covariance matrix between the family members depends on their IBD sharing at a location being tested for linkage. This approach can be applied to any type of pedigree structure. This approach is very popular and is used extensively. The major issue with this approach is that it can produce elevated type I error and biased parameter estimates in the case of nonnormal distribution of the trait or in case of selected sampling of phenotypes. Both of these issues are often encountered in real studies. The power of the VC method for linkage detection under these situations can be drastically reduced (Amos et al., 1996; Allison et al. 1999; Feingold, 2001, 2002; Tang & Siegmund, 2001). When there is nonnormality, one can perform a parametric transformation on the trait values to approximate normality in order to correct for the inflated type I error (Allison et al., 2000; Etzel et al., 2003). It is often difficult to find an appropriate transformation to ensure correct type I error for the VC approach. In this paper, we have studied the performance of two such transformations [Box–Cox transformation and rank-based inverse normal transformation (INT)] on the VC approach, but the main focus of the paper suggests an alternative powerful approach that always produces the correct type I error irrespective of the trait distribution, and thus avoids searching for appropriate transformation of the trait in order to produce correct type I error for the VC approach.

This issue with the VC approach has led to a number of attempts to propose different methods to handle situations where normality does not hold. Most of these are based on score statistics or regression-based statistics and attempt to achieve the power of the VC likelihood-based methods, while retaining the robustness and computational simplicity of the original Haseman–Elston regression (Haseman & Elston, 1972). A number of such approaches were discussed and studied extensively in the paper by Bhattacharjee et al. (2008). Among these approaches, the set of approaches that condition on the trait maintains correct type I error and also has comparable or better power to the VC approach and other existing approaches (Bhattacharjee et al., 2008). This is generally accomplished by fitting a model (usually multivariate normal) for the trait values, but conditional on the trait values in the sample. A statistic of this type has the correct type I error (at least asymptotically) no matter what the underlying trait distribution or the sampling scheme, although the power still depends on the correctness of the assumed model (Feingold, 2001). Many of these QTL mapping approaches (Haseman & Elston, 1972; Alcais & Abel, 1999; Dudoit & Speed, 2000) are restricted to nuclear families. A very promising approach in this category, which is applicable to general pedigrees is the regression approach proposed by Sham et al. (2002). An important assumption of this approach is that most of the linkage information in a pedigree can be summarized by pairwise relationships. The pairwise IBD sharing at a location *x* is regressed on the pairwise squared differences and squared sums for the trait. This approach produces statistics whose type I error is robust to the selected sampling or nonnormality of the trait, while having similar power with the VC approach if the multivariate normal model is correct.

In this paper, we have proposed a likelihood-based approach that belongs to the above category of approaches that condition on the trait data. This model is an extension of the approach proposed by Basu et al. (2010) for linkage analysis of a discrete trait and has similarities with the approach proposed by Sinsheimer et al. (2000), which is primarily developed for testing of association. Our approach specifies an explicit alternative model for the transmission of founder alleles among individuals in the pedigree conditional on the trait data of the individuals, and performs a likelihood-ratio test against the null hypothesis of no linkage; that is, the transmission of founder alleles is independent of trait. Our model conditions on the trait data and does not make any assumption on the distribution of the trait. Hence it produces the correct type I error irrespective of the underlying trait distribution or the sampling scheme. Moreover, a key advantage of this approach is that it models the segregation of founder alleles directly as opposed to modeling the pairwise IBD sharing in Sham et al. (2002), which could be more informative for extended pedigrees. This approach is also capable of dealing rigorously with the fact that IBD sharing is not observed directly, but must be inferred (with uncertainty) from the observed marker data. Our model is aimed primarily at extracting this multigenerational information, which current models struggle to capture. However, it is applicable to pedigrees of any size.

Hence, the potential advantage of this proposed approach is that it can be used for linkage analysis of any trait, without worrying about the type I error. The approach provides a computationally feasible solution even for extended pedigrees and models explicitly the transmission of alleles from founders to nonfounders in a pedigree. We compared the performance of our model with the VC approach (Almasy & Blangero, 1998) on transformed and untransformed data and the regression-based approach (Sham et al., 2002) through simulation studies for nonnormally distributed traits. We also studied the performance of our method on the National Heart Lung and Blood Institute (NHLBI) Family Heart Study (FHS) dataset (Higgins et al., 1996). The simulation study demonstrated that our method has better power than the majority of these approaches for the nonnormal traits we studied in the paper. Moreover, unlike the VC approach, our proposed approach always produced the correct type I error. The simulation study and the real data analysis demonstrated that our proposed approach is an attractive alternative for QTL analysis in general pedigrees, especially the in case of nonnormally distributed traits or selected sampling of phenotypes.