#### 2.1. The Modeling Approach

The starting point of our methodology is nonparametric mixture modeling for the collection of response distributions, which are indexed by dose level , where, typically, , as dose is commonly recorded on a logarithmic scale. With replicated binary responses at a number of observed dose values, the natural mixture structure involves mixing of Bernoulli distributions,

- (1)

where is the standard normal distribution function, and thus . Given the mixture setting, the choice of the link function for the kernel probability is driven mainly by convenience in model implementation where the probit link offers advantages. We aim to develop flexible nonparametric modeling for the response distributions , and for the implied dose–response relationship, , , retaining however the traditional assumption of monotonicity for the dose–response curve. To this end, the model needs to be completed with an appropriate nonparametric prior for the collection of mixing distributions, .

Given the nature of quantal bioassay studies, we expect response distributions associated with nearby dose levels to be more similar than those which are far apart. Under our mixture model setting, this situation gives rise to the need for a prior which relates the mixing distributions across dose level *x* to varying degrees. A powerful option for this modeling problem is the dependent Dirichlet process (DDP) prior (MacEachern, 2000). The DDP prior is motivated by the (almost sure) discrete representation of DP realizations (Sethuraman, 1994), where, in full generality, both the stick-breaking weights and the atoms may evolve with *x*. Because the general DDP prior is complicated to implement and requires large data sets to sufficiently learn about its hyperparameters, simpler versions are typically employed in applications. Under minimal conditions, Barrientos et al. (2012) have established full support (according to weak convergence) for both of the simplified DDP prior specifications discussed next.

In particular, a “single-*p*” DDP prior involves a countable mixture of realizations from a stochastic process over , with weights matching those from the standard DP, that is, . Here, the are i.i.d. realizations from a base stochastic process, , over , and the weights arise from stick-breaking: , for , with the i.i.d. from a Beta distribution (independently of the ). Under a single-*p* DDP prior for the mixing distributions in (1), the dose–response curve, , . This model does not enforce monotonic dose–response relationships, although an increasing trend can be incorporated in prior expectation. This structure can be achieved through a Gaussian process for with constant variance, , and linear mean function, + , with . Then, the distribution for , induced by the Gaussian process at , is stochastically ordered, and thus is increasing in *x* being the expectation of the increasing function with respect to (depending on the context, we use N for either the normal distribution or density with mean *m* and variance ).

Contrarily, we may consider a DDP prior structure where only the weights evolve with *x*, that is, , with the i.i.d. from a base distribution on , independently of the stochastic mechanism that generates the . This “single’” DDP formulation presents a formidable complication with regard to the main inferential objectives for bioassay experiments. A monotonically increasing dose–response relationship, at least in prior expectation, is imperative in terms of prediction at unobserved dose levels to anchor the inference with an increasing trend. In the case of the single’ DDP prior, there is no means to force such a trend for the dose–response curve, . More specifically, using the monotone convergence theorem and letting (where a finite mean is assumed for ), we obtain , for any . Thus, the prior expectation of the dose–response curve is constant in *x*, rendering interpolation and extrapolation inference practically useless.

Hence, the single-*p* DDP prior emerges as the preferred choice for dose–response inference built from modeling dependent response distributions. The general version of the single-*p* DDP prior can be used for settings where one seeks the extra flexibility of non-monotonic dose–response functions with the increasing trend only in prior expectation. We have studied such DDP mixture modeling for developmental toxicity experiments, which involve clustered categorical responses and multiple dose–response curves for distinct endpoints (Fronczyk and KottasFronczyk and Kottasin press, Kottas and Fronczyk, 2013). For these more involved experiments, the capacity of the DDP mixture model to uncover non-monotonic dose–response relationships is a practically relevant feature. However, for the simpler quantal bioassay setting, we seek a more structured nonparametric prior to incorporate the standard monotonicity assumption for the dose–response curve with prior probability one rather than only in prior expectation.

The essential observation from the construction of the linear-DDP prior is the correspondence of and , where DP with . In particular, the atoms , sampled originally from , and the weights, , in the stick-breaking construction of *G* induce the atoms in conjunction with the same weights, , in the stick-breaking construction of , for all *x*. Hence, the linear-DDP mixture model, , where , can be equivalently formulated as a DP mixture model

The corresponding dose–response curve is now defined by

- (2)

Hence, under the more structured linear-DDP mixture model, the restriction , implemented with a distribution supported by , yields a sufficient condition for dose–response curve realizations to be increasing with prior probability one.

For most bioassay experiments, it is natural to view the observed binary response as a proxy for a latent continuous response variable. This procedure offers a tractable way to estimate the parameters of the proposed DP mixture model. In particular, with the restriction , we can reparameterize the Bernoulli kernel, and, consequently, the underlying latent response distribution, to a location-scale mixture of normal distributions. Specifically, let and , and denote by *z* the latent -valued response associated with the binary outcome *y*. Then, we obtain

where the final probability arises under a location-scale normal DP mixture for the latent tolerance distribution, , where with an appropriate distribution on . Because the parameterization expedites prior specification, we work with the equivalent location-scale normal DP mixture formulation (suppressing the notation in the following).

#### 2.2. Hierarchical Model Formulation

As a computational tool, we use a DP truncation approximation replacing *G* with . Truncation from the outset provides closed form full conditionals for posterior simulation via Gibbs sampling and a straightforward means for obtaining inference for all objectives. Here, the , given , are i.i.d. , and the weights arise from a truncated version of the stick-breaking construction: , , , and , with the i.i.d., given , from Beta. The truncation level can be chosen using distributional properties for the tail probability of the stick-breaking weights, . More specifically, and (Ishwaran and Zarepour, 2000). These expressions can be averaged over the prior for to estimate and . Given a tolerance level for the approximation, the former expression yields the corresponding value *L*. For both data examples of Section 4, we used a gamma prior for . Consequently, we set the truncation level to , which, after averaging over the prior for , results in , and .

To represent the mixture component with which each data point is associated, configuration variables are introduced. Then, the hierarchical model for the data, augmented with the latent continuous responses, can be written as follows:

- (3)

where the prior distribution for is given by . Here, , , and . In addition to the gamma prior for , we place normal and priors on and , respectively, and an exponential prior on with mean .

#### 2.3. Prior Specification

Regarding the DP base distribution parameters, , empirical evidence suggests robustness to their hyperprior choice. In general, we recommend setting the shape parameter, *c*, of the inverse gamma distribution for the , and of the inverse gamma prior for to values that yield relatively dispersed distributions, but with finite variance. Then, working with the expressions for the marginal mean, , and variance, , of the latent responses under a single component of the mixture model, we are able to define the hyperpriors through only the range of the dose levels. In particular, , and thus can be specified by the midrange of the doses. Analogously, + + + + + . Therefore, with *c* and specified, the variance components, , , and , can be determined through a proxy for the marginal variance obtained by dividing the range of the dose levels by 4 (or 6) and squaring.