### Abstract

- Top of page
- Abstract
- 1. INTRODUCTION
- 2. MODEL-INDEPENDENT BENCHMARK ANALYSIS
- 3. BMDL PERFORMANCE EVALUATION
- 4. EXAMPLE: FORMALDEHYDE CARCINOGENESIS IN LABORATORY ANIMALS
- 5. DISCUSSION
- ACKNOWLEDGMENTS
- REFERENCES

Estimation of benchmark doses (BMDs) in quantitative risk assessment traditionally is based upon parametric dose-response modeling. It is a well-known concern, however, that if the chosen parametric model is uncertain and/or misspecified, inaccurate and possibly unsafe low-dose inferences can result. We describe a nonparametric approach for estimating BMDs with quantal-response data based on an isotonic regression method, and also study use of corresponding, nonparametric, bootstrap-based confidence limits for the BMD. We explore the confidence limits’ small-sample properties via a simulation study, and illustrate the calculations with an example from cancer risk assessment. It is seen that this nonparametric approach can provide a useful alternative for BMD estimation when faced with the problem of parametric model uncertainty.

### 3. BMDL PERFORMANCE EVALUATION

- Top of page
- Abstract
- 1. INTRODUCTION
- 2. MODEL-INDEPENDENT BENCHMARK ANALYSIS
- 3. BMDL PERFORMANCE EVALUATION
- 4. EXAMPLE: FORMALDEHYDE CARCINOGENESIS IN LABORATORY ANIMALS
- 5. DISCUSSION
- ACKNOWLEDGMENTS
- REFERENCES

Previously [32] we studied the performance of our model-independent BMDL, _{100BMR}, via a series of Monte Carlo simulations. The BMDL exhibited generally stable coverage characteristics, but some undercoverage—i.e., coverage rates below the nominal % level—occurred for very small sample sizes and very shallow dose-response patterns. Our earlier focus was on introducing the isotonic estimation method and exploring its theoretical properties, however, and so our study was limited to a small, illustrative selection of dose-response functions. We recognized that more complete evaluations were needed to study the operating characteristics of our model-independent BMDL. Here, we expand upon our previous results to study how the isotonic method operates over a broader range of possible dose-response models/shapes.

#### 3.1. Simulation Design

For the underlying dose-response patterns, we employ models corresponding to a variety of functions available from the U.S. EPA's BMDS software program for performing BMD calculations.[44] Table I provides a selection of four two-parameter and four three-parameter dose-response models. These are taken from a collection of dose-response forms chosen by Wheeler and Bailer[36] in their studies of (parametric) estimators for the BMD. (Wheeler and Bailer did not include the log-logistic model (3B) in all their calculations, although they did present it as a possible data generating model. They also presented a possible data generating model based on the gamma cumulative distribution function (c.d.f.), but did not use it in their calculations. In a similar vein, we do not consider the gamma model here.) In our earlier work, [32] we studied a subset of the models in Table I: the two-stage (3A), log-probit (3C), and Weibull (3D). Thus, for these three models the simulation outcomes we give here replicate the results we reported previously. Notice that certain models impose constraints on selected parameters; these are listed in Table I to correspond with typical constraints we find in the environmental toxicology literature. In the table, for the log-probit model (3C) we define . Note that the quantal-linear model (2A) may also be referred to as the “one-stage” model (a form of “multi-stage” model) or as the “complementary-log” model. This may equivalently appear as , where

Table I. Selected Quantal Dose-Response Models Common in Environmental ToxicologyModel | Code | | Constraints/Notes |
---|

Quantal-linear | 2A | | , |

Quantal-quadratic | 2B | | , |

Logistic | 2C | | — |

Probit | 2D | | is the N(0,1) c.d.f. |

Two-stage | 3A | | , |

Log-logistic | 3B | | , |

Log-probit | 3C | | , |

Weibull | 3D | | , |

Table II. Models and Configurations (Including True BMD, ξ_{10}, at BMR = 0.10) for the Monte Carlo Evaluations | Configuration: | A | B | C | D | E | F |
---|

Constraint | *R*(0) = | 0.01 | 0.01 | 0.10 | 0.05 | 0.30 | 0.10 |

| *R*() = | 0.04 | 0.07 | 0.17 | 0.30 | 0.52 | 0.50 |

| *R*(1) = | 0.10 | 0.20 | 0.30 | 0.50 | 0.75 | 0.90 |

Model | Parameters | | | | | | |

Quantal-linear (2A) | β_{0} | 0.0101 | 0.0101 | 0.1054 | 0.0513 | 0.3567 | 0.1054 |

| β_{1} | 0.0953 | 0.2131 | 0.2513 | 0.6419 | 1.0296 | 2.1972 |

| ξ_{10} | 1.1056 | 0.4944 | 0.4193 | 0.1641 | 0.1023 | 0.0480 |

Quantal-quadratic (2B) | γ_{0} | 0.0100 | 0.0100 | 0.1000 | 0.0500 | 0.3000 | 0.1000 |

| β_{1} | 0.0953 | 0.2131 | 0.2513 | 0.6419 | 1.0296 | 2.1972 |

| ξ_{10} | 1.0514 | 0.7032 | 0.6475 | 0.4052 | 0.3199 | 0.2190 |

Logistic (2C) | β_{0} | −4.5951 | −4.5951 | −2.1972 | −2.9444 | −0.8473 | −2.1972 |

| β_{1} | 2.3979 | 3.2088 | 1.3499 | 2.9444 | 1.9459 | 4.3944 |

| ξ_{10} | 1.0401 | 0.7773 | 0.5535 | 0.3974 | 0.1619 | 0.1700 |

Probit (2D) | β_{0} | −2.3263 | −2.3263 | −1.2816 | −1.6449 | −0.5244 | −1.2816 |

| β_{1} | 1.0448 | 1.4847 | 0.7572 | 1.6449 | 1.1989 | 2.5631 |

| ξ_{10} | 1.0476 | 0.7372 | 0.5331 | 0.3567 | 0.1606 | 0.1575 |

Two-stage (3A) | β_{0} | 0.0101 | 0.0101 | 0.1054 | 0.0513 | 0.3567 | 0.1054 |

| β_{1} | 0.0278 | 0.0370 | 0.0726 | 0.5797 | 0.4796 | 0.1539 |

| β_{2} | 0.0675 | 0.1761 | 0.1788 | 0.0622 | 0.5501 | 2.0433 |

| ξ_{10} | 1.0602 | 0.6756 | 0.5911 | 0.1783 | 0.1818 | 0.1925 |

Log-logistic (3B) | γ_{0} | 0.0100 | 0.0100 | 0.1000 | 0.0500 | 0.3000 | 0.1000 |

| β_{0} | −2.3026 | −1.4376 | −1.2528 | −0.1054 | 0.5878 | 2.0794 |

| β_{1} | 1.6781 | 1.8802 | 1.7603 | 1.3333 | 1.9735 | 3.3219 |

| ξ_{10} | 1.0648 | 0.6676 | 0.5848 | 0.2083 | 0.2439 | 0.2760 |

Log-probit (3C) | γ_{0} | 0.0100 | 0.0100 | 0.1000 | 0.0500 | 0.3000 | 0.1000 |

| β_{0} | −1.3352 | −0.8708 | −0.7647 | −0.0660 | 0.3661 | 1.2206 |

| β_{1} | 0.7808 | 0.9794 | 0.9456 | 0.8189 | 1.2261 | 1.9626 |

| ξ_{10} | 1.0711 | 0.6575 | 0.5789 | 0.2267 | 0.2608 | 0.2794 |

Weibull (3D) | γ_{0} | 0.0100 | 0.0100 | 0.1000 | 0.0500 | 0.3000 | 0.1000 |

| β_{0} | −2.3506 | −1.5460 | −1.3811 | −0.4434 | 0.0292 | 0.7872 |

| β_{1} | 1.6310 | 1.7691 | 1.6341 | 1.0716 | 1.4483 | 1.9023 |

| ξ_{10} | 1.0634 | 0.6716 | 0.5874 | 0.1852 | 0.2072 | 0.2025 |

#### 3.2. Infinite BMDs

An unusual artifact we uncovered while conducting our Monte Carlo computations was that for some very shallow dose-response patterns, calculation of a nonparametric BMD can sometimes break down. One obvious case is when the BMR is set too high, so that the isotonic extra risk estimator never reaches the desired benchmark response over the range of the doses, i.e., , for all . If so, there is no solution to the BMD-defining relationship . Of course, if the extra risk were estimated using a fully parametric (nondecreasing) function one would simply extrapolate the function outside of the dose range to find the solution. To imitate this strategy with our nonparametric estimator, suppose that from Equation (3) is linear and strictly increasing along its final segment between and . Then if , we simply extend this final line segment past until it crosses the horizontal BMR line, and solve for at that intersection point. While admittedly one should apply any such extrapolations past the range of the data with great caution, this strategy nonetheless allows us to report an objective estimate for the BMD in this unusual case.

The extrapolative estimate for the BMD will still fail if the final line segment from Equation (3) is flat, i.e., if (or when , etc.). When this occurs, the data are in effect telling us that the observed dose response cannot attain the BMR, no matter how large *x* grows. Correspondingly, in such an instance we simply drive the estimator to ∞, or, equivalently, report it as undefined. In the extreme, this also occurs if the s all equal each other. In this case, Equation (3) will produce , so we again are forced to drive to ∞.

Somewhat perniciously, this issue of undefined or infinite BMDs was not uncommon with some of the very shallow dose-response configurations in Table II, especially configurations “A” and “C.” Particularly at the smaller sample sizes, such shallow configurations could even produce nonmonotone response patterns in the simulated data, despite the fact that the underlying function was strictly increasing. This forced the PAV algorithm to “flatten out” the estimated extra risk over a large portion of the dose range, and when this occurred near the upper end of the range we encountered the infinite-BMD phenomenon.

#### 3.3. Simulation Results

We summarize the empirical coverage results from our Monte Carlo study in a series of tables. Recall that we fix BMR = 0.10 and operate at nominal 95% coverage. Each initial table displays empirical coverage results recorded for all model configurations and sample sizes except the quantal-linear model (2A); we will discuss model 2A in greater detail below. Table III presents the results for the geometric four-dose design with 0, 0.25, 0.50, 1.0: therein, coverages lie near and generally within Monte Carlo sampling variability of the nominal 95% level. Some undercoverage is observed at the lowest per-dose sample size of and/or with the (shallow) response configurations “A” and, to a lesser extent, “B”; however, conservative overcoverage is at least as prevalent. Indeed, averaged across the seven models (2B–2D and 3A–3D) in Table III, the empirical coverages as a function of sample size are at or near the 95% nominal level: 94.22% for , 94.76% for , 95.84% for , and 96.95% for 1,000.

Table III. Empirical Coverage Rates of Nonparametric Bootstrap BMDL _{10} from Monte Carlo Evaluations Under Geometric Four-Dose Design for Selected Dose-Response Models Given in Table I | | Configuration | |
---|

Model Code | Sample Size, *N* | A | B | C | D | E | F | Row Means |
---|

Notes |

2B | 25 | 0.9355 | 0.9470 | 0.9820 | 0.9640 | 0.9780 | 0.9340 | 0.9568 |

2B | 50 | 0.8955 | 0.9655 | 0.9900 | 0.9740 | 0.9750 | 0.9500 | 0.9583 |

2B | 100 | 0.9185 | 0.9730 | 0.9870 | 0.9820 | 0.9715 | 0.9715 | 0.9673 |

2B | 1000 | 0.9270 | 0.9970 | 0.9930 | 0.9895 | 0.9720 | 0.9895 | 0.9780 |

2C | 25 | 0.9330 | 0.9705 | 0.9680 | 0.9640 | 0.9305 | 0.9405 | 0.9511 |

2C | 50 | 0.8980 | 0.9845 | 0.9765 | 0.9725 | 0.9340 | 0.9565 | 0.9537 |

2C | 100 | 0.9125 | 0.9925 | 0.9760 | 0.9785 | 0.9465 | 0.9755 | 0.9636 |

2C | 1000 | 0.9050 | 1.0000 | 0.9675 | 0.9875 | 0.9605 | 0.9950 | 0.9693 |

2D | 25 | 0.9340 | 0.9640 | 0.9635 | 0.9555 | 0.9295 | 0.9440 | 0.9484 |

2D | 50 | 0.8945 | 0.9780 | 0.9740 | 0.9640 | 0.9345 | 0.9570 | 0.9503 |

2D | 100 | 0.9170 | 0.9865 | 0.9735 | 0.9710 | 0.9325 | 0.9765 | 0.9595 |

2D | 1000 | 0.9125 | 1.0000 | 0.9570 | 0.9820 | 0.9605 | 0.9945 | 0.9678 |

3A | 25 | 0.9370 | 0.9290 | 0.9725 | 0.8970 | 0.9260 | 0.9475 | 0.9348 |

3A | 50 | 0.9060 | 0.9545 | 0.9820 | 0.9125 | 0.9350 | 0.9605 | 0.9418 |

3A | 100 | 0.9185 | 0.9650 | 0.9820 | 0.9275 | 0.9450 | 0.9765 | 0.9524 |

3A | 1000 | 0.9295 | 0.9960 | 0.9760 | 0.9335 | 0.9620 | 0.9970 | 0.9657 |

3B | 25 | 0.9375 | 0.9220 | 0.9700 | 0.9045 | 0.9305 | 0.9370 | 0.9336 |

3B | 50 | 0.9085 | 0.9505 | 0.9780 | 0.9330 | 0.9360 | 0.9495 | 0.9426 |

3B | 100 | 0.9230 | 0.9595 | 0.9785 | 0.9470 | 0.9505 | 0.9685 | 0.9545 |

3B | 1000 | 0.9370 | 0.9885 | 0.9695 | 0.9740 | 0.9580 | 0.9915 | 0.9698 |

3C | 25 | 0.9375 | 0.9075 | 0.9695 | 0.9270 | 0.9460 | 0.9310 | 0.9364 |

3C | 50 | 0.9155 | 0.9445 | 0.9815 | 0.9235 | 0.9525 | 0.9500 | 0.9446 |

3C | 100 | 0.9290 | 0.9525 | 0.9805 | 0.9465 | 0.9560 | 0.9705 | 0.9558 |

3C | 1000 | 0.9465 | 0.9725 | 0.9620 | 0.9710 | 0.9495 | 0.9900 | 0.9653 |

3D | 25 | 0.9385 | 0.9230 | 0.9700 | 0.8935 | 0.9270 | 0.9520 | 0.9340 |

3D | 50 | 0.9035 | 0.9515 | 0.9805 | 0.9170 | 0.9410 | 0.9595 | 0.9422 |

3D | 100 | 0.9205 | 0.9600 | 0.9795 | 0.9355 | 0.9615 | 0.9775 | 0.9558 |

3D | 1000 | 0.9360 | 0.9940 | 0.9735 | 0.9500 | 0.9760 | 0.9960 | 0.9709 |

Table IV presents the results for the geometric six-dose design with 0, 0.0625, 0.125, 0.25, 0.50, 1.0: coverage patterns therein appear slightly more stable on average than those seen in Table III, although there are also more extreme cases of undercoverage. These again occur at the lowest per-dose sample size and/or with the (shallow) response configuration “A.” Overall, average empirical coverages across the seven models 2B–2D and 3A–3D are again close to nominal: 94.61% for , 95.96% for , 96.88% for , and 97.15% for 1,000. As might be expected, decreasing the per-dose sample sizes while increasing the number of doses yields greater stability at larger sample sizes, but this effect appears to reverse somewhat when sample sizes drop.

Table IV. Empirical Coverage Rates of Nonparametric Bootstrap BMDL _{10} from Monte Carlo Evaluations Under Geometric Six-Dose Design for Selected Dose-Response Models Given in Table I | | Configuration | |
---|

Model Code | Sample Size, *N* | A | B | C | D | E | F | Row Means |
---|

Note |

2B | 16 | 0.8805 | 0.9120 | 0.9795 | 0.9685 | 0.9900 | 0.9730 | 0.9506 |

2B | 33 | 0.8685 | 0.9480 | 0.9940 | 0.9790 | 0.9950 | 0.9820 | 0.9611 |

2B | 66 | 0.9050 | 0.9715 | 0.9950 | 0.9900 | 0.9945 | 0.9825 | 0.9731 |

2B | 666 | 0.9320 | 0.9985 | 0.9965 | 0.9930 | 0.9915 | 0.9830 | 0.9824 |

2C | 16 | 0.8765 | 0.9505 | 0.9690 | 0.9720 | 0.9550 | 0.9735 | 0.9494 |

2C | 33 | 0.8700 | 0.9770 | 0.9885 | 0.9800 | 0.9780 | 0.9855 | 0.9632 |

2C | 66 | 0.9025 | 0.9905 | 0.9835 | 0.9860 | 0.9755 | 0.9830 | 0.9702 |

2C | 666 | 0.9130 | 1.0000 | 0.9705 | 0.9910 | 0.9565 | 0.9690 | 0.9667 |

2D | 16 | 0.8785 | 0.9415 | 0.9680 | 0.9740 | 0.9565 | 0.9665 | 0.9475 |

2D | 33 | 0.8705 | 0.9680 | 0.9890 | 0.9815 | 0.9770 | 0.9790 | 0.9608 |

2D | 66 | 0.9035 | 0.9850 | 0.9825 | 0.9885 | 0.9725 | 0.9770 | 0.9682 |

2D | 666 | 0.9205 | 1.0000 | 0.9725 | 0.9865 | 0.9610 | 0.9625 | 0.9672 |

3A | 16 | 0.8800 | 0.9235 | 0.9760 | 0.9355 | 0.9700 | 0.9755 | 0.9434 |

3A | 33 | 0.8750 | 0.9475 | 0.9915 | 0.9635 | 0.9850 | 0.9845 | 0.9578 |

3A | 66 | 0.9070 | 0.9685 | 0.9915 | 0.9675 | 0.9845 | 0.9860 | 0.9675 |

3A | 666 | 0.9330 | 0.9940 | 0.9855 | 0.9495 | 0.9560 | 0.9825 | 0.9668 |

3B | 16 | 0.8825 | 0.9180 | 0.9715 | 0.9460 | 0.9705 | 0.9810 | 0.9449 |

3B | 33 | 0.9080 | 0.9385 | 0.9900 | 0.9615 | 0.9850 | 0.9890 | 0.9620 |

3B | 66 | 0.9170 | 0.9585 | 0.9885 | 0.9710 | 0.9885 | 0.9920 | 0.9693 |

3B | 666 | 0.9420 | 0.9895 | 0.9830 | 0.9590 | 0.9680 | 0.9955 | 0.9728 |

3C | 16 | 0.8815 | 0.8980 | 0.9740 | 0.9485 | 0.9740 | 0.9830 | 0.9432 |

3C | 33 | 0.8735 | 0.9295 | 0.9895 | 0.9645 | 0.9885 | 0.9895 | 0.9558 |

3C | 66 | 0.9110 | 0.9510 | 0.9885 | 0.9695 | 0.9855 | 0.9920 | 0.9663 |

3C | 666 | 0.9520 | 0.9775 | 0.9760 | 0.9625 | 0.9740 | 0.9975 | 0.9733 |

3D | 16 | 0.8815 | 0.9210 | 0.9725 | 0.9390 | 0.9745 | 0.9755 | 0.9440 |

3D | 33 | 0.8740 | 0.9430 | 0.9900 | 0.9640 | 0.9850 | 0.9845 | 0.9568 |

3D | 66 | 0.9085 | 0.9635 | 0.9895 | 0.9675 | 0.9895 | 0.9865 | 0.9675 |

3D | 666 | 0.9390 | 0.9915 | 0.9835 | 0.9500 | 0.9760 | 0.9880 | 0.9713 |

Table V. Empirical Coverage Rates of Nonparametric Bootstrap BMDL _{10} from Monte Carlo Evaluations Under Modified Six-Dose Design for Selected Dose-Response Models Given in Table I | | Configuration | |
---|

Model Code | Sample Size, *N* | A | B | C | D | E | F | Row Means |
---|

Note |

2B | 16 | 0.8840 | 0.9085 | 0.9710 | 0.9230 | 0.9750 | 0.9570 | 0.9364 |

2B | 33 | 0.8815 | 0.9360 | 0.9890 | 0.9465 | 0.9855 | 0.9760 | 0.9524 |

2B | 66 | 0.9100 | 0.9595 | 0.9850 | 0.9635 | 0.9900 | 0.9735 | 0.9636 |

2B | 666 | 0.9335 | 0.9875 | 0.9810 | 0.9560 | 0.9855 | 0.9705 | 0.9690 |

2C | 16 | 0.8795 | 0.9445 | 0.9680 | 0.9330 | 0.9325 | 0.9530 | 0.9351 |

2C | 33 | 0.8760 | 0.9695 | 0.9860 | 0.9480 | 0.9680 | 0.9715 | 0.9532 |

2C | 66 | 0.9065 | 0.9865 | 0.9810 | 0.9645 | 0.9590 | 0.9705 | 0.9613 |

2C | 666 | 0.9185 | 0.9995 | 0.9695 | 0.9485 | 0.9480 | 0.9580 | 0.9570 |

2D | 16 | 0.8840 | 0.9365 | 0.9690 | 0.9455 | 0.9325 | 0.9520 | 0.9366 |

2D | 33 | 0.8810 | 0.9590 | 0.9830 | 0.9620 | 0.9665 | 0.9690 | 0.9534 |

2D | 66 | 0.9100 | 0.9740 | 0.9830 | 0.9700 | 0.9590 | 0.9635 | 0.9599 |

2D | 666 | 0.9220 | 0.9980 | 0.9660 | 0.9635 | 0.9465 | 0.9580 | 0.9590 |

3A | 16 | 0.8885 | 0.8895 | 0.9640 | 0.9230 | 0.9355 | 0.9565 | 0.9262 |

3A | 33 | 0.8845 | 0.9325 | 0.9835 | 0.9495 | 0.9730 | 0.9705 | 0.9489 |

3A | 66 | 0.9135 | 0.9525 | 0.9785 | 0.9540 | 0.9665 | 0.9670 | 0.9553 |

3A | 666 | 0.9385 | 0.9715 | 0.9645 | 0.9520 | 0.9560 | 0.9545 | 0.9562 |

3B | 16 | 0.8895 | 0.8810 | 0.9660 | 0.9240 | 0.9640 | 0.9920 | 0.9361 |

3B | 33 | 0.8835 | 0.9220 | 0.9845 | 0.9505 | 0.9810 | 0.9955 | 0.9528 |

3B | 66 | 0.9145 | 0.9450 | 0.9810 | 0.9545 | 0.9785 | 0.9975 | 0.9618 |

3B | 666 | 0.9425 | 0.9640 | 0.9690 | 0.9445 | 0.9770 | 1.0000 | 0.9662 |

3C | 16 | 0.8840 | 0.8745 | 0.9695 | 0.9115 | 0.9740 | 0.9920 | 0.9343 |

3C | 33 | 0.8935 | 0.9140 | 0.9845 | 0.9495 | 0.9880 | 0.9980 | 0.9546 |

3C | 66 | 0.9165 | 0.9385 | 0.9860 | 0.9470 | 0.9865 | 0.9990 | 0.9623 |

3C | 666 | 0.9370 | 0.9585 | 0.9835 | 0.9520 | 0.9845 | 1.0000 | 0.9693 |

3D | 16 | 0.8885 | 0.8835 | 0.9655 | 0.9265 | 0.9435 | 0.9515 | 0.9265 |

3D | 33 | 0.8845 | 0.9285 | 0.9845 | 0.9485 | 0.9690 | 0.9710 | 0.9477 |

3D | 66 | 0.9150 | 0.9470 | 0.9805 | 0.9605 | 0.9635 | 0.9685 | 0.9558 |

3D | 666 | 0.9400 | 0.9685 | 0.9720 | 0.9445 | 0.9600 | 0.9575 | 0.9571 |

On balance, our simulation results exhibit generally stable large-sample coverage characteristics for the model-independent, bootstrapped limit _{100BMR} at the standard level of BMR = 0.10. Slight undercoverage is evidenced in selected instances, more so when sample sizes are small. We previously identified a possible explanation for this behavior, [32] where we found that the PAV-based estimator in Equation (4) exhibits slight negative bias when applied to convex response patterns. Negative bias in the estimator can translate to more conservative lower confidence bounds for the BMDL. On a relative scale the bias was not exceptional, however, and in fact is not wholly unexpected: bias can be a recurring issue with isotonic regression estimators. [47] Indeed, the effect moderated as sample sizes increased. Nonetheless, this suggests that the method should be applied when sufficient data are available to help validate its asymptotic motivation.

We also conducted Monte Carlo coverage evaluations for the case of BMR = 0.01. While less common, this smaller BMR may be employed in practice when sufficient data are available to support inferences at extreme low doses. [14, 9] Our results (not shown) were generally similar to those seen above; in particular, the coverage rates again drove toward the nominal 95% level as the sample size increased. At the lowest sample sizes, however, they appeared much more variable, and often in the direction of undercoverage. We encountered a few empirical coverage rates that dropped below 50% with some of the low-response-rate models, especially configurations “A” and “B.” This is, in fact, consistent with practical benchmarking experience: when response rates are very small at low doses, and if the s do not counter by being fairly large, insufficient information will be available to perform effective inferences if the BMR is set very low. We therefore urge caution in practice when employing these methods with very small sample sizes, particularly with low-response patterns.

#### 3.4. Simulation Results for the (Concave) Quantal-Linear Model

This feature impacts our linear interpolator. A concave response function will, more often than not, produce concave dose-response patterns in the s from Equation (1), and linearly interpolating a concave-increasing pattern can lead to underestimation of the extra risk. This translates as overestimation of . The corresponding lower confidence bounds would, in turn, be driven up. If variation in the data is tight this could push them past the true, underlying value of , collapsing the coverage rates. With very small numbers of doses, *m*, and large per-dose sample sizes, , the rates can drop well below their nominal level. In theory the issue would quickly be remedied by increasing *m*, since the theoretical properties of Equation (4) obtain as both and *m* grow large without limit. With small *m*, however, the potential undercoverage can be dramatic.

This effect was evidenced in our Monte Carlo coverage evaluations. Table VI presents Model 2A's small-sample coverage rates for our model-independent, bootstrap BMDL _{100BMR} at the standard level of BMR = 0.10, and with nominal coverage set to 95%. Notice the large numbers of entries below 0.95 (the zero coverage value, 0.0000, for the four-dose design under configuration “F” at 1,000 is *not* a typographical error). In Table II, one can determine that concavity in Model 2A increases as the configuration index moves from “A” to “F.” Thus two clear trends emerge in Table VI: (i) coverage performance worsens as the concavity of the model increases, and (ii) adding more doses improves the performance more often than it debilitates it. In addition, comparing the two six-dose designs shows that as more sample information is placed closer to the true BMD—which of course is impossible in practice without knowledge of that true value—the coverage locates closer to its nominal level.

Table VI. Empirical Coverage Rates of Nonparametric Bootstrap BMDL _{10} from Monte Carlo Evaluations for Quantal-Linear Dose-Response Model (2A) Given in Table I | | Configuration | |
---|

Model Code | Sample Size, *N* | A | B | C | D | E | F | Row Means |
---|

Note |

| | Geometric Four-Dose Design | |

2A | 25 | 0.9330 | 0.9160 | 0.9570 | 0.8425 | 0.9110 | 0.7465 | 0.8843 |

2A | 50 | 0.9120 | 0.9205 | 0.9630 | 0.9045 | 0.9115 | 0.6335 | 0.8742 |

2A | 100 | 0.9305 | 0.9330 | 0.9575 | 0.9215 | 0.9115 | 0.4785 | 0.8554 |

2A | 1000 | 0.9540 | 0.9560 | 0.9545 | 0.9145 | 0.8445 | 0.0000 | 0.7706 |

| | Geometric Six-Dose Design | |

2A | 16 | 0.8885 | 0.8860 | 0.9680 | 0.9250 | 0.9315 | 0.8665 | 0.9109 |

2A | 33 | 0.8950 | 0.9155 | 0.9835 | 0.9605 | 0.9630 | 0.9140 | 0.9386 |

2A | 66 | 0.9270 | 0.9430 | 0.9810 | 0.9615 | 0.9525 | 0.9205 | 0.9476 |

2A | 666 | 0.9495 | 0.9400 | 0.9680 | 0.9465 | 0.9470 | 0.9280 | 0.9465 |

| | Modified Six-Dose Design | |

2A | 16 | 0.8945 | 0.8790 | 0.9530 | 0.9190 | 0.8035 | 0.8595 | 0.8848 |

2A | 33 | 0.9150 | 0.9075 | 0.9695 | 0.9470 | 0.9010 | 0.9010 | 0.9235 |

2A | 66 | 0.9330 | 0.9355 | 0.9685 | 0.9505 | 0.9000 | 0.9065 | 0.9323 |

2A | 666 | 0.9455 | 0.9390 | 0.9485 | 0.9465 | 0.9395 | 0.8715 | 0.9318 |

The coverage results in Table VI warn that our PAV-based benchmark estimator should be applied to concave dose-response patterns with caution. For shallow response patterns (configurations “A”–“C”), coverage is roughly similar to that seen in Tables III–V. With greater concavity comes greater instability, however, leading to extreme degradation with the extremely concave configuration “F.”

Specifically, given PAV-based estimates from Equation (1), write the slope between each adjacent pair as . The change in these slopes is . Then, e.g., the average change in slope, quantifies concavity in the isotonically estimated response: smaller (more negative) values of indicate greater concavity. Note, however, that concavity far away from the BMD has little effect on the coverage for the BMDL _{100BMR}. More pertinent for our purposes is a measure of *local* concavity near the estimated benchmark point. For instance, suppose the calculated BMD lies between two dose values. Then, to measure local concavity we might average the two corresponding values of associated with those bracketing doses. That is, define a measure of local concavity, , as:

- (5)

The latter specification for when is based on our suggestion in Section 'Infinite BMDs' to extrapolate along a straight-line segment to define beyond the upper range of the data. A possible alternative in this case is to take . In any case, similar to , smaller (more negative) values of indicate greater local concavity.

Table VII. Local Change in Slope, from Equation (5), Under Quantal-Linear Dose-Response Model (2A) Given in Table I Across Configurations from Table IIConfiguration |
---|

Design | A | B | C | D | E | F |
---|

Geometric four-dose | −0.0128 | −0.0511 | −0.0635 | −0.3341 | −0.5768 | −2.5721 |

Geometric six-dose | −0.0128 | −0.0615 | −0.0765 | −0.5112 | −0.8273 | −3.7932 |

Modified six-dose | −0.0127 | −0.0499 | −0.0618 | −0.4365 | −0.7736 | −3.5018 |

### 4. EXAMPLE: FORMALDEHYDE CARCINOGENESIS IN LABORATORY ANIMALS

- Top of page
- Abstract
- 1. INTRODUCTION
- 2. MODEL-INDEPENDENT BENCHMARK ANALYSIS
- 3. BMDL PERFORMANCE EVALUATION
- 4. EXAMPLE: FORMALDEHYDE CARCINOGENESIS IN LABORATORY ANIMALS
- 5. DISCUSSION
- ACKNOWLEDGMENTS
- REFERENCES

Formaldehyde, CH_{2}O, is a well-known industrial compound, exposure to which can be extensive in a variety of occupational and environmental settings. To explore the toxic and carcinogenic potential of the chemical, Schlosser *et al*.[48] reported on nasal squamous cell carcinomas observed in laboratory rats after chronic, two-year, inhalation exposure. The CH_{2}O exposure dose, *x*, is actually a concentration (in ppm) here, and so technically we will compute BMCs based on the quantal carcinogenicity data. Six CH_{2}O concentrations were studied: 0.0, 0.7, 2.0, 6.0, 10.0, and 15.0 ppm. Since intercurrent mortality can occur in such chronic-exposure studies, the final tumor incidences were adjusted for potential differences in animal survival. Table VIII lists the survival-adjusted proportions.

Table VIII. Formaldehyde Carcinogenicity Data[48]Exposure Conc. (ppm), | 0.0 | 0.7 | 2.0 | 6.0 | 10.0 | 15.0 |
---|

Adjusted tumor incidence, | 0 | 0 | 0 | 3 | 21 | 150 |

Animals at risk, | 122 | 27 | 126 | 113 | 34 | 182 |

### 5. DISCUSSION

- Top of page
- Abstract
- 1. INTRODUCTION
- 2. MODEL-INDEPENDENT BENCHMARK ANALYSIS
- 3. BMDL PERFORMANCE EVALUATION
- 4. EXAMPLE: FORMALDEHYDE CARCINOGENESIS IN LABORATORY ANIMALS
- 5. DISCUSSION
- ACKNOWLEDGMENTS
- REFERENCES

Herein, we consider a model-independent, nonparametric method for estimating BMDs in quantitative risk analysis. Placing emphasis on cancer risk assessment, we describe an approach for estimating the BMD without call to any specific parametric dose-response models, relying only on the assumptions that the underlying response is continuous and monotone nondecreasing. [31, 30] Lower confidence limits (BMDLs) using this estimator are derived from nonparametric bootstrap methods, building upon previous explorations into bootstrap resampling for benchmark inference. [49, 37] Based on a Monte Carlo study, we find that the BMDLs exhibit relatively stable coverage for reasonably large sample sizes, but that some undercoverage can occur for very small sample sizes and very shallow dose-response patterns. Since the method is based on linearly interpolating the nonparametrically estimated risk function, the undercoverage is exacerbated if the dose-response pattern is highly concave and the number of doses is small; in such cases we cannot recommend use of this procedure. (But, see below.)

When sufficient data are available to support the nonparametric constructions, risk analysts can apply our results to build inferences on the BMD that avoid concerns over parametric model adequacy, expanding past the many, varied parametric models seen in practice. This extended operability can lead to improved risk analytic decisionmaking in carcinogenicity testing and other adverse-event risk assessments.

Of course, some caveats and qualifications are in order. The percentile method we used for finding the BMDL _{100BMR} is a basic approach for constructing bootstrap inferences. While the method appears generally stable at larger sample sizes, its mixed performance with very small samples might be improved by moving to more-complex bootstrapping strategies. For instance, the bias-corrected, accelerated (BCa) bootstrap [35] is a well-known alternative to the percentile method, so we evaluated the BCa approach under the standard four-dose design using our convex-model Monte Carlo configurations from Section 'Simulation Design'. We found that the resulting BMDLs exhibited slightly tighter empirical coverages at smaller sample sizes and with the more problematic, shallow, response patterns. Improvements were not seen across all configurations studied, however. (Details are available in a separate document.[50]) From this, we can recommend use of BCa-based BMDLs when samples sizes are very small and/or for shallow response patterns; however, the basic percentile method appeared to operate adequately in the majority of cases we studied.

Comparison of the FMA BMDL with our nonparametric BMDL _{100BMR} is possible here: fortunately, Piegorsch *et al*. employed the exact same models and parametric configurations as in our Table II for their own small-sample simulation study of 's coverage characteristics. (They considered only the geometric four-dose design, but they did use the same BMR = 0.10 and nominal 95% confidence level as we have here.) Thus, it is possible to explicitly compare the empirical coverage rates they achieved under their parametric FMA BMDL with those we find under our nonparametric BMDL. We plot in Fig. 1 the associated pairs of empirical coverages for all 192 model/configuration/sample size combinations represented in Table III and for Model 2A, with the geometric four-dose design (Table VI). In the figure, empirical coverage for the FMA BMDL is plotted on the vertical axis and that for our nonparametric BMDL is plotted on the horizontal axis. (Note the different axis scales.) If both methods operated perfectly, we would expect a tight cluster of points to lie at the crossing of the two “95% nominal coverage” lines in the figure. Instead, the graphic portrays a more complex scenario. It corroborates the near-to-nominal coverage for our nonparametric BMDL seen in Table III, but it also highlights the few distressingly low coverage values under Model 2A in Table VI: see the points lying far to the left of the vertical rule marking nominal coverage. Indeed, the horizontal scale is lengthened by these few low-coverage points, making it difficult to visualize the larger comparison. To compensate, Fig. 2 repeats the plot with all the Model 2A points removed. The finer detail illustrates that by excluding Model 2A, very few cases occur where both methods produce significant undercoverage (since the lower left quadrant in the plot is almost empty), and that the FMA BMDL is often more conservative than our nonparametric BMDL (since the upper right quadrant in the plot is the most dense). Indeed, the upper half of the plot is far more populated, indicating the more-conservative coverage of the FMA BMDL. (This was, in fact, explicitly recognized by Piegorsch *et al*.:[19] their method included a conservative approximation for to avoid cases of severe undercoverage, and this is evident in Fig. 1: to within Monte Carlo sampling error, only a handful of points lie below the nominal 95% level.) This conservatism does provide for some added “safety” in use of : in contrast to the nonparametric BMDL studied here, the FMA BMDL does not fall victim to problems of benchmark overestimation with concave dose-response patterns. As Piegorsch *et al*. readily admitted, however, their FMA BMDL is dependent on proper development of a pertinent uncertainty class of parametric models. In cases where this is not possible, the nonparametric BMDL we suggest here can serve as a model-robust alternative, when employed cautiously with concave-increasing data (as described in Section 'Simulation Results for the (Concave) Quantal-Linear Model').

Referring to our Monte Carlo results in Table III, we see that with doses the nonparametric BMDL's coverage characteristics are relatively stable if sufficient numbers of subjects/dose, *N*, are employed. Still, one questions whether we can gain greater information about the pattern of dose response, and therefore about , if we increase the number of doses. Somewhat more-stable coverage patterns were seen with doses in Tables IV and V. So, can increasing *m* to, say, 10 doses improve the small-sample operating characteristics of the BMDL if resource constraints force the s down to perhaps only 10 subjects/dose? For example, Bhattacharya and Lin[53] were able to show that an adaptive nonparametric estimator based on inverting Equation (2) proved competitive in such a setting. To examine how this extends to BMD estimation with extra risk functions, we repeated our Monte Carlo evaluations for all eight models in Table I with doses spaced evenly between and . We again studied constant per-dose sample sizes, *N*, using 10, 20, 40, and 400 to compare with the total sample sizes in Table III. (All other aspects of the calculations for the bootstrap BMDL _{10} remained the same.) The results appear in Table IX; note the inclusion of Model 2A at the top of the table.

Table IX. Empirical Coverage Rates of Nonparametric Bootstrap BMDL _{10} from Monte Carlo Evaluations Under Equi-Spaced 10-Dose Design for Dose-Response Models Given in Table I | | Configuration | |
---|

Model Code | Sample Size, *N* | A | B | C | D | E | F | Row Means |
---|

Note |

2A | 10 | 0.8430 | 0.8810 | 0.9845 | 0.9420 | 0.9900 | 0.9215 | 0.9270 |

2A | 20 | 0.8930 | 0.9170 | 0.9950 | 0.9645 | 0.9895 | 0.9555 | 0.9524 |

2A | 40 | 0.9275 | 0.9295 | 0.9970 | 0.9720 | 0.9880 | 0.9675 | 0.9636 |

2A | 400 | 0.9560 | 0.9505 | 0.9805 | 0.9655 | 0.9805 | 0.9615 | 0.9658 |

2B | 10 | 0.8155 | 0.9540 | 0.9830 | 0.9380 | 0.9995 | 0.9970 | 0.9478 |

2B | 20 | 0.8855 | 0.9445 | 0.9960 | 0.9785 | 1.0000 | 0.9980 | 0.9671 |

2B | 40 | 0.9215 | 0.9765 | 0.9985 | 0.9715 | 1.0000 | 0.9990 | 0.9778 |

2B | 400 | 0.9370 | 0.9980 | 1.0000 | 0.9735 | 0.9990 | 0.9950 | 0.9838 |

2C | 10 | 0.8105 | 0.9495 | 0.9855 | 0.9540 | 0.9910 | 0.9870 | 0.9463 |

2C | 20 | 0.8840 | 0.9615 | 0.9960 | 0.9850 | 0.9930 | 0.9870 | 0.9678 |

2C | 40 | 0.9065 | 0.9860 | 0.9980 | 0.9825 | 0.9920 | 0.9865 | 0.9753 |

2C | 400 | 0.9260 | 1.0000 | 0.9950 | 0.9675 | 0.9780 | 0.9795 | 0.9743 |

2D | 10 | 0.8130 | 0.9500 | 0.9860 | 0.9690 | 0.9905 | 0.9795 | 0.9480 |

2D | 20 | 0.8845 | 0.9580 | 0.9955 | 0.9890 | 0.9930 | 0.9825 | 0.9671 |

2D | 40 | 0.9220 | 0.9840 | 0.9980 | 0.9910 | 0.9915 | 0.9835 | 0.9783 |

2D | 400 | 0.9285 | 1.0000 | 0.9950 | 0.9820 | 0.9810 | 0.9815 | 0.9780 |

3A | 10 | 0.8165 | 0.9530 | 0.9840 | 0.9580 | 0.9970 | 0.9935 | 0.9503 |

3A | 20 | 0.8840 | 0.9500 | 0.9950 | 0.9750 | 0.9975 | 0.9955 | 0.9662 |

3A | 40 | 0.9240 | 0.9750 | 0.9985 | 0.9790 | 0.9970 | 0.9950 | 0.9781 |

3A | 400 | 0.9395 | 0.9965 | 0.9970 | 0.9715 | 0.9910 | 0.9930 | 0.9814 |

3B | 10 | 0.8205 | 0.9555 | 0.9840 | 0.9750 | 0.9990 | 1.0000 | 0.9557 |

3B | 20 | 0.8850 | 0.9440 | 0.9955 | 0.9900 | 0.9995 | 1.0000 | 0.9690 |

3B | 40 | 0.9240 | 0.9670 | 0.9975 | 0.9925 | 0.9995 | 1.0000 | 0.9801 |

3B | 400 | 0.9435 | 0.9930 | 0.9970 | 0.9905 | 1.0000 | 1.0000 | 0.9873 |

3C | 10 | 0.8265 | 0.9590 | 0.9865 | 0.9800 | 1.0000 | 1.0000 | 0.9587 |

3C | 20 | 0.8960 | 0.9360 | 0.9930 | 0.9930 | 1.0000 | 1.0000 | 0.9697 |

3C | 40 | 0.9190 | 0.9560 | 0.9975 | 0.9930 | 0.9995 | 1.0000 | 0.9775 |

3C | 400 | 0.9470 | 0.9915 | 0.9965 | 0.9885 | 0.9995 | 1.0000 | 0.9872 |

3D | 10 | 0.8200 | 0.9545 | 0.9840 | 0.8935 | 0.9990 | 0.9935 | 0.9408 |

3D | 20 | 0.8855 | 0.9465 | 0.9950 | 0.9170 | 0.9990 | 0.9960 | 0.9565 |

3D | 40 | 0.9245 | 0.9715 | 0.9980 | 0.9355 | 0.9990 | 0.9940 | 0.9704 |

3D | 400 | 0.9425 | 0.9950 | 0.9950 | 0.9500 | 0.9975 | 0.9915 | 0.9786 |

The patterns of coverage in Table IX appear roughly comparable to those in Tables III–VI: only with configuration “A” at does the coverage consistently weaken. Indeed, a highly encouraging indication is the improved stability in coverage for Model 2A. With doses the 2A rates now appear commensurate with patterns seen for most other models—even at the more-concave configurations—extending the indications seen in Table VI. At least as regards estimation and inferences in benchmark risk assessment, this provides strong encouragement to include larger numbers of doses for characterizing the response when designing modern dose-response studies. [54, 55, 22]