Statistical modeling to relate technology readiness to schedule

Projects from U.S. National Aeronautics and Space Administration and Department of Energy technology development activities provide historical data on project completion timelines versus technology readiness levels that were assessed during the life cycle of a program. A statistical analysis was performed to develop a method to forecast future project completion timelines based on technical maturity assessments. The goodness‐of‐fit of the model used for forecasting also was evaluated, and the null hypothesis that the data follows the probability distribution used for the forecasting model could not be rejected at a Type I error level of 0.05. Several potential extensions to the model using product forecasting methods, semi‐Markov processes, and Bayes nets are presented. The limitations of the statistical modeling and of the potential extensions are also discussed.


INTRODUCTION
During the early phase of system development, trade studies are performed to select among different alternative system concepts by comparing performance, cost, and schedule attributes of the alternative concepts.A quantitative method of forecasting development timelines can be used to support trade studies that investigate which technologies will support meeting the overall requirements for deploying an operational system.Once trade studies are completed and a baseline has been established for the system concept, forecasting of the system development timelines is still useful to perform effective risk identification and risk mitigation planning.
The critical path of system development schedules typically is defined by the development schedules of key component technologies that can range in technical maturity (TM) from ideas being tested in a laboratory to items that are readily available "off the shelf".The U.S. National Aeronautics and Space Administration (NASA) Technol-was the first use of NASA TRL data from Peisen, Schulz, Golaszewski, Ballard, and Smith 16 to estimate the parameters of the log-Student's t-distribution and use the results for schedule risk analysis.Subsequently, Crépin, El-Khoury, and Kenley 17

BACKGROUND
Research in the fields of systems engineering and project management has addressed project forecasting via multivariate schedule network models.One example of a detailed multivariate model that uses empirical data is Adler, Mandelbaum, Nguyen, and Schwerer, 18 which uses simulation to forecast project completion time based on a detailed task planning and execution model with parameters estimated from historical data.Browning 19 derives detailed multivariate relational models that were extended to project forecasting models by Browning and Eppinger 20 using a network of individual project activities that exchange deliverables.They use subjective expert opinion to provide inputs for schedule durations of the individual project activities.
Brady 21 developed dependency structure matrix analysis to understand the effects of each change and decision made early in a projects lifespan to predict the future stages and later operations.
Some researchers have noted certain shortfalls presented by using TRLs and have attempted to mitigate this by developing alternative assessment methods.Kasser 22 notes these deficiencies and proposes a holistic thinking approach that uses the technology availability window of opportunity (TAWOO) to estimate technology maturity.According to Kasser, while the rate of change of TRL levels have been used to predict the early stages of a technologies lifecycle, the TAWOO is meant to be a dynamic variable capable of spanning the entire lifecycle that would help people make better decisions regarding projects.
One approach to providing input to multivariate schedule network models is to estimate the project duration based on TM assessment for the individual project activities.Mankins 1 published the seminal paper on TRL scales.In a review of the history of these scales, Mankins indicates that the original TRL scales had either six or seven levels and were developed at NASA in the mid-1970s. 23For US Department of Defense (DOD) projects, the TM assessment method is derived from program risk assessment methods that originally used ordinal scales to assess the risk of completing software projects. 22 regression analysis of historical data to provide a basis for project forecasting provides a means to counteract biases obtained when using direct assessments by subject matter experts of times to completion and uncertainty ranges for project tasks.
Olechowski, Eppinger, and Joglekar 34(p.2085)indicate that it is common for technologists to inflate TRLs to encourage technology adoption to secure continued funding.They also state that, "for complex technologies it can be difficult for assessors to agree on precisely what level has been achieved.Assessors may have different expertise, experience, and preconceptions 34(p.2091)p.2091)" They support these claims with opinions from various industry professionals who utilize TRLs frequently."An engineering manager from John p.2091)" An executive at BP summed up this challenge well, saying "if the three of us were working on a project together, do you think we would assess all the equipment at the same TRL?

METHODOLOGY
This paper provides an analysis of the combination of historical data from NASA and DOE obtained from a variety of projects.The dataset from NASA was collected to provide an annual assessment of potential impacts of NASA technologies on national aerospace goals at the time. 16The data was collected through literature reviews and interviews with NASA and industry personnel, which produced TRL data for eighteen civil aeronautical products that included airframes, propulsion, flight systems, and ground systems.The original NASA-supported effort focused on collecting data and investigating the causes of variation in the length of time to achieve TRLs.
The DOE data was first analyzed by Kenley and Creque to predict a schedule for the project depending on the TM of its components.

Review of NASA's TRL scale
The NASA TRL scale is a metric that evaluates the technological maturity of specific technologies with respect to their being successfully incorporated into an operational system.The discrete scale ranges from 1 to 9, with 9 referring to the highest level of maturity.The scale is an ordinal scale, and as indicated by Conrow, 36 performing mathematical operations on the scale values can lead to erroneous results.The definitions for the TRLs and associated R&D activities shown in Table 1 are adapted from those used by Peisen, Schulz, Golaszewski, Ballard, and Smith. 16

Review of the DOE's TM scale and translation to TRL
The TM of a technology is a function of the maturity of its different aspects, ranging from its hardware equipment to facilities and safety.Table 2, obtained from Crépin, El-Khoury, and Kenley, 17 provides the scales that define the four parameters.
Crépin, El-Khoury, and Kenley 17 concluded that using only the PM and EQ scales was the preferred method for mapping the TM scales to TRL and provided an algorithm for the mapping the PM and EQ scale values for each of the projects into the TRL scale value.Table 3 shows a sparse correspondence table for the PM and EQ values to TRL values using this algorithm.It is to be noted that the highest TRL scale value did not exceed 7, because the DOE projects were developed as one-of-a-kind prototypes.The level of testing and quality assurance for the systems to be manufactured in large quantities was not required for these prototypes, and hence the TRL levels of 8 and 9 were not achieved.

Review of TRL data from NASA programs
The NASA-sponsored effort investigated how long it takes for technologies to go from an initial concept to marketable product and was based on NASA-defined TRLs shown in Table 1.Abbreviations: EQ, equipment maturity; PM, process maturity; TRL, Technology Readiness Level.
specific system and achieving a higher TRL is not meaningful, or because they were not yet integrated into a system at the time they were surveyed.
Table 5 shows the time in years that were needed to progress from one TRL to the next for the projects.Entries of zero are assumed to be those cases for which the experts who were surveyed indicated that the TRL bypassed a level of R&D activity on the way to achieving TRL 9.

Review of TRL data from DOE
After mapping TM scores into their corresponding TRL scores, we see the initial TRL values for each of the 25 projects in Table 6.The DOE project was focused on identifying technologies for converting and stabilizing the nuclear materials in safe storage.Using the initial TRL score and the time taken to become operational and attain a TRL value of 7, we can calculate the time to transition between the two levels.
Table 7 shows the transition time between the two stages for each project.

Review of combined TRL data from NASA and DOE
Table 8 shows the minimum and maximum times to progress from one TRL to the next for the combined NASA and DOE data.Note that all the technologies listed are NASA projects.The same technology was not always the fastest to complete a step, nor was the same technology always the slowest.In fact, Flow Visualization took the most time to progress from TRL 1 to 2; and the least time, from TRL 6 to 7.
Table 9 shows the minimum and maximum times to progress from TRL 1 to subsequent TRLs.Note that all of the technologies listed are NASA projects, because none of the DOE projects started at TRL 1.
The Surface Movement Adapter progressed most quickly from TRL 1 to TRL 6, 7, and 8, even though it was not fastest to transition for any individual TRL step.The 22-year transition from TRL 5 to 6 for the Tilt Rotor resulted in it having the slowest transition from TRL 1 to TRL 6 and higher, even though the Supercritical Wing had quite a long transition from TRL 7 to 8.

SCHEDULE-FORECASTING MODEL
The schedule-forecasting model is based on a log-Student's tdistribution that is fit using the data.The selection of a log-Student's t-distribution is based on a two-step thought process.First, fitting a probability distribution to the logarithm of the raw data ensures that only non-negative values of time are produced when calculating the inverse of the fitted distribution to generate forecasts.Second, fitting a Student's-t distribution provides "fat tails" for the fitted distribution to ensure that the extreme data points, for example, Direct To and Tilt Rotor, are weighed more in the curve fitting than would be the case if a Gaussian or other "thin-tailed" distribution that emphasizes the data points near the average.The logic is very similar to the choice of the log-Student's t-distribution to model stock returns in a way that does not under-represent the extreme data points. 37ere are 36 ordered pairs of transitions from TRL I to TRL j for i = 1,. . .,8 and j = i+1 to 9 and each pair has a data set of transition times.For each of the 36 data sets, the sample mean X, and the sample standard deviation S are calculated For our model, we assume that the statistic T = is distributed as Student's t-distribution with n-1 degrees of freedom.In section 5, we perform hypothesis tests for this assumption.Table 10 shows an example for fitting the probability distribution that is used for modeling the time to transition for two of ordered pairs, TRL 2 to TRL 7 and TRL 5 to TRL 7. First, the natural logarithm of the raw data for the time to transition is calculated.Note that there are only13 data points for the transition from TRL 2 to TRL 7; whereas there are 30 data points for TRL 5 to TRL 7.For TRL 2 to TRL 7, the degrees of freedom are 12; and for TRL 5 to TRL 7, 29.For example, starting with an initial TRL of 2 and ending with a final TRL of 7, the statistic is distributed as Student's t with 12 degrees of freedom.
Student's t is symmetric about zero, so the both the mean and the Using the t-distribution with the degrees of freedom for each corresponding transition, it is possible to identify a table of percentiles for the time to transition from the start TRL to the end TRL.Table 11 below shows a percentile table for years to achieve TRL 7 from all TRLs below it, with Figure 1 depicting it graphically as a probability distribution function.
Similarly, if time to transition from one TRL to another is a log-Student's t-distribution, the probability that the time to transition is greater than a given duration τ is Here τ refers to the mean time to transition between the two TRLs considered, SD is the standard deviation of that transition, n is the number of data points available for that transition, i is the initial TRL and j the final TRL.The h ij (x) function is the probability density function of the t-distribution between two TRL levels i and j.The probability density function h(x) of the t-distribution for a degree of freedom ν is given TA B L E 5 Raw data from NASA study of time to achieve Technology Readiness Levels.

Technology name Description
Initial TRL

Bagless transfer system-FB
A standardized design and procurement project to provide a packaging capability through a bagless transfer system.
Calciner with full batch TGA The calciner works to minimize the volatility of the reactive material, with the extent of calcination assessed using thermogravimetric analysis (TGA).

Cementation
Treatment to eliminate reactive species and moisture content in sand, slag, and crucible inventories, that have caused storage container failures.

Charcoal treatment
Treatment for processing charcoal containing absorbed 233 UF 6 , F 2 , and other compounds.

Digital radiography
Process to nondestructively gain information on metal, oxide, and residues in storage.

Electrolytic decontamination
Process for decontamination of the packaging using electropolish.

HB Phase II solution conversion
Phase II of the HB line is used to create plutonium and neptunium oxides from nitrate solutions.
Nitric acid soluble bags Technique to use acid-soluble bag materials for the stabilization of sand, slag, and crucibles.
Pipe component An additional layer of containment that could be used within a (DOT)−17C drum to optimize the use of the TRUPACT II shipping container.

Polycube Pyrolysis
Process involving the pyrolysis of polycubes to destroy the styrene matrix and stabilize plutonium.

Precipitation-Magnesium Hydroxide
Processing and stabilizing plutonium bearing solutions into forms that meet storage criteria through precipitation with hydroxide.
Precipitation-MgOH HAN Processing and stabilizing plutonium bearing solutions into forms that meet storage criteria through precipitation with hydroxide.

Pretreatment of RFETS SS&C-SRS
Process of removing interfering elements from plutonium-bearing solutions using chromatography.

Pretreatment of RFETS SS&C-RFETS
Process of removing interfering elements from plutonium-bearing solutions using chromatography.

Pu238 Storage Container-SRS/LANL
Container ensuring the safe storage of the Pu238 material in the plant through a surveillance program.

Pu239 Standard Container-RFETS
Procuring a container test lot for the standard storage of Pu239.
PuSPS-Packaging-RFETS An end-to-end automated system that will result in stabilized and packaged plutonium oxides and metals in a standardized package.
PuSPS-Packaging-LLNL An end-to-end automated system that will result in stabilized and packaged plutonium oxides and metals in a standardized package.

Pyrochemical salt oxidation
Process to eliminate reactive species, bound water, and most chemically bound water from salt residues.

Scrub alloy processing
Processing of scrub alloys using a standard dissolution/purification flow sheet.

SS&C Stabilization
Process to dissolve SS&C residues, converting Pu to metal and eliminating reactive species and moisture Thermal Stabilization-HAN Treatment to eliminate reactive species and moisture content in sand, slag and crucible inventories using thermal stabilization techniques.

Thermal Stabilization-RFETS
Treatment to eliminate reactive species and moisture content in sand, slag and crucible inventories using thermal stabilization techniques.

Trapping of Uranium Hexafluoride
A system for trapping uranium hexafluoride as a means to stabilize uranium and remove them from the off-gas system.

Vertical calciner
System to convert plutonium solutions into a stable, storable solid that minimizes personal exposure.

TA B L E 7
Raw data from DOE study of technologies for stabilization.Abbreviation: TRL, Technology Readiness Level.

Bagless
as: where Γ( .) is the gamma function given by Thus, the probability that the time to transition from TRL i to TRL j is lesser than τ is: To calculate the probability density function, we have: To calculate this derivative, we have the Leibniz Integral rule: The functions a() and b() mentioned in the Leibniz integral rule are the limits of the integral: Using the Leibniz rule, we can calculate the probability density function as: have pdf peaks greater than 1, integration over the domain leads to a cumulative probability of 1.
p.214)For this reason.the 50 th and 95 th percentile of the distributions are shown in Table 12 along with the degrees of freedom.
As seen in Figure 3, the probability density functions for transition-

EVALUATING GOODNESS OF FIT FOR MODEL
The Kolmogorov-Smirnov test is used to test if a sample comes from a population with a specific distribution.We evaluated the goodnessof-fit of our combined data from NASA and DOE to each estimated  Quindimil. 39In this test, the null and alternative hypothesis are given as: The data follows a t-distribution model H a : The data does not follow a t-distribution model If the null hypothesis cannot be rejected, we conclude there is insufficient evidence that the alternative hypothesis is true.Not rejecting the null hypothesis allows us to be confident in using log-Student's t-distribution to model the TRL transition times.
The means, standard deviations, and degrees of freedom for the logarithm of each transition time is collected to obtain the Z-values for Equations ( 13) and ( 14), which are the t-distribution values under our null hypothesis.These values are then arranged in ascending order and used to evaluate the Kolmogorov-Smirnov and Cramer-von-Mises distance values using the equations below: Kolmogorov-Smirnov distance: Cramer-von-Mises distance: Using these statistics, we created a bootstrapped simulation of 1000 samples under the null hypothesis.The t-distribution is generated for  The bootstrapped data was also fitted to the true t-distribution from each transition to graphically show how closely the data resembled a t-distribution, seen in Figure 5a,b for TRL transitions 2 to 7 and 5 to 7 respectively.

POTENTIAL EXTENSIONS TO THE MODEL
This section presents three potential extensions to the model that are based on product forecasting methods, semi-Markov processes, and Bayes nets.For each extension, it provides an overview of the theoretical framework, citations to key references in the literature, and a brief description as how the extension might be applied.

Extension 1: A concept for using the TRL-based method for technology forecasting to manage a product's life cycle
In their book on technology forecasting methods, Millett and Honton 40 define a model of the R&D process as follows: F I G U R E 5 (A) Fitting the bootstrapped sample for transition between TRL 2 and 7. (B) Fitting the bootstrapped sample for transition between TRL 5 and 7. TRL, Technology Readiness Level.
1. Early theorizing and conceptualization.This is the "light bulb" phase when someone comes up with an intuitive bright idea.The idea at this stage has little form and substance, but much enthusiasm.
2. Exploratory research.Will the idea work?What form will it take?At this stage the researchers need to achieve some critical parameters.Scientific methods, plus a lot of trial and error, are used to manifest the early idea, which is often modified in the process.Millett and Honton use this model to define the context for technology forecasting and product forecasting as follows 40 (p.5) : Technology forecasting and strategy analysis methods can be employed at virtually any one of these stages, but they may have different applications.In the first three stages, the forecast is most likely to conform to the strict definition of technology forecasting.At the later stages, forecasts and analyses more likely take the shape of product, market, and economic forecasts.By Stage 7 we are strictly faced with product forecasts rather than technology forecasts.The manager needs to appreciate the stage of technological innovation in which the forecast is conducted.The analyst also needs to understand this context for the forecast and appreciate the decision needs of the manager who will be using the forecast.
The Bass model 41 and its successors provide a forecast of market share against time, which is the measure of operational success for a product, and are the principal models used for technology forecasting for stage 6 and beyond.For technology forecasting up to stage 5, the TRL-based schedule forecasting method described in this paper provides a data-driven, quantitative method for technology forecasting.
The decisions that are made during these early and middle stages of development are often relegated to the realm of tactical decision making, when they are in fact the vital link between the strategic decisions to screen technologies for inclusion in the R&D portfolio at stages 1 and 2 and to bring a product to market at stage 6.The TRL-based model enables us to predict when the Bass curve starts in the life cycle of the product as shown in the conceptual life cycle model in Figure 6.The method and results presented in this paper are important to understanding the regime of component and prototype development that is necessary to progress from research to production and is useful for both the strategic and tactical aspects of new product development.

Extension 2: Using semi-Markov modeling methods
The modeling approach used in this paper is essentially a semi-Markov model, 42 and it is possible to extend the modeling approach to take advantage of all of the features available using semi-Markov modeling methods.Following Howard, 42(p.16)whenever a process achieves TRL state i, we imagine that it determines the next TRL state j to which it will move according to TRL state i's transition probabilities p ij for j = 1,. . ., 9.   For all values of j < i, p ij = 0, which means that we are assuming that the TRL does not decrease over time.We can allow p ij = 0 when i = j, which means we are assuming that there are circumstances for which a technology cannot advance beyond TRL i.After the next TRL state j has  the holding time mass function by applying our schedule-forecasting model estimation approach respectively to the data collected for the transition times from TRL 6 to 7 and for the transition times for transitions from TRL 6 to 8 (without an intermediate transition to TRL 4). Figure 8b shows numerical values for the labels on the arcs using our schedule-forecasting model results based on NASA and DOE.The estimates for the transition probabilities are p 67 = 0.9, p 68 = 0.1, and p 78 = 0.9167.The estimated probability p 78 is not 1, because the data shows a direct transition from TRL 7 to 9, which we have not depicted in Figure 8b.
Once the semi-Markov process is constructed with the available data, it can be used to calculate the conditional probability p ij (m) of a transition from TRL i to TRL j given the process has been at TRL i for a given amount of time m (Howard, 2007, p. 28): There are many other predictive statistics described by Howard that can be calculated using the semi-Markov process such as For our forecasting model, we modeled conditional probability of the time to transition using a log-Student's t-distribution.To model the conditional distribution, we would have to use a multivariate log-Student's t-distribution 38 with different degrees of freedom along each dimension, 44 and factor the distribution to derive the conditional distribution of  23 given  12 .The algorithm for factoring the multivariate log-Student's t-distribution would be analogous to the algorithm to convert a covariance matrix to a Gaussian influence diagram. 45 validate thar using a Bayes net would be an enhancement, we examined a selection of the correlations between the length of previous TRL transitions to future TRL transitions for transitions beginning at TRL 1 and ending at TRL 9, as shown in Table 15.Some of the correlations are between 0 and 0.3, indicating a weak positive linear relationship 46 ; some are between 0. dom along each dimension by calculating a covariance matrix between all of the allowable transition times.
To estimate the parameters for this circumstance of when TRL state i to TRL states other than i + 1, we would have to collect data on these transitions and develop an approach that is analogous solving least squares problems with equality constraints consistent with deterministic equations associated with nodes such as  13 in Figure 9.
Once the Bayes net is constructed with the available data, it can be used to provide adaptive predictions as to what is achievable during the remainder of a product development cycle when the actual time required to achieve various TRL levels are either ahead of or behind the historical average.These predictions would use methods similar to those developed by Shachter and Kenley, 45 which follow the general algorithms for probabilistic inference for influence diagrams. 47

DISCUSSION
This section presents a summary of the findings and describes how the

Conclusions
The

Limitations
The data only has technologies which successfully transitioned to high levels of technology maturity and there are no technologies which failed to make these transitions.This may lead one to assume that technologies will not be abandoned in early stages of development and will always transition to higher levels of technology maturity, Also, the missing data for abandoned technologies may cause bias in estimating the transition times for earlier stages.This limitation could be addressed in future efforts by collecting data on abandoned technologies, which is feasible for DOD projects now that reporting of TRL is required for DoD-funded projects.
Other factors such as level of effort and funding during various phases was not available for analysis.There are some technologies in the data set that did not transition from one TRL to the next for nearly 5 decades, which is an indication that they may not have had sufficient effort and funding during these periods.Collecting information on effort expended and funding would allow for an expanded statistical micro-level model that accounts for these and other additional relevant factors.The modeling approach in this article is a macrolevel model, that is, the factors not considered are assumed to be randomly distributed across all of the data.One justification for using macro-level modeling for forecasting for a specific project is that the most influential factors may be political, economic, social, technological, environmental, and legal forces that are dynamic and uncertain and that project management is neither able to control nor predict with certainty, It is possible that improvements in engineering tools and processes have improved substantially since the data used for this article was collected.which would suggest that it is possible to mature technologies more rapidly than when the data for this article was collected.
Also, the combined NASA and DOE data presented in this article may not be representative of industrial settings where R&D is not funded by the government.Under these circumstances the log-Student's tdistribution model can still be used with data sets collected from the relevant R&D environment to estimate the model parameters, and the goodness-of-fit can be evaluated using the method described in this article.If the goodness-of-fit null hypothesis that the data follows the estimated probability distribution cannot be rejected, the model with the updated estimates along with the extensions proposed in this article can be used for forecasting and risk management.
Two statistical approaches that could be used instead of a Bayes Net for project forecasting with TRL data are time series modeling and structural equation modeling.These approaches assume that the underlying uncertainty follows a multivariate normal distribution.Under this assumption, it has been shown that both time series modeling 48,49 and structural equation modeling 50 can be represented as graphical Bayes Net.Using a multivariate log-Student's t-distribution with different degrees of freedom along each dimension as proposed in this article allows higher probabilities for the TRL transition times in the tails of the distribution compared with models based on the multivariate normal distribution.In addition, using a Bayes Net can provide adaptive predictions via Bayesian updating once the actual times required to achieve various TRL levels are realized.
developed an algorithm to transform the TM scores from the DOE nuclear materials data into TRL scores and performed a statistical analysis that concluded transition times for developing aerospace and nuclear technologies are very similar.This article presents new results for statistical modeling by combining the NASA and DOE data to estimate the parameters of the log-Student's t-distribution.It also presents hypothesis testing of the goodness-of-fit of the statistical model to the observed data and potential extensions to the model using product forecasting methods, semi-Markov processes, and Bayes nets.Using the parameters estimated from the combined NASA and DOE data, the log-Student's t-distribution model and proposed extensions can be used to forecast the schedule for projects, resulting in better allocation of funds and resources.They also can be used to manage schedule risk and overruns in terms of schedule.Section 2 of this article provides a review of the literature on project forecasting methods.Section 3 describes the methodology used to develop the model presented in this article.It reviews the NASA TRL scale, the DOE TM scale and translating the DOE scale to TRL, the NASA TRL data used for modeling, the DOE TRL data used for modeling, and the combined TRL data from NASA and DOE used for modeling.Section 4 presents the schedule-forecasting model based on the log-Student's t-distribution.Section 5 evaluates the goodness of fit of the log-Student's t-distribution model to the data.Section 6 presents the three potential extensions to the model.Section 7 discusses the results and potential extensions, comparing them with other approaches to project forecasting and statistical modeling.It also describes possible limitations of the approach presented in this article.
Under the Strategic Defense Initiative, GE Aerospace developed hardware and software ordinal scales to generate numerical TM scale values from 0 to 10 that were used to calculate a probability of failure to meet program objectives.24Among the multiple assessment scales developed by GE Aerospace, one scale called the State of Technology is almost identical to the TRL scale.The GE Aerospace method uses a closed-form expression score that is a weighted average of input scores from multiple assessment scales.Use of such scales was encouraged by DOD25 and the US Air Force26 to determine ordinal (Low, Medium, or High) evaluation of failure probabilities and consequences from which are derived relative risk levels (Low, Medium, or High).Subsequently, the DOD published its own TRL scales that have been promulgated throughout the defense-funded technology development community and were mandated by policy in the year 2002.27In other research, TM assessment scales have not been directly used to calculate probabilities but are used as inputs to methods that quantify risk.Dechoretz and Palmer 28 use analytic hierarchy methods29 to convert the ordinal values to an uncertainty rating.Bellagamba30 uses fuzzy logic as an alternative to probability calculations to develop risk ratings.The correlation of TM scales to project costs and schedules also has been investigated.Hoy and Hudak 31 use regression analysis to correlate these scales to historical cost overruns on DOD programs.Dubos and Saleh 32 use linear regression of TRL levels to predict future schedule slippage and risk.Kenley and Creque 3 introduced the use of regression analysis to analyze the correlation of these scales to actual project time to completion for US DOE projects.When regression analysis is used, it is straightforward to obtain estimates of the cost overruns and the time to complete R&D projects along with confidence intervals for the estimates.Ma 33 estimated cost and schedule by identifying causal variables using the Technology Cost and Schedule Estimating (TCASE) tool as another form of linear regression.Using scores were both used to approximate the probability of failure, that is, a TM of 0 indicates that the probability of failure of 0/10 = 0 when deploying a technology that is already successfully in use and a probability of failure of 10/10 = 1 when there is no available technology to meet the operational requirements or the requirements are undefined.

12 )Figure 2
Figure 2 below shows the probability density function for each transition to TRL 7. The figure shows that for the lower TRL levels the density increases slowly towards the peak around the mean time to transition, and then gradually decreases again.As the initial TRL level ing from TRL 5 to 6 for each data set are unimodal with each having a slight right skew.DOE data has the widest range, with a 50 th percentile of 1.26 years and a 95 th percentile of 2.47 years.NASA data is skewed the most with a 50 th percentile of 1.00 years and a 95 th percentile of 1.94 years.Finally, the combined NASA + DOE data is in the middle with a 50 th percentile of 1.03 years and the 95 th percentile of 1.48 years.For the transition between TRLs 6 and 7, as seen in Figure 4, the probability density functions for each data set are also unimodal with each having a very slight right skew.The NASA data has a very wide range, with a 50 th percentile of 1.70 years and a 95 th percentile of 2.82 years.The DOE data has a 50 th percentile of 1.60 years and a 95 th percentile of 2.214 years.Finally, the combined NASA + DOE data is the least skewed of the three, with a 50 th percentile of 1.66 years and a 95 th percentile of 2.212 years, which is almost exactly the same as the 95 th percentile for the DOE data set.

F I G U R E 3
Probability density functions to transition from TRL 5 to 6 for Varied Data Sets.TRL, Technology Readiness Level.t-distribution model using a KS Test with simulation-based bootstrap procedure based on the steps explained in Stute, Manteiga, and

2 *. 2 *
these samples and the Kolmogorov-Smirnov and Cramer-von-Mises distances are calculated as D * n and W n If the values of D * n and W n are greater than the values of D n and W 2 n , we would fail to reject the null hypothesis.The values of D n , W 2 n , D * n , W n 2 * , and the results of the Kolmogorov-Smirnov test for all transitions to TRL 7 are tabulated below in Table 13.Transitions 3-7 and 4-7 have the same number of samples, leading to the same W 2 n value.F I G U R E 4 Probability density functions to transition from TRL 6 to 7 for varied data sets.TRL, Technology Readiness Level.TA B L E 1 3 Goodness-of-Fit tests for the t-distribution model.

3 .
Component development.If the second stage produces promising results, the next stage is to develop and fabricate the parts that will make the emerging technology work.4. Prototype development.If the third stage succeeds in its goals, the components will be assembled into the prototype product. 5. Testing of prototype.This may involve both physical testing of the assembled technology and market testing for possible consumer demand.6.Initial manufacturing and marketing of the product.7. Consumer acceptance or rejection.

8 .
Product modification and improvements.9. Product maturity and decline.

F I G U R E 6
The life cycle for product marketing.FI G U R E 7 (A) Semi-Markov process for transitioning from TRL 6 to 7 and TRL 7 to 8. (B) Semi-Markov process for transitioning from TRL 6 to 7 and TRL 7 to 8 with estimated parameters.TRL, Technology Readiness Level.been selected, but before making this transition from state i to state j, the process "holds" for a time τ ij in state i.The holding times τ ij are random variables each governed by a probability mass function h ij (⋅) called the holding time mass function for a transition from TRL state i to TRL state j.A depiction of the semi-Markov process for transitioning TRL 6 to 7 and from TRL 7 to 8 is shown in Figure7a.Note that there is no transition arc drawn from TRL 6 to TRL 8, which means that p 68 = 0, that is, a direct transition from TRL 6 to 8 is not possible.The labeling on the arcs in Figure7aare ordered pairs of the transition probability and the holding time mass function.Figure 7b shows numerical values for the labels on the arcs using our schedule-forecasting model results based on NASA and DOE.The state transition probability is 1, that is, our schedule forecasting model assumes that the only allow-F I G U R E 8 (A) Semi-Markov process where a single-step transition from TRL 6 to 8 is allowed.(B) Semi-Markov process where a single-step transition from TRL 6 to 8 is allowed with parameters estimated.TRL, Technology Readiness Level.able transitions are from TRL state i to TRL state i + 1, and the holding time mass function is estimated based on the probability density from Equation (12).

Figure
Figure8adepicts of the semi-Markov process where a transition from TRL 6 to 8 that does not pass through TRL 7 is allowed.To estimate the parameters for this circumstance, we would have to collect data on the instances for transitions from TRL 6 to 7 and TRL 6 to 8 (without an intermediate transition toTRL 7).We would estimate the state transition probabilities for p 67 , p 68 , and p 78 based on the proportion of transitions of each type that were observed and estimate

Figure 9
Figure 9 show a Bayes net for TRL transition.The net shown is abbreviated and could be extended to include all possible TRL transitions.The nodes that are shown as single circles are probabilistic and the one shown as double circles are deterministic.For probabilistic nodes, the incoming arcs for a node determine the conditional probability that is represented by the node.For example, the node  12 represents the unconditional probability of the time to transition from TRL 1 to TRL 2, which we modeled based on a log-Student's t-distribution to develop a forecasting model.The node  23 represents the conditional probability of the time to transition from TRL 2 to TRL 3, given that the time to transition from TRL 1 to TRL 2 is known.For deterministic nodes, the incoming arcs for a node represent the independent variables for the deterministic equation that is represented by the node.For example, node  13 represents the equation log-Student's t-distribution model and the proposed extensions can be employed to provide information in support of R&D management.It also describes limitations with regard to the data sets and how the log-Student's t-distribution model could be applied to other settings with different data sets.Finally, several useful statistical approaches beyond the scope of this article are identified as possible candidates that could overcome any limitations.
project forecasting model developed in this paper for analyzing the timing of operational availability of new products are based upon straightforward assessment of the TRL of component technologies from NASA and the DOE.Parameter estimates derived from statistical analysis when used in conjunction with the model provide forecasts of the time required to make a product operationally available, and indicate which areas have the largest impact on the product development timeline.The goodness-of-fit testing indicates that the modeling approach is valid.Extensions to the model using product forecasting methods, semi-Markov processes, and Gaussian processes can be employed to extend the forecasting to the full life cycle of a product, predict the impact of skipping a readiness level during product development, and provide adaptive predictions as to what is achievable during the remainder of a product development cycle when the time required lower TRL levels are either ahead of or behind the historical average.

Hardware equipment maturity (EQ) Facility readiness (FAC) Operational safety readiness (SAFT) Process maturity (PM)
Scales of the four parameters used to measure technical maturity for Department of Energy projects.

Table 4
shows the 18 technologies surveyed and a description.Some of the technologies only achieved a TRL of 6, either because they were component technologies not intended to be integrated into a TA B L E 3 Correspondence table for process maturity and equipment maturity values into Technology Readiness Level values.
Summary of technologies surveyed by NASA.
50 th percentile of this statistic is zero.Thus, if the time to transition from TRL 2 to 7 is modeled as a log-Student's t-distribution, the TA B L E 4 Minimum and maximum transition times for individual Technology Readiness Level steps.Minimum and maximum transition times starting from Technology Readiness Level 1. Example of data used to fit Log-Student's t-distribution.
Abbreviation: TRL, Technology Readiness Level.F I G U R E 1 Cumulative probability distribution functions for time to transition to TRL 7. TRL, Technology Readiness Level.TA B L E 1 0 Time to transition to achieve Technology Readiness Level 7.
F I G U R E 2Probability density function for transition to TRL 7. TRL, Technology Readiness Level.

to 7 Data set 50 th Percentile 95 th Percentile Degrees of Freedom 50 th Percentile 95 th Percentile Degrees of Freedom
Statistics for Technology Readiness Level transitions for various data sets.
TA B L E 1 2Abbreviations: DOE, Department of Energy; NASA, National Aeronautics and Space Administration; TRL, Technology Readiness Level.
Testing goodness-of-fit using p-values.
have proof to reject the null hypothesis.Table14 hasp-values from each transition and the critical values at α = 0.05.Due to the same number of samples, transitions 3-7 and 4-7 have the same values.
If the only allowable transitions are from TRL state i to TRL state i + 1, it is reasonably straightforward to estimate the parameters for a multivariate log-Student's t-distribution with different degrees of free- 3 and 0.7, indicating a moderate positive linear relationship; and none are between 0.7 and 1.0, indicating a strong positive linear relationship.The presence of moderate correlations indicate that a Bayes net would enhanceforecasting.Also, it should be noted that all of the correlations were positive, that is, longer times for the first transition indicate that transition times for the second transition are likely to be longer as well.