Validation of a new in vitro Sun Protection Factor method to include a wide range of sunscreen product emulsion types

Abstract In 2017, Cosmetics Europe performed a double‐blinded ring test of 24 emulsion‐type sunscreen products, across 3 in vivo test laboratories and 3 in vitro test laboratories, using a new candidate in vitro SPF test method. Based on the results of this work, an article was published showing how data derived from a new lead candidate method conform to new International Standards (ISO) acceptance criteria for alternative SPF test methods (Any alternative method should consider the matrix effect and if required, specify the matrix applicability of the method; Criterion 1a: Systematic differences between methods should be negligible: 95% of all individual results of an alternative method are within the range of ±2× reproducibility standard deviation of the in vivo method, that is overall bias must be below 0.5× reproducibility standard deviation of the in vivo method; Criterion 1b: Measurement uncertainty of an alternative method should be below the measurement uncertainty of the in vivo method. Candidate method predicted values must fall within the full ‘funnel’ (SPF 6‐50+) limits proposed by Cosmetics Europe (derived from the same minimum test design, that is using the ISO24444 Method to measure at least 24 products across at least 3 laboratories using at least 5 test subjects/laboratory, in a blinded fashion).). Of the 24 sunscreen products tested, the majority of emulsions were of the oil‐in‐water (O/W) type, whereas only one was water‐in‐oil (W/O) and there were no products with a mineral‐only sun filter system. In order to confirm the scope of this method, therefore, a new study was conducted that included 73 W/O (12 mineral + organic, 44 mineral only and 17 organic only) and 3 O/W mineral‐only, emulsion‐type sunscreen products (a total of 76 new sunscreen products). When combined with the previous 24 products (tested in 3 different laboratories), this yielded a new data set comprising a total of 100 emulsion‐type sunscreen products, with SPF values ranging from 6 to 50+ (with a total of 148 data points). These products were tested using the double‐plate in vitro SPF test method and compared with the ISO TC217/WG7 acceptance criteria for alternative SPF test methods. Over 95% of paired in vitro: in vivo SPF values lay within the upper and lower limits of the ISO acceptance criteria funnel, with no bias. This new in vitro SPF test method, therefore, meets the minimum requirements for an alternative SPF test method to ISO24444:2010, for emulsion‐type sunscreen products (which make up the majority of marketed sunscreen products).


Abstract
In 2017, Cosmetics Europe performed a double-blinded ring test of 24 emulsion-type sunscreen products, across 3 in vivo test laboratories and 3 in vitro test laboratories, using a new candidate in vitro SPF test method. Based on the results of this work, an article was published showing how data derived from a new lead candidate method conform to new International Standards (ISO) acceptance criteria for alternative SPF test methods (Any alternative method should consider the matrix effect and if required, specify the matrix applicability of the method; Criterion 1a: Systematic differences between methods should be negligible: 95% of all individual results of an alternative method are within the range of AE29 reproducibility standard deviation of the in vivo method, that is overall bias must be below 0.59 reproducibility standard deviation of the in vivo method; Criterion 1b: Measurement uncertainty of an alternative method should be below the measurement uncertainty of the in vivo method. Candidate method predicted values must fall within the full 'funnel' (SPF 6-50+) limits proposed by Cosmetics Europe (derived from the same minimum test design, that is using the ISO24444 Method to measure at least 24 products across at least 3 laboratories using at least 5 test subjects/laboratory, in a blinded fashion).). Of the 24 sunscreen products tested, the majority of emulsions were of the oilin-water (O/W) type, whereas only one was water-in-oil (W/O) and there were no products with a mineral-only sun filter system. In order to confirm the scope of this method, therefore, a new study was conducted that included 73 W/O (12 mineral + organic, 44 mineral only and 17 organic only) and 3 O/W mineral-only, emulsion-type sunscreen products (a total of 76 new sunscreen products). When combined with the previous 24 products (tested in 3 different laboratories), this yielded a new data set comprising a total of 100 emulsion-type sunscreen products, with SPF values ranging from 6 to 50+ (with a total of 148 data points). These products were tested using the double-plate in vitro SPF test method and compared with the ISO TC217/WG7 acceptance criteria for alternative SPF test methods. Over 95% of paired in vitro: in vivo SPF values lay within the upper and lower limits of the ISO acceptance criteria funnel, with no bias. This new in vitro SPF test method, therefore, meets the minimum requirements for an alternative SPF test method to ISO24444:2010, for emulsion-type sunscreen products (which make up the majority of marketed sunscreen products).
R esum e En 2017, Cosmetics Europe a r ealis e un ring test en double aveugle de 24 produits de protection solaire de type emulsion, dans 3 laboratoires de test in vivo et 3 laboratoires de test in vitro, en utilisant une nouvelle m ethode de test SPF in vitro. Sur la base des r esultats de ces travaux, un article a et e publi e montrant comment les donn ees d eriv ees de cette nouvelle m ethode sont conformes aux nouveaux crit eres d'acceptation des normes internationales (ISO) pour les m ethodes de test SPF alternatives. Sur les 24 produits de protection solaire test es, la majorit e des emulsions etaient du type huile dans l'eau (H / E), tandis qu'un seul etait de l'eau dans l'huile (E / H) et il n'y avait aucun produit contenant uniquement des min eraux. Afin de confirmer cette m ethode, une nouvelle etude a donc et e men ee comprenant 73 produits E/ H (12 produits contenant des filtres min eraux + organiques, 44 produits contenant des filtres min eraux uniquement et 17 produits contenant des filtres organiques uniquement) et 3 produits H / E contenant des filtres min eraux uniquement, tous de type emulsion (donc un un total de 76 nouveaux produits de protection solaire). Combin e aux 24 produits pr ec edents (test es dans 3 laboratoires diff erents), cela a donn e un nouvel ensemble de donn ees comprenant un total de 100 produits de protection solaire de type emulsion, avec des valeurs SPF allant de 6 a 50+ (avec un total de 148 points de donn ees) . Ces produits ont et e test es a l'aide de la m ethode de test SPF in vitro double approche et compar es aux crit eres d'acceptation de l'ISO TC217 / WG7 pour les m ethodes alternatives du SPF in vivo. Plus de 95% des valeurs de SPF appari ees in vitro: in vivo se situent dans les limites sup erieure et inf erieure de l'entonnoir des crit eres d'acceptation ISO, sans biais. Cette nouvelle  (Table 1).
Of the 24 sunscreen products tested, the majority of emulsions were of the oil-in-water type (O/W), whereas only one was waterin-oil (W/O) and there were no products with a mineral-only sun filter system (comprising varying ratios of micronized zinc and titanium dioxide).
In order to confirm the scope of this method, therefore, a new study was conducted that included 73 W/O (12 mineral + organic, 44 mineral only and 17 organic only) and 3 O/W mineral-only, emulsion-type sunscreen products (a total of 76 new sunscreen products), which were tested according to the protocol published previously [2].

Sunscreen products
In addition to the 24 commercial primary, emulsion-type, sunscreen products already used for the initial validation, 73 W/ O (12 mineral + organic, 44 mineral only and 17 organic only) and 3 O/W mineral-only emulsion sunscreens were chosen to represent the entire range of SPF categories defined by European Commission Recommendation 2006/647/EC [3] (namely 6, 10, 15, 20, 25, 30, 50 and 50+; see Table 1 for details).
It should be noted that in vivo data from the 76 new sunscreen products were not used to adjust the ISO acceptance criteria 'funnel'.

In vivo SPF test method
The 76 new sunscreen samples were tested on a minimum of 5 subjects using the current ISO24444:2010 In Vivo SPF test protocol [4], using a variety of test laboratories (according to the parent Company's choice; the laboratories were previously audited by the Company supplying the test products).

In vitro SPF test method
The method used in this study was exactly the same as described previously [5][6]. The protocol is summarized below: (1) Preparation of reagents and materials (2) Product application on substrates and robot automatic spreading (3) Measurement of initial absorbance using two plate types (290 nm to 400 nm).

Results and discussion
In the previous article, where we described the results of in vitro and in vivo testing of 24 products in 3 separate test laboratories [2], we showed that only 3 data points from the in vitro/in vivo relationship (out of a total of 72; 4.2%) lay outside the ISO acceptance criteria funnel, with no significant bias (see Fig. 1). The 95% confidence intervals of the slope of the in vitro/ in vivo relationship (0.85-1.17) included the expected value (that is, a perfect slope = 1.0), with a non-significant intercept (À1.48; P = 0.62). Although these data met the requirements for the ISO Acceptance Criteria (95.8% of data points within the upper and lower limits of the funnel), this data set did not account for the full range of emulsion-type sunscreen products in the marketplace (as they included a majority of O/W products and only one W/O product). When the data from testing the 73 W/O (12 mineral + organic, 44 mineral only, 17 organic only) and 3 O/W mineral-only products were added to this plot (see Fig. 2), 7 data points (out of a new total of 148; 4.7%) lay outside the upper/lower limits of the acceptance criteria funnel.
A matched-pairs analysis was performed on the 148 pairs of data (In Vivo vs. In Vitro) which showed no significant bias ( Fig. 3; difference average equal to À 0.80, 95% CI À2.44 to 0.84, Student's test P-value = 0.34 and Wilcoxon's test Pvalue = 0.39).
In summary, therefore, the combined data sets from the previous study and this new study show that, across a full range of sunscreen product emulsion types (O/W and W/O; 100 products in total, covering the full range of SPF values marketed within Europe (6-50+), the In Vitro SPF Test Method still meets ISO Acceptance Criteria for alternative SPF test methods to the in vivo reference ISO24444:2010 SPF method.
Although we strongly believe that the ISO Acceptance Criteria funnel represents a robust model for testing the validity of alternative SPF test methods, we acknowledge that there are some who may prefer other means of demonstrating equivalence, such as a 'Bland-Altman' plot [7] (a difference plot sometimes used in the fields of analytical chemistry or biomedicine to analyse the degree of agreement between two different assays). The resulting graph is an XY scatter plot, where the y-axis represents the difference between two paired measurements (A-B) and the x-axis represents the average of these measures ((A + B)/2). In other words, the difference of the two paired measurements is plotted against the mean of the two measurements. Bland and Altman [8] recommended that 95% of the data points should lie within AE 2 SD of the mean difference. For completeness, therefore, we used the new data set to construct a Bland-Altman plot (see Fig. 4).
As 96.6% of the data points are contained with the upper/lower limits of the plot, the new In Vitro SPF test method meets of the success criteria for this method also.
Discussing these results further, it is interesting to observe that, when unrealistically high/low data points are added to the data set ( Fig. 5), the 'funnel' model rejects an hypothesis of agreement between the two methods (as 11 data points from the in vitro/ in vivo relationship, 7.4% of a total of 148, lay outside the upper/ lower limits).
In contrast, when these values are added to the data set and analysed using the Bland-Altman approach, only 4 data points now lie outside the upper/lower limits. As this equates to 2.7% of the data set, the Bland-Altman method (wrongly) accepts an hypothesis of agreement (Fig. 6). This is because the introduction of these new unexpected high/low values drives a significant increase in standard deviation and, thus, a change in the upper/ lower limits of the model.
These new observations, combined with questions raised by other researchers [9][10], lead us to believe that, whereas the Bland-Altman method is suitable for comparing homoscedastic methods, it does not provide additional useful information for methods exhibiting heteroscedastic behaviour and, in some cases, may lead to erroneous conclusions.

Conclusion
When a total of 100 emulsion-type sunscreen products (spanning SPF6 to 50+, comprising W/O, O/W and products with a mineralonly sun filter system) were tested using the new double-plate in vitro SPF test method, over 95% of paired in vitro: in vivo SPF values lay within the upper and lower limits of the ISO acceptance criteria funnel, with no bias. This new in vitro SPF test method, therefore, meets the minimum requirements for an acceptable alternative SPF test method (to the current in vivo reference method, ISO24444:2010) for emulsiontype sunscreen products (which make up the majority of marketed sunscreen products).