Interobserver variability for retreatment indications after Ranibizumab loading doses in neovascular age-related macular degeneration

Abstract.

Purpose:  To assess the interobserver variability (IOV) in indicating retreatment for neovascular Age-related macular degeneration 4 weeks after three Ranibizumab loading doses using spectral domain OCT (SD-OCT) as the primary objective diagnostic tool.

Material and methods:  Four observers decided for or against 4th Ranibizumab injection in 108 patients by six different rating rounds (RR) based on the SD-OCT findings after the loading doses. Postoperative OCT images were supplemented consecutively with information from a chart review as the ‘patients subjective estimation of vision (SE)’, the course of best-corrected visual acuity (BCVA) and the preoperative OCT as well as all information collectively. Agreement rates (AR) and Kappa statistics were calculated.

Results:  Based on post-treatment OCT findings only (RR1), mean reinjection rate of all observers was 37.5%. Adding supplementary information, mean reinjection rate decreased to 20% when all information was available reflecting the ‘real’ situation (RR 6). Interobserver agreement rates varied from 66.7% to 90.7% depending on rating rounds and interobserver pairs. Mean AR and Kappa values (KV) were as following: AR 81.6%, KV 0.61 (RR1: ‘only post-OP OCT’); AR 76.7%, KV 0.33 (RR2: post-OP OCT + SE); AR 80.3%, KV 0.45 (RR3: post-OP OCT + BCVA); AR 80.7%, KV 0.46 (RR4: pre- and post-OP OCT); AR 82.2%, KV 0.49 (RR5: post-OP OCT + SE + BCVA); and finally AR 83.6%, KV 0.47 (RR6: pre- and post-OP OCT + SE + BCVA). The overall mean agreement rate was 80.9% with a Kappa of 0.47.

Conclusion:  IOV for indicating retreatment after three Ranibizumab loading doses reveals only moderate agreement in Kappa statistics, which seems to be too low considering the high costs for retreatments. More concise guidelines based on the post-treatment OCT scans as the presumably most sensitive and noninvasive objective tool to follow choroidal neovascularization activity by judging the course of sub- and intraretinal fluid are necessary.

Introduction

Age-related macular degeneration (AMD) and especially its neovascular form characterized by an abnormal growth of choroidal neovascularizations (CNVs) into the subretinal space is still the leading cause of blindness in the developed countries in people ageing over 50 years (Bressler 2004; Eye Disease Prevalence Research Group 2004).

Since the introduction of Ranibizumab (Lucentis®, Novartis, Basel, Switzerland), it is possible for the first time to treat neovascular AMD achieving gain of visual acuity. Ranibizumab is a recombinant monoclonal antibody fragment neutralizing all active forms of the vascular endothelial growth factor (VEGF) A and has been shown in large clinical trials to improve the course of neovascular AMD significantly (Brown et al. 2006; Regillo et al. 2008; Rosenfeld et al. 2006). Ranibizumab is FDA approved for all subtypes of CNVs because of AMD and in Germany currently injected intravitreally thrice during a period of 2 months (loading doses). After this phase, patients are usually re-examined four to 6 weeks later and, depending on the clinical findings, further treatment might be administered (Stellungnahme RG-DOG-BVA 2009).

As the logistics to examine and sufficiently treat all patients with AMD are extremely enhanced because of the new treatment modality using intravitreal Ranibizumab, the focus is nowadays laid on noninvasive and fast diagnostic tools to determine the course of treated AMD objectively and to decide properly whether retreatment is necessary or not. For this, the technique of optical coherence tomography (OCT) is – beside invasive fluorescein angiography – widely used, and the centre of attention in clinical studies is mainly laid on the regression of retinal oedema after therapy as objectively judgeable by OCT (Golbaz et al. 2009; Keane et al. 2008; Patel et al. 2009; Rothenbuehler et al. 2009). With the advent of the high-resolution spectral domain optical coherence tomography technique spectral domain OCT (SD-OCT), a much-improved differentiation of the retinal structures and the retinal pigment epithelium and also of the intraretinal or subretinal fluid in AMD became possible (Kiss et al. 2009; Sayanagi et al. 2009). Using SD-OCT in the course of treated neovascular AMD potential, oedema regression can be determined in an enhanced noninvasive and fast manner. However, even with the SD-OCT technique – as the supposedly most sensitive indicator for the treatment success after the Ranibizumab loading doses – the indication for retreatment is certainly always individual but might be heterogeneous between different examiners, even if the course of visual acuity or the subjective impression of the patient is considered. Herein, also the guidelines of Ophthalmic Societies seem to be imprecise in determining proper retreatment rules (Stellungnahme RG-DOG-BVA 2009). However, in view of the relatively high costs of this treatment modality, clear strategy rules for retreatment seem to be necessary. Thus, the aim of this study was to evaluate the interobserver variability in determining retreatment with Ranibizumab after the loading doses in neovascular AMD using SD-OCT.

Material and Methods

SD-OCT

Fundus and OCT images were acquired with a combined SD-OCT and scanning laser ophthalmoscope (SLO) imaging system (Spectralis HRA+OCT; Heidelberg Engineering, Heidelberg, Germany). The system is able to acquire en face SLO images in angiographic, AF and reflectance imaging modes as well as cross-sectional SD-OCT images. An 870-nm super luminescent diode is used for OCT imaging. In SD-OCT mode, the retina is scanned at 40 000 A-scans per second presenting highly detailed images of the retinal structure. The OCT depth resolution (FWHM) is 7 μm. The images of the SLO and OCT modes can be overlaid and automatically spatially co-registered. Images can also be co-registered over time for different visits of a patient. This set-up is usually used in our routine clinical examination.

Image acquisition

In a routine clinical setting, all consecutive patients, who were first time diagnosed for neovascular AMD (with no prior retinal treatment) and assigned to standard intravitreal Ranibizumab therapy (Stellungnahme RG-DOG-BVA 2009), underwent preoperative and postoperative SD-OCT examinations. The standard intravitreal Ranibizumab therapy consisted of a fixed monthly dosing (three times) followed by a pro-re-nata phase with variable monthly dosing if CNV activity is suggested as derived from the clinical findings. The postoperative examination took place usually 4 to 6 weeks after third injection of Ranibizumab. OCTs were performed with the cross-sectional technique using six different sections; same sections were performed during the follow-up to ensure matching sections for evaluation. Images were displayed and printed out using the Heidelberg Eye Explorer software (Heidelberg Engineering, Heidelberg, Germany) and collected for interobserver variability evaluation (no informed consent necessary).

Patients, whose fundus photographs from the database revealed e.g. new retinal haemorrhages – which is a clear indicator for retreatment – were not considered in this evaluation. Also other factors influencing visual acuity, e.g. cataract surgery during the Ranibizumab loading phase, were not considered herein; thus, the course of visual acuity was always based on macular changes.

Interobserver variability

Representative post-treatment OCT sections were displayed to four observers from the same University Eye department in a completely retrospective setting only based on OCT images and supplementary information of the course of AMD extracted from a chart review. Each of the observers is experienced over years in examining patients with AMD and indicating for intravitreal injections (IVIs) in neovascular AMD as well as for retreatments after the loading phase and further during the course of disease. Within the regular clinical setting, our treatment decisions based on the guidelines of the German Ophthalmologic Societies (Stellungnahme RG-DOG-BVA 2009) considering funduscopy, OCT and best-corrected visual acuity (BCVA) for retreatment decision after the loading phase. Each observer was asked to decide for retreatment (4th injection 4–6 weeks after the loading phase) in 108 consecutive patients. Therefore, a representative section of the SD-OCT from this time-point had to be judged in each patient. This was the base information, which was supplemented by different information about patients’ condition in six rating rounds. The following rating rounds were considered:

• 1
   Post-treatment OCT section without additional information.
• 2
   Post-treatment OCT section and ‘subjective estimation of vision’.
• 3
   Post-treatment OCT section and objective course of BCVA.
• 4
   Post-treatment OCT section and co-registered preoperative OCT section.
• 5
   Post-treatment OCT section and ‘subjective estimation of vision’ and BCVA.
• 6
   Post-treatment OCT section and co-registered preoperative OCT section as well as ‘subjective estimation of vision’ and BCVA (all information).

Regarding the ‘subjective estimation of vision’, the visual acuity was judged by the patient as being ‘stable’, ‘better’ or ‘worse’. Regarding the objective course of visual acuity, best BCVA (Snellen visual acuity) was determined immediately before first Ranibizumab injection and 4–6 weeks after third Ranibizumab injection. This information was given to the observers according to the rating rounds described earlier. The same OCT sections were presented when pre- and post-treatment OCT evaluation was required.

Statistical evaluation

The agreement rates as well as the Kappa values were calculated for all pairs of observers (six pairs) and all rating rounds (six rounds) leading to 36 primary results, from which mean values were extracted. Additionally, calculations from rating round six including all important information for retreatment indication were compared with the ‘real-life’ situation. Statistics were performed based on the recommendations of Viera (Viera & Garrett 2005). Thus, for all observer pairs, results were calculated using a suitable interobserver table (Table 1).

Table 1.   Interobserver variation between Observer 1 and Observer 2 regarding retreatment after Ranibizumab loading phase in neovascular AMD. Herein, values for rating round 1 (only post-treatment OCT section was presented) are displayed. The agreement rate in this example was 82.4% [(34 + 55)/108 = 0.824]. The expected agreement was 0.52 and the Kappa value 0.63, which is considered a ‘substantial agreement’ (see Stellungnahme RG-DOG-BVA 2009).

Expected agreement: (42/108 * 45/108) + (66/108 * 63/108) = 0.52. Kappa value: (0.824−0.52)/(1−0.52) = 0.63. AMD = Age-related macular degeneration.
	Observer 1
	Yes	No	Total
Observer 2
Yes	34	11	45
No	8	55	63
Total	42	66	108

Results

In our interobserver observation, the patients ‘subjective estimation of vision’ correlated with the objective course of BCVA in only 67.7%, whereas it correlated in 80.6% with the true given 4th injection after completing three loading doses (retreatment if vision got ‘worse’). BCVA correlated in only 71.3% with 4th injection (retreatment if BCVA decreased). Thus, ‘subjective estimation’ was superior to objective BCVA leading to the following treatment indications; however, only 2 out of 3 patients were able to estimate the course of vision adequately compared to BCVA after the Ranibizumab loading phase. Consecutively, BCVA and ‘subjective estimation’ do not seem to be very valuable criteria for retreatment indications.

Concerning the six rating rounds, highest number of retreatment indications was observed in RR1 (Table 2). Herein, the observers decided in 37.5% (23.1 – 46.3%) for 4th injection based on post-treatment OCT scans only. With supplementary information to the OCT scans, reinjection rate decreased to a mean of 22.8% (RR2: 22.7%, RR3: 23.8%, RR4: 22.0%, RR5: 22.7%). If all information was available (RR6), retreatment was indicated in 19.9% (Table 2), which was nearly the same rate as effectively observed from the chart review (19.4%).

Table 2.   Statistics of indicated retreatments of all four observers in dependence of the six rating rounds (see Material and methods) in all patients (n = 108); ‘real’ indications for reinjections were 21/108 = 19.4%.

	RR I	RR II	RR III	RR IV	RR V	RR VI
Observer 1	45 (41.6%)	34 (31.5%)	32 (29.6%)	44 (40.7%)	29 (26.8%)	28 (25.9%)
Observer 2	42 (38.9%)	6 (5.6%)	11 (10.2%)	8 (7.4%)	8 (7.4%)	6 (5.6%)
Observer 3	25 (23.1%)	20 (18.5%)	23 (21.3%)	21 (19.4%)	22 (20.4%)	22 (20.4%)
Observer 4	50 (46.3%)	38 (35.2%)	37 (34.3%)	22 (20.4%)	39 (36.1%)	30 (27.8%)
Mean	40.5 (37.5%)	24.5 (22.7%)	25.8 (23.8%)	23.8 (22.0%)	24.5 (22.7%)	21.5 (19.9%)

Table 3 lists the results of the agreement rates and Kappa statistics of all six pairs of observers from the six rating rounds. Herein, highest mean agreement rate from all six pairs of observers was found for RR6 (83.6%). In this rating round, all information as pre- and post-treatment OCT, the ‘subjective estimation’ and the course of BCVA were available for decision reflecting the ‘real life’ situation. However, the Kappa value for RR6 was only 0.47 indicating moderate agreement. Highest Kappa value of 0.61 (substantial agreement) was found for RR1, in which decision was derived from the post-treatment OCT scan only. The mean overall agreement rate was 80.9% with a Kappa value of 0.47 (Table 3).

Table 3.   Agreement rates and Kappa statistics for all pairs of interobservers and all six rating rounds (see Material and methods). Kappa values are: <0 (less than chance agreement); 0.01–0.20 (slight agreement); 0.21–0.40 (fair agreement); 0.41–0.60 (moderate agreement); 0.61–0.80 (substantial agreement); and 0.81–0.99 (almost perfect agreement).

	Obs. 1 – 2	Obs. 1 – 3	Obs. 1 – 4	Obs. 2 – 3	Obs. 2 – 4	Obs. 3 – 4	Mean
RR I	82.4% 0.63	80.6% 0.57	84.3% 0.68	82.4% 0.60	83.3% 0.66	76.6% 0.51	81.6% 0.61
RR II	72.2% 0.18	83.3% 0.56	80.6% 0.56	81.5% 0.16	66.6% 0.10	76.0% 0.41	76.7% 0.33
RR III	77.8% 0.33	84.3% 0.59	84.3% 0.64	85.2% 0.45	70.4% 0.20	79.6% 0.50	80.3% 0.45
RR IV	66.7% 0.23	77.8% 0.50	80.6% 0.56	86.1% 0.42	85.2% 0.41	88.0% 0.63	80.7% 0.46
RR V	78.8% 0.29	89.4% 0.71	85.2% 0.66	87.0% 0.48	73.2% 0.28	79.6% 0.50	82.2% 0.49
RR VI	79.6% 0.30	87.0% 0.64	90.7% 0.76	83.3% 0.30	75.9% 0.20	85.2% 0.59	83.6% 0.47
Mean	76.3% 0.33	83.7% 0.60	84.3% 0.64	84.3% 0.40	75.7% 0.31	80.8% 0.52	80.9% 0.47

Derived from Table 2, it was noticed that observer 2 had significant lower retreatment rates than the other observers; thus, Kappa values were lowest for these interobserver statistics (mean agreement rate: 78.8%; mean Kappa: 0.34; fair agreement) (Table 3). When removing observer 2 from statistics, the remaining mean agreement rate was 82.9% with a mean Kappa of 0.58 (moderate agreement) (Table 3).

When comparing the Kappa statistics of RR6 providing all information with the ‘real-life’ decision for retreatment, the following results were calculated: observer 1: agreement of 81.5% with a Kappa of 0.47; observer 2: agreement of 82.4% with a Kappa of 0.23; observer 3: agreement of 82.4% with a Kappa of 0.45; and finally observer 4: agreement of 78.7% with a Kappa of 0.41. This led to an overall mean agreement rate of 81.3% with a Kappa of 0.39 indicating only ‘fair agreement’.

Discussion

Repeated IVIs of Ranibizumab are one of the most expensive treatments in ophthalmology nowadays undoubtedly leading to great success rates of preserving vision in AMD. In contrast to former PDT treatment (Bressler 2001), all subgroups of neovascular AMD can be treated using Anti-VEGF strategies. However, before initial treatment, angiography is mandatory to determine the subgroup in neovascular AMD adequately, because not all kind of CNVs requires treatment. Many of less active occult CNVs might only be observed because visual deterioration is not expected on a fast track (Stellungnahme RG-DOG-BVA 2009, Schneider et al. 2005). The guidelines for the initial treatment based on visual loss, macular oedema, haemorrhages and the above-named angiographic classification of CNV are clear and well accepted (Stellungnahme RG-DOG-BVA 2009); however, especially guidelines for retreatment indication after the loading phase seem to be inhomogeneous. Beside the subjective estimation of the patient (e.g. Amsler chart) and the objectively determined BCVA, the most sensitive indicator of retinal oedema changes because of AMD presently seems to be the SD-OCT. Thus, it might be a reasonable view to use SD-OCT as the predominant tool to determine the therapeutic success after the Ranibizumab loading doses in an objective and noninvasive manner and to use it as a base for retreatment indications (Leydolt et al. 2009; Schaal et al. 2009).

The fact that in this study, the correlation between the patients’ subjective estimation of visual acuity and the objective BCVA was only about 68% underlines the demand of a firm and objective diagnostic tool. Even the course of BCVA (71%) and the subjective estimation (81%) correlated although higher but not well with the indication for Ranibizumab retreatment. Regarding the guidelines of the German Ophthalmologic and Retinologic Society, criteria for retreatment were considered as haemorrhage, increase of exsudates and oedema or increase of lesions size (Stellungnahme RG-DOG-BVA 2009). Patients with new appearing haemorrhage were ruled out from this observation; consequently, the first step in indicating retreatment certainly is still mydriatic funduscopy; the other factors however – even lesion size – can be determined by SD-OCT adequately (Framme et al. 2010). Thus, it was interesting to see what impact pre- and postoperative SD-OCT combined with the other information had on retreatment indications between different observers.

The overall agreement rate in this interobserver study was 80.9% with a Kappa of 0.47 indicating only moderate agreement. Highest agreement (83.6%) was obtainable for RR6 providing all information; however, Kappa was also only 0.47. This indicates the high variability in decision finding for retreatment in a ‘regular’ clinical setting having all desired information even for experienced observers. Also the comparison to the chart-extracted ‘real-life’ decision showed an agreement of 81.3% but only an even lower Kappa of 0.39 suggesting only ‘fair’ agreement.

The highest Kappa in this observation was found for RR1 with 0.61 (substantial agreement), when the post-treatment OCT scan alone was available for decision finding. This also reflects the value of an instrument as the OCT from which decisions can be derived in a relatively objective manner. In fact, a pure OCT-depending reinjection scheme as presented by Schaal et al. using Bevacizumab injections seems to sufficiently stabilize anatomical and functional features in treating AMD (Schaal et al. 2009). In that study, retreatment was considered in new or persisting sub- or intraretinal fluid. Similarly, in the PrONTO study using Ranibizumab, reinjection was given mainly on the basis of OCT findings; in fact, retreatment was also considered if there was any persistent fluid after completing the loading doses (Lalwani et al. 2009). However, in another study using again Bevacizumab, retreatment after the loading phase was only considered if OCT examination showed – and this in contrast to PrONTO – accumulation of fluid (Bashshur et al. 2009). Thus, no retreatment indication was seen in persisting fluid as described for the former studies. This reveals even a great variability in defining the exact value of OCT features as a base for retreatment. However, in consideration of the huge logistics in monitoring each patient adequately, a plausible set of criteria for retreatment should be postulated (Spaide 2009). Regarding the low correlation of the visual acuity course with the retreatment indications in our study, even in the cited studies, no correlation was found between the number of injections given and the visual acuity response (Bashshur et al. 2009; Lalwani et al. 2009; Spaide 2009).

Thus, if OCT – and especially SD-OCT – rationally will be considered as first-line tool in objectively determining the course of neovascular AMD after the loading phase, a consensus of interpreting the OCT findings is necessary. As seen in this study, especially observer 4 had the tendency to reinject most frequently derived from the OCT alone (RR1). The observer’s aim was to achieve complete remission of subretinal fluid. Even in the following rounds, the tendency for reinjection was enhanced in comparison with the other observers. In contrast, observer 2 showed lowest reinjection rates, especially when having supplementary information to the post-treatment OCT scan (RR2-RR6). That aim was predominantly to prevent the accumulation of fluid, which is a considerable contrast to the intention of observer 4. Derived from Tables 2 and 3, observers 1 and 3 also seem to have the general tendency to achieve remission of subretinal fluid by considering the course of visual acuity more dominantly. Figs 1–3 give examples for cases with: relatively homogenous interobserver decisions through all rating rounds for ‘non-treatment’ (Fig. 1) and for ‘re-treatment’ (Fig. 2) but absolutely inhomogeneous decisions, if OCT was improving (reduction of subretinal fluid) and visual acuity was decreasing (Fig. 3). No improvement in Kappa statistics was also observed, if pre- and post-treatment OCT was available determining the course of subretinal fluid remission most adequately.

Figure 1.  (A,B) Spectral domain OCT -based appearance of retinal pathology because of neovascular Age-related macular degeneration before and after three Ranibizumab loading doses. Subjective estimation was ‘better’, and best-corrected visual acuity increased from 0.2 to 0.3. OCT revealed subretinal fluid in a choroidal neovascularization with classic component before and completely ‘dry’ architecture after treatment. In this case, all four observers considered ‘no’ fourth injection in all rating rounds.

Figure 2.  (A,B) In another case, subjective estimation after the Ranibizumab loading phase was ‘stable’ but best-corrected visual acuity decreased slightly from 0.16 to 0.1. OCT revealed subretinal fluid in an occult choroidal neovascularization before and small reduction of fluid after treatment. In this case, observers 1,3 and 4 considered fourth injection in all rating rounds, whereas observer 2 did in 3 out of the six rating rounds. In none of all 108 cases, observers rated ‘4th injection’ in all rating rounds.

Figure 3.  (A,B) In this case, subjective estimation was ‘stable’ and best-corrected visual acuity changed from 0.5pp to 0.3. OCT revealed reduction of subretinal fluid but there was still a considerable amount of oedema. There was an enormous heterogeneity between observers in rating for or against the next Ranibizumab injection because of OCT improvement but reduction of visual acuity.

Derived from these results, SD-OCT nowadays seems to be an adequate tool to objectively determine the course of neovascular AMD during follow-up, it is superior to the invasive time- and cost-consuming fluorescein angiography and it might give reasonable criteria for retreatment indication. However, the aim of therapy needs to be clarified in adequate guidelines. Thus, in our opinion, the predominant aim should be defined as complete remission of subretinal fluid (Fig. 4), which is an indicator for the activity of the CNV. This can usually be obtained in about 60–70% of all treated patients after the loading phase, and no further reinjection is required at this time-point (Framme et al. 2010; Schaal et al. 2009). If subretinal fluid decreases after the Ranibizumab loading phase but persisting fluid is still present, reinjection should be considered independently of BCVA course. Reinjections should certainly also be considered in cases of fluid accumulation during further follow-up. However, if no fluid reduction or even fluid accumulation is noticed after the loading phase (nonresponder), reinjection using another drug might then be considered (Stellungnahme RG-DOG-BVA 2009). However, no clear statement is given whether ‘off-label’ Bevacizumab – which might currently be the most reasonable drug change – should be considered in these cases. No reinjection should be considered in patients with ‘dry’ OCT conditions even if visual acuity might have decreased. Frequently seen intraretinal cysts should not be necessarily treated, especially if scar formation (no subretinal fluid) is apparent. In such cases, fluorescein angiography might be performed to adequately determine CNV activity. However, in cases of reapparent subretinal fluid in scar formation, retreatment should be considered. Intraretinal cysts more than subretinal fluid play a major role in RAP lesions. For this entity, retreatment might then be considered in a comparable manner for intraretinal fluid rather than for subretinal fluid.

Figure 4.  Flow chart for spectral domain OCT -based indications for reinjection (4th injection) in neovascular Age-related macular degeneration after completing the Ranibizumab loading doses. The indication is only based on the post-treatment OCT scan, which revealed highest Kappa values in our observation. best-corrected visual acuity (BCVA) should only be a valuable criterion during further follow-up if persistent fluid cannot further be reduced (reinjection if BCVA decreases).

In this retreatment model, BCVA plays no role before, e.g. 5th Ranibizumab injection in cases of persistent fluid (the patient was initially a ‘responder’ and subretinal fluid reduced incompletely), because all treatment decisions are derived from OCT alone. Further treatments after 4th injection should only be administered in these cases, if BCVA decreases while subretinal fluid stays stable (Fig. 4).

If such suggested ‘easy’ treatment ‘rules’ as a kind of a standard operating procedure (SOP) are followed, retreatment indications might be more reproducible basing on the defined objective criteria. For this, SD-OCT presently seems to be the most suitable tool giving the needed base information on the retinal condition in treated AMD. It might be speculated that using the suggested SOP it is possible to enhance the agreement rates and Kappa statistics for different examiners while stabilizing frequency of reinjections and final visual acuity in patients with AMD.

Acknowledgements

This study was financially supported by Novartis Pharma GmbH, Nuremberg, Germany.