Lesion detection in epilepsy surgery: Lessons from a prospective evaluation of a machine learning algorithm

To evaluate a lesion detection algorithm designed to detect focal cortical dysplasia (FCD) in children undergoing stereoelectroencephalography (SEEG) as part of their presurgical evaluation for drug‐resistant epilepsy.

aspects, allowing access to algorithm code (that can be validated on new data) and data sets (to compare algorithm performance). 6In terms of effectiveness, robust prospective evaluation is essential.Traditional phases of clinical trial design may be inadequate for machine learning technology and require adaptation.The Idea, Development, Exploration, Assessment, and Long-Term Follow-Up (IDEAL) framework was developed to assess surgical innovation and devices but is also applicable to the machine learning technology associated with surgical procedures. 7,8n this IDEAL phase 1/2a study, we report the results and lessons from a prospective evaluation of a focal cortical dysplasia (FCD) lesion detection algorithm to identify areas of cortical abnormality consistent with FCD in children undergoing stereoelectroencephalography (SEEG) at a single centre.SEEG is a surgical technique to allow intracerebral recording of electroencephalography (EEG) and, in selected cases, forms part of the evaluation of children with drug-resistant epilepsy.It is undertaken when a patient is considered a potential candidate for resective epilepsy surgery but when non-invasive tests (e.g.magnetic resonance imaging [MRI], EEG video telemetry, and neuropsychological evaluation) do not identify the putative source of the seizures, termed the seizure-onset zone (SOZ), with sufficient degree of certainty.
The FCD lesion detection algorithm uses structural MRI (volumetric T1 and fluid-attenuated inversion recovery [FLAIR]) to identify areas of cortical abnormality consistent with FCD, the most common abnormality in children undergoing epilepsy surgery. 9The algorithm was developed in-house and has been validated retrospectively on a cohort undergoing SEEG, with colocalization of putative clusters with the SEEG-defined SOZ in 62% of cases and 86% of pathologically confirmed FCDs. 9,10Details of its development and evaluation are provided in the FCD lesion detection algorithm model card (Appendix S1).
While simultaneously addressing reproducibility and transparency aspects through the development of a large international data set with which to train the algorithm through the international Multicentre Epilepsy Lesion Detection (MELD) project, 10,11 we sought to undertake a prospective evaluation of the utility of the original FCD lesion detection algorithm 9 to identify areas of cortical abnormality in children undergoing SEEG.

M ET HOD
This MELD as an adjunct for SEEG trajectories (MAST) trial, a prospective, single-arm, interventional (IDEAL phase 1/2a) study, was approved by the UK Health Research Authority (Integrated Research Application System project ID: 275840) via the London-Riverside Research Ethics Committee and registered on Clini calTr ials.gov(ID: NCT04383028).Each patient or caregiver provided written informed consent for participation in the study according to UK law.Those older than 16 years with capacity provided their own consent and parents or caregivers provided consent for those aged below 16 years and those over 16 years without capacity.Every effort was made to explain the study to children under 16 years (including via ageappropriate information sheets); if they wanted, assent was also provided.It is reported in accordance with the TRIPOD and IDEAL 1/2a reporting guidelines (Appendices S2 and S3). 12,13n line with the IDEAL recommendations, the aim of this stage 1/2a study was not to establish a formal efficacy or effect size.Rather, it was to prospectively document firstin-human use and development of the protocol, to serve as a learning process for future larger implementation studies.Therefore a sample size of 20 was chosen.

Participants
Children (aged 3-19 yearss) undergoing SEEG as part of their routine clinical care at Great Ormond Street Hospital were eligible for participation.SEEG was undertaken in patients who were undergoing presurgical evaluation for drug-resistant epilepsy and was offered only when the non-invasive presurgical investigations did not sufficiently delineate a surgical target for resection.The decision for SEEG was made by a specialist epilepsy surgery multidisciplinary team (MDT) and was not affected by study involvement.The team is an experienced SEEG team, performing approximately 20 cases a year; in total, over 100 SEEG evaluations have been performed since 2015.Patients with tuberous sclerosis, large structural non-dysplastic abnormalities (e.g. a large perinatal stroke), or previous resective epilepsy surgery were excluded from participation because the algorithm was not developed or validated on these cohorts.Patients were also excluded if they had insufficient imaging or did not provide informed consent for participation.All patients during this study were scanned on the same Siemens MAGNETOM Prisma 3.0 T MRI scanner at Great Ormond Street Hospital, equipped with a 20-channel head coil, the same as used in the algorithm development and retrospective evaluation studies. 9,10Scans were acquired for routine clinical purposes and no additional study-specific imaging was required.

What this paper adds
• The focal cortical dysplasia detection algorithm collocated with the seizure-onset zone (SOZ) in 4 out of 19 patients.• The algorithm changed the resection boundaries in 1 of 19 patients undergoing stereoelectroencephalography for drug-resistant epilepsy.• The patient with an altered resection due to the algorithm was seizure-free 1 year after resective surgery.• Overall, the algorithm did not increase the proportion of patients in whom SOZ was identified.

Protocol
The protocol was designed to assess whether the FCD lesion detection algorithm provided additional information that would aid the identification of the SOZ and has been published previously; this publication included an IDEAL stage 0 evaluation of the risks and potential harms of incorporating the algorithm into prospective clinical use. 14The primary objective was to assess the proportion of patients who had additional electrode contacts (i.e.extra electrodes implanted into unsampled algorithm-identified clusters) in the SEEG-defined SOZ.A sample size of 20 was chosen in keeping with the IDEAL recommendations for phase 1/2a studies. 7etails of algorithm development, validation, and performance are provided in the FCD lesion detection algorithm model card (Appendix S1).Specifically, the algorithm used in this study was the first iteration of the algorithm, developed and validated on a small cohort of 22 patients and 28 typically developing controls at a single centre only.The retrospective study and stage 0 evaluation were all performed on the same version of the algorithm. 9,10,14he patient pathway is outlined in Figure S1.After informed consent was obtained, the SEEG implantation plan was formulated according to the routine clinical pathway; this was performed by a specialized SEEG planning MDT involving neurologists, neurosurgeons, neurophysiologists, and neuroradiologists.All patients underwent EEG video telemetry, 3 T epilepsy protocol MRI (including 3D T1 magnetization-prepared rapid gradient echo, 3D FLAIR, and 2D T2 in at least two planes) and neuropsychological evaluation.Positron emission tomography was acquired at the discretion of the MDT.Ictal single-photon emission computed tomography and EEG source localization were not part of our presurgical evaluation at the time this study was conducted.All preoperative investigations were taken into consideration during SEEG planning.Once the plans were agreed, the patient's MRI scans (volumetric T1 and FLAIR sequences) were run through the FCD lesion detection algorithm.Artefactual clusters (as identified by a paediatric neuroradiologist with a special interest in epilepsy) and contralateral clusters, in patients with an otherwise presumed unilateral SOZ, were excluded.
After this process, the top three remaining clusters, which were ranked based on the skew of the per-vertex classifier predictions within each cluster, were coregistered with the planned SEEG trajectories to assess whether there was already a plan to implant an electrode at the location of the cluster.If not, up to three extra electrodes could be added to the implantation plan to sample from each of the clusters (one electrode per cluster).
After this, implantation (using a robot-assisted method that has been detailed previously 15 ), recording, subsequent interpretation, and surgical treatment were carried out according to routine clinical practice.

Outcome measures
The primary outcome measure was the proportion of patients that had additional electrode contacts (i.e.extra electrodes implanted into unsampled identified clusters) in the SEEG-defined SOZ (Table 1).This provided a measure of the added benefit of the algorithm over routine clinical care.Our previous work suggested that 10% (3 of 30) patients may benefit from this technology. 10Therefore, we hypothesized that two patients would benefit in this small sample size and that the probability of no patients benefitting from this if the true rate was 10% would be 12.2%.Secondary objectives encompassed safety and efficacy that would allow a transparent assessment of the implementation of the new technology (Table 1).In addition, workflow optimization and any technical modifications are also reported in line with the IDEAL stage 2a reporting guidelines.

Statistical analysis
Given the nature of the study, the primary and secondary outcome measures are reported as descriptive statistics only.Missing data were excluded from the analysis.Statistical analyses were performed using non-parametric methods (Kruskal-Wallis tests) in an exploratory fashion.

R E SU LTS
Recruitment commenced in September 2020 and ended in May 2022.During this period, 35 patients underwent SEEG implantation, of which 22 were eligible for inclusion and 20 were enrolled in the study; the reasons for exclusion are shown in Figure S1.One patient consented to enrolment in the study and then chose not to undergo SEEG evaluation; therefore, the ensuing results are presented for 19 patients.
Overall, a SOZ was identified in 11 out of 19 patients undergoing SEEG implantation, from which six were offered resective surgery and five were offered laser interstitial thermal therapy ablation of the SEEG-identified SOZ.Details of SEEG outcome, treatment, and 1-year postoperative outcomes are shown in Table 2.

Primary outcome
The primary outcome of an extra electrode being part of the SEEG-identified SOZ was achieved in one patients.This patient went on to have an extended temporal lobectomy with amygdalohippocampectomy, with a histological diagnosis of FCD type 2a and an Engel class 1 outcome 1 year after resective surgery.Without involvement in the study, the patient would have had a more conservative temporal lobectomy, with unknown impact on his eventual seizure freedom.
Two other patients had a cluster that was part of the SOZ but it was already implanted by the clinical team, while another patient's SOZ was localized to a cluster that was not thresholded as one of the top three clusters.
Overall, there was overlap between algorithm-identified clusters and the SOZ in 21.1% (4 of 19 patients).If limited to those in whom a SOZ was found, this proportion increases to 36.4% (4 of 11 patients).In those with histologically confirmed FCDs, this proportion is 75.0%(3 of 4).

Secondary outcomes
Overall, 63 clusters were identified across the 19 patients, with a median of 2 and a range of 0 to 11 clusters per patient.There were no significant differences in the number of clusters detected in cases where a SOZ was identified compared to cases where a SOZ was not identified (median clusters 3 vs 2, p = 0.53) or by indication for SEEG (p = 0.94).Excluding artefactual and contralateral clusters and after thresholding to the top three clusters, 23 eligible clusters were identified, of which five already had an electrode recording from them and two could not be implanted.Sixteen additional electrodes were therefore implanted in nine patients.There were no significant differences in the number of electrodes planned by the clinical team (p = 0.55) or additional electrodes (p = 0.50) between cases where a SOZ was and was not identified.
Preimplantation confidence as to whether a SOZ would be identified was assessed independently by the treating neurologist, neurophysiologist, and neurosurgeon on a 7point Likert scale with 1 indicating 'definitely not' and 7 indicating 'definitely yes'.This was done to assess whether the FCD detection algorithm was more useful in 'easier' or 'more difficult' cases but, given that only one patient satisfied the primary outcome, statistical analysis was not appropriate.

Blinded neurophysiology and neurosurgical assessment of resection boundaries
As per the protocol, we undertook a blinded neurophysiological assessment in the nine patients who had additional electrodes.The blinded assessor (RT) was a clinical neurophysiologist with significant SEEG experience who was not involved in SEEG planning or interpretation for any of the included patients.Given the lack of data to support high interrater reliability in the visual assessment of SEEG data, 16 the assessment specifically asked two questions: (1) was a focal SOZ identified? and (2) if so, were the additional electrodes involved in this SOZ?The independent rater agreed with the clinical team on all nine cases, with a Cohen's κ = 1 for both questions.
Neurosurgical assessment was limited to the one patient in whom the additional electrode was part of the SOZ.As shown in case study no. 1 and Figure 1a, the resection boundary was altered by the additional electrode, potentially contributing to the seizure-free outcome.

Technical and workflow modifications
The most important learning point from this early phase of the study was the choice of inclusion and exclusion criteria.While some patients undergoing SEEG may indeed have occult FCDs, there are other indications for SEEG and many of these may fall outside the population in which the algorithm was developed and validated.
The pipeline was modified minimally over the course of the first cases to streamline the workflow.First, after review of the clusters by a neuroradiologist, the clusters were thresholded using fslmaths to ensure that only the relevant clusters were visible to the neurosurgeon responsible for planning the additional electrodes.The clusters were visible in monochrome only on the neuroinspire neurosurgical planning software (Renishaw, Wotton-under-Edge, UK); therefore, it was useful to provide screenshots of the clusters in bright colours to the neurosurgeon.

Safety
There were no safety issues from the additional electrodes.
No patients experienced any bleeding events related to any of the clinical or additional MAST study electrodes.

Case studies
To illustrate the utility of the algorithm, we provide two case studies, one of the patient with a positive primary outcome and another where the implanted cluster was not thresholded.
Case study number 1 (patient 7) Age at SEEG was 4 years.Age at seizure onset was 6 months.Semiology consisted of wake from sleep, oromotor automatisms, and right arm extended and left arm flexed to ear.MRI revealed extensive left hemispheric dysplasia, particularly in the temporal lobe, insula, and frontal operculum.EEG telemetry revealed lesions interictally localized to the left temporal region, ictally lateralized to the left, without localizing features.Positron emission tomography revealed hypometabolism of the left temporal lobe, insula, and frontal operculum.Neuropsychology revealed belowaverage performance, particularly in expressive language.
The patient was resistant to multiple antiseizure medications and the ketogenic diet.Extensive multilobar abnormality was present, although right-sided weakness was not established.The MDT discussed offering SEEG to assess whether a more limited intervention was possible, rather than hemispherotomy.
The MDT planned a left-sided fronto-temporo-insular SEEG implantation consisting of 18 electrodes (Figure 1a).The FCD lesion detection algorithm identified 10 clusters, of which six were on the left side and none of which were judged to be artefacts.Of the top three clusters, one was already implanted and two additional electrodes were implanted, one in the superior temporal gyrus and one in the fusiform gyrus (Figure 1a, yellow).
Seizure onset was found in the temporal pole and fusiform gyrus, with rapid spread to the hippocampus.During stimulation, a seizure was stimulated from the deep contacts of the fusiform gyrus (additional) electrode (Figure 1b, green), corresponding to the clusters (Figure 1b, red).Therefore, the standard anterior temporal lobectomy was modified to include the fusiform gyrus electrode as part of the resection (Figure 1a, red).Histology showed FCD type 2a; 1 year after surgery, the patient was seizure-free (Engel class IA) and off all medication.
Case study number 2 (patient 5) Age at SEEG was 5 years 6 months.Age at seizure onset was 18 months.Semiology revealed facial grimace with dysarthria, gulping, and breathing changes followed by left arm and face weakness, and subsequent left-sided hypermotor movements.The MRI showed right hemispheric abnormality involving the right temporal lobe and frontal operculum.EEG video telemetry showed ictal and interictal localization to right frontal region (F8, F10).The positron emission tomography scan showed hypometabolism of the right insula, frontal operculum, and Heschl's gyrus.Neuropsychology revealed broadly average performance but difficulties with hyperactivity and attention.The patient was resistant to multiple antiseizure medications.The MDT discussed subtle MRI abnormality involving the frontal operculum but were unable to determine the extent of the lesion to offer SEEG to map the extent of the lesion.
The MDT planned a right-sided fronto-temporo-insular SEEG implantation consisting of 15 electrodes (Figure 2a).The FCD lesion detection algorithm identified nine clusters, all of which were on the right.Only two of the top three clusters were implanted for technical reasons, one in the temporal pole and another in the temporal base (Figure 2a, yellow).
Seizure onset was found mainly in the orbitofrontal region, extending to the anterior insula.During stimulation, a seizure was stimulated from the orbitofrontal cortex (Figure 2b).On retrospective review, this corresponded to a cluster that was not within the top three (Figure 2b, red).An extensive orbitofrontal and anterior insula resection was performed.Histology showed FCD type 2b.The patient experienced seizure recrudescence 6 months after surgery, although there was improvement in seizure frequency and general behaviour, the patient being more alert, communicative, and interactive (Engel class III).There is a plan to undertake a further SEEG exploration to delineate the source of the ongoing seizures.

DISCUS SION
In this IDEAL phase 1/2a trial of 20 patients, we report that the FCD lesion detection algorithm, designed to detect FCD from volumetric T1 and FLAIR MRI sequences, provided additional information that altered management in 5.3% of children undergoing SEEG as part of their invasive presurgical evaluation.In addition, 15.8% had clusters that were part of the SOZ but already implanted by the clinical team.When limited to patients in whom SEEG identified a focal SOZ, the algorithm identified the SOZ in 4 out of 11 and 3 out of 4 in whom histology confirmed FCD.Given the small sample size, there is uncertainty around whether this reflects a true benefit, although the results show promise.SEEG planning is complex; in addition to the expertise of an experienced MDT, it relies on the amalgamation of multiple layers of information, including clinical semiology, EEG video telemetry, MRI, and other adjunctive investigations. 14It is therefore likely that any new technology will not transform the process but provide incremental benefit to a well-established workflow.8][19] The FCD lesion detection algorithm did not modify the proportion of patients in whom a SOZ was identified but led to an alteration in the subsequent resection boundaries in one patient, which may have contributed to subsequent seizure freedom.
This study sets the precedent for integration of the algorithm into routine clinical workflows and further prospective evaluation.Importantly, since the inception of this study, the algorithm has undergone further iteration, with the integration of multicentre data from 580 patients from 20 centres around the world as part of the MELD project and an update on the machine learning algorithm, which provides individual patient reports that increase the interpretability of the identified clusters. 11,20These individual reports overcome one of the technical modifications required as part of this study because they provide clear screenshots for the location of the clusters that will be useful to the neurosurgeon planning the implantation.
The biggest learning point from this early-phase study was that the inclusion criteria may have included patients without potential FCDs, a population in whom the FCD lesion detection algorithm has not been validated.For example, some of the patients included in this study had nodular heterotopias or scarring after encephalitis, which, although potentially epileptogenic and associated with adjacent cortical abnormalities, are not FCDs.There were only six classical 'lesion-negative' patients in the 19 included, three of whom did not have a localized SOZ and three of whom had posterior insula localization.We propose that future studies should have more stringent inclusion criteria to only include children who have a suspected or presumed FCD based on the non-invasive preclinical evaluation.This may improve the yield.
Given the safety of its integration in this study, one could continue with a multicentre evaluation of its efficacy, either as a randomized or single-arm study to proceed further down the IDEAL evaluation pathway to stages 2b and 3, in a more select group of children with suspected or presumed FCD only. 13However, this construct requires withholding of the additional layer of information (the algorithm output) until the electrodes have already been planned by the clinical team.In addition, it would require a prespecified threshold of what would be considered a significant difference between the arms to power it adequately.
An alternative would be to integrate the algorithm into the clinical information available at the time of SEEG planning, using retrospective cohorts for comparison; other studies have shown, for example, that integrating positron emission tomography and MRI led to a more focused SEEG implantation, reducing the mean number of electrodes to 7.1. 21Such an integration would also allow subtle changes in electrode locations as, for example, an electrode adjacent to a lesion could be adjusted to better sample it.In addition, the thresholding to the top three clusters in this study was somewhat arbitrary and it may be of more benefit to give teams access to all the identified clusters, with individual salient features, to allow a more nuanced interpretation before planning the SEEG trajectories.This may give increased relevance to clusters that support the electro-clinico-radiological hypothesis while decreasing the relevance of, for example, contralateral or distant clusters.Reduction of false positives may also be improved by improved algorithm performance, which is outside the scope of the current study.
In the sphere of presurgical evaluation, lesion detection algorithms also have the ability to influence decisionmaking before the SEEG stage, for example, whether a patient proceeds to resection, SEEG, or no surgery.Indeed Foged et al. 22 elegantly showed that electrical source imaging provided non-redundant information and altered management in 34% of the 82 patients included in their study, predominantly by changing the plans for invasive recordings.However, such interpretations may change based on the team that is interpreting the output of the algorithm; therefore, we designed this first prospective clinical evaluation to be as objective as possible.We report it with a relatively small sample of 20 patients in line with its prespecified design and the phases of the IDEAL framework; this is an important step before larger evaluations. 14nother key learning point is how to assess the riskbenefit balance.In this case, risk amounts to the additional risk of adding extra electrodes, which amounted to 0.8 electrodes per patient (16 electrodes across the 19 patients).Given the established small additional bleeding risk associated with each additional electrode, 23 a theoretical question may be raised as to what primary outcome rate (5% as in this study, higher, or lower) is required to make the additional electrodes worthwhile.

Conclusion
Robust evaluation is necessary to assess the impact of integrating machine learning technology into routine clinical care.Using the established IDEAL approach, we showed an early prospective integration of an FCD lesion detection algorithm to aid the planning of SEEG trajectories and alteration of subsequent surgical strategy in patients undergoing invasive presurgical evaluation for drug-resistant epilepsy.We conclude that future studies of machine learning algorithms require careful design and evaluation to effectively quantify the real-world impact of their integration into clinical workflows.These lessons make such early-phase studies (IDEAL phase 1/2a) crucial to shaping future evaluations.

AC K NOW L E D G M E N T S
This project was supported by the EANS-Stryker Research Grant 2020 and the Rosetrees Trust (A2665).AC is supported by a Great Ormond Street Hospital (GOSH) Children's Charity Surgeon Scientist Fellowship.This work has been supported by GOSH and the National Institute of Health Research Biomedical Research Centre.
AC, SA, KW, RT, and MT conceived and designed the study.AC was responsible for recruiting patients into the study and all authors were involved in the clinical care and additional study steps of the patients involved.RT performed the blinded neurophysiological analysis.AC, SA, and KW drafted the manuscript; all authors were involved in editing the manuscript and approved the final version before submission.

T A B L E 2
Details of patients included in MAST trial.after informed consentThis includes SEEG exploration, identification of SOZ, and post-SEEG treatments.From the total clusters detected, artefactual and contralateral clusters were excluded and clusters were thresholded to the top three clusters to provide the implantable clusters.Abbreviations: FCD, focal cortical dysplasia; LiTT, laser interstitial thermal therapy; LN, lesion-negative; LP Disc, lesion-positive, discordant non-invasive investigations; LP EOL, lesion-positive, to define extent of lesion; MAST, MELD as an adjunct for SEEG trajectories; ML, multiple lesions; PNH, periventricular nodular heterotopia; SEEG, stereoelectroencephalography; SMA, superior mesenteric artery; SOZ, seizure-onset zone.

F I G U R E 1
Case study no. 1.(a) 3D view of the cortical surface with the SEEG electrodes shown in green.Yellow identifies the additional electrodes implanted as part of the MAST trial.The red shaded area outlines the posteroinferior extension of a standard temporal lobectomy deemed necessary because of the additional electrode.(b) Axial, sagittal, and coronal views of the T1 magnetic resonance imaging scan with the MAST clusters visible in red.The additional basal temporal electrode contacts are highlighted in green.Abbreviations: MAST, MELD as an adjunct for SEEG trajectories; SEEG, stereoelectroencephalography.

F I G U R E 2
Case study no. 2. (a) 3D view of the cortical surface with the SEEG electrodes shown in green.Yellow identifies the additional electrodes implanted as part of the MAST trial.(b) Axial, sagittal, and coronal views of the T1 magnetic resonance imaging scan with the MAST clusters visible in red.A significant cluster identified in the orbitofrontal cortex did not meet the threshold for being one of the top three clusters but was identified as part of the seizure-onset zone through the electrode show in the coronal image.It was removed as part of subsequent anterior frontal resection.Abbreviations: MAST, MELD as an adjunct for SEEG trajectories; SEEG, stereoelectroencephalography.