Toward automation of initial chart check for photon/electron EBRT: the clinical implementation of new AAPM task group reports and automation techniques

Abstract Purpose The recently published AAPM TG‐275 and the public review version of TG‐315 list new recommendations for comprehensive and minimum physics initial chart checks, respectively. This article addresses the potential development and benefit of initial chart check automation when these recommendations are implemented for clinical photon/electron EBRT. Methods Eight board‐certified physicists with 2–20 years of clinical experience performed initial chart checks using checklists from TG‐275 and TG‐315. Manual check times were estimated for three types of plans (IMRT/VMAT, 3D, and 2D) and for prostate, whole pelvis, lung, breast, head and neck, and brain cancers. An expert development team of three physicists re‐evaluated the automation feasibility of TG‐275 checklist based on their experience of developing and implementing the in‐house and the commercial automation tools in our institution. Three levels of initial chart check automation were simulated: (1) Auto_UMMS_tool (which consists of in‐house program and commercially available software); (2) Auto_TG275 (with full and partial automation as indicated in TG‐275); and (3) Auto_UMMS_exp (with full and partial automation as determined by our experts’ re‐evaluation). Results With no automation of initial chart checks, the ranges of manual check times were 29–56 min (full TG‐315 list) and 102–163 min (full TG‐275 list), which varied significantly with physicists but varied little at different tumor sites. The 69 of 71 checks which were considered as “not fully automated” in TG‐275 were re‐evaluated with more automation feasibility. Compared to no automation, the higher levels of automation yielded a great reduction in both manual check times (by 44%–98%) and potentially residual detectable errors (by 15–85%). Conclusion The initial chart check automation greatly improves the practicality and efficiency of implementing the new TG recommendations. Revisiting the TG reports with new technology/practice updates may help develop and utilize more automation clinically.

TG-315 also recommends that each institution establishes its local standard format for treatment prescription.
Current initial chart checks still rely heavily on human inspection and evaluation of various aspects of treatment plans. However, many studies have called for the improvement of pretreatment physics review performance by introducing initial chart check automation. For example, to quantify the potential effectiveness of different quality control measures, some studies have used departmental incident learning systems, 6 which involve reporting any near-misses and incidents that occur in the practice of radiation oncology. A study conducted at the University of Washington Medical Center (Seattle, WA) documented 522 potentially severe or critical near-miss events within an institution-wide incident learning system over 3 years. 7 The majority of errors that were not detected could have been identified if automation of specific physics checks had been in place.
Concerning the increasing reliance on initial chart check automation, TG-275 provides an estimation of the types of checks that might be automated in the future, based on a review of prior publi- | 235 photon/electron EBRT initial plan/chart review checks. Eight ABR-certified medical physicists in our department with differing clinical experience (2-20 years) were invited to participate in a study based on their clinical experience. Manual initial chart check times were evaluated for six different tumor sites (prostate, whole pelvis, lung, breast, head and neck, and brain cancers) based on each physicist's experience. Three types of plansintensity-modulated radiation therapy (IMRT)/volumetric-modulated arc therapy (VMAT), 3D, and simple calculationwere evaluated (depending on their applicability in each cancer site).
Two derived checklists were created for TG-315 and TG-275 to eliminate some uncommon check items for different scenarios. In this article, "TG-315 recommended checklist" refers only to the 36 recommended items in Table 4

TG-275 checklists
Our expert development team including three physicists re-evaluated the TG-275 automation feasibility. This expert team has been developing automation tools for various EBRT procedures in our institution during the past 7 years. Our in-house automation tool for initial chart check includes the sophisticated scripts that can access electronic documents, treatment plan DICOM files, and record and verify (R&V) system. Table 1 shows the functions that were available at the time of publication for this in-house tool, which includes many items in the TG-275 checklist. Besides, a commercial tool -Mobius3D (Varian; Palo Alto, CA) has been used in our initial chart check procedures for years. Our experience with our in-house and commercial tools helps classify the feasibility of automating TG-275 items. Note, to be consistent with TG-275, full automation refers to "can potentially be fully automated" and partial automation refers to "can potentially automate whether particular information is present (e.g., a document exists) but not whether the information in it is correct." 2.C | Automation level simulation for initial chart check In this work, three levels of chart check automation, that is, Auto_UMMS_tool, Auto_TG275, and Auto_UMMS_exp, were simulated for the four checklists. Auto_UMMS_tool refers to the automation level that automates some checklist items by using our in-house tool ( Table 1) and Mobius3D. Auto_TG275 refers to the automation level that automates some checklist items fully or partially as indicated in TG-275 Table S1. A.ii 4 . Auto_UMMS_exp refers to the automation level that allows fully or partially automated checklist items as re-evaluated by our expert development team.
Our UMMS tool is composed of the in-house automation tool and Mobius3D, and both use DICOM files for CT image, RT structures, RT plan, and RT dose as input data. The in-house tool was designed to automatically review the items in Table 1. It compares all plan parameters in a DICOM RT-Plan file from the OIS to those in its counterpart DICOM RT-Plan file from the TPS. A comparison PDF report can be generated as a patient EMR (electronic medical record) document. In the report, the hospital name, patient name, ID, plan Name/Label, and approval status in TPS and ARIA OIS are listed. Any difference in monitor units (MUs), multileaf collimator (MLC) shape, energy, collimator angle, gantry angle, gantry rotation, couch angle, source-skin distance jaw position, isocenter, segment weights, wedge, bolus, patient position, or applicator can be highlighted if that difference exceeds the predefined tolerance. For the majority of plan parameters, the predefined tolerance is zero. Nonzero predefined tolerance for some plan parameters is mainly due to rounding errors while importing/exporting plans between different systems. More information, including plan name, beam name, radiation type, tolerance table, isocenter coordinate, and treatment machine name, is also compared. The commercial software The potential for automation of chart check items was mentioned in TG-275 Table S1. A.ii 4 . Some check items are regarded as potentially fully automated, including physician intent/prescription vs treatment plan), optimization or calculation parameter checks (target and organ at risk objectives, algorithms, dose grid size, etc.), and data transfer from the TPS to a third-party OIS. For some items, automation may be possible only to determine whether a specific document or item is present, not whether the information in that item is correct (e.g., most patient assessment and simulation checks). The remaining items require manual inspection. Most of them are related to free-typing or handwriting documents, such as consult note, physics consult, patient consent documents.

2.D | Benefit evaluation
Benefit evaluation for different automation levels was performed based on two aspects: (1) manual check time saving and (2) avoidance of errors as a result of automation.
T A B L E 1 Five categories of automated initial chart check items covered by our current in-house tool (=Auto_UMMS_Tool excluding the commercial tool) that has been used clinically for years. The corresponding TG-275 checklist items of patient assessment (PA), simulation (Sim), and treatment planning (TP) in Table S1.A.ii are also listed.

UMMS tool check items
Corresponding TG-275 items in Table S1.A.ii

Prescription
Prescription consistency with our institutional practice guidelines PA-Q1-1, PA-Q1-9, Sim-Q1-2, TP-Q2a-10   Table S1. A.ii 4 was a result of the survey from the AAPM members. Here, our expert development team re-evaluated the automation feasibility based on our experience of developing and implementing our in-house automation tool and the commercial product across our institution. Among the 71 items that were deemed not fully automated in the TG-275 report, the automation feasibility of 69 checks was re-evaluated differently. Table 3 lists some of our results versus TG-275: 35 items as "Full", 17 as "Full/Partial", 6 as "Partial", 2 as "No". The additional feasibility option "Full/Partial" means that automation can be partially implemented but full automation can be realized with certain conditions. For example, our institution is still using a scanned patient consent form. If an electronically fillable or online patient consent form is used, all the essential information could be retrieved by our in-house tool. However, using an electronically fillable or online patient consent form requires our current clinical procedure and policy to be altered, which may take a long process. Therefore, the feasibility for "patient consent" in Table 3 was re-evaluated as "Full/Partial" given the fact  Table 3 like "Prescription (respect to standard of care or institutional clinical guidelines," "Utilization of immobilization and ancillary devices" [ Fig. 1(b)], "Special Considerations for radiotherapy," and "Utilization of other treatment modalities," "Request for in vivo dosimetry," and "Parameters and setup for specialized devices." Another example is about "Insurance approval." Our automation tool queries the Varian ARIA SQL database to check if the care path task "Billing Approval" is completed by the billing office [ Fig. 1 T A B L E 3 An example of our expert team's re-evaluation vs TG-275 survey results on the automatic feasibility of EBRT initial chart check items in TG275. The automation feasibility was categorized as "Full," "Full/Partial," "Partial," or "No." The re-evaluation was based on the existing automation tools Auto_UMMS_Tool (that is already being used clinically) plus those scripts and programs that are under development by our expert development team. the planning system DICOM files, such as isocenter positioning with respect to the target, the CT image properties, and the plan quality, which also yield more advanced automation feasibility compared to the TG-275 survey results.
As more automation is introduced, manual time can be reduced significantly. Data in Fig. 2    tions, but we believe such variation will be less and less significant with more prevalence of automation tools.
The results suggest a strong need for the development of automated initial chart checks for the sake of efficiency and efficacy.
Introducing a high level of initial chart check automation may be the best solution to significantly ease the human workload and reduce human error. This is particularly important as our treatment techniques become more complex within the framework of precision radiotherapy. We believe that by introducing automation tools into initial chart checks for different levels of errors, from simple to sophisticated can be rapidly detected without human manual inspection. Regardless of the automation level used, we believe that human vigilance is always needed, particularly when it comes to the prevention of a medical event.
Check items in TG-275 that were considered beyond the clinical training and responsibility of a medical physicist as in TG-315 could be re-examined when we are equipped with automated tools.
According to the International Organization for Medical Physics' Pol- Once an automation tool is clinically implemented, each user must fully understand its limitations and outputs. Lack of such understanding might lead to adverse consequences in patient care.
In assessing the status of automation tool development, it seems likely that lower dimensional problems, such as treatment parameter comparison, can be easily handled by scripts/programs. Higher dimensional problems in physician order error, including disease staging and treatment modality decision, may be taken care of by machine learning, such as a k-means clustering algorithm, 8 random forest methods, 37 or Bayesian networks as proposed by Kalet et al. 38 and further developed by Luk et al. 31 As Kalet et al. 39 and Pallai et al. 40 pointed out, machine learning still faces many challenges and must be quality assured before introduction into the clinic. The breakthrough of automation tools or machine learning beyond low-level checks will take some time.
A limitation of this report is in quantitative analysis; our study data from physicists may be biased by their familiarity with current checklists and hardware/software, as well as by nonfamiliarity with the new checklists. After becoming accustomed to the new checklists, physicists may spend less time on TG-315 or TG-275 items.
However, we believe our results can provide insight into the process of evolving current initial chart check procedures to be consistent with the latest national guideline. While TG-315 is very similar to the current checklist in many clinics, it may add extra safety to also include most of the recommended items in TG-275. As pointed out in TG-275, items suggested in the report may be applied after considering each institution's workload. More automation tools will boost initial chart check efficiency, and then, it becomes practically feasible to include more check items suggested by TG-275.

| CONCLUSION
Without automated initial chart checks, the implementation of new guidelines, particularly TG-275, involves significant human work.
Automated initial chart checks can significantly reduce manual check time and detect more potential errors. With the evolution of automation techniques, it is foreseeable that more automated checks will be available to further improve practicality and efficiency in the clinical implementation of the new TG recommendations. Revisiting the TG reports with new technology and practice updates may help develop and utilize more potential automation for clinical use.

ACKNOWLEDG EMENT
We confirm that all coauthors contributed this work and agreed with the submission of this manuscript to JACMP.
T A B L E 4 The benefit of reduction in residual potentially detectable issues for the manual check when using full TG-275 checklist and automation levels Auto_TG275 and Auto_UMMS_Exp.