A process approach to quality management doubles NEON sensor data quality

A quality management system is critical for ensuring that the data and services provided by an organization meet the needs of its mission. With a mission to collect long‐term open‐access ecological data to better understand how US ecosystems are changing, the National Ecological Observatory Network (NEON) is a highly standardized measurement network distributed across the United States and Puerto Rico collecting data on the biosphere and its interfaces with the pedosphere, hydrosphere and atmosphere. In order to achieve high‐quality, comparable data across the network, a quality management system was developed by applying the seven ISO 9001:2015 principles of quality management: customer focus, leadership, engagement of people, process approach, improvement, evidence‐based decision making and relationship management. The resultant system is integrated throughout NEON's organizational structure with an approach that connects people and operational processes throughout the data life cycle (process approach). We describe the system with respect to sensor data (automated measurements), demonstrating its effectiveness through examples, lessons learned and a continuous history of improvement towards quality goals, including a doubling of data quality in NEON's meteorological and soil datasets since 2015 and substantial gains in other sensor datasets. Owing to a focus on quality management principles and particularly the interconnectedness of human and information systems, NEON's quality management system can serve as a model for networks with a variety of organizational structures and sizes.


| INTRODUC TI ON
Environmental measurement networks present the opportunity to understand broad-scale patterns and processes by synthesizing information collected across many discrete locations. One of the largest single-provider ecological measurement networks in the world, the National Ecological Observatory Network (NEON) harnesses a high level of standardization to collect consistent, comparable, highquality data across many sites and over many years in order to better understand how US ecosystems are changing. Achieving and maintaining a highly standardized network presents many challenges.
Not only is NEON large and diverse in the number and type of distributed measurements, but also in organizational structure and operational processes. Objectives and activities must be coordinated, and procedural updates, problems and solutions must be communicated across a distributed, multi-faceted workforce. Meeting these challenges requires a quality management system (QMS) that connects people, information and operational processes throughout the data generation chain, from sensor preparation through field data collection, data processing and publication.
Data quality is the degree to which data are fit for use by data consumers (Wang & Strong, 1996). Standardization has long been recognized as essential for meeting quality requirements of distributed environmental measurements. Beginning in the 18th century, meteorological societies developed standardized, calibrated instrumentation and regulated observation procedures (Kington, 1974).
Organizations like the World Meteorological Organization (WMO; established 1950) carry this work forward today, publishing technical standards and guidelines for generating high-quality, comparable sensor measurements (e.g. WMO, 2008). Since the late 1900s, cross-disciplinary networks such as FLUXNET (Baldocchi et al., 2001) and the Long Term Ecological Research program (LTER; Hobbie et al., 2003) have harnessed standardization to share and synthesize data collected at independently run stations to understand large-scale and long-term ecosystem dynamics. Individual site investigators coordinate on measured and derived quantities, general processing steps and data formats. NEON builds off of these historical efforts, incorporating standards and guidelines from relevant scientific communities into its design and operational requirements and additionally unifying sensors and measurement infrastructure, collection and maintenance protocols and processing algorithms.
Beyond a foundation of standardization, the vast majority of modern-day scientific literature concerning quality in sensor networks has been devoted to detecting and rejecting poor-quality outcomes (quality control; QC), and largely in post-processing (e.g. Campbell et al., 2013;Pastorello et al., 2014). As a result, automated data QC methods have become quite sophisticated, involving numerous algorithms (e.g. Hubbard et al., 2005;Leigh et al., 2019), objective choice of test thresholds (e.g. Durre et al., 2008), and decision structures for data flagging (e.g. Smith et al., 2014). However, data QC is a vital but small fraction of an effective QMS. To achieve acceptable data quality, controls (QC) must be applied throughout the data generation chain and then verified to work effectively over time (quality assurance; QA). A QMS provides a structured framework of interlinked QC and QA processes that collectively ensure the final quality of a product or service. Yet, there is comparatively very little scientific literature on QMS frameworks for sensor networks and how they are applied effectively and efficiently at the network scale (although some examples exist, e.g. Fiebrich et al., 2010;McCord et al., 2021).
This paper aims to address this gap, demonstrating how a process approach to quality management yields an integrated, end-toend QMS that enables achieving standardized, high-quality data across a national-scale network. Our approach draws heavily on quality management principles developed in industry, which has long addressed the challenge of meeting quality requirements across large and diverse organizations. The International Organization for Standardization (ISO) was initially established out of efforts to unify industrial standards in the mid-20th century and provides seven foundational principles applicable to any organization: customer focus, leadership, engagement of people, process approach, improvement, evidence-based decision making and relationship management (ISO, 2015a). NEON's QMS focuses on these ISO principles rather than specific tools or techniques in order to create an evolving system that continuously improves and maintains relevance through time. It also extends the applicability of NEON's QMS to networks narrower or larger in scope and with different organizational structures. We begin with an overview of different approaches to environmental measurement networks in order to set the context in which NEON's QMS operates. We then provide the general framework of the QMS, followed by descriptions of each QMS component with examples, lessons learned and/or metrics that demonstrate its effectiveness. Finally, we discuss overlap with and relevance to other networks and future improvements.

| ORG ANIZ ATIONAL APPROACHE S OF ENVIRONMENTAL ME A SUREMENT NE T WO RK S
Environmental measurement networks have a wide range of organizational structures that fall on a spectrum from bottom-up to top-down. Bottom-up networks consist of a coalition of independently managed stations, whereas top-town networks are directed by a single entity that manages all sites and data infrastructure.
Organizational structure has implications for the scope and distribution of QMS components (Table 1), and also plays a significant role in research capabilities and culture, both of which propagate to the network's quality programs (Peters et al., 2014).

| The National Ecological Observatory Network
Although much of the QMS we describe in this paper applies to all of NEON's measurement systems, we focus here on the sensor systems, which collect automated measurements. Sensors at NEON's 47 terrestrial sites measure soils, meteorology, surfaceatmosphere exchange, atmospheric chemistry and phenology (Metzger et al., 2019). Sensors at NEON's 34 aquatic sites measure physical, biological and chemical properties of water and a basic suite of meteorological measurements. Together, about 68 different types and over 8,000 total sensors are deployed at any one time across the network, producing upwards of 75 data products hosted on the NEON Data Portal (https://data.neons cience. org/). Both terrestrial and aquatic sensor suites are collocated and coordinated with NEON's airborne remote sensing observations (Musinsky et al., 2022) and in-situ sampling bouts (Parker & Utz, 2022). Furthermore, a mobile deployment platform can be deployed in rapid response to natural phenomena and can be requested by the research community for separate investigations. Requirements are the backbone of NEON's top-down architecture that fully integrates the sensor systems, resulting in standardized observations designed for inter-site comparability and analysis of feedbacks across disciplines, spatial and temporal scales. This is achieved through decomposing the scientific objective (here: addressing grand-challenge questions in the environmental sciences; National Research Council, 2000) into requirements for architecture, hardware, software and operations (Metzger et al., 2019). Preoperational planning and setup of the network has already occurred according to these requirements, including critical QA elements such as siting and exposure, instrument selection and engineering design, which were verified during the observatory commissioning process and are not covered here. Some requirements are routinely checked during nominal operation and are noted throughout remaining sections. A list of documents with expanded details of NEON's quality system are provided in the supplemental material.

| FR AME WORK OF NEON ' S QUALIT Y MANAG EMENT SYS TEM
The seven ISO 9001:2015 principles of quality management (ISO, 2015a(ISO, , 2015b guide both the design and evolution of NEON's QMS ( Table 2). The combined application of the principles is also expressed in the Plan-Do-Check-Act cycle , an iterative cycle to control and continuously improve operational processes and products. While the Plan-Do-Check-Act cycle is evident in much of NEON's QMS, we refer to the principles themselves (in italics) as they are demonstrated throughout the remainder of this TA B L E 1 Characteristics of bottom-up, hybrid and top-down measurement networks Characteristic
Quality-related processes are distributed among NEON departments

Customer focus
The primary focus of quality management is to meet customer requirements and to strive to exceed customer expectations Create confidence in the data and services by following accepted standards and best practices for collecting, processing, and serving data. Solicit and respond to user (i.e. customer) feedback

Leadership
Leaders at all levels establish unity of purpose and direction and create conditions in which people are engaged in achieving the organization's quality objectives Promote data quality as a network goal. Organize people such that goals and strategies propagate across dispersed field teams and centralized personnel. Provide training and tools that facilitate knowledge transfer and efficient use of time and resources Engagement of people Competent, empowered and engaged people at all levels throughout the organization are essential to enhance its capability to create and deliver value Distribute quality-related procedures and training among all roles (e.g. technicians, scientists, software developers, managers). Define pathways to communicate issues and make improvements in every role

Process approach
Consistent and predictable results are achieved more effectively and efficiently when activities are understood and managed as interrelated processes that function as a coherent system Understand and take advantage of how materials, information and operational processes are connected throughout the data generation chain. Safeguard and monitor their integrity and establish robust connections between them

Successful organizations have an ongoing focus on improvement
Create processes that continuously identify, prioritize and act upon opportunities for improvement

Evidence-based decision making
Decisions based on the analysis and evaluation of data and information are more likely to produce desired results Generate performance metrics for critical operational processes and data quality (e.g. maintenance frequency, data flagging rate, user satisfaction). Use performance metrics along with information from other trusted organizations and the scientific literature to inform assessments and improvements

Relationship management
For sustained success, an organization manages its relationships with interested parties, such as suppliers Identify and communicate with internal and external stakeholders (e.g. suppliers, data users, internal departments) to ensure awareness and buy-in of changes, problems, and solutions. Invite collaboration (e.g. through working groups, open-source software) to identify and enact improvements NEON's Quality Manager who directs audits and assessments to review performance and ensure the overall effectiveness of the QMS (evidence-based decision making, improvement). Adjacent to these decision-making bodies are advisory groups appropriate to each level made up of external experts and stakeholders that provide input and advice on user community needs and best practices F I G U R E 1 NEON's departmental organization (lower portion) and decision-support framework (upper portion). Blue boxes with white text show departments and their quality-related functions. Black arrows and text depict the flow of quality-related materials among departments. Centralized issue management and configuration and version management systems are shown in light orange with brown and grey text respectively. Wide arrows with the same colour scheme depict how these systems centralize their functions. The upper portion shows NEON's hierarchy of decisionsupport bodies that bridges departments as well as the external scientific community to enable a coordinated approach to achieving quality goals. Decision bodies and decision flow are in grey ovals and arrows respectively. Dashed arrows depict non-binding input and advice.
(customer focus, relationship management). These include Scientific Technical Working Groups (TWGs; see Section 4), NEON's Science and Technical Advisory Committee, and at the highest level the National Science Foundation.
The following sections describe the QMS processes involved in the observatory functions and connections displayed in Figure 1.  Figure 1).

| LE VER AG ING THE SCIENTIFI C COMMUNIT Y IN QUALIT Y MANAG EMENT
TWG objectives include advising NEON on technical and methodology issues as well as priority areas and solutions for improving data accessibility, usability and quality despite time and budget limitations (customer focus). As an example, NEON sought input from the Soil Sensor TWG on the most desirable treatment of data collected by an early version of the throughfall precipitation sensor which was prone to jamming, making it difficult to distinguish dry periods from a jammed sensor ( Figure 2). In addition to targeted advice, TWGs also serve as a direct injection point for broader scientific community endeavours into NEON, maintaining NEON's relevance and responsiveness to evolving scientific questions and priorities (customer focus).

| S ENSOR PREPAR ATI ON
Sensors are the source of all of NEON's instrumented data products.
Critical attention is therefore paid to sensors well before they are deployed to the field, including the management of suppliers, configuration and tracking, calibration and validation.

| Supplier management
NEON's supplier management process, directed by the Quality Assurance Department, defines the requirements for the qualification of suppliers, including those that provide sensors and components (relationship management). Quality requirements flow to suppliers through a procurement statement of work.
Upon approval, suppliers are added to a critical supplier list and assigned a risk classification. Review and re-qualification occur every 1-3 years depending on risk classification, and issues are addressed using NEON's corrective action process (see Section 8.3).
One lesson learned in refining supplier management was the need for recurring dialogue with suppliers in order to prepare for changes such as firmware upgrades and phasing out of sensor F I G U R E 2 Options discussed with a technical working group to mitigate a quality issue in published data. The TWG recommended (and NEON adopted) the solution outlined in red.

Issue
Difficulty disƟnguishing dry periods from jammed sensors for all throughfall data (>400 site-months)

| Sensor configuration and tracking
To safeguard basic measurement integrity, the Instrumentation Department inspects and configures all sensors prior to shipment to the field (process approach). The configurable options are researched, selected and controlled via the configuration management process in order to capture measurements that fulfil NEON requirements. Sensors are tracked throughout their life cycle using a small memory chip embedded in each sensor cable that stores the sensor's unique identifier, the data expected from the sensor, and other important metadata used by the data acquisition system at each site to identify and validate the connected device (process approach).

| Calibration and validation
Routine calibration and validation of instruments ensures that sensors meet performance requirements throughout their lifetime (process approach). NEON's in-house metrology laboratory within the Instrumentation Department performs these functions, meeting ISO requirements for testing and calibration laboratories (ISO, 2017).
During calibrations, all traceability, uncertainties and performance for sensor measurements are recorded and evaluated against requirements (Csavina et al., 2017). Approximately 8%-10% of sensors and dataloggers are removed from circulation each year for failing requirements, exemplifying this critical quality control mechanism and the need for robust supplier management. For sensors requiring field calibration, the metrology laboratory provides version-controlled protocols and training materials for the Field Science department (relationship management). Calibration activities result in machine-readable files that contain inputs for processing algorithms to convert raw readings to calibrated measurements and/or generate uncertainty estimates and quality flags (see Section 7).

| FIELD DATA COLLEC TI ON AND MAINTENAN CE
Operations become dispersed at great distances as sensors are deployed to the field. Maintaining standardization and adherence to requirements across a national-scale network thus requires strong links and feedbacks between dispersed field teams and centralized personnel and resources, including training, documentation and data systems for sensor installation and maintenance.

| Training
Training ensures competency of staff (leadership)  to focus on when not all tasks can be completed during a site visitsee Section 6.3). This challenged subject matter experts to think about the whole system and not just the individual components (process approach).

| Sensor installation and location metadata
Accurate metadata on a sensor's physical location on the geoid and its spatial relationship to other sensors, to various degrees of precision, is critical for many environmental analyses. In addition to physical installation, each sensor is virtually installed by linking its unique identifier to that of a location in a relational database. The layout of the entire network is represented by a hierarchy of locations that represent its structure (e.g. domain → site → tower → measurement level → boom arm → sensor). Each location is associated with properties used to validate and restrict sensor installation according to the site design as well as store important metadata for processing and dissemination, such as a complete history of its geolocation (process approach). Accommodating changing site configurations and sensor geolocations in metadata storage, processing and publication has been an important lesson learned. In this scenario, technicians use a prioritization hierarchy (Figure 4) to complete the most important maintenance in the time available.

| Maintenance
The prioritization logic aims to minimize the overall negative impact to NEON's scientific objectives (leadership), taking into account the redundancy of a measurement, risk of failure and the scope of impact of a particular maintenance task (process approach).

| DATA PRO CE S S I N G
Raw sensor data streaming from field sites to NEON headquarters requires processing to yield products useful for research. Successful utilization of NEON's data products highly depends on a design that fulfils user needs, robust software to produce them and serving them in a manner that facilitates their use. These multi-level indicators of data quality allow for propagation of quality information into higher-level data products and also aid efficient identification and tracing of poor-quality data to their root cause (process approach). Here again, this collaborative approach was a lesson learned from past misunderstandings of needs, priorities and limitations across previously disconnected groups and departments. Bringing stakeholders together for prioritizing shared resources instead promotes this understanding and fosters a shared sense of ownership that reduces internal conflict and elevates the most pressing improvements for the network as a whole. An example of this process occurred when NEON scientists discovered significant instrument drift in CO 2 concentration measurements. Utilizing NEON's decision-support framework (Figure 1)

| Data processing and publication
Data processing has evolved due to lessons learned over NEON's history to minimize any transformation of raw data before it is stored centrally within Data Infrastructure, enabling the correction of calibration, installation, location metadata or software errors when they are inevitably discovered (process approach). Another lesson learned has been to use structured files, such as Avro, Parquet and HDF5 formats. Such structures allow inclusion of provenance information, including calibrations, location information and QC thresholds, which are critical to building confidence in data quality (customer focus) and for tracing quality problems back to their source (process approach).
Data publication follows FAIR data principles (Wilkinson et al., 2016)-findable, accessible, interoperable and reusable, all fulfilling a customer focus. After a provisional period in which adjustments may be made as necessary, data are included in annual releases that provide static, permanently available data packages with findable DOIs. Released data packages as well as provisional data are accessible using standardized communication protocols via the NEON Data Portal or application programming interface (API).
Interoperability is facilitated through standardized vocabularies and metadata following the requirements for the Ecological Metadata Language (EML). Reusability is aided through a data usage policy and provenance documentation provided with data downloads, including the ATBD, definitions of data terms, location information and an issue log that describes known problems and data changes.

| MONITORING , PROB LEM TR ACKING AND ISSUE RE SOLUTION
Much of the utility of instrument data comes from their continuity, that is, availability at frequent, regular intervals. Gaps in data availability and periods of poor quality diminish this benefit. NEON's QMS organizes people and information to monitor operational processes throughout the data generation chain and promptly communicate, trace and resolve issues. monitoring data to provide high-level overviews of the entire network down to specific errors and quality concerns ( Figure 5).

| Site, sensor and data monitoring
Monitoring tools developed organically as systems came online and challenges were discovered. This often resulted in overlapping and uncoordinated efforts, and in many cases monitoring tools are still 'owned and operated' by specific departments. An effort is underway to consolidate tools in a centrally accessible location so that they are discoverable across the organization (leadership, process approach).

| Data quality trouble tickets
Despite best efforts, some quality issues evade automated detection. Although potentially involving any operational process in the data generation chain, these issues typically involve a sensor set being out of science requirements but still producing plausible data, such as an animal nest blocking a precipitation gauge ( Figure 6). Data Quality Trouble Tickets (DQTTs) are designed to handle these situations, leveraging vigilance and expertise across the observatory (engagement of people) to identify, review and resolve quality concerns as well as a formal pathway to communicate them in published data (process approach, customer focus). Anyone may report a DQTT through the central issue management system (including through the NEON website). It is then reviewed by a subject matter expert and, if warranted, the affected data are indicated as suspect by manually raising the final quality flag. Details and justification for manually raised flags are recorded in machine-readable format and provided to end users in publication metadata (customer focus). Since 2018, an average of 52 DQTTs have been reported each month across the sensor systems, highlighting their importance for accurately communicating quality and also for identifying areas to improve or where automated tests may be needed.

| Issue tracking and resolution
The central ship), a cross-functional team is formed to perform immediate containment, conduct root-cause analysis, and work with IPTs and/or TWGs as necessary to implement actions to reduce or eliminate the cause(s) of the issue. Corrective action reporting is provided

Focus Time interval Functionality
Site status Real-time Services critical to data collection (power, communications, device command and control). Active alerts

Sensor connectivity
Real-time Sensor connectivity, data transmission and database ingest.

F I G U R E 5
The NEON sensor health monitoring dashboard shows (a) colour coded, high-level summaries of sensor data availability and quality. Users can drill down and sort sensors (b) to identify problematic data streams. Automated email alerts (c) provide supporting information, the ability to link to a ticket in the centralized issue management system, and pre-determined guidance for how to handle issues.

| AUD ITS AND A SS E SS MENTS
Ascertaining the competency and degree of compliance of the data's provenance is critical to the reliable assessment of data quality (Snee et al., 2014). This premise guides NEON's audits and assessments, which perform periodic checks and analyses to determine whether the observatory is effectively delivering the intended data and services (evidence-based decision making, customer focus).
Audits assure observatory integrity by comparing actual conditions with NEON requirements and quality standards (e.g. ISO 9001, ISO 17025). For example, NEON's field audit program verifies site and sensor configuration, location and orientation. Assessments evaluate NEON's operational performance, often aggregating user surveys (Crall, 2018) or monitoring data (see Section 8.1) and comparing it to target values. Both audits and assessments are collaborative (relationship management) and adverse findings are addressed through the corrective action process (improvement).
High-level assessments are presented to advisory groups for review and advice on the most pressing or impactful improvements to meet user needs (customer focus). Major improvement initiatives are then coordinated down through the decision-support hierarchy into specific actions by IPTs, TWGs and departments (Figure 1; leadership, improvement, process approach).
Indicators of published data quality, such as completeness DOC.004764 in Table S1).
(2) validity = actual published records passing all quality checks actual published records .
(3) We propose the development of any QMS begin with the organization of people in a decision-support framework (e.g. Figure 1;  An effective QMS did not arrive at NEON on day 1 and nor is it complete, as evidenced by the history and trajectory of continued improvement in complete and valid instrument data (Figure 8).
Slated improvements focus on strengthening the embodiment of quality management principles throughout the data generation chain. (a) A sensor obsolescence strategy is being developed based on obsolescence risk assessment best practices (Rojo et al., 2012) in order to maintain standardization over time as sensors become obsolete or no longer available (relationship management).
(b) Sensor drift is being analysed to optimize re-calibration intervals to increase resource efficiency and minimize damage due to shipment and installations (evidence-based decision making). (c) Targeted, more concise maintenance recording aims to improve compliance in recording preventive tasks (relationship management), which will enable preventive maintenance frequency to be optimized by explicitly linking it to data quality (evidence-based

| CON CLUS ION
A quality management system is the mechanism by which a measurement network may achieve the data and services that fulfil its mission. We have demonstrated how an effective end-to-end quality management system has been developed for a national-scale network by applying the seven ISO principles of quality management: customer focus, leadership, engagement of people, process approach, TA B L E 4 General overlap between NEON's QMS and published comprehensive quality systems of other distributed environmental networks. Individual details of system components may differ

Literature reference
Decision-support framework Shafer et al. (2000); Schultz et al. (2015) Integrated workflows among teams and operational processes Abeysirigunawardena et al. (2015); Hudson et al. (1999);McCord et al. (2021); Shafer et al. (2000) Proper sensor procurement, configuration and routine calibration traceable to applicable standards Fiebrich et al. (2010); Schultz et al. (2015) Training program and standardized protocols for data collection and maintenance Fiebrich et al. (2010); Hudson et al. (1999); Schultz et al. (2015) Monitoring and swift routing of site and sensor health issues Lakkala et al. (2005); Shafer et al. (2000) Standardized data processing, publication and data management Boden et al. (2013); Isaac et al. (2017);Fiebrich et al. (2020) Data quality control with flagging framework that aggregates the results from several quality analyses and allows manual adjustment and/or annotation of the quality assessment Assessments and audits against performance targets Hudson et al. (1999); Schultz et al. (2015) improvement, evidence-based decision making and relationship management. A special focus on the interconnectedness of human and information systems (process approach) enables a distributed workforce and operations to act as a coordinated unit to produce standardized, high-quality data through space and time. Its application has resulted in a doubling of data quality in NEON's meteorological and soil datasets since 2015 and substantial gains in other sensor datasets. This framework can be adapted as needed for networks of varying size and structure.

AUTH O R S ' CO NTR I B UTI O N S
C.S. led the writing of the manuscript. All authors contributed critically to the development and operation of NEON's quality management system, writing and editing of the drafts and gave final approval for publication.

ACK N OWLED G EM ENTS
The National Ecological Observatory Network is a program sponsored by the National Science Foundation and operated under cooperative agreement by Battelle. This material is based in part upon work supported by the National Science Foundation through the NEON Program.

CO N FLI C T O F I NTE R E S T
The authors declare no conflict of interest.

PE E R R E V I E W
The peer review history for this article is available at https://publo ns.com/publo n/10.1111/2041-210X.13943.