These criteria were conceived from (1) well-known methodological and conceptual frameworks in urban climatology (e.g. Landsberg, 1970; Oke, 1976; Lowry, 1977; Goldreich, 1984; Wanner and Filliger, 1989; Szymanowski, 2005); (2) World Meteorological Organization (WMO) guidelines for meteorological observation (e.g. WMO, 1983; Oke, 2004) and (3) classical interpretations of scientific method (e.g. Hempel, 1966; Valiela, 2001). Included in (3) are the hallmark features of science: the problem statement, consisting of a conceptual model, operational definitions, and research hypotheses; and systematic measurement, consisting of a defined study area and controlled and repeated observations.
2.3.1. Scientific criteria
Having the stated or understood aim of measuring UHI magnitude or intensity in the canopy-layer, each study must invoke a suitable test of these concepts. The operational test required of the canopy-layer heat island model is surface-air temperature measurement in urban and rural, city and country, or otherwise built and non-built environments. This model is implicit in Howard's (1833) historical analysis of London's heat island, but is developed and systematised more formally by Oke (1976, 1982, 1988) in modern literature. Studies that fail to measure air temperature at approximately shelter height (1–2 m agl), or at least below roof level, and at field sites broadly defined as urban and rural, are poorly aligned with their conceptual model. These studies met Criterion 1 unsuccessfully. If sufficient detail of instrument height was not found in a report, or could not be inferred from its text, tables or figures, Criterion 1 was graded ‘unknown’.
Operational definitions translate concepts into procedures. Investigators must therefore contrive and communicate appropriate ad hoc procedures of their own to quantify the magnitude of a canopy-layer UHI. Criterion 2 requires two conditions of an operational definition: it must stipulate (1) the location and number of field sites used to quantify UHI magnitude, and (2) the measurement variables obtained at those sites. In passing Criterion 2, a study must satisfy both conditions. If an operational definition was not stated in a heat island report, or if the measurement variables or field sites chosen to represent UHI magnitude were not sufficiently explained or illustrated, Criterion 2 failed.
The WMO is unequivocal in its stance on measurement precision: ‘No statement of the results of a measurement is complete unless it includes an estimate of the probable magnitude of the uncertainty’, which is normally expressed as the interval of values ‘within which the true value of a quantity can be expected to lie’ (WMO, 1983). UHI investigators must be explicit in disclosing the measurement precision of their temperature sensors. If measurement precision was stated in a report, as was instrument type, Criterion 3 passed. If instrument type was stated but with no reference to its precision, Criterion 3 failed. Finally, if sufficient detail of instrument mounting (including shielding) was not found in a report, or could not be inferred from its text, tables or figures, Criterion 3 failed.
Site metadata are appropriately detailed in the report. Metadata include a local- or regional-scale map, sketch or photograph of the study area, and one or more quantitative indicators of micro- or local-scale surface exposure, roughness or cover at the field sites used to quantify UHI magnitude.
According to WMO guidelines on climate metadata, all meteorological measurements should include specification of station identity, geographical location, local environment, instrumentation, observing practices, data processing and station history (Aguilar et al., 2003). Supplementary WMO guidelines for meteorological measurements in urban areas stress that local environment and historical events are especially important due to the complex and dynamic nature of cities (Oke, 2004). The conditions of Criterion 4 are relaxed from these guidelines, which are too inclusive for a single heat island report. The first condition stipulates that site metadata include a local- or regional-scale illustration (e.g. plan map, site sketch, aerial photograph) of the study area. The illustration must portray major physical and cultural features of the region, such as mountain ranges, valleys, water bodies, transportation routes, built-up areas and other terrain features that are relevant to local and regional surface climate. Also expected of this, or another, illustration are the relative locations of the field sites used to quantify UHI magnitude. The second condition of Criterion 4 stipulates that site metadata include one or more measurable and climatologically relevant indicators of micro- or local-scale surface exposure, roughness or cover of the field sites used to quantify UHI magnitude. Possible indicators include sky view factor, aspect ratio of buildings or trees, fractional coverage of built and natural surfaces, and thermal admittance of built or natural surfaces. If either of these two conditions was not met in a heat island report, Criterion 4 failed. If both conditions were met, Criterion 4 passed.
The micro-scale settings of the field sites used to quantify UHI magnitude are approximately representative, in surface materials, geometry and human activity, of the local-scale surroundings.
The role of scale in Criterion 5 is paramount. UHI investigators are expected to place shelter-height instruments in areas where the local-scale fetch, or ‘circle of influence’, is relatively homogeneous in surface cover, geometry and human activity. The radius of this circle is difficult to estimate because it changes with building density and atmospheric stability. However, empirical evidence suggests that, as a general rule, the radius is no more than a few hundred metres (Chandler, 1964; Oke, 2004; Runnalls and Oke, 2006). If the micro-scale (<102 m) setting of a thermal sensor at 1–2 m agl is reasonably uniform, but the local-scale (102–103 m) surroundings are conspicuously varied or more heterogeneous, then the measured temperatures are not spatially representative, or accurate, beyond the micro-scale area. Investigators who extrapolate temperatures beyond regions of uniformity into wider, more diverse and more complex surroundings are confusing the scales of influence behind their measurements. ‘Confusion of scales’ is a common flaw in UHI investigation and it amounts to failure of Criterion 5.
In each heat island report, Criterion 5 was judged not on rigorous statistical measures but on qualitative evidence from site maps, photographs, sketches, station names and locations, and descriptions of the study area and its individual field sites. If evidence was sufficient to conclude that investigators used instrument sites approximately representative of the local-scale environment, Criterion 5 passed. If evidence was insufficient to conclude that the sample sites quantifying UHI magnitude were locally representative, the study was graded ‘unknown’.
In judging the representativeness of each study's field sites, special attention was given to a controversy known among research reviewers as the ‘expectancy effect’. The expectancy effect arises in primary research when investigators induce, through contrived means, a desired or exaggerated response from an experimental test (Hunt, 1997). In empirical UHI studies, the tendency to quantify UHI magnitude with field sites known a priori to exhibit maximum temperature differences, regardless of their representativeness, is a legitimate example of the expectancy effect. Evidence of the expectancy effect in a primary UHI report is adequate warning that field sites may not be representative. Insufficient metadata to allay this warning constitutes failure of Criterion 5.
Regular and repeated measurement provides control over random variation, and increases the probability of obtaining representative values of a desired effect at a chosen time and place (Valiela, 2001). Regular measurement also gives reliable basis to inferences. Judgement of Criterion 6 is based on the success with which a study's sample size, or number of repeated heat island observations, is aligned with its aims. Studies boasting large sample sizes were not automatically judged superior to ones with small sample sizes. However, studies with extremely small samples, such as one or a few nights of observation, failed Criterion 6 regardless of their stated aims. If the number of observations in a study could not be found, or could not be deduced from its discussion or presentation of data, Criterion 6 was graded ‘unknown’.
The extraneous effects of weather on UHI magnitude are passively controlled. Computations of UHI magnitude use temperatures measured in relatively steady-state weather: no passing fronts, strong advection, or precipitation.
UHI investigators must passively control weather to reduce the risk of confounding ‘real’ heat islands caused by urban effects with ‘fictitious’ ones caused by precipitation or air mass advection (Lowry, 1977). Passive control of weather can be gained through preconceived sampling designs or through post hoc data selection. Preconceived sampling avoids frontal or unsettled weather conditions, such as precipitation or strong advection, during data retrieval. Post hoc selection excludes data retrieved during non-steady weather from computations of UHI magnitude, or at least acknowledges weather effects on reported UHI magnitudes. Each of the sample studies was inspected for evidence of non-stationary or unsettled weather in its UHI dataset. If the investigators avoided, removed, or acknowledged the effects of frontal weather in their computations of UHI magnitude, and this effort was explicitly stated, the paper passed Criterion 7. If evidence suggested that frontal weather, especially precipitation and strong advection, had occurred during a measured heat island event, but weather was neither acknowledged as a confounding effect nor excluded from computations of UHI magnitude, the study failed Criterion 7. If neither the observed weather conditions during the heat island events nor any attempts to avoid, remove or acknowledge weather effects were reported, the study was graded ‘unknown’.
The extraneous effects of surface relief, elevation and water bodies on UHI magnitude are made sufficiently small through planned sampling design, or made sufficiently known through discussion and recognition of their influences on observed heat island magnitudes.
The effects of surface relief, elevation, and water bodies are difficult to avoid in most UHI studies (Landsberg, 1970; Wanner and Filliger, 1989). Investigators must therefore adopt an appropriate design strategy to counteract unwanted surface influences, otherwise the perceived ‘urban heat islands’ may not be sufficiently urban-induced to warrant use of this term. Experimental design is critical in eliminating or avoiding the extraneous effects of relief, elevation and water bodies. Placing urban and rural field sites at similar elevation and within relatively uniform local to meso-scale settings is essential for isolating the urban contribution to observed heat islands. Instruments should be sited away from slopes, gullies, cliffs, or ridges, and configured parallel—not perpendicular—to elongated surface features such as valleys and coastlines. These site configurations greatly reduce variable surface effects across a sampled area.
Most urban and rural locations have unwanted surface effects that cannot be avoided, in which case corrective measures can be performed on the data after they have been collected. Two post hoc techniques can improve isolation of the urban effect in complex terrain (Goldreich, 1984). The first technique regresses temperature against height to determine a representative lapse rate for a particular study area. The observed temperatures can then be normalised to a standard level using the measured lapse rates. The second technique regresses temperature against distance inland to determine a representative sea-land profile for a particular study area. Variable sea effects on urban and rural temperatures can then be reduced by normalising the observed temperatures to a standard distance from the shoreline. Both of these post hoc techniques, however, have serious drawbacks—namely, the instability of regression equations—and should be used cautiously, if at all, to correct estimates of UHI magnitude.
Each study was assessed of its success with Criterion 8 on evidence gathered from its discussion and illustration of the study area and on the individual field sites used to quantify UHI magnitude. If, through planned sampling, UHI investigators were unable to avoid the disturbing surface features of a particular study area, they should instead account for the surface factor in other ways. At minimum, they should qualify their estimates of UHI magnitude by appropriately recognising unwanted surface effects on measured heat island magnitudes. Recognition of these effects may include one or more of the post hoc regression techniques previously discussed. Post hoc correction by itself, however, does not constitute a passing grade—it must be part of a broader treatment of the surface factor that qualifies the purported ‘urban’ heat island estimates as over- or under-estimates by way of unavoidable land-surface features.
Given the difficulty and uncertainty of establishing control over the effects of surface relief, elevation and water bodies on UHI magnitude, qualitative treatment alone of the topo-climatic effect constitutes a passing grade for Criterion 8. Similarly, if, by planned sampling design, investigators sufficiently reduced or eliminated surface relief, elevation and water body effects from measured UHI magnitudes, passing grades were earned. Investigations that disregarded extraneous surface effects altogether from their study areas, and thus failed to discriminate a reasonably accurate urban factor, met Criterion 8 unsuccessfully. If a report did not describe or depict the surface features of a study area in sufficient detail, or did not disclose the locations of the field sites used to quantify UHI magnitude, Criterion 8 was graded ‘unknown’.
Criterion 9 highlights the importance of time control during UHI measurement. If the temperatures used to quantify UHI magnitude are not synchronous, or adjusted so as to be synchronous, urban-induced heat islands may be confounded with time-induced heat islands. If regional temperature change during mobile data-collection was said or shown to be significant by the investigators, and temperature-time adjustments were carried out, Criterion 9 passed. If temperature-time adjustments were judged to be necessary, but were not acknowledged in the investigation, Criterion 9 was graded ‘unknown’. Investigations that used temperature minima to quantify UHI magnitude were likewise expected to apply temperature-time corrections to their data. Temperature minima yield unreliable estimates of UHI magnitude because they are not normally synchronised across a spatial network of instruments, especially over complex urban and rural topography or in non-steady weather (Oke and Maxwell, 1975; Szymanowski, 2005). Investigations that failed to acknowledge or execute temperature-time corrections, which were judged to be necessary, did not pass Criterion 9.