Next generation personalized display systems employing adaptive dynamic‐range compression techniques to address diversity in individual circadian visual features

Perceptually natural standard‐dynamic‐range (SDR) images reproduced under normal viewing conditions should retain enough information for the human observer to estimate the time at which the actual high‐dynamic‐range (HDR) scene was captured without recourse to artificial information. Currently, both global‐ and local‐tone mapping operators (TMOs) seem to have comparable levels of performance. Therefore, we first consider the constraints created in the actual human visual system by eye movement, and buttress a hypothesis with a demonstration. We briefly review the imperceptible illuminance effects yielded by the personal circadian clock suggested by chronophysiological research and other related effects, because our previous study suggested that the characteristics of the human visual system dynamically varies depending on the individual's circadian pattern. Finally, we conduct two psychophysical experiments based on the hypothesis that the human visual system employs several global TMOs at the first stage for information compression that depend on individual‐circadian‐visual‐features (ICVF). The results suggest that (1) no participant can perceive actual‐capture‐time (ACT) and (2) sensitive observers can discriminate reproduced images based on virtual‐shooting‐time (VST) effects induced by different types of global TMOs. We also discover that the VST‐based discrimination differs widely among people, but most are unaware of this effect as evidenced by daily conversations.


| INTRODUCTION
In the real world, human observers perceive various objects under a wide range of luminance values from low to high; sunlight at noon can be 10 8.5 times more intense than star-light as described in the Ohta and Robertson's textbook (see Fig. 1.8). 1 Although the dynamic range that can be perceived simultaneously by human observers (e.g., more than approximately 10 3.5 in our previous experiment in the laboratory 2 ) is much less than the above-mentioned value, it is still much greater than that of a standard-dynamic-range (SDR) display (i.e., approximately 10 2 ).Images that try to reproduce the same range end up looking overexposed, underexposed, or both.This is because our visual systems can capture a full range of tones in high contrast scenes.To achieve greater realism, high-dynamic-range (HDR) technologies are emerging in not only the display field but also television system standards, that is, Recommendation ITU-R BT. 2100-2 (2018). 3To display HDR images or videos, quite dark rooms are conventionally required to avoid the picture quality degradation caused by viewing flare from illumination sources in the room. 2,4The next step is, therefore, converting HDR images/videos into SDR images/videos that retain their Perceptual Reality (PR) or Perceptual Fidelity (PF), that is, reducing the impairments caused by flare, so that HDR contents can be comfortably viewed in daily life often. 2,4one mapping operators (TMOs) are now being used to compress HDR into SDR images.Many types of methods employing either global-or local-TMOs have been proposed. 5Tone mapping performance comparisons of proposals based on deep learning have also been made. 6hree examined methods that employ Local-TMO, [7][8][9] and two methods that employ global-TMO 10,11 were selected as the best five from among 24 methods tested; the implication is that they were nearly comparable.
Therefore, in preparation to understand the general characteristics of visual perception in HDR environments, we need to analyze why both methods are simultaneously acceptable for human observers even though they appear to be radically different.
In this paper, we describe (1) why global processing is essential in the first stage of human visual system processing in Section 2, and then (2) the results of two experiments that confirm our hypothesis are given in Sections 3 and 4. Saccadic eye movement is constantly exhibited, except when pursuing steadily moving objects, for acquiring visual information distributed across a wide field.Therefore, rapid eye movements are inherent in the visual system input process.For example, R. Gregory described in his textbook 12 the following: (1) "Some of the data processing needed for perception takes place in the eye, which is thus an integral part of brain.This preprocessing funnels 120 000 000 receptors down to 1 000 000 optic nerve fibers, no doubt reducing the thickness and stiffness of the optic nerves so that eye movements are possible" (in p. 55; i.e., the reduction rate is approximately 1/ 100), and (2) "Why does the world remain stable when F I G U R E 1 Our hypothesis of two-step human visual perception.
we move our eyes?… Eye-head systems do not work by detecting actual movement of eyes, but from commands to move them" (in pp.101-105; i.e., feed-forward control system is employed).
This clearly suggests that the human visual system has no spare capacity to process the additional local data created by the new information yielded by saccadic eye movement before combining it with previously integrated information in the egocentric coordinate system.Therefore, the data processing configuration, that is, adaptation status, in the retina before input to early vision must be kept not only temporally but also spatially constant regardless of the saccadic eye movements evoked.In other words, this implies that the well-known local differential operation in the receptive field in each retinal cell is employed not for constructing adaptive local-tone-mapping within the first stage, that is, retina, but for constructing a global-tone-mapping with the objective of keeping the total retinal receptive field consistent from the peripheral to fovea centralis.
This also suggests that several illusions that are based on local operations, like the well-known Mach-band (usually explained by lateral inhibition within the retinal cells) 13 or checker-shadow-illusion (one of the same luminance parts in shadow region is perceived to be brighter than the other in non-shadow), 14 are created outside the retina (i.e., in the brain).The reason is because the above mentioned gradual global luminance changes do not significantly impact the relative luminance relationship between the parts, except for saturated parts.
These two notes, based on our analysis, are very important in explaining (1) the compatibility between the existence of stable percepts, not only spatial but also brightness, independent of eye-movements and (2) the causality of above-mentioned several illusions that are based on local operations.In other words, it shows a key consequence of saccadic eye movements, which are driven by feed-forward control, and have been quite ignored in the image processing research field.
Here, there is a caveat that must not be forgotten as a supplementary explanation.In special cases, frequent changes of adaptation level are required by the human visual system, which looks as if the retina was employing local tone mapping.It is suggested that, according to our previous experiment, in excessively bright surrounding regions, that is, backlit environment, the brightness adaptation level was clearly changed with changes in the locus of attention accompanying saccadic eye movements. 2Such changes are perceptible because the low reaction speed (reaction-time was approximately hundreds of milliseconds to the order of seconds).Therefore, the difference between the two situations is noticeable.
Figure 1 shows our hypothesis of two-step human visual perception. 15In the first step, named NVP (normalized visual percept), the human visual system employs only scene-independent Global-TMO, for acquiring compressed images that are passed to early vision.Note that we cannot perceive or understand this image (i.e., NVP) directly, because we can perceive only the image output by the second step.In the second step, that is, complex image processing stage in or after the early vision, individual-scene-dependent Local-TMOs are employed.
2.2 | Fundamental discussion with demonstration about the reason why global-and local-TMOs are comparable in performance in current image processing technologies although the former is essential

General confirmation about the difference in captured images
Figure 2 shows examples of images captured from an HDR scene with mixed light source environment under various exposure levels and processing methods.Panels (a) to (d) are conventional SDR images with different exposure levels taken by Nikon Mirrorless camera (Z-9), where the color balance was set to daylight default (T = 5210 K).Exposure level was varied in 1 stop intervals and total difference was 3 EV (exposure value), that is, ratio was 8 times.They can be used to check the physical ground-truths within the constraint of not overexposure or underexposure.Panel (e) is an SDR image compressed by our proposed method, where Global-TMO was employed and gradation tone was set at middle (TC-3; see Section 4 and Figure 10), and Panel (f) is partial enlargement of (e).Panel (g) is an example of taken by a current smartphone (iPhone SE 2nd generation), presumably a Local-TMO was employed, and Panel (h) is partial enlargement of (g).
In this scene, checkpoints for comparison are labeled A to G. Letters with prime (A 0 to C 0 , and E' to G') show the same thing as the panel without prime.In other words, their images were recursively processed in one more cycle using the same exposure and processing, like the so-called picture-in-picture (PiP).
First, note that the luminance level of the display (IO-DATA EX-LDC161DBM) that already presented the same scene (i.e., recursive capture) was lower than that of the real scene.Therefore, human observer felt that the real scene in front was blown out while the display was slightly dim.In the real scene, the upper part of "F," which presented the brightest part (white with just slight tint of blue) of the thick paper was approximately 700 cd/ m 2 while the maximum white luminance of the display was 150 cd/m 2 .Therefore, the luminance ratio between two was approximately 4.7 (i.e., ΔEV = 2.2).The image in the display, which was recursively processed (i.e., PiP), should be approximately two stops darker than that of the direct capture if the image was naturally reproduced by perception.Similarly, the "X" portion (shown only in panels (e) and (g)) was recursively processed twice, and therefore the part should become approximately four exposure stops darker than that of the non-recursively processed portion (i.e., ΔEV ≈ À4).
Second, in general, the merit of image quality checking by recursive processing is countering error accumulation.Therefore, not only brightness tone but also Examples of images captured from an high-dynamicrange (HDR) scene with mixed light source environment under various exposure levels and processing methods.
richness of color (i.e., saturation) can be checked easily.If the reproduced image is perceptually natural, regardless of physical ground-truth, all recursive changes should be at least consistent.
Finally, before discussing Global versus Local in the next section, we check the above two points by using four reference SDR images with different exposure levels.The most important index is "ColorChecker Passport Photo 2" (see "A"). 16The image represented in Panel (a) (+2 EV) is clearly over exposed but preserves information of darker parts.The images in Panels (b) (+1 EV) and (c) (0 EV) keep at least five F-stops' dynamic-range (i.e., all six gray scale patches are clearly separated) and best exposure level in SDR shot maybe intermediate between (b) and (c).The image in Panel (d) (À1 EV) is clearly under exposed but preserves most of the information in highlighted parts.
Summarizing the above examinations, (1) in the real HDR environment, human observer could perceive both highlighted and shadowed areas simultaneously unlike any of the four SDR images (i.e., observer adapted to the HDR environment), (2) highlighted parts gradually became saturated in actual perception and highlighted parts of more distant table lamp (shown as "G"; actual luminance was approximately 2200 cd/m 2 ) which is basically intermediate between panels (c) and (d) (i.e., observer's perception was saturated a little), (3) darkest parts were similar to Panel (a), and (4) these SDR images had larger luminance contrast and higher saturation than actual human perception.

Prominent differences between Global and Local compression
A simple observation can confirm that there are some remarkable differences in image reproduction between panels (e) and (g).
First, Panel (e) seems to have less color saturation in general compared with SDR images.However, natural color was reproduced, especially in the color checker's tone (see "A"), and the luminance of recursively processed parts darkened with the increase in the number of recursed times, similar to real scenes (note that the content was reproduced adequately, even the dark parts; see panel (f )).
For supplemental information, the left side in "X" (X displays content of the original shot) was created by SDR (ΔEV = 0) while the right side was created by the proposed method.Therefore, even in smartphone's shot (see panel (h)), the displayed image showed no effect of local processing.
On the other hand, Panel (g) has distinct and quite saturated colors in general compared with SDR images.Flashy (i.e., highly saturated) colors everywhere were reproduced, especially the highly saturated color patches on the color checker (see "A") with no effect of reflection glare.The luminance of recursively processed parts also remained nearly constant and did not become darker with the increase in the number of recursed times, as if the installed display had the maximum luminance as same as that of real HDR scene, resulting in a difference from the real situation (see panel (h)).
Second, furthermore, in panel (g), you can find that some clear artifacts (i.e., false and differences due to insufficient reproduction) were created by the embedding of color information, where (1) you can check the ground truth by checking panes (a) to (d) in which appropriate exposure parts are present and (2) independent of ground truth, you can check the differences between corresponding non-recursed and recursed parts within this panel.For example, (1) the ground-truth color of "B" was blue that of "C" was purple (correctly captured in non-recursively processed image), but the color of "B 0 " and "C 0 " in recursively processed parts is highly saturated blue (see also panel (f)) and ( 2) the color of the thick paper was basically white but clearly reproduced as bluish in recursively processed parts (see also panel (f)).Note that it is also very difficult to discern the above difference by yourself in ordinary situations.
In conclusion, it is suggested that the SDR images yielded by our global processing proposal, based on human perception, offer perceptual fidelity with smooth and well-balanced gradation.On the other hand, it is suggested that the SDR images yielded by local processing offer easy-to-understand impressions with bright and salient colors, at the cost of artifacts in both luminance and color reproduction that actually exist but are subjectively indiscernible.Therefore, the choice of image processing depends on the desired objectives, that is, the former suits taking photographs that preserve perceptual fidelity, while the latter is better for creating personally favored pictures with artificial enhancement (this assumes that users know of the tradeoff made).
For supplemental information, note that conventional SDR images, especially film-based and captured by recent well-constructed digital camera, usually involve both global-and local-characteristics simultaneously.In particular, (1) they have global TMOs in general (i.e., nonclipping area), but, on the other hand (2) in highlight zones, they have soft clipping characteristics, but their range still fails to match that of the human visual system, resulting in dependence of the clipping characteristics of captured contents involving colors (i.e., "local as a result" figuratively speaking).Easy-to-follow examples include, reproduction of sky-blue (hue shift from light blue to cyan; see Figure 4 (a) in our previous study 15 ), highly saturated flower colors, autumnal tints, etc.Therefore, people conventionally used auxiliary lighting (e.g., electronic flash, etc.) or reflector board to acquire better image quality, especially for professional use.In addition, as mentioned above (in "local" explanation), even though in these (i.e., locally saturated) situations, observers usually were unaware except for the under-exposure cases or especially interesting objects.

| Simple subjective tests for demonstration
Observers Participants were just two: S0, who was the first author of this paper, who created the proposed globaltone-mapping curves, had never participated any other formal experiment, and S51, who could make accurate judgments (chosen by experimental results shown in Section 4 and Table 1 #8).

Methods
Experimental images were the six shown in Figure 2, except for enlarged images.Observers were asked to judge picture quality (i.e., which one is more closely resemble the real experimental setup) by pairwise comparison, using a five-grade comparison scale.All 15 pairs of images were viewed sequentially in randomized order on a display in an ordinary office environment with ceiling fluorescent lights (outside light was shaded to prevent glare), just after observing the original experimental setup in a booth.Before making the comparisons, observers could change pairwise images alternatively (a gray image was inserted between pairwise images to prevent afterimages) until they satisfied.They were also allowed to reconfirm the real setup during this experiment.

Experimental results analyzed by pairwise comparison and MDS
Analyses were carried in two different ways.One was Scheffé's Method (Nakaya's modification) 17 for overall effects and the other was multidimensional scaling (MDS) 18 for individual effects, where we employed Bell-Curve's plug-in for Excel. 19irst, the pairwise comparison revealed two significant effects "Main" (i.e., experimental images) (p < 0.001) and "Main x Individual" (p < 0.01), respectively.As shown in Figure 3, TC-3 (global) was highest, where SDR (0EV) and Smartphone (Local) yielded no significant difference, because they were located within Yardstick.However, two SDR images (±1 EV) were judged poor compared with only TC-3 (global) with significant difference (p < 0.05).The worst one was an SDR image (+2 EV).
Second, to confirm the observers' characteristics, we employed MDS analysis.Results are shown in Figure 4.As suggested by the pairwise comparison, the two observers had different results.
For both observers, the first axis seemed to represent an evaluation of basic quality, where the right side the axis was positive.
However, the second axis was assigned differently.In the case of observer S0, it appears that the second axis plotted total balance of brightness impression within proper exposure duration, where upper direction was positive.Two reasons for this were, surprisingly, introduced approx.50 years ago.First, because S0 was well experienced with positive slide films, which demand very precise exposure due to high gamma values over 2, such as Kodachrome II or Kodachrome 25, 20 (the first experience was around 1973-1974), he judged that brighter was better within appropriate exposure area.Second, however, S0's evaluation of over exposure was severe, because positive films are quite weak to over exposure (i.e., this evaluation related to 1st axis).This suggests that two logically different or opposite judgments are required within and outside appropriately exposed areas, that is, non-linear judgments (brighter vs. darker).By the way, S0 became aware of this effect for first time in his life by this analysis.In the case of observer S51, it is suggested that the second axis represented the color salience or differences in saturation, where lower direction was positive.There are two reasons for this; he selfreported that the color of TC-3 was observed to be pale during the experiment, and he generally chose high contrast as natural in the previous formal experiment (see Section 4).
Therefore, overall favorite directions were upper right in case of S0, but lower right in the case of S51.Note that both observers accepted both global-and local-compressions as being effective compared with conventional SDR images, despite the difference in terms of their judgment criteria.
T A B L E 1 Participants' attributes and experimental results, sorted by cluster analysis result employing Ward's method (See also Appendix Figure A1).  .

Brief summary
The above results of the simple subjective experiment suggest that both global-and local-tone mapping offer useful information compression, even though the evaluation standard was "looks like real HDR scene"; their outputs were judged as being higher in quality than the equivalent conventional SDR images even though their basic processing concepts were completely different.This result shows that human observers can accept both photographs (i.e., in the same way as their perception) and pictures (i.e., art, where usually some parts are enhanced or deformed following the painter's whim) similarly, even though they have clear objective difference.In other words, both representations offer some form of acceptable "illusions," unlike the SDR images (i.e., conventional photographs consisting of physical data captured within normal dynamic range) and the acceptance of the illusions is usually unconscious.It should be also considered that the two illusions assist the human observer by making scene understanding easier in different ways.One is "global" operation contributing to "compression of dynamic range" before early vision (i.e., output signal of retina) and the other is "local" operation contributing to "perception related to objects" in or after early vision.
Here, if the former operates correctly in the generation of SDR images from HDR images (i.e., implementing the alternative role of real processing in retina when viewing real HDR scenes), the second compensation should be attenuated because perceived processing in viewing the presented image automatically adds this operation in or after early vision.In other words, the first stage focusing on only previous captured HDR image (i.e., it is equivalent to local processing) should be bypassed and the first stage global compression in recursion step should be performed by retina instead.On the other hand, if the former was ignored or not consistent with human perception in the first stage, the later compensation should be emphasized to enable easy scene understanding, even though this likely to create artifacts.The authors see the current general situation as following the second assumption.

| Perceptible but generally unconsciously occurring illuminance effects related to diversity of circadian characteristics as suggested by our previous study
The discussions in Sections 2.1 and 2.2 confirmed that global processing is essential as a first stage operation even though it is hard for the individual to discern.However, the discussion did not address the periodic variation in perceptual contrast over 1 day.The prior discussions considered only the impact of the difference in lighting condition on dynamic range, like SN (sunny: standard percept in afternoon outside), DR (in daylight room), or CD (cloudy outside).
Coincidentally, this finding was triggered by some discussions by two authors (Ohtsuka and Orita) and another former member of our lab.(Nakamura) a few years ago that focused on the difference in supporting NVP curves, that is, CD and DR, in the morning outside viewing environment.Figure 5 demonstrates the difference; CD images have higher contrast than SN, whereas DR have lower contrast.The authors were very surprised at the unexpected large individual differences because this visual environment was very common and easily compared with situations demanding severe lighting adaptations, for example, twilight or night, from the human visual system.
Therefore, here we consider the possibility that the NVP depend on temporally sourced personal characteristics, that is, diversity of human circadian characteristics, rather than differences in illumination dynamic-range.

| Imperceptible illuminance effects in human circadian rhythm suggested by chronophysiological research and other related effects
First, in the chronophysiological field, the circadian rhythm is a well-known effect; however, it has not been generally considered that it impacts visual perception directly.For example, S. Higuchi (2013) wrote "Physiological polytypism is very difficult keyword, but it is important to clear this keyword" from the point of "Nonvisual effects of light and circadian rhythm." 21][24] In separate research, M. Takao et al. (2009) demonstrated that neither photoperiod nor season of birth modulated diurnal preference in the Japanese population, 25 even though diurnal preference in Canadian populations was modulated by the season of birth. 26These findings suggest that these ethnic differences might reflect circadian photosensitivity in infancy.It was also reported that two biological differences are reported to exist between Caucasians and Asians: polymorphisms of circadian clock genes and difference in ocular photosensitivity.
It is considered that the onset of melatonin secretion, that is, the dim light melatonin onset (DLMO), has been a common tool for determining the phase of the circadian timing system in the above research field.Recently, Kennaway (2023) revealed that DLMO across ages correlated to the Morningness Eveningness Questionaire (MEQ). 27Here, MEQ is self-assessment questionnaire for determining individual circadian preference, that is, chronotype. 28econd, MEQ has been utilized in healthcare field for more than four decades.Here, note that the chronotype can be defined from self-assessment questionnaires, resulting that people can be explicitly aware of their subjective circadian preferences without receiving any physiological tests.Therefore, for example, although it was suggested that chronotype can create some type of physical symptoms among adolescents, 29 or psychiatric disorders in general, 30 in relation to melatonin suppression time, the relationship between chronotype and relating physiological mechanisms might be quite complex as mentioned in articles. 29,30In addition, note that MEQ does not explicitly evaluate the outdoor spending time in daytime.However, previous research revealed that (1) it was affected by the amount of time spent outdoors in broad daylight significantly affected the timing of sleep and (2) for precise analysis of genetic chronotype, many other factors should be considered. 31Furthermore, recent research on myopia revealed that the time spent outdoor in childhood affected to reducing risk of myopia as an adult. 32It is suggested that not only chronotype in childhood, that is, subjective morning-evening preference, but also the location where child usually spent either indoor or outdoor even in case of morning type, could affects physiological visual feature in adults.
In summary, it is considered that imperceptible illuminance effects in human circadian rhythm have complex physiological mechanisms.Therefore, it is also considered that they are affected by many other parameters, especially the amount of time spent outdoors in daylight, although it correlated to subjectively judged preference by MEQ.2.3.2 | Example of clearly perceptible visual effects associated with chronotype: an illusion linked to mixed color temperature condition of illumination, that is, "The dress" In contrast to the invisible effects mentioned in above section, the interindividual difference in subjective color is a particular and perceptible illusory situation connected to higher order perception which is impacted by top-down interpretation.For example, there has been considerable interest in stimuli ("the dress") that yield starkly divergent subjective color percepts between observers.Wallisch (2017) 33 showed that (1) assumptions about the illumination of the dress-that is, whether the stimulus was illuminated by natural or artificial light or whether it was in a shadow-strongly affects the subjective interpretation of observers, compared to demographic factors, such as age or gender, which have a relatively smaller influence, and (2) interpreted these findings in a Bayesian framework by also showing that prior exposure to long-or short-wavelength lights due to circadian type by using morningnesseveningness self-judgment shapes the subjective experience of the dress stimulus in theoretically expected ways. 33Using more plain words without using color temperatures or wavelength of lights, it was suggested that this phenomenon was based on individual location preference differences in daily life, that is, the color judgment of people who prefer outside tends to focus on bright parts, whereas people who prefer inside focus on dark parts, given mixed color lighting conditions.
Here, note that our preliminary previous study with its single lighting conditions suggested related interesting results 15 : If in conditions without sunlight shining in, like underground shopping malls, average participants (i.e., in Japan, it was considered that they preferred outside) had a tendency to change their NVP from outside daylight to night, even during the daytime.This result newly suggested that color judgment might be consistent within individuals regardless of location change because both optical lighting environments and overall percepts, that is, from the first stage (i.e., NVP) to higher order color judgment, changed simultaneously.Therefore, when considering lighting condition, it is suggested that single color temperature lighting conditions are normal (i.e., frequently occurred) and mixed ones are specific; it is considered to be reasonable assumption for human evolution.
In addition, Gregory described in his textbook that visual illusions offer good cues for elucidating about how perception normally works and why it sometimes fails with many examples. 12As an easy understanding example, he described that the Poggendorff illusion (see Plate 6 in final page of the textbook 12 ) disappeared in 3D binocular condition even though clearly observed in 2D condition.Our previous experiments 34,35 revealed that the cause was the existence of monocular regions in binocular stereoscopic vision and a correction mechanism in alignment perception worked well in 3D.On the other hand, in case of 2D geometrically correct picture or photographs, there were no monocular regions, resulting in miss-correction by the human visual system.In other words, it is considered that 3D binocular stereoscopic occluding environments are more familiar, or frequently experienced, compared with directly overlapped objects like 2D conditions, given the long evolutional history of human beings.
From examples in this section, it is estimated that the human physiological system including visual perception have adapted to common problems well, with individual variety, whereas infrequent problems remain a challenge.The latter is, as a result, clearly perceived as illusions.It is considered, therefore, that the illusion of "the dress" could likely be the tip of the iceberg in visible effects in relation to circadian type, with consideration that so many research results involve the invisible in many situations as mentioned in Section 2.3.1.This creates the new question as to why no visible effects in normal environments have yet to be observed in relation to circadian type.

| New possibilities: Existence of diversity of intermediate circadian effects between visible and invisible named "individual-circadian-visual-features (ICVF)"
According to the discussions in Sections 2.3.1 and 2.3.2, it remains a mystery why no effects have been observed at the intermediate level, that is, normal visible or perceptible effects, between both situations until now.
Recently, however, in addition to the above discussion regarding the effects induced by melatonin, Saito (2018) revealed that the effects of ipRGC (intrinsically photosensitive retinal ganglion cell), which contains melanopsin, inside the blind spot (i.e., scotoma) affected luminosity sensitivity threshold of other visible parts. 36ubota and colleagues (2022) also revealed that a headmounted device with different light incident angles (55 vs. 28 ) altered (1) the magnitude of nocturnal melatonin suppression, that is, lower angles were significantly effective, and (2) light-evoked pupillary constriction mediated by ipRGCs, that is, with lower angles offering significantly greater effects, in healthy young subjects. 37his suggests that (1) the morning sun light affects the human visual system directly via photoreceptor cells in the eyes and (2) the validity of our above consideration that the human visual system may control its perceived contrast transfer function according to the lighting condition may also be driven in part by the circadian cycle with the trigger being morning sun light.

| Objectives of new experiments in this paper
From the discussion in Section 2.3 there is a strong suggestion that there are three categories of imperceptible-, perceptible-but-unconscious-, and perceptible-illuminance effects related to circadian rhythm, that are affected by personal characteristics with the middle being generally unknown or overlooked.In other words, therefore, the objective in this paper is to reveal at least one phenomenon connected to perceptible but generally unconsciously occurring illuminance effects deemed to have statistical significance in psychophysical experiments.The objective in this study is not to fully reveal the relationship between visual characteristics related to diversity in circadian preferences, either.
To make clear the definition or extent of visible circadian effects, we use the new term of individual-circadianvisual-features (ICVF).We also stop directly using the terminology of "circadian rhythm" used in our previous study, 38,39 to avoid confusion as suggested by reviewers' comments.
Therefore, we examined the remaining mysterious effects by psychophysical tests.To this end, we had to create precise global tone curves (see Section 4 and Figure 10) and conduct a formal experiment.

| EXPERIMENT 1: INDIVIDUAL DIFFERENCES IN PERCEPTUAL TONE IN RELATION TO IMAGE CAPTURE TIME AND DIFFERENCES IN DISCRIMINATION ABILITY
This experiment was triggered by some discussions by the two authors and others as mentioned in Section 2.3; the difference in their supporting NVP curves, that is, CD and DR, in the morning outside viewing environment. 38,39The aim of this experiment was to confirm these differences.

| Experimental environment
A 24-in.LCD sRGB color display (EIZO ColorEdge CS2410) was employed in a light room with natural ambient LED lights, which replicated normal illuminance of approximately 300 lx at the display surface.Display resolution was 1920 Â 1200.Maximum white luminance was approximately 120 cd/m 2 , with approximate contrast ratio of 100:1 or more.Viewing distance was approx.60 cm.

| Stimuli
Nine kinds of images taken in either early-morning or late-afternoon for EM/LA discrimination experiments, and nine kinds of images taken in either morning or afternoon for MG/AN discrimination experiments were selected.
All original images were taken by Nikon Digital SLR cameras (D500 or D800E) with 14-bit luminance resolution (i.e., HDR raw images), and SDR images for evaluation were converted from HDR by using our proposed method (i.e., CD, SN, or DR for MG/AN, and CD + ST, SN + ST, or DR + ST for EM/LA). 15

| Participants
A total of 24 subjects participated (17 males and seven female) ranging in age from 20-to 60.All were confirmed as CNOs by Ishihara tests.

| Actual image capture time versus perceived time
Analysis of the paired comparison test results revealed that there no observers were able to directly estimate actual capture time.Here, actual time means that the observer could discern that the capture time was "morning" independent of which NVP processing method was used (i.e., CD, SN, or DR), if actual capture time was in the "morning."It appeared that they judged, or more precisely described, the captured time based on the difference in tone mapping curves (i.e., gradation).

| Comparison of two kind cluster analyses
We subjected the paired comparison test results to two cluster analyses.First, observer classification based on discrimination ability was examined.In this case, the input data of each observer was created based on significant difference between methods A and B (1) or no-significance (0).The other formed classes of observers based on favorite tone curve.In this case, the input data of each observer was created based on significant preference for method A (+1), no significance, method B was significantly preferred (À1).Analyses were conducted using the Ward method and a free library. 40igures 6-8 show the analysis results.In summary, two cluster analyses suggest that (1) gradation-sensitive and insensitive groups were generally separated, but their distribution was spectrum-like, (2) there were individual differences in significant tone-mapping pairs, (3) especially in early-morning and/or morning conditions, two classes of individuals existed, those who preferred highor low-contrast, which reflected the difference in individual daily visual characteristics.

F I G U R E 6 Cluster analysis result 1 (by binary state).
F I G U R E 7 Cluster analysis result 2 (comparison to Figure 6).

| EXPERIMENT 2: LARGE SCALE SURVEY OF PERSONAL DIFFERENCES
Following the results of Experiment 1, we carried another experiment exploring perceptible illuminance effects in ICVF, observing daytime images relatively closer to noon (i.e., not early morning or not late afternoon), that have similar physical properties, using participants with a greater variety of attributes. 41,42One of the main purposes was confirming the characteristics of the so-called "Night owl type," usual students in the younger generation.

| Capturing new images: Reflecting a sense of seasonality for easy judgments
As Japan has four distinct seasons due to the Asian monsoon, a sense of seasonality is important to Japanese in terms of their perception of the outside world.Our previous study showed that people could not perceive differences in reproduced image tones when they had insufficient experience of landscapes in the real world (e.g., snowy landscape or autumn leaves in case of Kagoshima). 15herefore, we selected a clear sunny day in early spring (late March) that was not contaminated by PM2.5 and took images by a Nikon Digital SLR D500 (captured by 14 bit raw) for subjective tests.The visibility was more than 40 km.Mt.Sakurajima, which gives Kagoshima residents a sense of direction, was included in some of the images.Images are shown in Figure 9.

| Deriving new precise tone curves for examining perceptual differences in sunny day time
Considering the precise reproduction of luminance gradations from the HDR daytime/outdoor environments with extreme highlights to SDR images (i.e., soft clipping), we re-created a series of five new tone curves (TC-1 to TC-5; See Figure 10) with minor revisions from previously used tone curves (CD, SD, SN, DR, and NM). 15 Figure 11 shows examples of the images so reproduced.The curve order reflect progression in contrast, where smaller numbers have higher contrast.This work was only performed by only the first author.The working environment for retouching used indoor fluorescent lighting, where illuminance was approximately 200 lx.An sRGB 24-in.monitor (EIZO ColorEdge CS2410) was used for image display.Because the contrast characteristics of the images perceived on the monitor itself varied significantly depending on the time of day (i.e., it is suggested to reflect the influence of ICVF), final confirmation of the results was conducted in the early afternoon when visual characteristics are expected to be empirically stable.

| Conversion from self-luminous display image to illuminated reflection image for observation in normal living space
We needed both the displayed images and the printed images to be available for participant observation.Here, images were printed on photo-matte paper to prevent glare by specular reflections.The difference in visibility, that is, perceived colors, between self-luminous displays and printed media (illuminated reflection print) has been discussed often.With recent advances in color management technology, it is believed that valid harmonization of visibility under standard viewing conditions can be fully realized.
First, therefore, we confirmed that there are usually no problems in using color management (sRGB/ICM) in the inkjet printer environment used in this study (EPSON EW-M873T, the company's genuine photo-matte paper).Concretely speaking, we compared the print result in this configuration with a print and an original (TIFF, 16-bit) of portrait made by a reliable commercial photo studio that we had on hand and found no problems.In other words, it was judged that "the intention of the creator under the monitor work was correctly reflected in the printed result" in the standard environment.
However, it should be noted that "the environmental conditions during image capture (sunny outdoor HDR; illuminance of approximately 10 5 lx) and reproduction (indoor SDR; illuminance of approximately 10 2.5 lx) are quite different" as regards the tonal reproduction issue we are dealing with here.For this reason, although details are omitted, additional slight correction via global tone curves, that were different between AM and PM, was necessary at the time of printing (Details were reported at a conference 41 in Japanese and an overview was also reported in IDW2022 42 ).It is considered that the main reason was as follows.Unconscious locus of attention in observer's mindset probably slightly varied with the assumed time (i.e., virtual-shooting time; VST), AM or PM.Related to this large difference in viewing conditions, it is known that further partial dynamic-range compression in luminance tone for printing is required, as was discussed in our previous study. 15In addition, as a result of careful observation in this study, we found that the compression processing differed in relation to the VSTs.In other words, relatively brighter parts were important in the AM, while relatively darker parts in PM were important, although general observers might not be aware of this difference because this shift was quite slight.
Next, a comparative experiment was conducted to verify the validity of this print correction.The subjects were three co-authors (Iwaida, Hira, and Kashima: they were not informed of any retouching or other information at the time of their participation in the experiment).The subjects' task was to select the one of the five images that was closest to their own perception under conditions that allowed repeated checking (in the case of A4 prints, simultaneous comparison of all images, See Figure 12) with picking them up and without any restrictions on viewing distance.Three images were used, each captured in the morning and afternoon (as shown in Figure 9).An analysis of variance revealed no main effects other than those of the presented images, indicating that the print correction ensured a certain degree of universality and that it functioned appropriately. 41I G U R E 1 1 Demonstration of changing Tone (#101: captured in morning, and #201: captured in afternoon).

| Participants
All participants (total number of 29; 20 males and nine females), whose ages ranged from 19 to older social adults (we did not acquire accurate ages to ensure privacy), were confirmed as Color-normal-observers (CNOs) by Ishihara tests.They were separated into three groups: (a) Group-1; college student group living in Kagoshima area, ages under 26, (b) Group-2; social adult group living in Kagoshima-area, including above-mentioned three authors, and (c) Group-3; social adult group living in Kanazawa area, with ages over 28.The number of participants was 9 (Group-1), 12 (Group-2), and 8 (Group-3).

| Evaluation methods
Since the reality of the sense of season was very important in the evaluation process, (1) for the participants living in Kagoshima, the date and time of shooting were given as is, and (2) for those living in the Kanazawa area in Ishikawa prefecture (Northern central of Honshu), the arrival of spring was corrected (equivalent to late April, not late March) and the time concept was corrected (about 1 h earlier) due to the longitude shift against Japan standard-time, so that the sense of season could be reproduced.The task was performed in accordance with the two tasks described above.The participants' task was the same as that described in Section 4.1.3.Here, note that they were required to report "no choice" when there was none of the five images was suitable.The experiment was conducted in the early afternoon (generally from 1:00 p.m. to 6:00 p.m. in case of Kagoshima; the time when it is sufficiently bright outdoors in July), and all except one participant evaluated images on printed media.

| Experimental result (1): Cluster analysis based on participants' attributes
Table 1 shows the main attributes of the participants and their judgments, along with the results of a cluster analysis using Ward's method (See also Appendix Figure A1).Table 2 shows the results of a regression analysis using three explanatory variables: the amount (absolute value) of difference in average TC number between the morning and afternoon, the direction of change, and the mean value of overall tone, to determine the differences in the characteristics of the clusters.Figure 13 also shows the differences in perceived contrast change from AM to PM between clusters.Although a detailed discussion is omitted for reasons of space, we provide a summary as follows: (1) C1 and C2 differ in mean value, with C2 maintaining contrast characteristics similar to those at night even during the day (i.e., always low contrast; confirmed for the first time in this paper), (2) C1-1 and C1-2 differ in direction of change, with the latter being standard "Early bird type" as previously considered, (3) the former (C1-2) is classified into two groups (C1-1-1 and C1-1-2) by mean value difference, where C1-1-1 belongs to standard "Night owl type" as previously considered (low contrast in the morning and high contrast in the afternoon), (4) the latter (C1-2) is classified into two groups (C1-2-1 and C1-2-2) depending on the slope (confirmed for the first time in this paper), and (5) the latter (C1-2-2) always maintains high contrast (also confirmed here for the first time).
Here, note that several participants belonging to the C1-2-2 group voluntarily said that they had difficulty getting up in the morning during their youth (at that time, they felt very dazzled in the morning), but that they adapted to a morning routine after entering the workforce (suggesting a transition from C1-1-1 or C2 due to their social life).

| Experimental result (2): Differences in the composition of students and working adults in each cluster
Next, we examined the ratios of students and working professionals in each cluster.Table 3 shows the breakdown.As a whole, there was no significant difference between the ratios of students and workers in each cluster (χ 2 test).However, a significant difference (p < 0.05) was found between C1-1 and C1-2 by Fisher's exact test.In other words, the result of the analysis suggests that the proportion of students is lower in the "Early bird type" clusters, which is consistent with the common knowledge that "in the case of students around 20 years old, many reverse day and night due to late night activities."This tendency was also suggested by a recent analysis of DLMO distribution across age. 27

| Discussions
First, although many participants were torn between the two or three choices, no participants reported "nonselectable."This suggests that it is possible to select one from about five tone curves (such as those used in the experiment) if the experiment employed outdoor landscape images in sunny conditions during the daytime.In all cases, total dynamic range compression rate from HDR (≈10 3.5 :1) to SDR (≈10 2 :1) was approximately 1/30 (i.e., ≈2 5 = 5 f-stops) as shown in Figure 10.This value was for the case of display observation; the compression rate was generally higher for the print material. 41,42The difference in each was exposure value and contrast in T A B L E 3 Comparison of ratios between students and working professionals in each cluster.half tone areas, which are considered to be the general locus of attention for the observers.In other words, it is suggested that the human visual system employs enough large information compression in the first stage before scene understanding where the system needed local processing as generally known from the illusions, as mentioned earlier, even in normal daytime viewing.Note that this "dynamic range compression employed global-tonemapping" itself is also an "illusion" because the reproduced SDR image where reproduced color was changed dynamically (i.e., extent was different in each part) from physically correct SDR image (see Section 2) felt natural when viewed in a room environment.
In conclusion, the above results clearly show that there was no need for additional local tone mapping, which validates our hypothesis, that "Global-, not" "Local-," tone-mapping in the retina is important.
Second, the results from analyzing the attributes of participants (see Table 1, Figure A1, and Figure 13) showed that (1) various variations existed not only in the morning, which had previously been prominent, but also in the afternoon, and (2) it was difficult to simply ascribe individual differences to current lifestyle, and it can be presumed that these differences are actually based on the cumulative results of lifestyle habits from childhood.
Finally, Table 1 (right columns) shows that there were differences among observers as to the sensitivity to acquire image-independent tone mapping properties.Several participants had quite high accuracy (p < 0.01; 5 À4 ) in judgment in both AM and PM cases, whereas most participants had relatively lower accuracy.This supported the results of Experiment 1.

| GENERAL DISCUSSIONS
First, the most important thing to be discussed is to answer the question "why Global-and Local-TMOs are still competing in current image processing technologies even though the former is essential" (see Section 2.2) based on the results of recent experiments.
It was suggested that the reason for their "equivalence," is that Global-TMOs are dominant in the retina Local-TMOs active in the brain, so the association is collaborative is competitive.The result of Experiment 2 proved that the latter was not mandatory for display image processing (i.e., [1] no participants said "non-selectable" for any of the presented images and [2] sensitive observers could discriminate six different images consecutively by using only global-tones in five-choice, not the contents).In other words, there were no problems even though Local-TMO processing was active only in the brain.
This tandem structure of the human visual system offers not only very efficient and high-speed information processing, but also significant reductions in physical energy consumption.In other words, it is considered that it was huge contribution to winning the competition for survival because the brain, especially visual system, is the most energy consuming organ as is well-known.The issue of energy consumption saving is elucidated as follows.As you know in computer technology, global tone mapping can be implemented using only LUT (Lookup table) access with very little latency.On the other hand, local tone mapping requires high performance parallel processing techniques like GPU (Graphics Processing Unit) to achieve realistic processing speeds, which incurs high power consumption.This argument makes it clear how the high energy efficiency of human eyesight is achieved, and the system honed by evolution is highly sophisticated in that it integrates saccadic eye movements with stable and effective information acquisition from the wide viewing field.
In conclusion, it is suggested that superficial level perception is unable to separate either global or local processing results even though the processes have distinctly different physical origins, because our final perception is obtained after both processes are completed.The results also suggest that engineers should learn more about human information processing.In addition, it is considered that the role of the hypothalamus, which connects both retina and early vision, should be researched in more depth from both physiological and psychological aspects (this is not here due to space constraints).
Second, it is considered that information display systems can be made more sophisticated to address ICVF and other personal characteristics.This is in contrast to our previous understanding that the individual's image preferences were randomly determined, therefore the control system had to employ separate parameter settings like brightness, contrast, and color temperature.However, as mentioned above, the first stage process establishes systematic human characteristics.Therefore, it is considered that we will be able to improve color control in consumer display systems by developing simpler and more systematic methods that reflect the knowledge newly identified in this paper.

| CONCLUSIONS
Perceptually natural SDR images reproduced under normal viewing conditions should retain enough information, without creating artificial (i.e., not enhancing the real, but replacing the false) information, for the human observer to estimate the times at which the actual HDR scenes were captured.Here, it is considered that there are some individual differences in terms of visual perception in HDR environments because people have many types of images preferences as is generally known.
TMOs are now being used to compress HDR into SDR images and many types of methods employing either global-or local-TMOs have been proposed.Current technologies assign equal importance to global-and local-TMOs, even though their basic mechanisms are quite different; the reason for this equivalence has not been adequately addressed up to now.
To rectify the confusion, we first analyzed the constraints in the human visual system accompanying eye movement and introduced a hypothesis with a demonstration (see Sections 2 and 3).In conclusion, it was suggested that light-weight and switchable-predefined global TMOs, named NVPs (Normalized visual percept), part of our hypothesis, had an essential role in image compression at the first stage (i.e., alternative role of real processing in retina when viewing real HDR scenes), resulting that the possibility of skipping later local processing for image enhancement (i.e., that carried out only by the brain).It was also suggested, from the viewpoint of psychology, that top level perception could not separate either global or local processing results even though the processes were had diametrically different physical meaning, because our final perception is obtained after both processes cooperated for efficient scene understanding.Note that the result from the viewpoint of psychology was the same when local processing results included false-replaced information, by analogy of "Where's Wally?".Another important result is that the concept of "dynamic range compression via global-tonemapping" is itself a newly observed "visual illusion," because we felt natural (i.e., perceptually truth) when we viewed, in a room environment, NVP-processed SDR images whose reproduced color was changed locally (i.e., color changing status differed part by part depending on luminance) from physically correct SDR images (see Figure 2).
We then reviewed the imperceptible illuminance effects in circadian rhythm individuality suggested by physiological research briefly, because our previous study suggested that the characteristics of the human visual system dynamically depended on the individual's circadian pattern.
Finally, we conducted two psychophysical experiments based on the hypothesis that human visual system employs several global TMOs at the first stage for information compression depending on both individual-circadian-visual-features (ICVF) and other personal characteristics.The results suggested that (1) no participants could perceive actual-capture-time (ACT; i.e., this result suggested control only by reproduced luminance tone) and (2) sensitive observers could discriminate reproduced images based on virtual-shooting-time (VST) which was induced by different types of Global-TMOs (surprisingly, they could discriminate six different images consecutively by utilizing only global-tones, not contents, even in five-choice).We also discovered that the VSTbased luminance tone discrimination differed widely among people but most were unaware of this effect as evidenced by daily conversations (an example was shown in GTOC).It suggested that they significantly depend on individuality, that is, personal characteristics.
Applying this work to display systems will make it possible to create enhanced information display systems that respond to circadian rhythm and personal characteristics.More specifically, it is considered that color control methods in consumer display systems can be improved in a simpler and more systematic manner by reflecting the newly obtained knowledge.
Our experiments strongly suggest the existence of a new research field important to both academia, including healthcare, physiological and psychological aspects (e.g., research field of sleep, circadian rhythm, and visual illusion), and industry.

2 | 2 . 1 |
GLOBAL PROCESSING IS ESSENTIAL IN THE FIRST STAGE OF HUMAN VISUAL SYSTEM PROCESSING: PREPARATION FOR NEW EXPERIMENTS THAT ADDRESS DIVERSITY IN HUMAN CIRCADIAN CHARACTERISTICS Data compression model based on processing speed constraint: The path from retina to early vision

F
I G U R E 3 Experimental result of pairwise comparison (Scheffé's method modified by Nakaya).F I G U R E 4 Experimental results of multidimensional scaling (MDS) for checking individual differences.

F I G U R E 5
Example of images reproduced using CD, SN, and DR normalized visual percepts (NVPs).

F
I G U R E 8 Cluster analysis result 2 (summary).

F I G U R E 1 2 2 *
Experimental setup for printed materials (direct incident light from desk lamp was eliminated).T A B L E 2 Results of a regression analysis using three explanatory variables.Analyzed the significance of variables only if the regression variation is significant.*p < 0.05, and **p < 0.01.

1 3
Average perceived contrast in each cluster in the morning and afternoon.