Modelling and automated calibration of a general multi-projective camera

Recently, multi-projective cameras (MPCs), often based on frame-mounted multiple cameras with a small baseline and arbitrary overlap, have found a remarkable place in geomatics and vision-based applications. This paper outlines the geometric calibration of a general MPC by presenting a mathematical model that describes its unknown generic geometry. A modi ﬁ ed bundle block adjustment is employed to calibrate an industrial-level 360 ° non-metric camera. The structure of any MPC can be retrieved as a calibration set of relative and interior orientation parameters (as well as the pose of the MPC shots) using a calibration room which has been accurately determined by close range photogrammetry. To demonstrate the ef ﬁ ciency and precision of the model, a Panono camera (an MPC with 36 individual cameras) was calibrated. After the adjustment, sub-pixel image residuals and acceptable object-space errors were observed.


Introduction
WHEN WISHING TO VIEW MORE of the surrounding environment, a multi-camera system (MCS) offers great potential for geomatics instrumentation, robotics, car navigation, entertainment systems and even space applications (such as spacecraft docking navigation systems).Technology based on multiview geometry is inexpensive, accessible and highly customisable.A fixed-structure MCS consists of a set of cameras mounted on a solid platform and is generally simple and flexible enough to focus on the regions of interest.Today, MCSs exist using a wide variety of platforms, some as simple and cheap as 360°c ameras and small robots, others being expensive autonomous cars or complex satellite structures.
Recently published literature on MCSs highlights a wide range of real-world and futuristic applications.For example, robotic navigation by stereo vision was proposed by many researchers such as Grosso and Tistarelli (1993), Desouza and Kak (2002) or English et al. (2014); a multi-camera-based indoor navigation system was successfully demonstrated by Svoboda et al. (2002); Se et al. (2007) proposed a stereo-camera-based 3D modelling of space structures; a multi-camera-based outdoor simultaneous localisation and mapping (SLAM) was demonstrated by Schleicher et al. (2009); a stereo-camera tracking algorithm was successfully tested by Gasparini and Bertolino (2013); multi-camera-based navigation for autonomous cars was discussed and demonstrated by Paracchini et al. (2016); and submarine visual tracking by a stereo camera was shown by Pfingsthorn et al. (2016).
The MCSs that widely appear in recent literature can be categorised according to the architecture of the camera system; for example, categorisation based on the type of lenses incorporated in an MCS: (1) multi-omnidirectional cameras (MOCs), where the field of view is a full 360°; (2) multi-projective cameras (MPCs), where the component cameras are more conventional, having a restricted field of view; and (3) hybrid multiple cameras, with mixed lens mechanisms (for example, including both projective and omnidirectional lenses).
Note that, in this paper, conventional imaging systems which have a restricted field of view are termed projective cameras.It is very logical to have models that either work solely on one category or on more generic models.Many combinatorial cameras, employing different capturing domains, fit into the MPC category, such as multispectral projective cameras or thermal projective cameras.The first two categories simplify the foundation of a calibration process, because both categories only include combinations of the same type of camera, which therefore share a significant degree of calibration modelling, such as interior orientation parameters (IOPs).In contrast, the third category includes different combinations of sensors or camera types on a common platform for image capture.Since an MPC places a limit on the type of lens, a suitable model consequently describes all similar MPCs.In this work, the focus is only on the calibration modelling of a general MPC.
Single-frame Cameras and Stereo Cameras MCS n (n = 2) For many decades, conventional single-frame cameras (SFCs) and fixed-base stereo cameras (two cameras, often mounted on a bar with a known separation) (MPC 2 or MOC 2 ) have been successfully employed in both close range photogrammetry and computer vision applications, mainly because of their design simplicity and ease of calibration.Many diverse applications of stereo cameras have been proposed in the literature.Recent applications of SFCs and stereo cameras may be classified into three main categories: (1) 3D structure-and-motion estimation, for example, dense reconstruction (Furukawa and Ponce, 2007;Furukawa et al., 2010), sparse bundle adjustment and Internetbased reconstruction (Snavely et al., 2008), space structure estimation (Se et al., 2007), incremental structure from motion (Wu, 2013), stereo panoramas (Amini et al., 2014) and dual fluoroscopy imaging system calibration used for bone reconstruction (Lichti et al., 2015).(2) Navigation systems, for example, indoor localisation (Svoboda et al., 2002), stereo tracking for dental surgery (Wang et al., 2014), car trajectory estimation (Paracchini et al., 2016), underwater navigation (Pfingsthorn et al., 2016) and spacecraft precise docking (Tweddle, 2010).
In most above-mentioned applications (such as indoor positioning), MCSs offer the unique possibility of rapidly capturing 360°shots.The variety of commercially available sensors makes it feasible to measure several spectral bands of data as SFC output.A rigid body usually ensures that the relative locations of cameras (with respect to a local coordinate system) are fixed, at least during a measurement period.A brief look at the development of recent imaging systems reveals that MCSs are rapidly growing to become a leading imaging technology in many diverse areas.In order to achieve the best geometric use of a camera, the applications upon which the system is founded need to be carefully considered.However, because of the shared properties of MPCs, a universal paradigm could be defined for the calibration phase.

Multi-camera Systems
Multiview systems, where MCS n (n > 2), are usually developed for applications for which two-image stereo vision is not flexible enough to capture the environment surrounding a moving platform.A few examples of such cases are 360°multi-camera panoramic imaging sensors, vehicle autonomous navigation cameras, indoor cameras and multiview reconstruction.
Based on applications that an MCS n (n > 2) is designed for, cameras are mounted in a wide variety of geometric configurations; therefore, a system calibration method needs to be flexible enough to work with any configuration.In recent literature, two main configurations have been considered for a multiview MCS n (n > 2): (1) The cameras are mounted solidly on an object or a piece of apparatus (such as the walls of a building or a multi-camera rig).
(2) The cameras are mounted on a solid, but movable, frame (for example, a multicamera semi-panoramic system).
Since the topology and dynamic properties are different between these two groups, different calibration paradigms need to be defined for each group.In this work, the focus is on MPCs of the second group, where a movable MPC consists of a set of cameras fixed on a frame (usually pointing outwards).

State-of-the-art of MPC Calibration
A standard bundle block adjustment (BBA) has been proven to be an efficient tool for metric single/stereo camera calibration (for example, Granshaw, 1980;Fraser, 1997Fraser, , 2013;;Gruen and Huang, 2001).A photogrammetric BBA was then used to calibrate non-metric cameras in a very cost-effective manner, for example, Triggs (1998) proposed an automatic calibration approach coupled with planar coded targets for projective cameras.His method was able to run the calibration with five or more image exposures.His only assumption was that the IOPs and the structure remained fixed during measurement.Zhang (2000) introduced a planar chequerboard coded target to calibrate a projective camera, employing a corner detector to automatically find the coded target in an image set.A few images from different angles were usually enough to run the BBA.Bouguet (2011) later modified Zhang's toolbox to accept stereo cameras.Both these methods have demonstrated deficiencies when it comes to comprehensive, scene-independent photogrammetric (as opposed to computer vision) calibration.Camera calibration has been undertaken for many decades; Clarke and Fryer (1998) provide a useful historical overview from a photogrammetric standpoint.Some very early attempts to calibrate a stereo camera from a computer vision perspective appeared in works such as those of Zhuang (1995), who used distance measurements of a fixed-length moving object such as a utility pole to find the extrinsic (external orientation) parameters of a stereo camera.Using a different methodology, Lerma et al. (2010) employed distance constraints between camera perspective centres to calibrate an MPC 3 consisting of a stereo camera coupled with a thermal camera, using a rigid MPC body as a part of a multi-sensor system.In their work, high accuracies were achieved only by employing a full set of baseline distance constraints.Svoboda et al. (2002) used an easily detectable moving object (a laser track pointer) to calibrate a set of cameras that were fixed in an indoor environment.Their goal was to find interior and exterior orientation parameters (IOPs and EOPs) of a set of cameras that was specifically designed for the indoor localisation of moving objects.In the MPC n of Svoboda et al. (2002), the distances between cameras were relatively large, and the MPC n was supposed to stay geometrically fixed during the operational period of the localisation system (see the toolbox of Svoboda et al., 2011).Tommaselli et al. (2014) designed a terrestrial calibration field where 139 AURUCO coded targets (Garrido-Jurado et al., 2014) were installed on the floor and side walls of a room, modifying the collinearity equations to accept their proposed fisheye lens model.
The variation of IOPs and EOPs of a stereo camera and an MPC was reviewed by Habib et al. (2014).Modified collinearity equations for an MPC by enforcing the relative pose (position and orientation) of all cameras with respect to a reference camera/frame have been demonstrated, for example, in works of He et al. (1993), Tommaselli et al. (2013) and Habib et al. (2014).This formulation is listed in Habib et al. (2014) as a "one step procedure" where invariant relative orientation parameter (ROP) enforcement is applied at different times, and it "intensifies with the increase of the number of cameras"; however, they used the modified collinearity formulation for stability analysis of an experiment with an MPC.In their work, mean and standard deviation values of time-dependent calibration parameters were directly calculated from time series.Li et al. (2013) proposed an algorithm that was able to detect a partially visible chequerboard coded target.Consequently, their method was usable even for cases when the field of view of neighbouring cameras did not overlap.They mainly focused on projective and catadioptric cameras (that use a combination of lenses and curved mirrors) MPC n , and used geometrically known coded targets to connect neighbouring cameras, under the condition that no overlap exists.In their paper, an MPC 4 of projective cameras on a rig with a separation angle of 90°was calibrated.Urban et al. (2017) presented a BBA approach (called MultiCol) to calibrate an MOC n .They simulated an MOC 15 with a few randomly placed cameras in order to show the usefulness of their method and implementation.
The lack of a uniform photogrammetric paradigm for a multiview MPC n (n ≥ 2) is understandable.Diverse technical language has been used in recent literature.The most significant literature on this topic could not be seen as sufficient and general enough for the MPC n studied in this paper (Svoboda et al., 2002;Li et al., 2013;Habib et al., 2014;Urban et al., 2017).In some cases, the focus is limited to the special condition that a specific MPC/MOC is designed for (Svoboda et al., 2002).This current work, however, tries to continue the aforementioned work in terms of photogrammetric formulation integrity, as well as the practicalities that are involved in a real multiview calibration process with an MPC n (n ≥ 2).In this paper a modified BBA approach is demonstrated for calibrating a general MPC, based on the customised collinearity equations defined by He et al. (1993), Tommaselli et al. (2013) and Habib et al. (2014).The proposed formulation is employed to improve the methodology stated by Li et al. (2013) and Urban et al. (2017).The main improvements over this previous work are: (i) ease of calibration through automation; (ii) error propagation based on a standard statistical model; and (iii) formulating the problem as a standard photogrammetric paradigm.This work demonstrates an efficient, practical and precise methodology based on non-linear least squares estimation for calibrating a general MPC.
The method presented in this work is based on the following six factors: (1) Taking a set of images from a calibration room filled with coded targets with an SFC to precisely retrieve the positions of the targets and the corresponding uncertainties by employing a minimum-constraint BBA.
(2) Taking a second set of images of the calibration room with an MPC.
(3) Executing a minimum-constraint BBA for individual cameras of the MPC.(4) Approximating an initial structure for the MPC and the initial pose (locations and orientations) of images by combining the output from the individual BBA runs.(5) Enforcing the structure of the MPC in a standard BBA as relative orientation constraints within the collinearity model to build a customised BBA for a general MPC.(6) Discovering the internal structure of the underlying MPC, IOPs of individual cameras, positions and orientations of the MPC images by running the customised BBA whilst assuming the coded target locations fixed.
The structure of this paper is as follows.Theoretical aspects are described first, followed by specifications of the hardware (such as the calibration room and cameras).The method is presented subsequently.The results are demonstrated and discussed afterward, and finally the paper is summarised.Symbols and conventions that are used in this paper are listed in the Appendix.

Theoretical Aspects of the MPC Calibration
In this section, the required technicalities are expanded in photogrammetric terminology to address the MPC calibration problem.The intention is to keep the formatting standard and consistent with well-known photogrammetric work, such as that of Gruen and Huang (2001).This section starts with basic stereographic formulations and leads to a complete MPC calibration paradigm.

Coplanarity and Collinearity Equations
The coplanarity and collinearity equations are fundamental to photogrammetric adjustment.The coplanarity equation expresses an ideal assumption that, in a single stereopair, an object point, its two corresponding image points in the stereopair and the perspective centres (nodal points) of both camera positions lie on a plane.In contrast, the collinearity equations are defined for an individual image, whereby an object point, the perspective centre and the corresponding image point (on the image plane) lie on the same straight line.
Fig. 1(a) depicts a stereopair of an SFC whereby each image is captured with the same sensor but at a different time.The local coordinate system of an image is demonstrated in Fig. 1(b).In this work, it is assumed that the origin of the local coordinate system of an image is the perspective centre, where the x and y axes are parallel to the image plane, and The Photogrammetric Record Coplanarity Equation.The coplanarity condition, as depicted in Fig. 1(a), can be formulated as a vector product in the following form: where (X 0 ) ti is the position vector of the camera at time (ti).In equation ( 1), the products are the inner product (.; also termed the dot or scalar product) and the outer product (9; also termed the cross or vector product), respectively.Equation (1) could, equivalently, be rewritten as a matrix product: where [(X 0 ) t2 ] x is a 3 9 3 matrix of the form: For constructing a network of images with unknown locations and orientations, the coplanarity equation can be helpful in situations where the collinearity equations could easily erroneously converge (get stuck) to local optimums if the initial values are not close enough to global optimums.
Collinearity Equations.After finding sufficiently close approximations of the pose (location and orientation) variables, the collinearity equations play an essential role by removing all the unnecessary constraints and connecting all cameras throughout a uniform model.If we assume a simple central perspective (pinhole) camera model with a local coordinate system that looks downward, the collinearity condition is formulated as the following: X ¼ K:R ðx;/;jÞ In equation ( 4), Λ is an unknown scale factor and R ðx;/;jÞ t is the Euler rotation matrix of a camera at a certain time t.Using the above notion, an essential matrix E is defined as: If the unknown scale (Λ) in equation ( 4) is removed, observation equations are generated that will be employed in the body of the BBA: where X j is the corresponding object point for the image point x j , and M t ¼ R À1 t .In equations ( 4) to ( 6), perspective centre (pinhole) coordinates X 0 (Fig. 1) are used.The lens distortion and scale factor inherent in a real optical system are considered as a non-linear function that connects the actual pixel coordinates to corrected image coordinates (Brown, 1966;Fraser, 1982;Gruen and Huang, 2001): Rad is the radial term containing the symmetric radial distortion coefficients (K 1 , K 2 , K 3 ); P 1 and P 2 are the decentring distortion coefficients; (PP x , PP y ) and f are the location of the principal point and the principal distance in pixel units, respectively; and d, k are scale and shear factors, respectively.

The Normalised 8-point Algorithm
In this work estimating an essential matrix is achieved using the normalised 8-point algorithm (Hartley, 1997).If it assumed that x t 1 is the location of an image point in frame (t 1 ) in pixel units, this point is transferred to the corresponding corrected perspective centre (pinhole) image coordinates x t 1 by using equation ( 7).For a set of normal image points, normalised "normal image points" (forming normalised image points: see Hartley, 1997;Hartley and Zisserman, 2004) are calculated based on the centroid of the points and scaling to one: where M t 1 is the mean of the normalised image points and S is the scale normalising term.The coplanarity equation is therefore rewritten in the following form: The Photogrammetric Record The estimated essential matrix, E ðt 1 ;t 2 Þ , encodes the relative orientation connection between these two image frames.This equation is equivalent to a linear system of equations: One solution to the above equation is retrieved by finding the singular-value decomposition (SVD) of A. The estimated E is finally decomposed into a relative rotation matrix ½R t 2 : R T t 1 and a positional vector The Self-calibrating BBA of a Multi-projective Camera A single image of an MPC n is represented in this work by the following parameters: (1) Inner orientation parameters: (2) Relative orientation parameters: the first set of parameters f; g; w ð Þ i are the relative Euler angles of the ith sensor with respect to the first sensor in the MPC n , and the shift vector D x ; D y ; D z À Á i is the relative position of the ith camera with respect to the first camera in the local coordinate system of the MPC n .
(3) Pose: The location and orientation of the frame at the moment of image capture (R t ; X 0 ð Þ t ) with respect to the local coordinate system.
Therefore, the number of calibration parameters (nCalib) for an MPC n is: The results of "The Normalised 8-point Algorithm" section provide an initial estimation for the pose (locations and orientations) of a set of cameras with respect to a local object coordinate system.The next step is to find a more stable estimation by applying the least squares approach.The main equations that are employed for this are based on the collinearity of an object point, a perspective centre and the corresponding image point.Since a given object point potentially emanates to image points in multiple photographs, much better estimations are feasible compared to coplanarity estimations.Collinearity in its basic form (equations ( 4) and ( 6)) provides a non-linear function of the calibration parameters, object points and corresponding image points.In an MPC n , the observation equation ( 6) can be rewritten as: where M i,t = R i,t À1 and R i,t is the rotation matrix of the ith camera at time t when the orientation of the frame is x; /; j ð Þ t .The calculation of R i,t uses: In a similar way D i,t is defined as: where X 0 ð Þ t is the location of the MPC n at time t in object space.Subsequently, equation ( 14) boils down to a non-linear function of the form: F ðf ; PP; K; P; d; kÞ i¼1:n ; R ðf;g;wÞ i¼2:n ; D i¼2:n ; R ðx;/;jÞ t¼t 1 :t m ; X 0 ð Þ t¼t 1 :t m ; X ð1:nOÞ ; x ð1:nIÞ ¼ 0: ð15Þ The modified collinearity equations ( 12) enforce the internal structure of an MPC into collinearity equations representing the relative position and orientation of projective cameras with respect to the local coordinate system of the first camera in the MPC.Subsequently, equation ( 15) combines all non-linear observation equations to be the basis for the least squares estimation.As a result, the output reflects the structure of the MPC in addition to the inner orientation parameters, and positions and orientations of the camera stations.

Bundle Block Adjustment
In this work, the statistical model used for non-linear adjustment is based on the combined adjustment method that was discussed by Wells and Krakiwsky (1971).Primarily a set of non-linear functions is considered: where f is m non-linear functions of the observations l and unknowns x.A first-order Taylor expansion of f around an initial approximation is then expressed as: where df dx is the partial derivative of f with respect to x (Jacobian matrix with respect to the unknowns) and x 0 ; l l0 ð Þ are the initial estimations of the unknowns and observations.After linearisation, the mathematical model of the combined method comprises of m observation equations that can be expressed as: where A is the Jacobian matrix with respect to the unknowns, B is the Jacobian matrix with respect to the observations, v is the residual vector of observations and w is the misclosure vector: Either an analytical or a numerical estimation of the Jacobian matrix could be fed into equation ( 18); here, the latter form is formulated and employed.The optimisation step is formulated as: A variation function is defined, based on Lagrange coefficients, to carry out a least squares estimation.In the above system of linearised equations, the least squares estimation of the unknowns is a Newton step toward the location of a local extremum: where P is a positive definite matrix of observation weights.Sparsity in the Jacobian matrices A, B and weight matrix P of equations ( 18), ( 20) and ( 21) are employed to facilitate the weighted non-linear least squares solution, otherwise a considerable amount of memory is required to solve the adjustment.

Materials
In this section, different aspects of the experiment's materials, such as specifications of the calibration room, the SFCs and the MPC, are given.

Single-frame Cameras
In order to determine the geometry of the calibration room, an SFC was employed (Fig. 2(a)).The camera used was a Canon EOS 6D digital single-lens reflex (DSLR) camera, with a 20 Mpixel (5472 9 3648) complementary metal oxide semiconductor (CMOS) detector, a Canon EF 24 mm f/2Á8 IS USM lens with a principal distance (calibrated focal length) of 20Á650 mm AE 2 lm and a field of view (FOV) of 81Á82°AE °10 (Fig. 2(a)).A second SFC (Fig. 2(b)) was employed for cross-checking the calibration room's coded target locations.This was a Samsung NX300 mirrorless digital camera, again with a 20 Mpixel (5472 9 3648) CMOS detector, a Samsung ultra-wide-angle f/2Á4 lens with a principal distance of 16Á340 mm AE 3Á3 lm and a FOV of 71Á41°AE °10 (Fig. 2(b)).Approximate initial estimations of the IOs were used to initiate the BBA.

MPC Camera
A Panono camera, a commercially available MPC manufactured by Professional360 (Berlin, Germany), was chosen to show the capabilities of the underlying approach.A Panono is a throwable multi-projective semi-panoramic camera ball with 36 projective cameras that look outward (Fig. 3).The camera was introduced in 2014 as a successful start-up project.
After stitching using cloud software, the Panono provides stunning, non-metric, 108 Mpixel, 360°images.The output of the cloud software is mainly suitable for non-metric applications, such as panoramic visualisation.Each neighbouring pair of cameras has a very small (<10%) overlap.The MPC has three main structural surfaces that lock together to form a camera ball.Each surface contains 12 projective cameras, mounted on a plastic frame.Each projective camera consists of a red/green/blue (RGB) sensor with 2064 9 1552 pixels.The approximate focal length of each projective camera is the equivalent of about 1900 AE 30 pixels, the FOV is about 57°AE 0Á4°and pixel size is 3Á07 lm.A Panono (an Explorer Edition [2015]) was used for the MPC calibration.
For the calibration, a room measuring 356 cm 9 519 cm 9 189 cm was considered as the rigid structure.A total of 215 coded targets were printed and attached to the ceiling, floor, walls and a staircase railing.Since the attached coded targets were not strictly rigid, additional laminated dots were printed and fixed to ensure the rigidity of the targets (Fig. 4).

Image Datasets
Seven datasets, each containing between 50 and 90 images, were captured by the Canon EOS 6D camera with a ground sampling distance (GSD) of between 0Á41 and 1Á03 mm.Two cross-check datasets of 50 and 80 images were taken by the Samsung NX300 camera (GSD between 0Á20 and 0Á62 mm).

Calibration Room
For the Panono's calibration, two datasets were captured and analysed.The first of these contained 34 panoramic exposures, and thus 1224 (34 9 36) individual images at three different height levels (adjusted by a tripod); the GSD was between 0Á32 and 0Á86 mm.The second dataset contained 84 panoramic shots, providing 3024 (84 9 36) individual images at two different height levels; the GSD in this case was between 0Á20 and 0Á67 mm.At the  lowest height level, the camera was fixed on a line slider in such a way that every 5 to 10 shots were taken in a straight line.This physical constraint was later found useful in checking the adjusted motion.

Method
In this section, different aspects of the methodology are discussed, including initialisation steps together with hardware and software implementation.

MPC Bundle Block Adjustment Design
Firstly, by assuming the room to be a fixed structure, a standard BBA is performed for every individual projective camera of the Panono.The output of the BBA provided approximate motions of cameras with respect to the local coordinate system.Implicit scale bars were applied by employing fixed distances to the internal structure of the coded targets.Since the extracted motion of the projective cameras could be fairly noisy in the case of the Panono (with its non-metric projective cameras), weighting was applied in order to reduce the effect of noise.The estimated IOPs and EOPs of individual cameras were also a side product of running an individual BBA.By combining the resultant "projects" (containing 36 individual project files), a uniform project containing all 36 component cameras was formed.The next step was to estimate the initial structure of the Panono (MPC 36 ) by combining the resultant relative poses (orientation and position) of the cameras with respect to the reference projective camera.The initial structure was estimated by calculating the geometric mean of individual BBA outputs to reduce the effect of noise as much as possible: where m is the number of cameras and s represents the individual camera.
The resulting MPC 36 sensor resembled the initial structure of the underlying sensor.Finally, the MPC 36 structure was retrieved by optimising the least squares cost function by employing Newton's iterative solution (see the section "The Self-calibrating BBA of a Multi-projective Camera").The result of this process is a least square estimate of the underlying calibration parameters, their corresponding variance/covariance values and image residuals of the observations.These results depicted the structure of the underlying sensor.

Software and Hardware
Equation ( 15) was linearised and coded in C++.A 64-bit Microsoft C++ compiler was employed to build executable binaries with the possibility of accessing large portions of random-access memory (RAM).A computer with a Core i7 4712HQ processor with 16 GB of RAM was used for data processing.Qt (4.8), Coin3D (3.1.3)and SoQt (1.5) were employed for the graphical user interface (GUI) and 3D visualisations.An optimised implementation of LAPACK (Intel Parallel Studio IPP 2015) was used for the matrix operations such as multiplication and factorisation (SVD and QR decomposition).Whenever possible, multi-threading was employed within the code by adopting OpenMP tags.

Automatic Tie-point Extraction Approach
The customised coded targets employed in this work are based on Khoramshahi et al. (2017).The structural properties of the customised coded targets have enabled the implementation of the automatic detection algorithm in MATLAB.The main pipeline of this algorithm consists of: (1) Running the maximally stable extremal region (MSER) algorithm to find approximate locations of ellipses (that approximate MSER regions).
(2) Applying k-means clustering on the extracted region to find boundaries with subpixel accuracy.
(5) Using the topological structure of the coded target to find targets in clusters of ellipses.(6) Establishing a projective transformation between ideal 2D locations of coded targets and their corresponding location in images to automatically read the identification (ID) part.

Performance Assessment
The accuracy of the 3D-point reconstruction using the Panono camera was assessed using two different approaches.In the first approach, 2300 reference 3D check points were selected from Canon EOS 6D dataset 7. The standard deviations of the points from this dataset's adjustment were observed to be less than 0Á7 mm in all directions.The reference labels were excluded from the Panono adjustment; therefore, the result of this adjustment were independent of these points.By employing the retrieved EOP/IOP parameters of the Panono's adjustment, a corresponding set of 3D coordinates for these labels were estimated by spatial intersection of image points.Two sets of coordinates (from the Canon EOS 6D and the Panono) were then co-registered and the residual values were used as the first performance metric.In the second approach, 150 scale bars from intersected object points in Panono dataset 2 were compared with a priori known values.Since the coded-target structure was known, the longest distance could be calculated from the ideal size of an A4 piece of paper and used as the reference value for the scale bars.

Results and Discussion
In this section, the results of the calibration room's adjustment and the geometric calibration of the Panono camera (MPC 36 ) datasets 1 and 2 are presented.

The Calibration of SFCs and the Calibration Room's Geometry
A standard BBA was used to determine the geometry of the calibration room by employing the Canon EOS 6D and Samsung NX300 cameras, as well as for calibrating the two cameras.The self-calibrating BBA was performed by employing 10 parameters for the interior orientation of the SFCs: principal distance; principal point in the x and y directions; radial (K 1 , K 2 , K 3 ) and tangential (P 1 , P 2 ) lens distortions; scale and shear.
The estimated sensor information, including a posteriori variances, for the Canon EOS 6D and Samsung NX300 cameras are listed in Table I.Most of the estimated distortion values could be considered as significant when comparing the parameter values to their corresponding standard deviation values.Moreover, the significance of an individual distortion parameter was assessed by temporarily blocking it from the model to measure the largest displacement caused by it.The results of this test are demonstrated for the Canon EOS 6D and Samsung NX300 in Fig. 5.The scale factor in the standard BBA did not exhibit a significant effect on image residuals, which was also confirmed by the closeness of the standard deviation to the estimated value in Table I.The corresponding image displacement by removing this parameter (Fig. 5) also confirms the insignificance of this parameter; therefore, the SFCs were not noticeably affected by the conformity induced by this parameter.In the evaluation, all symmetric radial distortion parameters (K 1 , K 2 , K 3 ) were combined (as K i ) to avoid misleading the reader by large displacements caused by correlated parameters.The magnitude of the corresponding standard deviation values for scale and shear still suggested that the estimated values were meaningful, and thus all parameters were included in the final model.The results showed somewhat lower standard deviation values for the IOPs of the EOS 6D compared to those for the NX300.Important indicators of goodness of fit of the proposed model are the residuals of image observations; the average root mean square errors (RMSEs) in x and y directions are demonstrated in Fig. 8 for each image of the selected datasets.Most of the RMSE values for EOS 6D and NX300 were less than 0Á3 pixel.The averages for EOS 6D dataset 6 were 0Á15 and 0Á14 pixel in the x and y directions, respectively: the corresponding figures for NX300 dataset 1 were 0Á16 and 0Á15, indicating slightly lower RMSEs for the EOS 6D.
In Fig. 6, the extracted structure of the calibration room and the 3D error ellipsoids of the corresponding 3D points are demonstrated.The estimated error ellipsoids shown in Fig. 6(a), demonstrate an acceptable fit of the BBA model to the observed data.As expected, stronger imaging geometry led to smaller error ellipsoids; for example, in Fig. 6(a) the points on the left have a lower number of intersecting rays, leading to larger error ellipsoids.A cumulative representation of the percentage of points with standard deviation values in predefined intervals is presented in Fig. 7.This histogram demonstrates the precision of the calibration room's structure, since most of the points (>90%) have standard deviation values less than 60 lm in all directions.The corresponding error ellipsoids of the Canon EOS 6D are smaller than those of the Samsung NX300; however, both cameras demonstrate acceptable qualitiessuch as sub-pixel image residuals, low RMSEs and small (<100 lm) error ellipsoids for the coded targets.The employment of error ellipsoids visibly helped in understanding weak intersections, low-accuracy cases and blunders; these played a fundamental role in the various implementations, including the standard and modified BBA.The RMSE of the differences between scale bars in EOS 6D dataset 7 and the ideal value was 0Á2 mm, with a maximum difference of 0Á6 mm.
By co-registering several datasets captured with both SFCs, acceptable registration residuals (RMSEs of 0Á52, 0Á51 and 0Á67 mm in the X, Y and Z directions, respectively, with a confidence interval of 98%) were observed according to the standard Gaussian distribution that was assumed for the observations.The previous results presented precise geometric reconstruction of the calibration room with two SFCs.

MPC 36 Calibration
Fraser (1982) stated that correlated parameters could weaken geometrically fragile networks and so threaten the convergence of Newton's iterative method.Linear correlations are addressed in a standard photogrammetric adjustment of SFC data by assuming a minimum set of constraints, and taking shots with varying scales (different positions and orientations).Variation in the depth of object points also helps to improve linear correlations.For an MPC with many side-looking cameras, however, the linearly correlated equations in the solution lead to a singularity that affects the convergence of the algorithm.If all the parameters of the cost function of an MPC with non-overlapping cameras are The Photogrammetric Record

© 2018 The Authors
The Photogrammetric Record © 2018 The Remote Sensing and Photogrammetry Society and John Wiley & Sons Ltd considered free, A and B in equation ( 18) are singular specifically for a setting similar to that in this experiment where many cameras with opposing directions are attached to an MPC body.Other well-studied solutions to deal with this singularity, such as taking exposures with varying scale or depth variation, did not help in the case studied here.To overcome this problem, the selected solution was to reduce the number of degrees of freedom by considering the object points to be fixed and known, which was done by employing the calibration body.Thereby, the number of free parameters in equation ( 18) is decreased, which significantly improves the convergence of the algorithm.As a result, the modified BBA converged to a geometrically stable solution.
Initially, the primary structure of the underlying MPC n was estimated through geometric averaging of a combined solution (equation ( 22); Fig. 9(b)).To enable this, a separate run of the standard BBA was performed on the individual projective cameras of the Panono.The RMSE values after running a standard BBA was of order 0Á3 of a pixel.Since the calibrated structure that was recovered by the EOS 6D and NX300 proved to be of high quality and robust, the standard BBA was performed by considering the calibration object to be fixed and known.All 36 sets of camera location and orientations were combined into an initial MPC structure (Fig. 9(b)).This initial geometry deviated from reality (Fig. 9(a)) and was very noisy; the initial estimation of the sensor pose highlights the inability of the standard BBA to structurally estimate the underlying MPC 36 (Fig. 9(b)).After running the modified BBA with the proposed sensor model (equations ( 15) to ( 21)), the final structure (Fig. 9(c)) resembled the reality of the sensor well.The average RMSEs of the image residuals of Panono datasets 1 and 2 in the x and y directions are given in Figs.13(a) and (b), respectively.The magnitude of the Panono's RMSEs were slightly higher than those of the SFCs, with an average value of about 0Á5 pixel.Both Panono datasets had somewhat comparable RMSE values.Fig. 10 demonstrates the pose (location and orientation) of Panono's shots in Panono datasets 1 and 2. The geometric constraints that were considered in both datasets, such as fixed height levels, as well as taking shots on straight lines in dataset 2, were used as a verification check.Obviously, the cluster of cameras in dataset 2 converged into linear clusters.In dataset 1, three different height levels were considered for the verification check, which later confirmed the correctness of the recovered structure.
For Panono, the significance test achieved by temporarily blocking a distortion parameter from the model is demonstrated in Fig. 12.The impact was greater for some cameras with poorer point distributions.Fig. 11 presents estimated principal distances of the Panono (for better visualisation the standard deviations are magnified 10 times).Fig. 14 shows the principal point locations with the corresponding 2D error ellipses.Variation in standard deviation values is mainly because of the inconsistency in the distribution of the coded targets in the calibration room.More specifically, larger error ellipses were observed for the upper and lower projective cameras, whereas the side-looking cameras' values were estimated more accurately.This inconsistency in estimated precisions is due to fewer coded targets on the ceiling and floor of the calibration room compared with the number of targets that were attached to the walls; it is aimed to improve the set-up of the targets in future work.
The calibrated ROPs, including a posteriori variances, for the Panono are listed in Table II, where lower accuracy cases are highlighted in grey.As expected, indices with a lower quality in their point distribution resulted in lower standard deviations (i = 14, 19, 22, 23, 24) compared to the indices with a good point coverage (middle cameras).For most of the indices, a standard deviation of less than 1 0 was observed that indicates the   MPC 36 Performance Assessment The average RMSEs of image residuals in the x and y directions are given in Figs.13(a) and (b) for Panono datasets 1 and 2, respectively.They are approximately two times larger than those of the Canon and Samsung.The standard deviations of the estimated object points derived from the Panono were less than 7Á9 mm in all directions, compared to 0Á5 mm for the EOS 6D and NX300.The differences between 3D check points and their corresponding intersected points from image points, the estimated calibration parameters and EOPs are presented in Fig. 15.In this figure, the differences are plotted as histograms in three main directions, and 90% error intervals are indicated by vertical red lines.In the X direction, the 90% error region corresponds to error between À10Á60 and 11Á23 mm, with an absolute mean of 5Á41 mm and RMSE of 7Á05 mm.In the Y direction, the 90% error zone is between À10Á04 and 10Á41 mm, with an absolute mean of 4Á61 mm and RMSE of 6Á17 mm.In the Z direction, the 90% error zone is between À12Á39 and 13Á31 mm, with an absolute mean of 5Á91 mm and RMSE of 7Á89 mm.The errors were close to the estimated standard deviation of the object points.The RMSE of the differences between scale bars in Panono dataset 2 and the reference value was 1Á7 mm, with a maximum difference of 4Á1 mm.
All these results were consistent and they indicated approximately 10 times poorer accuracy for the object point measurement by the Panono than those obtained by the good-quality Canon EOS 6D and Samsung NX300 SFCs.Poorer quality of the object reconstruction by the Panono could be seen in the image residuals and the standard deviations of estimated object points.Potential reasons for these poorer results for the Panono include poorer image quality, a less than ideal FOV and poorer geometry of the image block.Furthermore, some inconsistencies remain in the sensor model and, in particular, some individual cameras having ROPs of poorer quality can cause a reduction in the overall accuracy.However, the results can be considered of appropriate quality, and the 360°coverage makes this inexpensive MPC an attractive target tool for many applications such as 3D reconstruction of indoor spaces and texturing.

Summary
This paper has presented the proposed photogrammetric paradigm for the MPC calibration as a consistent model with the standard BBA.The algorithm has proved to be an efficient model for estimating the geometrical structure of a general MPC n .The major contributions of this work include: (1) Offering a uniform photogrammetric paradigm for MPC n (n ≥ 2).
(2) Addressing the singularity in the calibration process of an MPC n n ! 2 ð Þ where no overlap exists by proposing a fixed calibration body (a calibration room).
(3) Modifying the well-known BBA to accept a more general camera model as a solid single unit.(4) Successfully demonstrating the real-case adjustment of a semi-panoramic MPC 36 by employing the proposed model.
The efforts of the authors' future research will extend the algorithm to accept more general classes of cameras as well as further development of the calibration room to obtain accurate calibration of all cameras with a minimum number of images.It will also investigate the utilisation of various 360°cameras in photogrammetric reconstruction when used for mobile reconstruction tasks.Zhuang, H., 1995.A self-calibration approach to extrinsic parameter estimation of stereo cameras.Robotics and Autonomous Systems, 15(3): 189-197.

R esum e
Les cam eras multi-projectives (MPC), souvent bas ees sur le montage de plusieurs cam eras sur un même support avec une faible base st er eoscopique et un recouvrement arbitraire, ont depuis peu fait leurs preuves dans les applications de la g eomatique et de la vision par ordinateur.Cet article traite de l' etalonnage g eom etrique d'une MPC g en erique en pr esentant un mod ele math ematique susceptible de d ecrire sa g eom etrie g en erique inconnue.Une compensation par blocs modifi ee est utilis ee pour etalonner une cam era 360°i ndustrielle non m etrique.La structure de n'importe quelle MPC peut être retrouv ee a partir de l'ensemble des param etres d'orientation relative et interne, ainsi que la position et l'orientation des cam eras, au moyen d'une salle d' etalonnage pr ealablement d etermin ee avec pr ecision par photogramm etrie rapproch ee.Afin de d emontrer l'efficacit e et la pr ecision du mod ele, une cam era MPC Panono (constitu ee de 36 cam eras individuelles) a et e etalonn ee.Au terme de l'ajustement, on observe des r esidus sub-pixellaires dans l'espace image et des erreurs acceptables dans l'espace objet.

©
2018 The AuthorsThe Photogrammetric Record © 2018 The Remote Sensing and Photogrammetry Society and John Wiley & Sons Ltd the z axis is perpendicular to the image plane and positive in the direction towards the object space.

Fig. 1 .
Fig. 1.Coplanarity, collinearity and a camera's local coordinate system.(a) A central perspective (pinhole) camera looking towards to an object point X at epochs t1 and t2the coplanarity equation is defined based on the intersection geometry.(b) The local coordinate system of an ideal (pinhole) camera.
KHORAMSHAHI and HONKAVAARA.Modelling and automated calibration of a general multi-projective camera © 2018 The Authors The Photogrammetric Record © 2018 The Remote Sensing and Photogrammetry Society and John Wiley & Sons Ltd

Fig. 5 .
Fig. 5. Effect of blocking individual interior orientation parameters of the Canon EOS 6D and Samsung NX300 cameras.(a) Radial (K i ) and decentring (P 1 , P 2 ) distortion.(b) Scale and shear.

Fig. 6 .
Fig. 6.The structure of the calibration room from the BBA.(a) Error ellipsoids, with the left-hand ellipsoids 300 times larger than those with better intersection geometry.(b) The location of points.

Fig. 9 .
Fig. 9. Panono calibration.(a) Panono (top view).(b) The initial structure of the individual cameras.(c) The adjusted structure of the individual cameras.

Fig. 14 .
Fig. 14.Estimated values and error ellipses for the principal points of the Panono's individual projective sensors (dataset 1).

Fig. 15 .
Fig. 15.Histograms of differences between ground truth from Canon EOS 6D dataset 7 and model points from Panono dataset 2. A total of 2300 3D check points were used after registration of model coordinate system to the ground in the X, Y and Z directions.
KHORAMSHAHI and HONKAVAARA.Modelling and automated calibration of a general multi-projective camera © 2018 The Authors The Photogrammetric Record © 2018 The Remote Sensing and Photogrammetry Society and John Wiley & Sons Ltd and HONKAVAARA.Modelling and automated calibration of a general multi-projective camera © 2018 The AuthorsThe Photogrammetric Record © 2018 The Remote Sensing and Photogrammetry Society and John Wiley & Sons Ltd The Photogrammetric Record © 2018 The Remote Sensing and Photogrammetry Society and John Wiley & Sons Ltd

Table I .
The calibrated parameters for the Canon EOS 6D and Samsung NX300; s.d.= standard deviation.