#+TITLE: How to run a calibration
#+OPTIONS: toc:t

Calibrating cameras is one of the main applications of mrcal, so I describe this
process in some detail here.

* Capturing images
** Calibration object
We need to get images of a [[file:formulation.org::#calibration-object][calibration object]], a board containing an observable
grid of points. mrcal doesn't care where these observations come from: any tool
that can produce a table of corner observations can be utilized. Currently the
recommended method is to use a classical /chessboard/ target, and to employ the
[[https://github.com/dkogan/mrgingham][mrgingham]] detector. The mrgingham documentation has a [[https://github.com/dkogan/mrgingham/raw/master/chessboard.14x14.pdf][.pdf of a chessboard
pattern]] that works well. See the [[file:recipes.org::#non-mrgingham-detector][recipes]] to use a non-mrgingham detector
and/or a non-chessboard object.

*** Chessboard shape
:PROPERTIES:
:CUSTOM_ID: chessboard-shape
:END:

The pattern should be printed and mounted onto a /rigid/ and /flat/ surface to
produce the calibration object. The [[file:formulation.org][optimization problem]] assumes the board shape
to be constant over all the calibration images, and any instability in the board
shape breaks that assumption. Depending on the resolution of the camera and the
amount of calibration accuracy required, even a slightly floppy chessboard can
break things. Don't take this for granted. A method for quantifying this effect
appears [[#residuals-chessboard-shape][below]].

Similarly, the optimization problem assumes that the calibration object is
mostly flat. Some amount of deformation is inevitable, so mrcal /does/ support a
simple [[file:formulation.org::#board-deformation][non-planar board]] model, with the board shape being computed as part of
the calibration solve. In my experience, if you try to make sure the board is
flat, this model is enough to fix most of the unavoidable small deflections. If
care isn't taken to ensure the board is flat, the current simple deformation
model is often insufficient to fully model the shape. Don't take this for
granted either. A method for quantifying this effect appears [[#residuals-chessboard-shape][below]] as well.

I usually use a square chessboard roughly 1m per side to calibrate my fisheye
lenses. Such a large chessboard often has issues with its shape. To mitigate
these, my chessboard is backed by about 1 inch of aluminum honeycomb. This is
expensive, but works well to stabilize the shape. The previous board I was using
is backed with the same amount of /foam/ instead. This is significantly cheaper,
and still pretty good, but noticeably less flat and less rigid. This foam-backed
board does warp with variations in temperature and humidity, but this drift is
slow, and the shape is /usually/ stable long-enough for a chessboard dance. If
you can make one, the aluminum honeycomb-backed boards are strongly recommended.

*** Chessboard size
:PROPERTIES:
:CUSTOM_ID: chessboard-size
:END:
There's a tradeoff to consider between large and small chessboards. According to
the [[file:tour-choreography.org::#choreography-distance][dance study]], we want close-ups that fill the imager, so we're choosing
between small chessboards placed near the camera and large chessboards placed
away from the camera.

- Small chessboards placed close to the camera produce two issues:

  1. the chessboard would appear out of focus. This is OK up to a point
     (out-of-focus blur is mostly isotropic, and this noise averages out
     nicely), but extreme out-of-focus images should probably be avoided
  2. Extreme close-ups exhibit [[file:formulation.org::#lens-behavior][noncentral projection behavior]]: observation rays
     don't all intersect at the same point. Today's mrcal cannot model this
     behavior, so extreme close-ups should be avoided to allow the central-only
     models to fit. The [[file:tour-cross-validation.org][cross-validation page]] shows how to detect errors caused
     by noncentral projection

- Large chessboards are better, but are difficult to manufacture, to keep them
  flat and stiff, and to move them around. Currently I use a 1m x 1m square
  board, which seems to be near the limit of what is reasonable.

For wide lenses especially, you want a larger chessboard. When calibrating an
unfamiliar system I usually try running a calibration with whatever chessboard I
already have, and then analyze the results to see if anything needs to be
changed. It's also useful to rerun the [[file:tour-choreography.org][dance study]] for the specific geometry
being calibrated.

** Image-capturing details
Now that we have a calibration object, this object needs to be shown to the
camera(s). It is important that the images contain clear features. Motion blur
or exposure issues will all cause bias in the resulting calibration.

If calibrating multiple cameras, mrcal will solve for all the intrinsics /and/
extrinsics. There is one strong requirement on the captured images in this case:
the images must be synchronized across all the cameras. This allows mrcal to
assume that if camera A and camera B observed a chessboard at time T, then this
chessboard was at /exactly/ the same location when the two cameras saw it.
Generally this means that either

- The cameras are wired to respond to a physical trigger signal
- The chessboard and cameras were physically fixed (on a tripod, say) at each
  time of capture

Some capture systems have a "software" trigger mode, but this is usually too
loose to produce usable results. A similar consideration exists when using
cameras with a [[https://en.wikipedia.org/wiki/Rolling_shutter][rolling shutter]]. With such cameras it is imperative that
everything remain stationary during image capture, even when only one camera is
involved. See the [[#residuals-sync][residual analysis of synchronization errors]] and
[[#residuals-rolling-shutter][of rolling shutter]] for details.

** Lens settings
Lens intrinsics are usually very sensitive to any mechanical manipulation of the
lens. It is thus strongly recommended to set up the lens it its final working
configuration (focal length, focus distance, aperture), then to mechanically
lock down the settings, and then to gather the calibration data. This often
creates dancing challenges, described in the next section.

If you /really/ need to change the lens settings after calibrating or if you're
concerned about drifting lens mechanics influencing the calibration (you should
be!), see the [[file:recipes.org::#lens-stability][recipes]] for a discussion.

** Dancing
:PROPERTIES:
:CUSTOM_ID: dancing
:END:

As shown in the [[file:tour-choreography.org][dance study]], the most useful observations to gather are

- close-ups: the chessboard should fill the whole frame as much as possible.

- oblique views: tilt the board forward/back and left/right. I generally tilt by
  ~ 45 degrees. At a certain point the corners become indistinct and the
  detector starts having trouble, but depending on the lens, that point could
  come with quite a bit of tilt. A less dense chessboard eases this also, at the
  cost of requiring more board observations to get the same number of points.

- If calibrating multiple cameras, it is impossible to place a calibration
  object at a location where it's seen by all the cameras /and/ where it's a
  close-up for all the cameras. So you should get close-ups for each camera
  individually, and also get observations common to multiple cameras, that
  aren't necessarily close-ups. The former will serve to define your camera
  intrinsics, and the latter will serve to define your extrinsics (geometry).
  Get just far-enough out to create the joint views. If usable joint views are
  missing, the extrinsics will be undefined, and the solver will complain about
  a "not positive definite" (singular in this case) Hessian.

A dataset composed primarily of tilted closeups produces good results.

As described [[#chessboard-size][above]], you want a large chessboard placed as close to the camera as
possible, before the out-of-focus and noncentral effects kick in.

If the model will be used to look at far-away objects, care must be taken to
produce a reliable calibration /at long ranges/, not just at the /short/ ranges
where the chessboards were. The primary way to do that is to get close-up
chessboard views. If the close-up range is very different from the working range
(infinity, possibly), the close-up images could be very out-of-focus. The
current thought is that the best thing to do is to get close-up images even if
they're out of focus. The blurry images will have a high uncertainty in the
corner observations (hopefully without bias), but the uncertainty improvement
that comes from the near-range chessboard observations more than makes up for
it. In these cases you usually need to get more observations than you normally
would to bring down the uncertainties to an acceptable level. In challenging
situations it's useful to re-run the [[file:tour-choreography.org][dance study]] for the specific use case to
get a sense of what kind of observations are required and what kind of
uncertainties can be expected. See the [[file:tour-choreography.org][dance study]] for detail.

Another issue that could be caused by extreme close-ups is a
[[file:formulation.org::#lens-behavior][noncentral projection behavior]]:
observation rays don't all intersect at the same point. Today's mrcal cannot
model this behavior, and the [[file:tour-cross-validation.org][cross-validation]] would show higher-than-expected
errors. In this case it is recommended to gather calibration data from further
out.

It is better to have more chessboard data rather than less. mrgingham will throw
away frames where no chessboard can be found, so it is perfectly reasonable to
grab too many images with the expectation that they won't all end up being used
in the computation. I usually aim for about 100 usable frames, but you may get
away with fewer, depending on your specific scenario. The mrcal uncertainty
feedback will tell you if you need more data.

Naturally, intrinsics are accurate only in areas where chessboards were
observed: chessboard observations on the left side of the image tell us little
about lens behavior on the right side. Thus it is imperative to cover the whole
field of view during the chessboard dance. It is often tricky to get good data
at the edges and corners of the imager, so care must be taken. Some chessboard
detectors (mrgingham in particular) only report complete chessboards. This makes
it extra-challenging to obtain good data at the edges: a small motion that
pushes one chessboard corner barely out of bounds causes the whole observation
to be discarded. It is thus /very/ helpful to be able to see a live feed of the
camera as the images are being captured. In either case, checking the coverage
is a great thing to do. The usual way to do this is indirectly: [[file:tour-uncertainty.org][by visualizing
the projection uncertainty]]. Or by [[#Visualization][visualizing the obtained chessboard
detections]] directly.

** Image file-naming convention
With monocular calibrations, there're no requirements on image filenames: use
whatever you like. If calibrating multiple synchronized cameras, however, the
image filenames need to indicate which camera captured each image and at which
time. I generally use =frameFFF-cameraCCC.jpg=. Images with the same =FFF= are
assumed to have been captured at the same instant in time, and =CCC= identifies
the camera. Naming images in this way is sufficient to communicate these
mappings to mrcal.

* Detecting corners
:PROPERTIES:
:CUSTOM_ID: corner-detector
:END:

A number of tools are available to detect chessboard corners (OpenCV is commonly
used, and different organizations have their own in-house versions). None of the
available ones worked well for my use cases, so I had to write my own, and I
recommend it for all corner detections: [[https://github.com/dkogan/mrgingham/][=mrgingham=]]. mrgingham is fast and is
able to find chessboard corners subject to very un-pinhole-like projections. At
this time it has two limitations that will be lifted eventually:

- It more or less assumes a regular grid of N-by-N corners (i.e. N+1-by-N+1
  squares)
- It requires /all/ the corners to be observed in order to report the detections
  from an image. Incomplete chessboard observations aren't supported

If these are unacceptable, any other detector may be used instead. See the
[[file:recipes.org::#non-mrgingham-detector][recipes]].

** Using mrgingham
Once mrgingham is [[file:install.org][installed]] or built from [[https://github.com/dkogan/mrgingham][source]], it can be run by calling the
=mrgingham= executable. The sample in the [[file:tour-initial-calibration.org][tour of mrcal]] processes [[file:external/2022-11-05--dtla-overpass--samyang--alpha7/2-f22-infinity/images][these images]]
to produce [[file:external/2022-11-05--dtla-overpass--samyang--alpha7/2-f22-infinity/corners.vnl][these chessboard corners]] like this:

#+begin_src sh
mrgingham --jobs 4 --gridn 14 '*.JPG' > corners.vnl 
#+end_src

mrgingham tries to handle a variety of lighting conditions, including varying
illumination across the image, but the corners must exist in the image in some
form.

At this time mrgingham returns /only/ complete chessboard views: if even one
corner of the chessboard couldn't be found, mrgingham will discard the entire
image. Thus it takes care to get data at the edges and in the corners of the
imager. A live preview of the captured images is essential.

Another requirement due to the design of mrgingham is that the board should be
held with a flat edge parallel to the camera xz plane (parallel to the ground,
usually). mrgingham looks for vertical and horizontal sequences of corners, but
if the board is rotated diagonally, then none of these sequences are clearly
"horizontal" or "vertical".

** Choice of calibration object
When given an image of a /chessboard/, the detector is directly observing the
feature we actually care about: the corner. Another common calibration board
style is a grid of circles, where the feature of interest is the center of each
circle. When given an image of such a grid of circles, the detector either

- detects the contour at the edge of each circle
- finds the pixel blob comprising each circle observation

and from either of these, the detector infers the circle center. This can work
when looking at head-on images, but when given tilted images subjected to
non-pinhole lens behaviors, getting accurate circle centers from outer contours
or blobs is /hard/. The resulting inaccuracies in the detections of circle
centers will introduce biases into the solve that aren't modeled by the
[[file:uncertainty.org][projection uncertainty routine]], so chessboards are /strongly/ recommended in
favor of circle grids.

Some other calibration objects use [[https://april.eecs.umich.edu/software/apriltag][AprilTags]]. mrcal [[file:formulation.org::#noise-model][assumes independent noise]] on
each point observation, so correlated sources of point observations are also not
appropriate sources of data. Thus using individual corners of AprilTags will be
un-ideal: their noise is correlated. Using the /center/ of an AprilTag would
eliminate this correlated noise, but maybe would have a similar inaccuracy
problem that a grid of circles would have.

If using a possibly-problematic calibration object such as a grid of circles or
AprilTags, double-check the detections by [[file:recipes.org::#reproject-to-chessboard][reprojecting to calibration-object
space]] after a solve has completed.

** Visualization
:PROPERTIES:
:CUSTOM_ID: Visualization
:END:
Once we have a =corners.vnl= from some chessboard detector, we can visualize the
coverage. From the [[file:tour-initial-calibration.org][tour of mrcal]]:

#+begin_example
$ < corners.vnl       \
  vnl-filter -p x,y | \
  feedgnuplot --domain --square --set 'xrange [0:6000] noextend' --set 'yrange [3376:0] noextend'
#+end_example

[[file:external/figures/calibration/mrgingham-coverage.png]]

Doing this is usually unnecessary since the [[file:tour-uncertainty.org][projection uncertainty reporting]]
shows the coverage (and more!), but it's good to be able to do this for
debugging.

* Model choice
:PROPERTIES:
:CUSTOM_ID: model-choice
:END:

Before calibrating we need to choose the model for representing the lenses. Use
[[file:splined-models.org][=LENSMODEL_SPLINED_STEREOGRAPHIC=]]. This model works very well, and is able to
represent real-world lenses better than the parametric models (all the other
ones). This is true even for long, near-pinhole lenses. Depending on the
specific lens and the camera resolution this accuracy improvement may not be
noteworthy. But even in those cases, the splined model is flexible enough to get
truthful [[file:uncertainty.org][projection uncertainty estimates]], so it's /still/ worth using. Today I
use other models only if I'm running quick experiments: splined models have many
more parameters, so things are slower.

[[file:splined-models.org][=LENSMODEL_SPLINED_STEREOGRAPHIC=]] has several [[file:splined-models.org::#splined-models-configuration-selection][configuration variables]] that need
to be set. The full implications of these choices still need to be studied, but
the results appear fairly insensitive to these. I generally choose =order=3= to
select cubic splines. I generally choose a rich model with fairly dense spline
spacing. For instance the splined model used in the [[file:tour-initial-calibration.org][tour of mrcal]] has
=Nx=30_Ny=18=. This has 30 spline knots horizontally and 18 vertically. You
generally want =Ny= / =Nx= to roughly match the aspect ratio of the imager. The
=Nx=30_Ny=18= arrangement is probably denser than it needs to be, but it works
OK. The cost of such a dense spline is a bit of extra computation time and more
stringent requirements on calibration data to fully and densely cover the
imager.

The last configuration parameter is =fov_x_deg=: the horizontal field-of-view of
the lens. The splined model is defined by /knots/ spread out across space, the
arrangement of these knots defined by the =fov_x_deg= parameter. We want the
region in space defined by the knots to roughly match the region visible to the
lens. A too-large =fov_x_deg= would waste some knots by placing them beyond
where the lens can see. And a too-small =fov_x_deg= would restrict the
projection representation on the edge of the image.

An initial estimate of =fov_x_deg= can be computed from the datasheet of the lens.
Then a test calibration should be computed using that value, and the
[[file:mrcal-show-splined-model-correction.html][=mrcal-show-splined-model-correction=]] tool can then be used to validate that
=fov_x_deg= parameter. In the [[file:tour-initial-calibration.org][tour of mrcal]] we get something like this:

#+begin_src sh
mrcal-show-splined-model-correction   \
  --set 'cbrange [0:0.1]'             \
  --unset grid                        \
  --set 'xrange [:] noextend'         \
  --set 'yrange [:] noextend reverse' \
  --set 'key opaque box'              \
  splined.cameramodel
#+end_src

[[file:external/figures/splined-models/splined-magnitude.png]]

This is about what we want. The valid-intrinsics region covers most of the
spline-in-bounds region without going out-of-bounds anywhere. In the [[file:tour-initial-calibration.org][tour of
mrcal]] we followed this procedure to end up with

- =LENSMODEL_SPLINED_STEREOGRAPHIC_order=3_Nx=30_Ny=18_fov_x_deg=150=

Getting this perfect isn't important, so don't spent a ton of time working on
it. See [[file:splined-models.org::#splined models field of view selection][the lensmodel documentation]] for more detail.

* Computing the calibration
:PROPERTIES:
:CUSTOM_ID: calibration
:END:

We have data; we have a lens model; we're ready. Let's compute the calibration
using the [[file:mrcal-calibrate-cameras.html][=mrcal-calibrate-cameras=]] tool. The invocation should look something
like this:

#+begin_src sh
mrcal-calibrate-cameras                                                         \
  --corners-cache corners.vnl                                                   \
  --lensmodel LENSMODEL_SPLINED_STEREOGRAPHIC_order=3_Nx=30_Ny=18_fov_x_deg=150 \
  --focal 1900                                                                  \
  --object-spacing 0.0588                                                       \
  --object-width-n 14                                                           \
  '*.JPG'
#+end_src

- =--corners-cache corners.vnl= says that the chessboard corner coordinates live
  in a file called =corners.vnl=. This is the output of the [[#corner-detector][corner detector]]. If
  this argument is omitted, or a non-existent file is given,
  [[file:mrcal-calibrate-cameras.html][=mrcal-calibrate-cameras=]] will run mrgingham, and write the results into the
  given path. Thus the same command would be used to both compute the corners
  initially, and to reuse the pre-computed corners in subsequent runs.

  As described above, the =corners.vnl= file can come from any chessboard
  detector. If it's a detector that produces a 4th column of /weights/ instead
  of a decimation level, pass in =--corners-cache-has-weights=

- =--lensmodel= specifies which lens model we're using for /all/ the cameras.
  See [[#model-choice][the "Model choice" section above]]. Not being able to select different
  models for different cameras is a current limitation of mrcal. If some very
  different lenses are present in the same calibration, I use a densely-spaced
  (high =Nx=, =Ny=) splined model with the =fov_x_deg= of the widest lens in the
  set. This wastes knots in the too-wide areas of the narrow lenses, but if the
  spline was dense-enough, there're enough knots remaining to fit the narrower
  lenses.

- =--focal 1900= provides the initial estimate for the camera focal lengths, in
  pixels. This doesn't need to be extremely precise, but do try to get this
  close if possible. The focal length value to pass to =--focal=
  ($f_\mathrm{pixels}$) can be derived using the [[file:lensmodels.org::#lensmodel-stereographic][stereographic model]] definition:

\[ f_\mathrm{pixels} \approx \frac{\mathrm{imager\_width\_pixels}}{4 \tan \frac{\mathrm{field\_of\_view\_horizontal}}{4}} \]

  This expression is a good initial estimate for both long and wide lenses. Note
  that the manufacturer-specified "field of view" and "focal length" values are
  usually poorly-defined: the former is different in all directions, and the
  latter is meaningless in wide lenses that are nowhere near the pinhole model.
  With a longer lens, we can assume pinhole behavior to get

\[ f_\mathrm{pixels} = f_\mathrm{mm} \frac{\mathrm{imager\_width\_pixels}}{\mathrm{imager\_width\_mm}} \]

  As with the =fov_x_deg= parameter described in [[#model-choice][the "Model
  choice" section above]], running a test calibration with a rough estimate, and
  then passing in the much-closer optimized value is a good strategy. The
  optimized focal length is the first two values of the =intrinsics= vector in
  the result. Those two values should be similar, and anything around there
  should work well for =--focal=.

- =--object-spacing= is the distance between neighboring corners in the
  chessboard. Even spacing, identical in both directions is assumed

- =--object-width-n= is the horizontal corner count of the calibration object.
  In the example invocation above there is no =--object-height-n=, so
  [[file:mrcal-calibrate-cameras.html][=mrcal-calibrate-cameras=]] assumes a square chessboard

After the options, [[file:mrcal-calibrate-cameras.html][=mrcal-calibrate-cameras=]] takes globs describing the images.
One glob per camera is expected, and in the above example /one/ glob was given:
='*.JPG'=. Thus this is a monocular solve. More cameras would imply more globs.
For instance a 2-camera calibration might take arguments

#+begin_example
'frame*-camera0.png' 'frame*-camera1.png'
#+end_example

Note that these are /globs/, not /filenames/. So they need to be quoted or
escaped to prevent the shell from expanding them: hence ='*.JPG'= and not
=*.JPG=.

Finally, to compute [[#cross-validation][cross-validation diffs (see below)]] it's necessary to run
several independent calibrations. I generally split the dataset into even/odd
chunks, and calibrate them independently. With this =zsh= snippet, for instance:

#+begin_src sh
for oddeven (odd even) {

    cmd=(mrcal-calibrate-cameras                                                       \
         --corners-cache corners.vnl                                                   \
         --lensmodel LENSMODEL_SPLINED_STEREOGRAPHIC_order=3_Nx=30_Ny=18_fov_x_deg=150 \
         --focal 1900                                                                  \
         --object-spacing 0.0588                                                       \
         --object-width-n 14)

    if [[ $oddeven = "even" ]] {
        globs=('frame*[02468]-camera'{0,1}'.png')
    } else {
        globs=('frame*[13579]-camera'{0,1}'.png')
    }

    cmd=($cmd $globs)

    $cmd

    for i (0 1) {
        mv camera-$i.cameramodel camera-$i-$oddeven.cameramodel
    }
}
#+end_src

** Troubleshooting broken solves
Usually the solver converges, and produces a result. Then we [[#interpreting-results][look at the
diagnostics to evaluate the quality of this result]] (next section). We can do
that if the errors are small, and the optimization completed successfully. If
the data issues are too large, however (pervasive sync errors, completely
insufficient lens model, etc), the solver will have trouble. It could

- Take a very long time to converge to /some/ solution
- Produce tool many outliers, possibly incrementally, producing output such as
  this:
  #+begin_example
mrcal.c(5413): Threw out some outliers. New count = 174/35280 (0.5%). Going again
mrcal.c(5413): Threw out some outliers. New count = 252/35280 (0.7%). Going again
mrcal.c(5413): Threw out some outliers. New count = 303/35280 (0.9%). Going again
  #+end_example
  This will slow down the solve dramatically. Unless the data is known to be
  funky, more than ~ 3% outliers should raise questions

If the solver has trouble like this, it is usually helpful to turn off the
outlier rejection by running =mrcal-calibrate-cameras --skip-outlier-rejection
....=, and then [[#residuals][examine the residuals as described below]]. Hopefully that would
provide a hint about the issues.

It also helps to simplify the problem, which is effective at isolating certain
issues. For instance, a nonconverging multi-camera solve should be attempted
monocularly, one camera at a time. If some cameras converge and some don't, that
points to the issue. If individually the solves converge, but together they
don't, there could be an issue with the [[#residuals-sync][camera synchronization]], or the
chessboard corner indices aren't consistent. Similarly solving with a subset of
images is often enlightening.

Another common problem is getting messages like this:

#+begin_example
mrcal.c(3758): WARNING: Board observation 157 (icam_intrinsics=0, icam_extrinsics=-1, iframe=104) had almost all of its points thrown out as outliers: only 0/100 remain. CHOLMOD is about to complain about a non-positive-definite JtJ. Something is wrong with this observation
mrcal.c(3758): WARNING: Board observation 158 (icam_intrinsics=1, icam_extrinsics=0, iframe=104) had almost all of its points thrown out as outliers: only 0/100 remain. CHOLMOD is about to complain about a non-positive-definite JtJ. Something is wrong with this observation
mrcal.c(5412): Threw out some outliers (have a total of 861 now); going again
libdogleg at dogleg.c:1115: CHOLMOD warning:
libdogleg at dogleg.c:1115:  not positive definite.
libdogleg at dogleg.c:1115:  file: ../Cholesky/t_cholmod_rowfac.c
libdogleg at dogleg.c:1115:  line: 430
libdogleg at dogleg.c:1115: 
#+end_example

The complaint is about a singular Hessian matrix. Usually this happens if some
variable in the optimization has no effect at all on the solution, and the
optimizer thus doesn't know what to do with that variable. Usually heavy outlier
rejection precedes this, and the missing data is causing the problem. Example:
all chessboard observations for a given frame were thrown out; thus moving the
chessboard pose at that time has no effect, and we get a singular Hessian. The
diagnostic technique is the same: disable the outlier rejection and examine the
residuals.

* Interpreting the results
:PROPERTIES:
:CUSTOM_ID: interpreting-results
:END:

Once we have a calibration, we should evaluate how well it represents reality.
The tour of mrcal shows a very detailed process: [[file:tour-initial-calibration.org::#opencv8-model-solving][a real-world fit using
=LENSMODEL_OPENCV8=]] and [[file:tour-initial-calibration.org::#splined-model-solving][a real-world fit using
=LENSMODEL_SPLINED_STEREOGRAPHIC_...=]]. The analysis sequence outlined there is
too thorough for everyday use, but it's good to look through those pages to get
a sense. A high-level process is

- [[#uncertainty][Examine the projection uncertainty to make sure we have enough good data in
  the right places]]
- [[#cross-validation][Examine the cross-validation diffs to confirm that the model fits and that the
  reported uncertainty is trustworthy]]
- [[#residuals][If these diffs are too high, examine the residuals to find out why]]

This is a /lot/, but you don't need to do all this every time. As noted in [[file:formulation.org::#noise-model][the
noise model description]], we want homoscedastic noise in our observations of
the chessboard corners. We will get that if and only if our models faithfully
describe the world; so the available diagnostics serve to pinpoint areas where
the models don't fit, so that those issues could be addressed.

I now describe each step.

** Projection uncertainty
:PROPERTIES:
:CUSTOM_ID: uncertainty
:END:

As described in detail in [[file:uncertainty.org][the projection-uncertainty page]], the projection
uncertainty computed by mrcal gauges the effect of sampling error. Since we
/always/ have noise in our chessboard observations, it's important that the
solution be insensitive to this noise. Otherwise a recalibration of the same
system would produce very different results due to new chessboard observations
containing new noise.

Projection uncertainty can be visualized with the
[[file:mrcal-show-projection-uncertainty.html][=mrcal-show-projection-uncertainty=]] tool. From the [[file:tour-uncertainty.org][tour of mrcal]]:

#+begin_src sh
mrcal-show-projection-uncertainty splined.cameramodel --cbmax 1 --unset key
#+end_src
#+begin_src sh :exports none :eval no-export
# THIS IS GENERATED IN tour-uncertainty.org
#+end_src

[[file:external/figures/uncertainty/uncertainty-splined.png]]

This is projection uncertainty at infinity, which is what I'm usually interested
in. If we care /only/ about the performance at some particular distance, that
can be requested with =mrcal-show-projection-uncertainty --distance ...=. That
uncertainty will usually be better than the uncertainty at infinity. Making sure
things work at infinity will make things work at other ranges also.

The projection uncertainty measures the quality of the chessboard dance. If the
[[#dancing][guidelines noted above]] were followed, you'll get good uncertainties. If the
uncertainty is poor in some region, you need more chessboard observations in
that area. To improve it everywhere, follow the guidelines: more observations,
more closeups, more tilt.

The projection uncertainties will be overly-optimistic if model errors are
present or if a too-lean lens model is selected. So we now look at the
cross-validation diffs to confirm that no model errors are present. If we can
confirm that, the projection uncertainties can be used as the authoritative
gauge of the quality of our calibration.

Since the uncertainties are largely a function of the chessboard dance, I
usually don't bother looking at them if I'm recalibrating a system that I have
calibrated before, with a similar dance. Since the system and the dance didn't
change, neither would the uncertainty.

** Cross-validation diffs
:PROPERTIES:
:CUSTOM_ID: cross-validation
:END:

If we have an acceptable projection uncertainty, we need to decide if it's a
good gauge of calibration quality: if we have model errors or not.

A good way to do that is to compute a cross-validation: we calibrate the camera
system twice with independent input data, and we compare the resulting
projections. If the models fit, then we only have sampling error affecting the
solves, and the resulting differences will be in-line with what the
uncertainties predict: $\mathrm{difference} \approx \mathrm{uncertainty}_0 +
\mathrm{uncertainty}_1$. Otherwise, we have an extra source of error not present
in the uncertainty estimates, which would cause the cross-validation diffs to be
significantly higher. That would suggest a deeper look is necessary.

To get the data, we can do two separate dances, or we can split the dataset into
odd/even images, as described [[#calibration][above]]. Note: if the chessboard is moved very
slowly, the even and odd datasets will be very qualitatively similar, and it's
possible to get a low cross-validation diff even in the presence of model
errors: the two samples of the model error distribution would be similar to each
other. This usually doesn't happen, but keep in mind that it /could/, so you
should move at a reasonable speed to make the odd/even datasets non-identical.

In the [[file:tour-cross-validation.org][tour of mrcal]] we computed a cross-validation, and discovered that there
indeed exists a model error. The cross-validation diff looked like this:

#+begin_src sh
mrcal-show-projection-diff           \
  --cbmax 2                          \
  --unset key                        \
  2-f22-infinity.splined.cameramodel \
  3-f22-infinity.splined.cameramodel
#+end_src

[[file:external/figures/cross-validation/diff-cross-validation-splined.png]]

And the two uncertainties looked roughly like this:

[[file:external/figures/uncertainty/uncertainty-splined.png]]

The diff is more than roughly 2x the uncertainty, so something wasn't fitting:
the lens had a non-negligible noncentral behavior at the chessboard distance,
which wasn't fixable with today's mrcal. So we could either

- Accept the results as is, using the diffs as a guideline to how trustworthy
  the solves are
- Gather more calibration images from further out, minimizing the unmodeled
  noncentral effect

Cross-validation diffs are usually /very/ effective at detecting issues, and I
usually compute these every time I calibrate a lens. In my experience, these are
the single most important diagnostic output.

While these are very good at /detecting/ issues, they're less good at
pinpointing the root cause. To do that usually requires thinking about the
[[#residuals][solve residuals (next section)]].

** Residuals
:PROPERTIES:
:CUSTOM_ID: residuals
:END:

We usually have a /lot/ of images and a /lot/ of residuals. Usually I only look
at the residuals if

- I'm calibrating an unfamiliar system
- I don't trust something about the way the data was collected; if I have little
  faith in the camera time-sync for instance
- Something unknown is causing issues (we're seeing too-high cross-validation
  diffs), and we need to debug

The residuals in the whole problem can be visualized with the
[[file:mrcal-show-residuals.html][=mrcal-show-residuals=]] tool. And the residuals of specific chessboard
observations can be visualized with the [[file:mrcal-show-residuals-board-observation.html][=mrcal-show-residuals-board-observation=]]
tool.

Once again. As noted in [[file:formulation.org::#noise-model][the noise model description]], we want homoscedastic noise
in our observations of the chessboard corners. So the residuals should all be
independent and should all have the same dispersion: each residual vector should
look random, and unrelated to any other residual vector. There should be no
discernible patterns to the residuals. If model errors are present, the
residuals will /not/ appear random, and /will/ exhibit patterns. And we will
then see those patterns in the residual plots.

The single most useful invocation to run is

#+begin_example
mrcal-show-residuals-board-observation \
  --from-worst                         \
  --vectorscale 100                    \
  solved.cameramodel                   \
  0-5
#+end_example

This displays the residuals of a few worst-fitting images, with the error
vectors scaled up 100x for legibility (the "100" often needs to be adjusted for
each specific case). The most significant issues usually show up in these few
worst-fitting chessboard observations.

The residual plots are interactive, so it's useful to then zoom in on the worst
fitting points (easily identifiable by the color) in the worst-fitting
observations to make sure the observed chessboard corners we're fitting to are
in the right place. If something was wrong with the chessboard corner detection,
a zoomed-in residual image would tell us this.

Next the residual plots should be examined for patterns. Let's look at some
common issues, and the characteristic residual patterns they produce.

For most of the below I'm going to show the usual calibration residuals we would
expect to see if we had some particular kind of issue. We do this by simulating
/perfect/ calibration data, corrupting it with the kind of problem we're
demonstrating, recalibrating, and visualizing the result. Below I do this in
Python using the [[file:mrcal-python-api-reference.html#-make_perfect_observations][=mrcal.make_perfect_observations()=]] function. These could
equivalently have been made using the [[https://www.github.com/dkogan/mrcal/blob/master/analyses/mrcal-reoptimize][=mrcal-reoptimize=]] tool:

#+begin_example
$ mrcal/analyses/mrcal-reoptimize \
    --explore                     \
    --perfect                     \
    splined.cameramodel

... Have REPL looking at perfect data. Add flaw, reoptimize, visualize here. 
#+end_example

This technique is powerful and is useful for more than just making plots for
this documentation. We can also use it to quantify the errors that would result
from problems that we suspect exist. We can compare those to the
[[file:tour-cross-validation.org][cross-validation errors]] that we observe to get a hint about what is causing the
problem.

For instance, you might have a system where you're not confident that your
chessboard is flat or that your lens model fits or that your camera sync works
right. You probably see high [[file:tour-cross-validation.org][cross-validation diffs]], but don't have a sense of
which factor is most responsible for the issues. We can simulate this, to see if
the suspected issues would cause errors as high as what is observed. Let's say
you placed a straight edge across your chessboard, and you see a deflection at
the center of ~ 3mm. mrcal doesn't assume a flat chessboard, but its deformation
model is simple, and even in this case it can be off by 1mm. Let's hypothesize
that the board deflection is 0.5mm horizontally and 0.5mm vertically, in the
other direction. We can then use the [[https://www.github.com/dkogan/mrcal/blob/master/analyses/mrcal-reoptimize][=mrcal-reoptimize= tool]] to quantify the
resulting error, and we can compare the simulated cross-validation diff to the
one we observe. We might discover that the expected chessboard deformation
produces cross-validation errors that are smaller than what we observe. In that
case, we should keep looking. But if we see the simulated cross-validation diff
of the same order of magnitude that we observe, then we know it's time to fix
the chessboard.

*** Poorly-fitting lens model
We saw this in [[file:tour-initial-calibration.org::#opencv8-solve-diagnostics][the tour of mrcal]]: we tried to calibrate a very wide lens with
=LENSMODEL_OPENCV8=, and it showed clear signs of the model not fitting. Read
that page to get a sense of what that looks like in the various diagnostics.
Broadly speaking, lens modeling errors increase as you move towards the edges of
the imager, so we would see higher errors at the edges. This often looks similar
to an [[#residuals-chessboard-shape][unmodeled deformation in the chessboard shape]].

*** Errors in the chessboard detector
What if the chessboard detector gets a small number of corners localized
incorrectly? If the shift is large, those corner observations will be thrown out
as outliers, and will not affect the solve. But if they're small, they may cause
a bias in the solution. What does that look like? Let's simulate it. We

- take the =LENSMODEL_SPLINED_STEREOGRAPHIC= solve from [[file:tour-initial-calibration.org][the tour of mrcal]]
- produce perfect observations at the computed optimal geometry
- add perfect independent gaussian noise to the observations

This data now follows [[file:formulation.org::#noise-model][the noise model]] perfectly. Then I add the flaw: I bump all
the observed corners on the chessboard diagonal in observation 10 to the right
by 0.8 pixels, and I look at the residuals. This requires a bit of code:

#+begin_src python
#!/usr/bin/python3

import sys
import mrcal
import numpy as np
import numpysane as nps

def add_imperfection(optimization_inputs):
    for i in range(14):
        optimization_inputs['observations_board'][10,i,i,0] += 0.8


model = mrcal.cameramodel('splined.cameramodel')
optimization_inputs = model.optimization_inputs()

mrcal.make_perfect_observations(optimization_inputs)
add_imperfection(optimization_inputs)
mrcal.optimize(**optimization_inputs)

mrcal.show_residuals_board_observation(optimization_inputs, 10,
                                       vectorscale =  200,
                                       circlescale =  0.5,
                                       title       =  "Effect of corner detection error at the center",
                                       paths       = optimization_inputs['imagepaths'])
#+end_src
#+begin_src python :exports none :eval no-export
#!/usr/bin/python3

import sys
import mrcal
import numpy as np
import numpysane as nps

def add_imperfection(optimization_inputs):
    for i in range(14):
        optimization_inputs['observations_board'][10,i,i,0] += 0.8


model = mrcal.cameramodel('/home/dima/projects/mrcal-doc-external/2022-11-05--dtla-overpass--samyang--alpha7/2-f22-infinity/splined.cameramodel')
optimization_inputs = model.optimization_inputs()

mrcal.make_perfect_observations(optimization_inputs)
add_imperfection(optimization_inputs)
mrcal.optimize(**optimization_inputs)

mrcal.show_residuals_board_observation(optimization_inputs, 10,
                                       vectorscale =  200,
                                       circlescale =  0.5,
                                       hardcopy =  "~/projects/mrcal-doc-external/figures/residuals/chessboard-detection-errors.png",
                                       title =  "Effect of corner detection error at the center",
                                       terminal =  'pngcairo size 1024,550 transparent noenhanced crop          font ",12"',
                                       paths   = optimization_inputs['imagepaths'])
mrcal.show_residuals_board_observation(optimization_inputs, 10,
                                       vectorscale =  200,
                                       circlescale =  0.5,
                                       hardcopy =  "~/projects/mrcal-doc-external/figures/residuals/chessboard-detection-errors.pdf",
                                       title =  "Effect of corner detection error at the center",
                                       terminal =  'pdf size 8in,6in noenhanced solid color   font ",16"',
                                       paths   = optimization_inputs['imagepaths'])
import os
os.system("pdfcrop ~/projects/mrcal-doc-external/figures/residuals/chessboard-detection-errors.pdf")

#+end_src

With a model on disk, the same could be produced with

#+begin_src sh
mrcal-show-residuals-board-observation                       \
    --vectorscale 200                                        \
    --circlescale 0.5                                        \
    --title "Effect of corner detection error at the center" \
    splined.cameramodel                                      \
    10
#+end_src

[[file:external/figures/residuals/chessboard-detection-errors.png]]

Here we see the residuals on the top-left/bottom-right diagonal be consistently
larger than the others, and we see them point in a consistent non-random
direction. This isn't easily noticeable without knowing what to look for. /But/
the usual method of zooming in to the worst-few points will make the error in
the detected corners visibly apparent.

*** Rolling shutter
:PROPERTIES:
:CUSTOM_ID: residuals-rolling-shutter
:END:

Many cameras employ a [[https://en.wikipedia.org/wiki/Rolling_shutter][rolling shutter]]: the images aren't captured all at once,
but are built up over time, capturing different parts of the image at different
times. If the scene or the camera are moving, this would produce images that
violate mrcal's view of the world: that at an instant in time I can describe the
/full/ chessboard pose and its observed corners. mrcal does not model rolling
shutter effects, so non-static calibration images would cause problems. Today
the only way to calibrate rolling-shutter cameras with mrcal is to make sure the
cameras and the chessboard are stationary during each image capture.

How do we make sure that no rolling-shutter effects ended up in our data? We
look at the residuals. Prior to the dataset used in [[file:tour-initial-calibration.org][the tour of mrcal]] I captured
images that used the "silent mode" of the Sony Alpha 7 III camera. I didn't
realize that this mode implied a rolling shutter, so [[file:external/2022-10-17--dtla-overpass--samyang--alpha7][that dataset]] wasn't useful
to demo anything other than the rolling shutter residuals. Looking at the worst
few observations:

#+begin_src sh
mrcal-show-residuals-board-observation \
  --from-worst                         \
  --vectorscale 20                     \
  --circlescale 0.5                    \
  splined.cameramodel                  \
  0-3
#+end_src
#+begin_src sh :exports none :eval no-export
D=~/projects/mrcal-doc-external/2022-10-17--dtla-overpass--samyang--alpha7/2-f22-infinity/
for i (`seq 0 3`) {
  mrcal-show-residuals-board-observation \
    --from-worst \
    --vectorscale 40 \
    --circlescale 0.5 \
    --title "Residuals from a rolling shutter camera" \
    --hardcopy "~/projects/mrcal-doc-external/figures/residuals/rolling-shutter-$i.png" \
    --terminal 'pngcairo size 1024,768 transparent noenhanced crop          font ",12"' \
    $D/splined.cameramodel \
    $i
  mrcal-show-residuals-board-observation \
    --from-worst \
    --vectorscale 40 \
    --circlescale 0.25 \
    --title "Residuals from a rolling shutter camera" \
    --hardcopy "~/projects/mrcal-doc-external/figures/residuals/rolling-shutter-$i.svg" \
    --terminal 'svg size 800,400 noenhanced solid dynamic font ",14"' \
    $D/splined.cameramodel \
    $i
  mrcal-show-residuals-board-observation \
    --from-worst \
    --vectorscale 40 \
    --circlescale 0.25 \
    --title "Residuals from a rolling shutter camera" \
    --hardcopy "~/projects/mrcal-doc-external/figures/residuals/rolling-shutter-$i.pdf" \
    --terminal 'pdf size 8in,6in noenhanced solid color   font ",16"' \
    $D/splined.cameramodel \
    $i
  pdfcrop ~/projects/mrcal-doc-external/figures/residuals/rolling-shutter-$i.pdf
}
#+end_src

[[file:external/figures/residuals/rolling-shutter-0.svg]]

[[file:external/figures/residuals/rolling-shutter-2.svg]]

[[file:external/figures/residuals/rolling-shutter-3.svg]]

I'm omitting the 2nd image because it qualitatively looks very similar to the
first. Note the patterns. Clearly something is happening, but it varies from
image to image, and a warped chessboard isn't likely to explain it. In addition
to that, the magnitude of all the errors is dramatically higher than before: the
vectors are scaled up 5x less than those in [[file:tour-initial-calibration.org][the tour of mrcal]].

*** Synchronization
:PROPERTIES:
:CUSTOM_ID: residuals-sync
:END:

What if we're calibrating a multi-camera system, and the image synchronization
is broken? You should have a hardware sync: a physical trigger wire connected to
each camera, with a pulse on that line telling each camera to begin the image
capture. If this is missing or doesn't work properly, then a similar issue
occurs as with a [[#residuals-rolling-shutter][rolling-shutter]]: images captured at allegedly a particular
instant in time haven't actually been captured completely at /that/ time.

This can be confirmed by recalibrating monocularly, to see if individually the
solves converge. And as expected, this can be readily seen in the residuals. A
time sync problem means that synced images A and B were supposed to capture a
chessboard at some pose $T$. But since the sync was broken, the chessboard
moved, so the chessboard was really at two different poses: $T_\mathrm{A}$ and
$T_\mathrm{B}$. mrcal was told that there was no motion, so it ends up solving
for some compromise transform $T_\mathrm{mid}$ that splits the difference. So
the residuals for image A would point from the observation at $T_\mathrm{A}$
towards the observation at the compromise pose $T_\mathrm{mid}$. And the
residuals for image B would point in the opposite direction: from the
observation at $T_\mathrm{B}$ towards the observation at the compromise pose
$T_\mathrm{mid}$.

Let's demo this with a little code. I grab the monocular solve from [[file:tour-initial-calibration.org][the tour of
mrcal]], add a second camera with perfect observations and noise, and add the
flaw: one of the chessboard observations was erroneously reported for two images
in a row. This happened in only one camera of the pair, so a time-sync error
resulted. This is a simulation, but as with everything else on this page, I've
seen such problems in real life. The incorrect observation is observation 40,
frame 20, camera 0.

#+begin_src python
#!/usr/bin/python3

import sys
import mrcal
import numpy as np
import numpysane as nps

def binocular_from_monocular(optimization_inputs):
    # Assuming monocular calibration. Add second identical camera to the right
    # of the first
    optimization_inputs['intrinsics'] = \
        nps.glue( optimization_inputs['intrinsics'],
                  optimization_inputs['intrinsics'],
                  axis = -2 )
    optimization_inputs['imagersizes'] = \
        nps.glue( optimization_inputs['imagersizes'],
                  optimization_inputs['imagersizes'],
                  axis = -2 )
    optimization_inputs['extrinsics_rt_fromref'] = np.array(((0,0,0, -0.1,0,0),), dtype=float)
    i = optimization_inputs['indices_frame_camintrinsics_camextrinsics']
    optimization_inputs['indices_frame_camintrinsics_camextrinsics'] = \
        np.ascontiguousarray( \
          nps.clump(nps.xchg(nps.cat(i, i + np.array((0,1,1),dtype=np.int32)), 0, 1), n=2))

    # optimization_inputs['observations_board'] needs to have the right shape
    # and the right weights. So I just duplicate it
    optimization_inputs['observations_board'] = \
        np.ascontiguousarray( \
          nps.clump(nps.xchg(nps.cat(optimization_inputs['observations_board'],
                                     optimization_inputs['observations_board']),
                             0,1),
                    n=2))
    del optimization_inputs['imagepaths']

def add_imperfection(optimization_inputs):
    iobservation = 40
    iframe,icam_intrinsics,icam_extrinsics = \
        optimization_inputs['indices_frame_camintrinsics_camextrinsics'][iobservation]
    print(f"observation {iobservation} iframe {iframe} icam_intrinsics {icam_intrinsics} is stuck to the previous observation")

    optimization_inputs    ['observations_board'][iobservation,  ...] = \
        optimization_inputs['observations_board'][iobservation-2,...]


model = mrcal.cameramodel('splined.cameramodel')
optimization_inputs = model.optimization_inputs()

binocular_from_monocular(optimization_inputs)
mrcal.make_perfect_observations(optimization_inputs)
add_imperfection(optimization_inputs)

optimization_inputs['do_apply_outlier_rejection'] = False
mrcal.optimize(**optimization_inputs)

mrcal.cameramodel(optimization_inputs = optimization_inputs,
                  icam_intrinsics     = 0).write('/tmp/sync-error.cameramodel')

for i in range(2):
    mrcal.show_residuals_board_observation(optimization_inputs,
                                           i,
                                           from_worst  = True,
                                           vectorscale = 2,
                                           circlescale = 0.5)
#+end_src
#+begin_src python :exports none :eval no-export
#!/usr/bin/python3

import sys
import os
import os.path
import mrcal
import numpy as np
import numpysane as nps

def binocular_from_monocular(optimization_inputs):
    # Assuming monocular calibration. Add second identical camera to the right
    # of the first
    optimization_inputs['intrinsics'] = \
        nps.glue( optimization_inputs['intrinsics'],
                  optimization_inputs['intrinsics'],
                  axis = -2 )
    optimization_inputs['imagersizes'] = \
        nps.glue( optimization_inputs['imagersizes'],
                  optimization_inputs['imagersizes'],
                  axis = -2 )
    optimization_inputs['extrinsics_rt_fromref'] = np.array(((0,0,0, -0.1,0,0),), dtype=float)
    i = optimization_inputs['indices_frame_camintrinsics_camextrinsics']
    optimization_inputs['indices_frame_camintrinsics_camextrinsics'] = \
        np.ascontiguousarray( \
          nps.clump(nps.xchg(nps.cat(i, i + np.array((0,1,1),dtype=np.int32)), 0, 1), n=2))

    # optimization_inputs['observations_board'] needs to have the right shape
    # and the right weights. So I just duplicate it
    optimization_inputs['observations_board'] = \
        np.ascontiguousarray( \
          nps.clump(nps.xchg(nps.cat(optimization_inputs['observations_board'],
                                     optimization_inputs['observations_board']),
                             0,1),
                    n=2))
    del optimization_inputs['imagepaths']

def add_imperfection(optimization_inputs):
    iobservation = 40
    iframe,icam_intrinsics,icam_extrinsics = \
        optimization_inputs['indices_frame_camintrinsics_camextrinsics'][iobservation]
    print(f"observation {iobservation} iframe {iframe} icam_intrinsics {icam_intrinsics} is stuck to the previous observation")

    optimization_inputs    ['observations_board'][iobservation,  ...] = \
        optimization_inputs['observations_board'][iobservation-2,...]


model = mrcal.cameramodel('/home/dima/projects/mrcal-doc-external/2022-11-05--dtla-overpass--samyang--alpha7/2-f22-infinity/splined.cameramodel')
optimization_inputs = model.optimization_inputs()

binocular_from_monocular(optimization_inputs)
mrcal.make_perfect_observations(optimization_inputs)
add_imperfection(optimization_inputs)

optimization_inputs['do_apply_outlier_rejection'] = False
mrcal.optimize(**optimization_inputs)

mrcal.cameramodel(optimization_inputs = optimization_inputs,
                  icam_intrinsics     = 0).write('/tmp/sync-error.cameramodel')

for i in range(2):
    mrcal.show_residuals_board_observation(optimization_inputs,
                                           i,
                                           from_worst  = True,
                                           vectorscale = 2,
                                           circlescale = 0.5,
                                           title       = "Residuals from a broken camera sync",
                                           hardcopy    = f"~/projects/mrcal-doc-external/figures/residuals/sync-errors-{i}.png",
                                           terminal    = 'pngcairo size 1024,550 transparent noenhanced crop          font ",12"')
    mrcal.show_residuals_board_observation(optimization_inputs,
                                           i,
                                           from_worst  = True,
                                           vectorscale = 2,
                                           circlescale = 0.25,
                                           title       = "Residuals from a broken camera sync",
                                           hardcopy    = f"~/projects/mrcal-doc-external/figures/residuals/sync-errors-{i}.svg",
                                           terminal    =  'svg size 800,400 noenhanced solid dynamic font ",14"')
    mrcal.show_residuals_board_observation(optimization_inputs,
                                           i,
                                           from_worst  = True,
                                           vectorscale = 2,
                                           circlescale = 0.25,
                                           title       = "Residuals from a broken camera sync",
                                           hardcopy    = f"~/projects/mrcal-doc-external/figures/residuals/sync-errors-{i}.pdf",
                                           terminal    =  'pdf size 8in,6in noenhanced solid color   font ",16"')
    os.system(os.path.expanduser(f"pdfcrop ~/projects/mrcal-doc-external/figures/residuals/sync-errors-{i}.pdf"))
#+end_src

[[file:external/figures/residuals/sync-errors-0.svg]]

[[file:external/figures/residuals/sync-errors-1.svg]]

We're plotting the two worst-residual observations, and we clearly see the
tell-tale sign of a time-sync error.

As before, we can make these plots from the commandline:

#+begin_src sh
mrcal-show-residuals-board-observation \
    --from-worst                       \
    --vectorscale 2                    \
    --circlescale 0.5                  \
    /tmp/sync-error.cameramodel        \
    --explore                          \
    0-1
#+end_src

Here we passed =--explore= to drop into a REPL to investigate further, which
gives us another clear indication that we have a time-sync error. The tool
reports some useful diagnostics at the top of the REPL:

#+begin_example
The first 10 worst-fitting observations (i_observations_sorted_from_worst[:10])

[41, 40, 97, 94, 95, 38, 99, 96, 98, 39]


The corresponding 10 (iframe, icamintrinsics, icamextrinsics) tuples
(indices_frame_camintrinsics_camextrinsics[i_observations_sorted_from_worst[:10] ]):

[[20  1  0]
 [20  0 -1]
 [48  1  0]
 [47  0 -1]
 [47  1  0]
 [19  0 -1]
 [49  1  0]
 [48  0 -1]
 [49  0 -1]
 [19  1  0]]
#+end_example

Since a time-sync error affects multiple images at the same time, we should see
multiple chessboard observations from the same frame (instant in time) in the
high-residuals list, and we clearly see that above. We corrupted frame 20 camera
0, so it's no longer syncronized with frame 20 camera 1. We expect both of those
observations to fit badly, and we do see that: the
[[file:mrcal-show-residuals-board-observation.html][=mrcal-show-residuals-board-observation=]] tool says that the worst residual is
from frame 20 camera 1 and the second-worst is frame 20 camera 0. High errors in
multiple cameras in the same frame like this are another tell-tale sign of sync
errors. We can also query the errors themselves in the REPL:

#+begin_example
In [1]: err_per_observation[i_observations_sorted_from_worst][:10]
Out[1]: 
array([965407.29301702, 864665.95442895,  11840.10373732,   7762.88344105,
         7759.5509593 ,   7618.23632488,   7190.35449443,   6621.34798572,
         6499.23173619,   6039.45522539])
#+end_example

So the two frame-20 sum-of-squares residuals are on the order of $900,000
\mathrm{pixels}^2$, and the next worst one is ~ 100 times smaller.

*** Chessboard shape
:PROPERTIES:
:CUSTOM_ID: residuals-chessboard-shape
:END:

Since flat chessboards don't exist, mrcal doesn't assume that the observed
chessboard is flat. Today it uses [[file:formulation.org::#board-deformation][a simple parabolic model]]. It /does/ assume the
board shape is constant throughout the whole calibration sequence. So let's run
more simulations to test two scenarios:

1. The chessboard is slightly non-flat, but in a way not modeled by the solver
2. The chessboard shape changes slightly over the course of the chessboard dance

**** Unmodeled chessboard shape
As before, we rerun the scenario from [[file:tour-initial-calibration.org][the tour of mrcal]] with perfect
observations and perfect, but with a flaw: I use the very slightly parabolic
chessboard, but tell mrcal to assume the chessboard is flat. The script is very
similar:

#+begin_src python
#!/usr/bin/python3

import sys
import mrcal
import numpy as np
import numpysane as nps

def add_imperfection(optimization_inputs):
    H,W = optimization_inputs['observations_board'].shape[1:3]
    s   = optimization_inputs['calibration_object_spacing']
    print(f"Chessboard dimensions are {s*(W-1)}m x {s*(H-1)}m")
    print(f"Chessboard deflection at the center is {optimization_inputs['calobject_warp'][0]*1000:.1f}mm , {optimization_inputs['calobject_warp'][1]*1000:.1f}mm in the x,y direction")
    optimization_inputs['calobject_warp'] *= 0.
    optimization_inputs['do_optimize_calobject_warp'] = False


model = mrcal.cameramodel('splined.cameramodel')
optimization_inputs = model.optimization_inputs()

optimization_inputs['calobject_warp'] = np.array((1e-3, -0.5e-3))
mrcal.make_perfect_observations(optimization_inputs)
add_imperfection(optimization_inputs)
mrcal.optimize(**optimization_inputs)

mrcal.show_residuals_board_observation(optimization_inputs,
                                       0,
                                       from_worst  = True,
                                       vectorscale = 200,
                                       circlescale = 0.5)
#+end_src
#+begin_src python :exports none :eval no-export
#!/usr/bin/python3

import sys
import os
import os.path
import mrcal
import numpy as np
import numpysane as nps

def add_imperfection(optimization_inputs):
    H,W = optimization_inputs['observations_board'].shape[1:3]
    s   = optimization_inputs['calibration_object_spacing']
    print(f"Chessboard dimensions are {s*(W-1)}m x {s*(H-1)}m")
    print(f"Chessboard deflection at the center is {optimization_inputs['calobject_warp'][0]*1000:.1f}mm , {optimization_inputs['calobject_warp'][1]*1000:.1f}mm in the x,y direction")
    optimization_inputs['calobject_warp'] *= 0.
    optimization_inputs['do_optimize_calobject_warp'] = False


model = mrcal.cameramodel('/home/dima/projects/mrcal-doc-external/2022-11-05--dtla-overpass--samyang--alpha7/2-f22-infinity/splined.cameramodel')
optimization_inputs = model.optimization_inputs()

optimization_inputs['calobject_warp'] = np.array((1e-3, -0.5e-3))
mrcal.make_perfect_observations(optimization_inputs)
add_imperfection(optimization_inputs)
mrcal.optimize(**optimization_inputs)

mrcal.cameramodel(optimization_inputs = optimization_inputs,
                  icam_intrinsics     = 0).write('/tmp/unmodeled-chessboard-shape.cameramodel')

mrcal.show_residuals_board_observation(optimization_inputs,
                                       0,
                                       from_worst  = True,
                                       vectorscale = 200,
                                       circlescale = 0.5,
                                       title       = "Residuals from a static chessboard shape error",
                                       hardcopy    = f"~/projects/mrcal-doc-external/figures/residuals/unmodeled-chessboard-shape.png",
                                       terminal    = 'pngcairo size 1024,550 transparent noenhanced crop          font ",12"')
mrcal.show_residuals_board_observation(optimization_inputs,
                                       0,
                                       from_worst  = True,
                                       vectorscale = 200,
                                       circlescale = 0.25,
                                       title       = "Residuals from a static chessboard shape error",
                                       hardcopy    = f"~/projects/mrcal-doc-external/figures/residuals/unmodeled-chessboard-shape.svg",
                                       terminal    =  'svg size 800,400 noenhanced solid dynamic font ",14"')
mrcal.show_residuals_board_observation(optimization_inputs,
                                       0,
                                       from_worst  = True,
                                       vectorscale = 200,
                                       circlescale = 0.25,
                                       title       = "Residuals from a static chessboard shape error",
                                       hardcopy    = f"~/projects/mrcal-doc-external/figures/residuals/unmodeled-chessboard-shape.pdf",
                                       terminal    =  'pdf size 8in,6in noenhanced solid color   font ",16"')

os.system(os.path.expanduser(f"pdfcrop ~/projects/mrcal-doc-external/figures/residuals/unmodeled-chessboard-shape.pdf"))
#+end_src

It says:

#+begin_example
Chessboard dimensions are 0.7644m x 0.7644m
Chessboard deflection at the center is 1.0mm , -0.5mm in the x,y direction
#+end_example

And the worst residual image looks like this:

[[file:external/figures/residuals/unmodeled-chessboard-shape.svg]]

It usually looks similar to the residuals from a poorly-modeled lens (chessboard
edges tend to be observed at the edges of the lens). In /this/ solve the
residuals all look low-ish at the center, much bigger at the edges, and
consistent at the edges.

**** Unstable chessboard shape

What if the chessboard flexes a little bit? I redo my solve using a small
parabolic deflection at the first half of the sequence and twice as much
deformation during the second half.

#+begin_src python
#!/usr/bin/python3

import sys
import mrcal
import numpy as np
import numpysane as nps

def add_imperfection(optimization_inputs, observed_pixel_uncertainty):
    Nobservations = len(optimization_inputs['observations_board'])
    observations_half = np.array(optimization_inputs['observations_board'][:Nobservations//2,...])

    optimization_inputs['calobject_warp'] *= 2.
    mrcal.make_perfect_observations(optimization_inputs,
                                    observed_pixel_uncertainty = observed_pixel_uncertainty)

    optimization_inputs['observations_board'][:Nobservations//2,...] = observations_half

    H,W = optimization_inputs['observations_board'].shape[1:3]
    s   = optimization_inputs['calibration_object_spacing']
    print(f"Chessboard dimensions are {s*(W-1)}m x {s*(H-1)}m")
    print(f"Chessboard deflection at the center (second half of dataset) is {optimization_inputs['calobject_warp'][0]*1000:.2f}mm , {optimization_inputs['calobject_warp'][1]*1000:.2f}mm in the x,y direction")
    print(f"Chessboard deflection at the center at the first half of the dataset is half of that")


model = mrcal.cameramodel('splined.cameramodel')
optimization_inputs = model.optimization_inputs()

observed_pixel_uncertainty = np.std(mrcal.residuals_board(optimization_inputs).ravel())

optimization_inputs['calobject_warp'] = np.array((1e-3, -0.5e-3))
mrcal.make_perfect_observations(optimization_inputs,
                                observed_pixel_uncertainty = observed_pixel_uncertainty)

add_imperfection(optimization_inputs, observed_pixel_uncertainty)
mrcal.optimize(**optimization_inputs)

print(f"Optimized chessboard deflection at the center is {optimization_inputs['calobject_warp'][0]*1000:.2f}mm , {optimization_inputs['calobject_warp'][1]*1000:.2f}mm in the x,y direction")


mrcal.cameramodel(optimization_inputs = optimization_inputs,
                  icam_intrinsics     = 0).write('/tmp/unstable-chessboard-shape.cameramodel')

mrcal.show_residuals_board_observation(optimization_inputs,
                                       0,
                                       from_worst  = True,
                                       vectorscale = 200,
                                       circlescale = 0.5)
#+end_src
#+begin_src python :exports none :eval no-export
#!/usr/bin/python3

import sys
import os
import os.path
import mrcal
import numpy as np
import numpysane as nps

def add_imperfection(optimization_inputs, observed_pixel_uncertainty):
    Nobservations = len(optimization_inputs['observations_board'])
    observations_half = np.array(optimization_inputs['observations_board'][:Nobservations//2,...])

    optimization_inputs['calobject_warp'] *= 2.
    mrcal.make_perfect_observations(optimization_inputs,
                                    observed_pixel_uncertainty = observed_pixel_uncertainty)

    optimization_inputs['observations_board'][:Nobservations//2,...] = observations_half

    H,W = optimization_inputs['observations_board'].shape[1:3]
    s   = optimization_inputs['calibration_object_spacing']
    print(f"Chessboard dimensions are {s*(W-1)}m x {s*(H-1)}m")
    print(f"Chessboard deflection at the center (second half of dataset) is {optimization_inputs['calobject_warp'][0]*1000:.2f}mm , {optimization_inputs['calobject_warp'][1]*1000:.2f}mm in the x,y direction")
    print(f"Chessboard deflection at the center at the first half of the dataset is half of that")


model = mrcal.cameramodel('/home/dima/projects/mrcal-doc-external/2022-11-05--dtla-overpass--samyang--alpha7/2-f22-infinity/splined.cameramodel')
optimization_inputs = model.optimization_inputs()

observed_pixel_uncertainty = np.std(mrcal.residuals_board(optimization_inputs).ravel())

optimization_inputs['calobject_warp'] = np.array((1e-3, -0.5e-3))
mrcal.make_perfect_observations(optimization_inputs,
                                observed_pixel_uncertainty = observed_pixel_uncertainty)

add_imperfection(optimization_inputs, observed_pixel_uncertainty)
mrcal.optimize(**optimization_inputs)

print(f"Optimized chessboard deflection at the center is {optimization_inputs['calobject_warp'][0]*1000:.2f}mm , {optimization_inputs['calobject_warp'][1]*1000:.2f}mm in the x,y direction")


mrcal.cameramodel(optimization_inputs = optimization_inputs,
                  icam_intrinsics     = 0).write('/tmp/unstable-chessboard-shape.cameramodel')

mrcal.show_residuals_board_observation(optimization_inputs,
                                       0,
                                       from_worst  = True,
                                       vectorscale = 200,
                                       circlescale = 0.5,
                                       title       = "Residuals from a dynamic chessboard shape error",
                                       hardcopy    = f"~/projects/mrcal-doc-external/figures/residuals/unstable-chessboard-shape.png",
                                       terminal    = 'pngcairo size 1024,550 transparent noenhanced crop          font ",12"')
mrcal.show_residuals_board_observation(optimization_inputs,
                                       0,
                                       from_worst  = True,
                                       vectorscale = 200,
                                       circlescale = 0.25,
                                       title       = "Residuals from a dynamic chessboard shape error",
                                       hardcopy    = f"~/projects/mrcal-doc-external/figures/residuals/unstable-chessboard-shape.svg",
                                       terminal    =  'svg size 800,400 noenhanced solid dynamic font ",14"')
mrcal.show_residuals_board_observation(optimization_inputs,
                                       0,
                                       from_worst  = True,
                                       vectorscale = 200,
                                       circlescale = 0.25,
                                       title       = "Residuals from a dynamic chessboard shape error",
                                       hardcopy    = f"~/projects/mrcal-doc-external/figures/residuals/unstable-chessboard-shape.pdf",
                                       terminal    =  'pdf size 8in,6in noenhanced solid color   font ",16"')
os.system(os.path.expanduser(f"pdfcrop ~/projects/mrcal-doc-external/figures/residuals/unstable-chessboard-shape.pdf"))
#+end_src

It says:

#+begin_example
Chessboard dimensions are 0.7644m x 0.7644m
Chessboard deflection at the center (second half of dataset) is 2.00mm , -1.00mm in the x,y direction
Chessboard deflection at the center at the first half of the dataset is half of that
Optimized chessboard deflection at the center is 1.66mm , -0.79mm in the x,y direction
#+end_example

[[file:external/figures/residuals/unstable-chessboard-shape.svg]]

It looks a bit like the unmodeled chessboard residuals above, but smaller, and
more subtle. But lest you think that these small and subtle residuals imply that
this doesn't affect the solution, here's the resulting difference in projection:

#+begin_src sh
mrcal-show-projection-diff                   \
  --no-uncertainties                         \
  --radius 500                               \
  --unset key \
  /tmp/unstable-chessboard-shape.cameramodel \
  splined.cameramodel
#+end_src
#+begin_src sh :exports none :eval no-export
mrcal-show-projection-diff \
  --no-uncertainties \
  --radius 500 \
  --unset key \
  /tmp/unstable-chessboard-shape.cameramodel \
  ~/projects/mrcal-doc-external/2022-11-05--dtla-overpass--samyang--alpha7/2-f22-infinity/splined.cameramodel \
  --hardcopy ~/projects/mrcal-doc-external/figures/residuals/unstable-chessboard-shape-diff.png \
  --terminal 'pngcairo size 1024,768 transparent noenhanced crop font ",12"'
#+end_src

[[file:external/figures/residuals/unstable-chessboard-shape-diff.png]]

So [[#chessboard-shape][take care to make your chessboard rigid]]. I can think of better visualizations
to more clearly identify these kinds of issues, so this might improve in the
future.

For both of these chessboard deformation cases it's helpful to look at more than
just 1 or 2 worst-case residuals. Look at the worst 10. In both cases the most
tilted chessboard observations usually show very consistent residual vectors
along the far edge of the chessboard.

