Camera Calibration
Obtaining the tool The camera calibration tool is not currently distributed with the platform. To request it, please contact us at aide.support@advantech.com.
Purpose of calibration
Imagery in the simulation environment is rendered by the simulator, and the position and rotation of the top camera are configurable. That set of values determines the field of view of the virtual camera, and hence the framing of every image in the generated dataset.
Where this field of view does not correspond to that of the physical camera, the simulated and physical datasets present two different framings of the same scene. A model trained on such data will still converge normally, but at inference on physical hardware the object appears at a position in the frame different from that seen during training, and performance degrades markedly. This is one of the contributing causes of the Sim-to-Real gap.
Camera calibration is the means of eliminating that divergence: the lens characteristics and mounting pose of the physical camera are measured, and the resulting values are entered into the virtual camera of the simulation environment, so that the framing of the two corresponds.
This procedure and arm calibration are of the same nature but applied to different subjects:
| Procedure | Purpose |
|---|---|
| Arm calibration | Makes the joint readings of the physical arm correspond to physical reality |
| Camera calibration (this chapter) | Makes the imagery of the virtual camera correspond to the physical camera |
The two sets of measured parameters
| Parameter | Content | When re-measurement is required |
|---|---|---|
| Intrinsics | Focal length, principal point, and distortion coefficients of the lens itself | On changing the camera or altering the resolution |
| Extrinsics | Position and orientation of the camera relative to the world origin | On moving the camera or altering its mounting position |
Intrinsics are determined by the camera hardware and are independent of placement; a single measurement per camera may be used indefinitely. Extrinsics describe where the camera is and where it points, and must be re-measured whenever the camera is moved.
Preparation
Checkerboard
Calibration uses a checkerboard as its measurement reference. The principle is as follows: the number of squares and the square edge length are known, so once the corners have been detected in an image, the lens parameters and the camera pose can be derived.
| Item | Specification |
|---|---|
| Pattern | OpenCV official pattern.png; the download link is given in the tool documentation |
| Inner corners | 9 × 6 (that is, 10 × 7 squares) |
| Square edge length | 25 mm |
Printing and mounting When printing, 100% actual size must be selected; no scaling may be applied, as otherwise the square edge length will not match the program setting and all measurements will be distorted proportionally. Once printed, the pattern must be mounted on cardboard or acrylic to keep it flat; any warping of the paper translates directly into deviation in corner positions.
Where a checkerboard of a different specification is used, the number of inner corners and the square edge length must be specified at execution.
Camera
A general-purpose USB 2D camera is sufficient for calibration; no depth camera is required. What is measured here is the imaging characteristics and mounting pose of the lens, for which depth information serves no purpose.
The camera must already be set up and managed with its connection confirmed. Extrinsics measure the camera's placement at the time of capture, and the camera must therefore be fixed in its final mounting position beforehand and must not be moved once calibration is complete.
Calibration procedure
Step 1: Capture images for intrinsics
Capture 15 to 20 checkerboard images using the calibration tool. The purpose of this step is to allow the program to observe a single known pattern from multiple viewpoints and thereby solve for the lens parameters; the variation between images is accordingly the source of data quality:
- Cover a range of angles: frontal, tilted left and right, pitched up and down
- Cover a range of distances, from near to far
- The checkerboard must appear in full, neither cropped by the frame edge nor occluded by objects
- Avoid overexposure, backlighting, and motion blur
Twenty images captured from an identical angle are equivalent in effect to a single image.
Step 2: Capture the image for extrinsics
Place the checkerboard flat at the position to be defined as the world origin and capture one image.
This placement establishes the datum for the entire coordinate system: the camera position subsequently computed is relative to the checkerboard position in this image. In practice it is recommended that the checkerboard be laid flat at a fixed position on the workbench relative to the arm base, and that the position be recorded so that it may be reproduced later.
Step 3: Run the calibration
Run the calibration program; intrinsics and extrinsics may be completed in a single pass. The program reports, in order:
- The number of valid images and the reprojection error
- The intrinsic matrix and distortion coefficients
- The position and orientation of the camera in world coordinates
- The converted values for direct entry into the simulation environment
Two files, intrinsics.json and extrinsics.json, are produced for subsequent use in pose replication.
Interpreting calibration quality
The reprojection error reported by the program is the sole acceptance criterion. Its meaning is as follows: the computed parameters are used to project the corner positions back onto the image, and the result is compared, in pixels, with the positions actually detected.
| Error range | Assessment | Action |
|---|---|---|
| < 0.5 px | Good | May be used as is |
| 0.5 – 1.0 px | Acceptable | May be used; capture additional images if greater precision is required |
| > 1.0 px | Unacceptable | Recapture required |
Where the error is high, the cause is generally one of three: the checkerboard was not flat, the capture angles were insufficiently varied, or some images were blurred or overexposed. Poor images should be excluded and the procedure repeated rather than proceeding with an unacceptable result — unacceptable intrinsics are inherited in full by the subsequent extrinsics computation.
Where the program reports that corners could not be detected in a number of images, the inner corner setting should first be confirmed. The figure required is the number of corners, not of squares: a 10 × 7 checkerboard corresponds to 9 × 6 inner corners.
Entering the values into the simulation environment
On completion, the output values are entered into the position and rotation fields of the Simulation Environment Settings → Top camera tab; see Simulation Environment & SDG Generation.
| Output item | Target field |
|---|---|
| Position X / Y / Z | Top camera position |
| Rotation X / Y / Z | Top camera rotation |
Visual confirmation is required after entry Once entered, the simulation view should be compared with the actual view of the physical camera to confirm that the framing corresponds. Coordinate system conversion involves several differences of convention, and residual deviation may remain after the values are entered; where an evident discrepancy is observed, the rotation values may be adjusted by small increments until the framing agrees. This visual confirmation is of the same nature as the confirmation of the virtual arm described under alignment of the virtual and physical.
The randomization range is unaffected by calibration and is configured according to the principles set out under randomization range of the top camera. What calibration determines is the centre value of that range.
Deployment across hosts: pose replication
When a trained model is deployed to further stations, camera mounting positions cannot be entirely identical between them, and the resulting differences cause post-deployment performance to diverge from the validated result.
The calibration toolset provides a further pose guidance function whose purpose is to replicate an existing camera placement at another site: intrinsics.json and extrinsics.json from the original site are loaded, and the tool compares the current camera pose against the target in real time, displaying the deviation on each axis and indicating by colour whether it lies within tolerance, so that adjustment may be guided until the two correspond.
The precondition for this function is that the position and orientation of the checkerboard relative to the arm at the new site must correspond to those at the original site. Extrinsics record the relative pose of camera to checkerboard; for the relation of camera to arm to correspond, the relation of checkerboard to arm must first correspond.
Completion criteria for this stage
- The reprojection error falls within the acceptable range (< 1.0 px)
intrinsics.jsonandextrinsics.jsonhave been produced and stored securely- The framing of the top camera in the simulation environment corresponds to that of the physical camera
- The camera has not been moved since calibration
The last of these is the most readily overlooked. Should the camera be displaced after calibration by cable management, cleaning, or accidental contact, the extrinsics are invalidated and Steps 2 and 3 must be repeated; the intrinsics are unaffected and need not be re-measured.
Next: Policy selection.