Camera Calibration (isaacsim.replicator.agent.camera_calibration)#

isaacsim.replicator.agent.camera_calibration provides camera metadata for cameras generated by IRA. It can be used to generate the stage layout image with each camera’s FOV polygons. Camera calibration data such as the camera’s direction, location, and FOV polygon information can also be captured and saved to a .json file.

Enable isaacsim.replicator.agent.camera_calibration#

  1. Open the extension manager by Window > Extensions. In the extension manager, search and enable the isaacsim.replicator.agent.camera_calibration extension.

  2. The IRA camera calibration UI will automatically be opened on the right side of the screen. It is accessible by Tools > Replicator > Calibration Tool.

Calibration Tool Components#

UI Layout#

Below is an image showing the Camera Calibration UI layout. It is accessible by Tools > Replicator > Calibration Tool.

an example of what the UI looks like for the camera calibration extension

Input Fields#

Place Info: A string that describes the location of the scene, including city, building, and room. It will be converted and stored in calibration.json. See the output example for more details.

  • Input Format: city=[city name]/building=[building name]/room=[room name]

  • Example:

    • Input: city=Santa Clara/building=NVIDIA Voyager/room=Visitor Lobby

    • Output: in calibration.json:

    {
    "place": [
        {
        "name": "city",
        "value": "Santa Clara"
        },
        {
            "name": "building",
            "value": "NVIDIA Voyager"
        },
        {
            "name": "room".
            "value": "Visitor Lobby"
        }
    ],
}

Scene Root Prim Path: The path to the root prim of the scene. This is used to approximate the top view camera’s position. The top view camera will look at the scene root prim’s center.

Top View Camera: This camera will be used to render the top_view images.

  • Create: Once Scene Root Prim Path is set, clicking this button will automatically generate a topview camera that looks at the scene root’s center. The top view camera will be generated under /World/Top_Camera.

  • Path: After clicking Create, the top view camera’s path will be shown here. You can also use this field to select existing top view cameras in the stage.

Note

  • The Top View Camera must be vertical to the ground and it must cover the position of all the calibration dots under World/Calibration_Dots and cameras under the World/Cameras.

  • The Top View Camera should have a rotation of [0,0,0] with the projection type set to “orthographic”.

Raycast Density: The density of the raycast. The higher this value, the more detailed the FOV contour will be. A density value of N indicates that N * N rays will be cast and they are uniformly distributed for each camera.

  • Default value: 100

Minimum FOV Polygon Edge Length (meter): The minimum length of edges in the polygon’s contour. Edges shorter than this length are ignored and the vertices are connected to the next point that meets this criteria. The unit is meter.

  • Default value: 0 (no simplification in drawing the contour)

Minimum Area of FOV Polygon Hole to Ignore: When generating data, holes in the FOV polygon that are smaller than this threshold value are ignored. Holes are the areas that are not included in the FOV polygon.

  • Default value: 0 (don’t ignore any hole in FOV polygons)

Create Camera View Images: Whether to include camera view images in the output folder.

Create FOV Polygon Images: Whether to render top view image with FOV polygon in debug data folder.

Show FOV Polygon: Whether to show FOV polygons from the currently selected camera.

Output Folder Path: The path to the output folder. Click on the folder icon to select the output folder path.

Button and Functions#

Note

Before start generating the calibration file, the following prerequisites must be met:

  • Top View Camera Path field is set up with a valid camera prim path to a valid top view camera.

  • The output folder path value must be valid.

  • The Place Info must have the correct format and input.

  • The cameras must be under /World/Cameras.

  • Create Dot Prims: Generate calibration dot prims for each camera. Calibration dots will be randomly generated and they are used to sample the polygons contour that each camera can view.

  • Generate Calibration File: Create calibration.json that stores all the camera calibration data.

    Note

    You must run Create Dot Prims before generating the calibration file.

  • Generate Top View Image: Generate the Top View image and store the image in the output folder. An imageMetadata.json file will be generated to store the image metadata.

    Note

    • If the Create FOV Polygon Images is checked, the FOV polygon is visualized on the topview layout. The FOV images are generated in a debug data folder within the output folder.

    • You must have the current camera viewport set to your top view camera.

Walkthrough#

Here is a walkthrough for using the Camera Calibration tool. We are using the Isaac Sim Full Warehouse for demonstration.

Note

  • Stage unit must be in meters.

  • A valid NavMesh is required.

Step 1 - Enable the Extension#

Enable isaacsim.replicator.agent.camera_calibration and its UI

Step 2 - Create Cameras#

Add two cameras under /World/Cameras as sample cameras. Ideally, the cameras should be able to view the walkable area of the scene.

Tip

To add cameras to the stage, follow the Isaac Sim Camera tutorial.
Alternatively, you can use IRA to spawn cameras.

Step 3 - Create Top View Camera#

This step is to generate a top view camera with the help of Section Tool extension (omni.kit.window.section). Section Tool uses a horizontal plane to slice the stage, and it can help remove the ceiling from the view, which may block the camera’s view of the floorplan.

  1. Adjust the perspective camera’s view so that you can overview the whole environment. This is for adjusting the section tool height in the next step.

  2. Search “Section” from Window > Extension UI and enable the omni.kit.window.section extension. The Section Tool UI will be enabled and docked on the right side of the screen. It can be re-enabled from Tools > Section.

  3. To create an optimal top view we need to exclude objects like the ceiling. We can do so by manipulate the transform of the Section Tool plane /SectionTools/Section_Tool_Object.

    • You can use the move tool to drag the plane up and down to slice the scene.

    • Alternatively, you can also directly set the transform of the plane from the Property window on the bottom right of the screen. Set the Transform for the plane to the same as the below iamge for optimal result.

    ../../_images/isim_4.5_full_ext-isaacsim.replicator.agent.camera_calibration-5.0.0_viewport_adjust_section_tool_plane_property_window.png

    Note

    You can hide the Section Tool plane by unchecking Options > Display Section Manipulator in its UI.

  4. Set the Scene Root Prim Path. The Top View Camera generated by this tool will look at this scene root. In this case, set it to /Root.

  5. Click Create and the top view camera will be generated and its path will be shown in the text field.

  6. Switch the viewport to the new top view camera to verify that it covers the floorplan.

Tip

To switch the viewport to the top-view camera, click the Camera icon, then click Cameras > Calibration_Top_Camera camera.

../../_images/isim_4.5_full_ext-isaacsim.replicator.agent.camera_calibration-5.0.0_viewport_switch_viewport.png

Step 4 - Set Up the Calibration Tool Attributes#

This step is to enter the information needed for camera calibration.

  1. Enter the place information in Place Info. In this case, it’s city=Santa Clara/building=Isaac Sim Warehouse/room=Warehouse.

  2. Set Raycast Density, Minimum FOV Polygon Edge Length, and Minimum Area of FOV Polygon Hole to Ignore. See the Input Field for more details. In this case, we are using the default values.

  3. Check the Create Camera View Images, Create FOV Polygon Images, and Show FOV Polygon boxes if these additional data is neeeded.

  4. Set Output Folder Path by either entering the path or clicking the folder picker icon.

Step 5 - Generate Calibration Dots#

Generate Calibration Dots for each Camera by clicking on the Create Dot Prims button.

Note

  • Calibration dot prims are generated under /World/Calibration_Dots/[Camera Name]/, where [Camera Name] is the name of the camera.

  • For each Camera prim under /World/Cameras, six calibration dots are generated. The dot prims are used to calculate the projection matrix for each camera.

  • You can switch your viewport to any camera’s view to check whether all the calibration dots are visible.

Step 6 - Generate Calibration File#

  1. Generate the calibration file by clicking on the Generate Calibration File button. This generates a calibration.json file to Output Folder Path.

  2. After the calibration.json file is generated. You can visualize the FOV in the stage by selecting the target camera.

Note

Your result may look different because it depends on the camera parameters. In our case, the translate of the camera is (-13.02311, 7.20828, 5.0), the orient is (-55.253, -56.035, -150.088), and the camera focal length is 20.94.

Step 7 - Generate Top View Image#

To visualize the generated FOV polygon top view image, generate the image by clicking on Generate Top View Image button.

The Top View Camera’s view will be rendered and output to the Output Folder Path]. An imageMetadata.json file is also generated to store image metadata.

Note

If Create FOV Polygon Images is checked, for each camera there will be an image with a white-shaded FOV polygon from the top view. The FOV polygon images will be generated under [Output Folder Path]/Debug/fieldOfViewPolygon.

../../_images/isim_4.5_full_ext-isaacsim.replicator.agent.camera_calibration-5.0.0_viewport_fov_polygon.png
On this page