Required inputs

The behavior tree generation workflow in omni.ai.behavior_tree_gen can be used in two ways:

• Manually, by filling in the UI.

• From bundled examples, where most file inputs are pre-populated automatically.

Base inputs for every run

These inputs form the first layer of a run:

• Scenario text: Natural-language task or goal, such as Anna picks up the CardBox_A and places it on the Table..

• Output folder: Writable folder where generated behavior trees and reusable cache data are stored.

• Model endpoint credentials: Read from the app environment during runtime preparation. The NVIDIA-hosted sample profiles use NVIDIA_API_KEY; unauthenticated local model servers need no token. Refer to Endpoint authentication.

• Context files: One or more JSON files containing the actor and object instances for the scene, including their metadata values.

• Node catalog files: One or more JSON files describing the behavior tree nodes the pipeline is allowed to use.

Note

Context files should be read together with the actor and object schema files. The context provides the concrete scene data, while the schema defines the metadata fields and structure that give that data planner-visible meaning.

Advanced direct inputs

These inputs shape metadata interpretation and runtime model routing:

• Actor schema file: Defines the structure of actor metadata used by the pipeline.

• Object schema file: Defines the structure of object metadata used by the pipeline.

• Model selection config: Selects the default chat and embedding profiles, or auto for automatic selection, and can define per-node overrides that drive runtime behavior.

• Embedding model configs: Defines the named embedding profiles referenced by the model selection config during RAG and retriever preparation.

• Named model configs: Defines the named chat-model profiles referenced by model selection and per-node routing.

• Node-to-model map: Provides explicit routing from planner nodes to named chat models when runtime behavior must be controlled per node.

Important

Changing the embedding model invalidates any vector store built with the previous one. Delete the rag_storage directory in the configured output folder before the next run so the store is rebuilt. If the replacement model happens to produce embeddings of the same dimension, a stale store loads without raising an error while comparing vectors from a different embedding space, which degrades retrieval silently instead of failing.

Model configuration and availability

Three JSON files select which models a run uses. In the UI, select them under Model Config > Model Configuration Files:

UI field	File	Purpose
Model Selection	model_selection_config.json	Selects the chat and embedding profile names and optional per-node overrides.
Embedding Models	embedding_model_configs.json	Registers the embedding endpoint and its discovery preference.
LLM Models	model_configs.json	Registers the chat endpoint, generation options, and discovery preference.

The bundled files are adaptable samples rather than working defaults: each contains an informational _comment describing what to adapt, which the loader ignores.

Automatic model selection

The sample profiles set "model": "auto" for both chat and embedding models; no remote model ID is pinned or used as a fallback. During runtime preparation, the planner queries each endpoint’s model catalog, ranks the candidates, and sends each one a small readiness request until a model answers or a 60-second budget expires. Catalog entries whose deployments are gone (for example, HTTP 404) are skipped; authentication and quota failures stop selection immediately. The selected models are recorded for the session, and the Planner status names them.

The planner needs an instruction-following chat model that can produce structured JSON with enough context for the scene and behavior catalog, and a text embedding model served through an OpenAI-compatible embeddings API. A successful readiness request does not guarantee output quality or sufficient context.

To pin a model instead, set model in the profile to an explicit model ID. Explicit IDs also work with endpoints that do not support model listing. A per-node override set to auto inherits the selected planner model.

Warning

Models hosted at integrate.api.nvidia.com can be retired at any time. Automatic selection moves on to another available model, but a pinned model ID stops working. For a deployment that must keep working, host the model yourself.

Example model configuration files

The values below only show the file format. Replace each <...> placeholder with the values for your own deployment.

Models hosted by NVIDIA

Export NVIDIA_API_KEY before launching Isaac Sim; the key is detected from the URL, so the profiles do not name it.

model_configs.json

{
  "models": [
    {
      "name": "<chat-profile-name>",
      "service": "nvidia",
      "model": "auto",
      "base_url": "https://integrate.api.nvidia.com/v1",
      "temperature": 0.0,
      "max_tokens": 16384
    }
  ]
}

embedding_model_configs.json

{
  "models": [
    {
      "name": "<embedding-profile-name>",
      "service": "nvidia",
      "model": "auto",
      "base_url": "https://integrate.api.nvidia.com/v1",
      "truncate": "END"
    }
  ]
}

Locally deployed models

Point base_url at the API root of your OpenAI-compatible servers. "auth": "none" sends no token.

model_configs.json

{
  "models": [
    {
      "name": "<chat-profile-name>",
      "service": "openai",
      "model": "auto",
      "base_url": "http://127.0.0.1:<chat-port>/v1",
      "auth": "none",
      "temperature": 0.0,
      "max_tokens": 16384
    }
  ]
}

embedding_model_configs.json

{
  "models": [
    {
      "name": "<embedding-profile-name>",
      "service": "openai",
      "model": "auto",
      "base_url": "http://127.0.0.1:<embedding-port>/v1",
      "auth": "none",
      "api_format": "openai"
    }
  ]
}

In both setups, the selection file references the profile names:

model_selection_config.json

{
  "llm": {
    "model": "<chat-profile-name>",
    "node_model_overrides": {}
  },
  "embedding": {
    "model": "<embedding-profile-name>"
  }
}

Replace "auto" in a profile with a served model ID to pin that model instead of discovering one.

Hosting your own model

Because each profile carries its own base_url, the pipeline can target any endpoint that exposes the OpenAI-compatible /chat/completions and /embeddings APIs. Chat and embeddings can share one endpoint if it serves both kinds of model. Automatic selection also requires /models; otherwise, replace "model": "auto" with the served model ID. The pipeline discovers and calls served models; it does not download weights or launch model servers. To run NVIDIA models locally, refer to the NIM getting started guide.

The pipeline has been benchmarked on lightweight, locally served chat models such as Qwen2.5-Coder-7B-Instruct.

1. Copy the three local templates from data/example/model_info/local/ in the omni.ai.behavior_tree_gen.bridge extension folder.

2. Set base_url in model_configs.json to your chat API and in embedding_model_configs.json to your embedding API. The templates use ports 8000 and 8001. Use the API root, such as http://127.0.0.1:8000/v1, including any API prefix but excluding /chat/completions, /embeddings, or /models. Adjust chat temperature and max_tokens for your served model and its context limit.

3. Load your scene or a bundled example first, then select your copies under Model Config > Model Configuration Files. Loading an example afterwards can replace these paths with the example’s defaults.

4. Leave Authentication Registry untouched for unauthenticated local profiles, confirm the remaining inputs, and click Run Pipeline.

To route a planner node to a separate chat profile, add that profile to model_configs.json and reference it in llm.node_model_overrides, for example "node_model_overrides": {"SimulationEventNode": "<your-chat-profile>"}. An empty map uses the selected chat profile for all nodes.

Endpoint authentication

Chat, per-node, discovery, and embedding requests read credentials from the current app environment; tokens are never stored in model configuration files or Kit settings. Each profile accepts:

• auth: auto (the default), environment to require a token, or none to send no token. Local loopback and private-network endpoints need no token under auto; set none for an unauthenticated self-hosted server whose hostname is not recognized as local.

• api_key_env: The name of the environment variable holding the token, never the token itself. Known hosted URLs select their standard variable automatically, such as NVIDIA_API_KEY, OPENAI_API_KEY, or GROQ_API_KEY. An authenticated custom gateway must set it, for example "api_key_env": "MY_SERVICE_API_KEY". Chat and embedding profiles can use different variables.

• api_format (embedding profiles only): auto, openai, or nvidia. auto adds NVIDIA-specific request fields only for NVIDIA’s hosted URL; a local NIM that requires them can choose nvidia.

These endpoints must expose the OpenAI-compatible API; native provider APIs are not adapted.

To set a token without restarting the app, expand Authentication Registry in the Model Config panel, enter the Environment Variable Name and the masked Value, and click Update Environment. The next run prepares the runtime again so the new token reaches model requests. Environment Tokens lists variable names with their set/unset status. Updates affect only the running app process; after a restart, the app inherits its launch environment again. The same environment tokens are shared with VLM Scene Captioning.

Planner step attempts

When a model reply is invalid or fails validation, each planner step asks the model again, up to Attempts per planner step (default 3, range 1-10) in Advanced Runtime Settings. After the last attempt, the step falls back to default values or reports the failure. Changes apply to the next planner step without preparing the runtime again, and the Planner status shows the current limit when a run starts. The value is stored in a persistent setting, which can also be set at launch:

--/persistent/exts/omni.ai.behavior_tree_gen.core.settings/behavior_tree_generator_setting/node_retry_max_attempts=5

Conditional inputs

The following input is optional, but important when the workflow depends on it:

• Blackboard file: Preloads blackboard metadata and variables into the workspace when blackboard-driven planning data should be available.

Relationship between schema and context

The schema files and context files work together as a connected input set:

• Context provides the instance data for actors, objects, identifiers, and metadata values.

• Schema provides the structure and meaning of that metadata.

• During workspace setup, the pipeline loads the actor and object schemas first, builds typed models from them, and then loads the context JSON into those typed models.

• During runtime preparation, schema-derived terms are reused for grounding, parameter generation, and Action IR preparation.

Where these inputs usually come from

• The examples window can load the bundled simple or warehouse scene and prefill the Behavior Tree Gen panels from a scene configuration JSON.

• The Context Cache Files panel is where context files, node catalogs, blackboard data, and metadata schemas are selected.

• The Model Config panel is where the model-related JSON files are selected and endpoint tokens can be updated.

• The Planner panel is where the scenario text is entered.

Bundled example inputs

For the built-in simple example, the extension automatically resolves inputs such as:

• actor and object context JSON files.

• actor and common node catalog JSON files.

• a blackboard JSON file.

• actor and object metadata schema JSON files.

• model selection, embedding-model, and named-model JSON files.

If you use the example flow, the main values you still need to confirm are the endpoint credentials (NVIDIA_API_KEY for the NVIDIA-hosted sample profiles), the output folder, the scenario text, and any additional runtime routing inputs required by your model selection setup.