# DeGirum
DeGirum offers quickstart guides for AI Hub and PySDK, plus resources for fast, efficient edge AI development.
At DeGirum, we’re focused on making AI development easy and efficient. Whether you’re evaluating models with DeGirum AI Hub or deploying AI locally using PySDK, our tools help you move quickly from prototype to production.
Explore these docs for guides, tutorials, and resources to support your edge AI projects. Let’s build the future of edge AI together.
## Get Started Quickly
AI Hub Quickstart
Prototype and evaluate AI models in your browser with this step-by-step guide
# Overview
Explore the DeGirum AI Hub—your platform for evaluating models, running cloud inference, and compiling models for deployment with your hardware.
The DeGirum *AI Hub* is a cloud platform for quickly prototyping edge AI applications. It provides cloud access to the *Device Farm*, a collection of real SoCs and AI accelerators hosted by DeGirum. This allows you to run model inference without the time and cost of hardware bring-up, either directly in the browser with the *Inference Dashboard* or programmatically through DeGirum *PySDK*.
*Public Model Zoos* offer curated, pre-compiled models for multiple supported hardware targets, so you can run them immediately without packaging or setup.
*Workspaces* add advanced capabilities such as role-based collaboration, the ability to bring and manage your own models, and device management.
The *Cloud Compiler* produces artifacts for supported hardware targets, which are saved in *Workspace Model Zoos*. AI Hub also includes a *Fleet Manager* for device oversight and administration pages for managing *Workspace Tokens*, *Workspace Settings*, and *Tasks*.
AI Hub integrates with PySDK, so applications prototyped in the cloud can be deployed on local hardware with the same code, typically requiring only changes to the device or host settings and artifact name.
Sign up for AI HubGet started
## AI Hub Features
Hardware Explorer
Begin prototyping without hardware setup or maintenance
# Quickstart
Learn to sign up, navigate AI Hub, and execute real-time inferencing to quickly bring your ideas to life.
## Sign Up and Log In
To begin using the AI Hub, go to the [AI Hub](https://hub.degirum.com/?utm_source=docs.degirum.com\&utm_medium=site\&utm_campaign=ai-hub-quickstart), click **Sign Up,** and create your account with your email.
AI Hub Sign Up Page
Once you verify your email, log in to access the main portal.
## Explore the AI Hub
The AI Hub provides a graphical user interface for managing your AI development assets.
After logging into the AI Hub, you can access these pages:
* **Hardware Explorer:** Explore hardware from the browser.
* **Public Models:** Browse models hosted on the AI Hub.
* **Workspaces**: Get access to advanced AI Hub features.
### Hardware Explorer
The [Inference Dashboard](/ai-hub/model-console) displays AI accelerators and processors hosted on the AI Hub by vendor. Use it to quickly preview AI accelerators, processors, and models by running inference in your browser.
{% hint style="success" %}
On the bottom-left corner of the window is a **Take a Tour** button. Click this button to get a quick tutorial of the AI Hub. In this tutorial, you'll learn how to evaluate hardware and models hosted on the AI Hub.
{% endhint %}
### **Public Models**
DeGirum maintains [public Model Zoos](/ai-hub/public-models) that all registered users can access for free.
### Workspaces
Workspaces are private, self-contained environments for building and deploying AI applications.
When you create or join a workspace, you can unlock features such as:
* [Cloud Compiler](/ai-hub/workspaces/cloud-compiler)
* [Creating Workspace Model Zoos](/ai-hub/workspaces/workspace-models#creating-a-model-zoo)
* [Managing Workspace tokens](/ai-hub/workspaces/workspace-tokens)
Click here to learn more about [Workspaces](/ai-hub/workspaces).
{% hint style="info" %}
Workspaces are currently in a free early access stage and may transition to a paid feature.
For more information, contact the DeGirum team.
{% endhint %}
# Hardware Explorer
Access a diverse range of AI hardware via DeGirum’s Device Farm on AI Hub. Prototype and experiment with cutting-edge accelerators without owning physical devices.
Use our Hardware Explorer to explore models on real-world hardware without needing local devices.
Click any of the tiles you see to open the [Inference Console](/ai-hub/hardware-explorer/inference-console). Then, select a model type and run inference.
When you select a model, the AI Hub automatically provisions hardware and manages the runtime environment. You can focus on prototyping—no manual setup required.
We currently provide access to these hardware and runtime combinations:
# Inference Console
Explore model and hardware combinations with the AI Hub Inference Console.
Go to the Hardware Explorer, click any tile, and you'll see the Inference Console.
Use our pre-defined models and example images so you can see hardware and models in action.
Inference Console using an object detection model on an example image with two birds.
## Using the Inference Console
All you need to do is select a model type, select an input image, and click **Run Inference**.
{% stepper %}
{% step %}
**Select a model type in the carousel**
When you open the Inference Console, you'll see a carousel featuring model types like object detection, instance segmentation, pose detection, and more. These are for use-cases like:
General:
* Object detection: Obtain bounding boxes with confidence levels.
* Instance Segmentation: Split an image into objects.
Specialized:
* Fire & Smoke Detection: Detect fire and smoke.
* PPE Detection: Detect personal protective equipment.
{% endstep %}
{% step %}
**Select an input file**
The image on the left is the input image that will be provided to the model. Use the arrows select one of our sample images, or click **Input File** to upload your own.
{% endstep %}
{% step %}
**Run the inference**
Click **Run Inference** to run the inference using the model type on your input file. It should at most a couple seconds for the inference to run.
{% endstep %}
{% step %}
**View Results**
When the inference is complete, you'll see the image overlayed with inference results on the right. Click **Json Result** to see the inference results in JSON format, and click **Image Result** to see the image again.
{% endstep %}
{% endstepper %}
# Public Models
Explore the public Model Zoo, a comprehensive collection of pre-optimized AI models for diverse applications.
The DeGirum AI Hub provides free access to collections of AI models.
In our public Model Zoos, we host a variety of models optimized for use-cases ranging from Face Detection to PPE Detection. To access public Model Zoos, click Public Models in the AI Hub.
Our public Model Zoos provide models for these hardware and runtime combinations:
|
| yolov8n\_relu6\_widerface\_kpts--640x640\_quant\_hailort\_hailo8l\_1 | Face Detection with Five Keypoints |
bbox: 26.715
kpts: 17.673
|
bbox: 79.181
kpts: 45.783
|
| yolov8n\_silu\_coco--640x640\_quant\_hailort\_hailo8\_1 | COCO Detection | | |
| yolov8n\_silu\_coco--640x640\_quant\_hailort\_hailo8l\_1 | COCO Detection | | |
| yolov8s\_coco--320x320\_quant\_hailort\_hailo8\_1 | COCO Detection | | |
| yolov8s\_coco--320x320\_quant\_hailort\_hailo8l\_1 | COCO Detection | | |
# DEEPX
This page features models for DEEPX devices currently available on the DeGirum AI Hub model zoo.
Models are located at the DEEPX model zoo on the AI Hub.
## Models by DeGirum
### DEEPXM1A
**Detection**
Model Name
Use Case
mAP 50‑95
mAP 50
yolov8n_coco--640x640_quant_deepx_m1a_1
COCO Detection
0.371
0.524
yolov8n_relu6_coco--640x640_quant_deepx_m1a_1
COCO Detection
0.351
0.503
yolov8s_coco--640x640_quant_deepx_m1a_1
COCO Detection
0.449
0.620
yolov8s_relu6_coco--640x640_quant_deepx_m1a_1
COCO Detection
0.438
0.603
yolov8n_relu6_hand--640x640_quant_deepx_m1a_1
Hand Detection
0.455
0.789
yolov8n_silu_hand--640x640_quant_deepx_m1a_1
Hand Detection
0.480
0.800
yolov8s_relu6_hand--640x640_quant_deepx_m1a_1
Hand Detection
0.471
0.838
yolov8s_silu_hand--640x640_quant_deepx_m1a_1
Hand Detection
0.486
0.828
yolov8n_relu6_person--640x640_quant_deepx_m1a_1
Person Detection
0.320
0.516
yolov8s_relu6_person--640x640_quant_deepx_m1a_1
Person Detection
0.309
0.513
yolov8n_relu6_ppe--640x640_quant_deepx_m1a_1
Personal Protective Equipment (PPE) Detection
0.359
0.678
yolov8s_relu6_ppe--640x640_quant_deepx_m1a_1
Personal Protective Equipment (PPE) Detection
0.571
0.824
# Axelera AI
This page features models for Axelera AI devices currently available on the DeGirum AI Hub model zoo.
Models are located at the [Axelera AI model zoo on the AI Hub](https://hub.degirum.com/public-models/degirum/axelera?utm_source=docs.degirum.com\&utm_medium=site\&utm_campaign=ai-hub-public-models-axelera-ai).
## Models by DeGirum
### Axelera Metis
**Detection**
| Model Name | Use Case | mAP 50‑95 | mAP 50 |
| ------------------------------------------------------------------ | --------------------------------------------- | -------------------------------------- | -------------------------------------- |
| yolov8n\_coco--640x640\_quant\_axelera\_metis\_1 | COCO Detection | 0.3561 | 0.5059 |
| yolov8n\_coco\_seg--640x640\_quant\_axelera\_metis\_1 | COCO Instance Segmentation |
# Model Console
Leverage DeGirum’s Model Console to run models directly in your web browser – upload inputs, view real-time results, and explore model details with ease.
When you view any model in AI Hub, the Model Console will allow you to run inference, view details about the model, download the model, and more.
In the Model Console, you can upload an input file, run inference on it, view the source code, examine the model JSON, check labels, read the Model Readme, and more.
## Run Inference
Most models can be run from the browser.
{% stepper %}
{% step %}
**Select an Input Image**
Use one of our sample images, or click the **Input File** button on the left side to upload your image. After you select an image, it will appear on the left side of the screen.
{% endstep %}
{% step %}
**Running Inference**
After uploading your image, click **Run Inference** on the right side to process the input.
{% endstep %}
{% step %}
**Results Display**
On the right side, the results of the inference will be overlayed onto the original image. You may also click **Json Result** to see the inference results in JSON format.
The results you get will vary depending on model and model type.
{% endstep %}
{% endstepper %}
## Code
Code tab in the Model Console.
Click **Code** to view sample code for running the model with DeGirum PySDK using the AI Hub and Python.
To use this code, you'll need to [create a Workspace token](/ai-hub/workspaces/workspace-tokens) and [install PySDK](https://docs.degirum.com/pysdk/installation). Then you'll be able to save the code to a .py file, edit it to use your cloud token and a file, and run the code.
## Model JSON
Model JSON tab in the Model Console.
Click **Model JSON** to see the model JSON file using our model JSON editor and validator.
With the model JSON editor and validator, you can:
* Edit the JSON file.
* Validate the JSON.
* Download or copy the JSON file.
## Labels
Labels tab in the Model Console.
Click **Labels** to see the contents of the labels file used by the model.
## Copy, download, and delete the model
The Model Console features action buttons to:
* Copy the model URL.
* Download the model as a .zip file.
* Copy the model to a model zoo in any Workspace you have joined.
* Delete the model from this model zoo.
# Workspace Plans
Learn what each workspace plan includes and how to upgrade, add users and devices, and manage billing and usage in AI Hub.
DeGirum Workspaces let you choose a plan, manage seats and devices, and control usage and add-ons for your team. For plan pricing and a full feature comparison, see the [Pricing page](https://degirum.com/pricing).
## Plans at a glance
**Free** is for individuals exploring. It includes 20k AI Hub inferences per month, 1 included user, and 1 local device license.
**Professional** is for developers building and testing. It includes 100k AI Hub inferences per month, 1 included user, 3 local device licenses, Cloud Compiler access, model storage, and community plus email support.
**Enterprise** is for teams deploying at scale. It includes 1M AI Hub inferences per month, 5 included users, 10 local device licenses, offline licensing, and priority support with SLAs.
### What the key terms mean
**AI Hub inferences**: AI Hub inferences are the monthly inference runs included with your plan.
**PySDK local runtimes**: all plans include access to PySDK local runtimes. Your plan’s local device licenses determine how many devices can run local runtimes under your Workspace.
**Local device licenses**: a local device license allows a device to run supported PySDK local runtimes under your Workspace.
* Free: 1 device license included
* Professional: 3 device licenses included, then $4.99 per additional device per month
* Enterprise: 10 device licenses included, then $1.99 per additional device per month
**Cloud Compiler**: Cloud Compiler is available on Professional and Enterprise and is billed at $4.99 per successful compile. A successful compile is a job that completes successfully and produces a usable compiled output artifact for the selected target. Failed, canceled, or timed-out compiles are not billed.
**Model storage**: model storage is the number of models you can store in your Workspace.
* Professional: 5 models included
* Enterprise: 15 models included
* Additional model storage: $1.99 per 5 models per month (Professional and Enterprise)
**Application packages**: Application packages are pip-installable packages with PySDK-style APIs for complete workflows.
* Free: [trial available upon request](https://forms.degirum.com/trial-application-packages)
* Professional and Enterprise: [see pricing](https://degirum.com/pricing)
**Offline licensing**: offline licensing is included with Enterprise. It supports license activation and periodic subscription validation without internet connectivity for air-gapped or restricted deployments, with customer-specific provisioning.
## Upgrade your Workspace plan
{% hint style="warning" %}
Only Workspace owners can upgrade a Workspace plan. If you do not see billing options, ask a Workspace owner to upgrade or to grant you owner access.
{% endhint %}
{% embed url="" %}
{% stepper %}
{% step %}
**Confirm you have a Workspace**
On the AI Hub home page, click the Workspace dropdown on the left-hand side and select the Workspace that you would like to upgrade. If you do not have a Workspace, click Create New.
{% hint style="info" %}
Workspace names must be globally unique. Use letters, numbers, underscores, and dashes.
Do not use spaces and avoid generic names like *test*, *example*, or *myspace*.
Choose a name the same way you would choose an organization name on GitHub or a username for email. If the name is already taken, you will need to pick a different one.
{% endhint %}
{% endstep %}
{% step %}
**Open Workspace Settings**
Click on Workspace Settings in the left-hand menu. Click on the Billing tab, and then click on Manage Billing to manage the Billing Portal.
{% endstep %}
{% step %}
**Add payment method**
Click on Payment Methods, add your payment information (e.g., credit card), and click Add.
{% endstep %}
{% step %}
**Manage your subscription**
Under Manage Subscriptions, click on your current plan to view Subscription Details. From Subscription Details, click on Edit Subscription.
{% endstep %}
{% step %}
**Change and confirm the upgrade**
From Edit Subscription Details, click Change and select your desired plan from the dropdown. Click Update. After reviewing the updated Subscription Details, click Update Subscription.
{% endstep %}
{% endstepper %}
## Manage users
Each plan includes a set number of users. Additional users are billed per user per month.
To add users:
{% stepper %}
{% step %}
**Open Workspace Settings**
Click on Workspace Settings in the left-hand menu. Click on the Members tab to view the Workspace members.
{% endstep %}
{% step %}
**Add new member**
Type the email of the new Workspace member and choose their permission level from the dropdown. Click Invite.
{% endstep %}
{% step %}
**Manage existing members**
Owners can manage existing members from the table and change permissions in the Permissions dropdown.
{% endstep %}
{% endstepper %}
# Workspaces
Read this page to learn about workspaces in the AI Hub. You must join a workspace to access unique AI Hub features.
When you create or join a Workspace, you can unlock advanced AI Hub capabilities such as:
* [Cloud Compiler](/ai-hub/workspaces/cloud-compiler)
* [Creating Model Zoos](/ai-hub/workspaces/workspace-models#creating-a-model-zoo)
* [Workspace tokens for AI Hub authentication](/ai-hub/workspaces/workspace-tokens)
* [Inviting people to the Workspace and assigning roles](/ai-hub/workspaces/workspace-settings)
{% hint style="info" %}
Workspaces are free during early access; charges begin after early access ends.
{% endhint %}
## Accessing Workspaces
Workspace dropdown in the left navigation with a Workspace selected.
To access Workspaces, click the **Create new** button or the name of your active Workspace in the left-hand navigation bar of the AI Hub. The dropdown menu that appears will display the "Create new" button and Workspaces your account has joined.
Click a Workspace to select it. When your Workspace is selected, you'll see a check next to the Workspace name in the selector, and you'll be able to use Workspace-specific features like the [Cloud Compiler](/ai-hub/workspaces/cloud-compiler).
## Creating Workspaces
Any AI Hub user can create a Workspace.
{% stepper %}
{% step %}
**Click Create new or the Workspace in the left-hand navigation bar**
When you click your account or Workspace, the dropdown menu will display the **Create new** button and the Workspaces your account has joined.
{% endstep %}
{% step %}
**Set a Workspace Name**
After clicking **Create new**, enter the Workspace name.
{% hint style="danger" %}
Workspace names cannot be changed after creation.
{% endhint %}
{% endstep %}
{% step %}
**Click Create**
After entering the Workspace name, click **Create**. You'll immediately join as the Workspace owner, and the Workspace will appear in the dropdown menu the next time you open it.
{% endstep %}
{% endstepper %}
## Managing a Workspace
To manage a Workspace, select a Workspace in which you are a Workspace owner, then select **Workspace Settings**.
Only owners of the Workspace can access the Workspace Settings page.
# Cloud Compiler
Port and optimize your custom AI models for various hardware platforms using DeGirum’s Cloud Compiler.
{% hint style="info" %}
Have any questions about the Cloud Compiler? [Click here](https://community.degirum.com/t/308?utm_source=docs.degirum.com\&utm_medium=site\&utm_campaign=ai-hub-workspaces-cloud-compiler) for our FAQ!
{% endhint %}
The DeGirum Cloud Compiler simplifies the process of preparing AI models for real-world deployment. It takes PyTorch checkpoints—specifically models trained using the Ultralytics repository—and compiles them into formats that run efficiently on supported edge AI hardware.
DeGirum Cloud Compiler interface.
Upload your PyTorch checkpoint to the Cloud Compiler and let it handle the conversion. You can adjust parameters to optimize performance, choose target runtimes and devices, and provide a custom dataset for quantization to meet your deployment requirements.
When a model is compiled, it can be loaded [online or offline with PySDK](/ai-hub/pysdk-integration) or run directly in the browser with the [Model Console](/ai-hub/model-console).
The Cloud Compiler currently supports models for these runtimes, devices, and precisions:
| Runtime | Devices | Precision support |
| -------- | ---------------------- | ----------------- |
| HAILORT | HAILO8, HAILO8L | Quant only |
| DEEPX | M1A | Quant only |
| AXELERA | METIS | Quant only |
| OPENVINO | CPU, NPU, GPU | Float and quant |
| MEMRYX | MX3 | Float only |
| TFLITE | CPU, EDGETPU | Float and quant |
| N2X | CPU, ORCA1 | Float and quant |
| RKNN | RK3566, RK3568, RK3588 | Float and quant |
### Using the Cloud Compiler
Access to the Cloud Compiler is on a per-workspace basis.
When your workspace has access to the Cloud Compiler, you will be able to access the Cloud Compiler. Without access, you can request early access for your workspace to the Cloud Compiler through the form available in the Cloud Compiler section of the AI Hub.
{% stepper %}
{% step %}
**Select a Workspace with Cloud Compiler access**
Click the Workspace dropdown menu in the left-hand navigation bar of the AI Hub, and select a Workspace with Cloud Compiler access.
{% endstep %}
{% step %}
**Create or get access to a Model Zoo**
The Cloud Compiler places compiled models into a model zoo. Ensure you have a model zoo where you can upload pre-compiled models.
To learn how to create Model Zoos, [click here](/ai-hub/workspaces/workspace-models#creating-a-model-zoo).
{% endstep %}
{% step %}
**Visit the Cloud Compiler page**
After ensuring you have access to a Model Zoo where you can upload models, click Cloud Compiler in the AI Hub navigation bar to visit the Cloud Compiler page.
{% endstep %}
{% step %}
**Upload a checkpoint file**
Click **Upload File** to submit a PyTorch checkpoint (.pt) for compilation.
{% endstep %}
{% step %}
**Fill out the Details section**
In the Details section, fill in details such as name prefix, version, image width, and image height to identify your model in the model zoo.
{% endstep %}
{% step %}
**Select a Model Zoo in the Details section**
Choose the Model Zoo where the compiled model will be published.
If the Model Zoo selector is empty, then recheck if you have created or can access a Model Zoo where you can upload pre-compiled mode.
{% endstep %}
{% step %}
**Select a runtime and device in the Target section**
In the Target section, choose the target runtime, device, and type from the dropdown menus. The Cloud Compiler uses your selections to build your model.
{% endstep %}
{% step %}
**Select advanced options (optional)**
Each runtime and device offers advanced options. View the advanced options to further optimize your model, such as by uploading a calibration dataset for quantization.
{% endstep %}
{% step %}
**Start model compilation**
Click **Compile** to start. The Cloud Compiler task will appear in the task list available when you click Tasks in the AI Hub navigation bar. When the process completes, your model is published to the selected model zoo, and you will receive an email confirming that the model has been compiled.
{% endstep %}
{% endstepper %}
{% hint style="info" %}
Some runtime, device, and type combinations support multiple devices. The device you select in the form will be the device in the resulting model .json's `DeviceType` field, while all supported devices using the same runtime will be listed in the `SupportedDeviceTypes` field.
[Click here](https://docs.degirum.com/pysdk/user-guide-pysdk/model-json-structure) for more information about the model .json file.
{% endhint %}
# Workspace Models
Read this page to learn about how to navigate and create model zoos in the DeGirum AI Hub. All users have access to the model zoo. The ability to create model zoos is on a per-workspace basis.
If your Workspace has permission, you can create Workspace Model Zoos for your Workspace.
## Creating a Workspace Model Zoo
If your Workspace has permission to create Model Zoos:
{% stepper %}
{% step %}
**Click Models in the navigation bar on the left**
You will see tiles for Model Zoos and All Models. Click the Model Zoos tile.
Model Zoos and All Models Tiles
{% endstep %}
{% step %}
**Click New Model Zoo**
After you click the Model Zoos tile, you will be brought to the list of model zoos.
New Model Zoo button when viewing Model Zoos
If enabled for your Workspace, there will be a New Model Zoo button. Click the New Model Zoo button.
{% endstep %}
{% step %}
**Set up the Model Zoo**
After you click the New Model Zoo button, a window opens with a menu to set up the model zoo.
New Model Zoo form
* Model Zoo Name: Name of the model zoo.
* Model Zoo Description: Description of the model zoo.
{% endstep %}
{% step %}
**Click Add New Model Zoo**
The Model Zoo will be created for your Workspace.
{% endstep %}
{% endstepper %}
## Viewing all models in your Workspace
You can easily view all models available in your Workspace.
Model Zoos and All Models Tiles
To view all models:
{% stepper %}
{% step %}
**Click Workspace Models in the navigation bar on the left**
You will see tiles for Model Zoos and All Models.
{% endstep %}
{% step %}
**Click All Models**
After clicking the All Models tile, you will see a directory of models. You will also see a search/filter panel to search and filter models.
{% endstep %}
{% endstepper %}
To view all models in a Workspace:
{% stepper %}
{% step %}
**Select a Workspace in the navigation bar on the left**
The Workspace you have selected will determine what models you can view.
{% endstep %}
{% step %}
**Click Workspace Models**
You will see tiles for Model Zoos and All Models.
{% endstep %}
{% step %}
**Click All Models**
After clicking the All Models tile, you will see a directory of models. You will also see a search/filter panel to search and filter models. This directory of models features only models in the Workspace you selected; it will not feature any models available in the public Model Zoo. Likewise, the search/filter panel will filter only models in the Workspace.
{% endstep %}
{% endstepper %}
## Models Directory Page
The Models Directory page displays information about models and provides a few action buttons.
Models Directory Page
* Name: Identifies each model. We provide a recommended naming convention for models in our documentation about [organizing models](https://docs.degirum.com/pysdk/user-guide-pysdk/organizing-models) for DeGirum PySDK.
* If you are viewing all models, the Models Directory Page will also indicate the workspace and zoo where the model lives.
* Devices: Hardware compatibility of the model. This information is retrieved from the model JSON file.
* Version: Version number of each model. This information is retrieved from the model JSON file.
* Action Buttons:
* Edit: Opens a window to edit the model readme, model JSON file, and labels JSON file. You will need the right permissions to use this action button.
* Download: Downloads a zip file containing the model, model JSON file, and labels JSON file.
* Copy: Copies the model to a different model zoo controlled by your workspace. You will need the right permissions to use this action button.
* Delete: Deletes the model from the model zoo. You will need the right permissions to use this action button.
If you are viewing a specific model zoo, the models page features:
* Model Zoo Action Buttons: Buttons to copy the Model Zoo AI Hub path, edit the Model Zoo, and delete the Model Zoo. The edit and delete action buttons are identical to the edit and delete action buttons available in the [Model Zoos](#model-zoos-page) page. You will need the right permissions to use this action button.
* New Model button: Allows you to upload a new model if your workspace created the Model Zoo. To upload a model, you must prepare a zip file containing the model, model JSON, and labels JSON all in one archive. The max size of this zip file is 1 GB. The Models Directory page will automatically identify information about the model based on the contents of the JSON file in the uploaded archive.
* Pagination Controls: Navigates through pages of models.
### Search/Filter Panel When Viewing the Models Directory Page
When viewing the Models Directory page, the search/filter panel enables searching by model name and filtering by device type, quantization, and output postprocessor type.
Search/Filter Panel for the Models Directory Page
Click Quick Search and enter text to search the model list for model names matching your search.
Click buttons for Device Type, Quantization, and Output Postprocessor Type to filter models.
* Device Type: Hardware compatibility of the model.
* Quantization: Whether the model is a floating point model or a quantized model. Quantized models typically perform much faster than floating point models.
* Output Postprocessor Type: The type of postprocessor file used by the model. Most models use postprocessors, but some models do not require postprocessors.
## Viewing Model Zoos
{% stepper %}
{% step %}
**Verify that you have your Workspace selected**
Your selected Workspace determines which model zoos you can view. Select your Workspace using the Workspace dropdown in the left navigation bar.
{% endstep %}
{% step %}
**Click Workspace Models in the Navigation Bar on the Left**
You will see tiles for Model Zoos and All Models. Click the Model Zoos tile.
{% endstep %}
{% step %}
**View Available Model Zoos**
After clicking the Model Zoos tile, you will see tiles for model zoos. You will also see a search/filter panel to search and filter model zoos.
{% endstep %}
{% endstepper %}
## Model Zoos Page
The Model Zoos page displays model zoo tiles and provides a few action buttons.
Model Zoos page
Each model zoo tile follows this format:
* Model Zoo Name: Identifies each model zoo. Written in bold.
* Workspace: The workspace that created the model zoo. Written in italics beneath the model zoo name.
* Action Buttons:
* Edit: Opens a window to edit the model zoo name and description. You will need the right permissions to use this action button.
* Visibility: Displays the model zoo's visibility.
* Delete: Deletes the model zoo. Opens a confirmation pop-up to confirm deletion of the selected model zoo. Deleted model zoos cannot be restored.
Additionally, the Model Zoos page features:
* Sort Controls: Sort the model zoo tiles by model count or alphabetically in descending order.
* Pagination Controls: Navigates through pages of model zoo tiles.
### Search/Filter Panel When Viewing Model Zoos
When viewing model zoos, the search/filter panel enables searching by model zoo name and filtering by organization and visibility.
Search/Filter Panel for the Model Zoos page
Click Quick Search and enter text to search the model zoos for model zoos matching your search.
Click buttons for organization and visibility to filter models.
* Organization: The workspace that created the model zoo.
* Visibility: Indicates whether the model zoo is **Private**, **Shared Private**, or **Public**. All zoos created by workspaces are private. Only DeGirum publishes public model zoos.
# Workspace Tokens
The AI Hub features Workspace tokens for accessing the AI Hub API. You can create Workspace tokens in the AI Hub as soon as you create an account.
{% hint style="danger" %}
Do not share Workspace tokens.
{% endhint %}
To access the AI Hub with PySDK, you'll need to create a Workspace token with the AI Hub. You can create Workspace tokens only if your account is part of a Workspace.
If you plan to use PySDK fully offline, or you plan to use models available in public Model Zoos, you don't need to create Workspace tokens.
Workspace tokens allow you to:
* Run cloud inference on models in public Model Zoos.
* Access to models in the Workspace Model Zoos for local and AI server inference.
* Run cloud inference on models in Workspace Model Zoos.
## Creating and managing tokens
AI Hub Tokens interface.
To create or manage tokens, navigate to [Workspace tokens](https://hub.degirum.com/workspace-tokens?utm_source=docs.degirum.com\&utm_medium=site\&utm_campaign=ai-hub-workspaces-workspace-tokens) in the AI Hub. You'll be able to:
* Generate new tokens: Click the **Generate Workspace Token** button, then set a description and expiration date for the token.
* View partial token keys: The Tokens interface will display part of the token key for existing tokens. You will not be able to view the entire token key through the interface.
* Copy token keys: Click the copy action button to the right of the token key to copy the token key.
* Delete tokens: Click the delete action button to the right of the token key to delete the key. A confirmation popup will appear to confirm if you want to delete the key.
If you have [PySDK ](https://docs.degirum.com/pysdk)installed, you may also use a PySDK CLI helper to manage tokens easily over a terminal. [Click here](https://docs.degirum.com/pysdk/user-guide-pysdk/command-line-interface#manage-ai-hub-tokens) for more information about the PySDK CLI helper.
# Workspace Settings
Learn about Workspace Settings. Change member roles, invite members, remove members, manage billing, and delete the Workspace.
When you are the owner of a Workspace, you can access **Workspace Settings**. In here, you can change member roles, invite members, remove members, manage billing, and delete the Workspace.
This page will not show up if you have a Workspace selected that you do not own.
Workspace Settings page.
## Members
The **Members** tab features a list of members, a search bar, permission levels, and the invite button.
### Permission levels
Workspaces have two permission levels: owner and member.
* Members have access to all features enabled for the Workspace.
* Owners can edit member roles and delete the Workspace.
### Inviting people to Workspaces
Only Workspace owners can invite people to Workspaces. To invite someone, you'll need their email address. The AI Hub sends a confirmation email to the new member.
{% stepper %}
{% step %}
**Select a Workspace**
Select a Workspace in which you are a Workspace owner.
{% endstep %}
{% step %}
**Enter the Workspace settings page**
Click **Workspace settings**.
{% endstep %}
{% step %}
**Ensure you are an owner of the Workspace**
In the **Members** tab of the selected Workspace, ensure that you are an owner of the Workspace. If you are an owner, then locate the **Invite** button in the bottom-right corner of the list of members.
{% endstep %}
{% step %}
**Set the email and permission level**
To the left of the **Invite** button, enter the email and default permission level of the person you would like to invite. You can invite anyone. They'll receive an invitation email with instructions to sign in. If they don't have an AI Hub account yet, the email will include steps to register.
{% endstep %}
{% step %}
**Click Invite**
After you click **Invite**, the AI Hub sends a confirmation email and immediately adds the recipient to your Workspace with the permission level you selected.
{% endstep %}
{% endstepper %}
### Removing members from Workspaces
To remove members from the Workspace, click their permission level then click Remove. You'll see a popup to confirm removal of this member.
## General Settings
General settings currently feature one item: deleting the Workspace.
### Deleting the Workspace
{% hint style="danger" %}
Deleting a Workspace is a permanent action and cannot be undone. This will irreversibly remove all data associated with this workspace including tasks, members, and settings.
{% endhint %}
To delete a Workspace, navigate to **General Settings**. You'll see a **Danger Zone** in **General Settings** with instructions on deleting the Workspace.
# PySDK Integration
Integrate DeGirum PySDK with AI Hub to keep your code concise while tapping into powerful cloud-hosted inference.
[DeGirum PySDK ](https://docs.degirum.com/pysdk)integrates with the AI Hub so you can write compact, easy-to-understand code. The `inference_host_address` parameter tells PySDK where to run your model. You can set it to `"@cloud"` to use DeGirum’s cloud-hosted hardware—no local setup required.
## Example PySDK Code
The snippet below connects to the remote device farm and runs inference on an image hosted on DeGirum's [PySDK Examples GitHub repository](https://github.com/DeGirum/PySDKExamples).
{% code overflow="wrap" %}
```python
import degirum as dg
# Connect to the AI Hub
inference_manager = dg.connect(
inference_host_address = "@cloud",
zoo_url = "degirum/degirum",
token = ""
)
# Load a model. In this case, an image detection model from degirum/degirum.
model = inference_manager.load_model("yolov8n_relu6_coco--640x640_quant_n2x_orca1_1")
# Run inference on an image
result = model("https://raw.githubusercontent.com/DeGirum/PySDKExamples/main/images/bikes.jpg")
# Print raw results
print(result)
```
{% endcode %}
# Application Package Licensing
Understand how Application Package licensing works in DeGirum AI Hub, including automatic license fetch, Workspace eligibility, and billing based on peak active usage.
Application Package licensing in AI Hub is designed to be automatic and Workspace-based. When you run an Application Package on a device, the package uses the Workspace Token installed on that device to fetch and activate a license, without any manual license keys. This page walks through the requirements to run an Application Package, what happens at launch, and how active licenses are counted for billing.
## Key ideas
* **Licenses are Workspace-scoped.** Each license is associated with a specific Workspace.
* **Tokens drive licensing.** The device running an Application Package must have a Workspace Token installed for the Workspace that the package is licensed under.
* **No manual license keys.** Licenses are fetched and activated automatically when the Application Package runs.
* **Billing uses peak active usage.** A Workspace is billed based on the highest number of licenses activated (or renewed) during a billing period.
## Workspace eligibility for licensing
By default:
* Pro Workspaces can generate Application Package licenses.
* Enterprise Workspaces can generate Application Package licenses.
Exception:
* In some cases, Free Workspaces can be temporarily enabled to generate Application Package licenses during a trial period.
If a Workspace is not entitled (or not trial-enabled), devices using that Workspace Token cannot fetch licenses for Application Packages tied to that Workspace.
## Requirements to run an Application Package on a device
To run an application package on a device, the user must:
1. Have access to the Application Package in AI Hub (typically through Workspace access and permissions).
2. Install a Workspace Token on the target device that belongs to the Workspace the Application Package is licensed under.
A simple way to think about it:
* The Application Package belongs to a Workspace.
* The device must use a token from that same Workspace.
## What happens when you run an Application Package
When you launch an Application Package on a device:
1. The application Package reads the Workspace Token installed on that device.
2. It attempts to fetch and activate a license automatically from AI Hub for that Workspace.
3. If successful, the Application Package runs normally.
### If the token is missing or not entitled
The Application Package cannot fetch a license if any of the following are true:
* No Workspace Token is installed on the device.
* The installed token belongs to a different Workspace than the one the Application Package is licensed under.
* The Workspace is not entitled (or not trial-enabled) for Application Package licensing.
* The token or user does not have permission in that Workspace to use Application Package licensing.
In these cases, the Application Package fails to start and the user sees an error indicating they do not have permission or the Workspace is not entitled.
## Billing and counting active licenses
Licensing is billed at the Workspace level. During each billing period, AI Hub tracks:
* The maximum number of licenses that were activated or renewed for that Workspace.
That peak active count is what the Workspace is billed for during the period.
**Example: peak active usage**
If a Workspace had:
* 3 devices running the Application Package early in the month,
* later 7 devices running at the same time,
* then it dropped back to 2 devices,
the billing basis for that period is 7.
### License renewal
Once an Application Package license is activated on a device, it is renewed periodically (approximately every 10 days) as long as the Application Package continues to run on that device using a valid Workspace Token. If the device can’t reach AI Hub during a renewal window, the Application Package may eventually fail to renew and raise a licensing error until connectivity (and permissions) are restored.
### How licenses stop being counted
There is no manual deactivation step.
To stop a device from being counted:
* Do not run the Application Package on that device for a full billing cycle.
After a full billing cycle of inactivity, the device’s license is automatically no longer counted toward active usage.
## Why licensing works this way
This design is intentional:
* **No manual activation flow**: Users do not need to generate keys or track which machine used which key.
* **Fewer support issues**: Tokens and automatic license fetch reduce misconfiguration and lost-key scenarios.
* **Usage-based fairness over time**: Billing reflects actual usage over time rather than licenses requested in the past.
* **Scales cleanly**: Teams can add or remove devices without extra license management steps.
## Troubleshooting checklist
If you see a licensing error, check:
* Is the Workspace plan Pro or Enterprise, or a trial-enabled Free Workspace?
* Is a Workspace Token installed on the device?
* Is the token for the same Workspace as the Application Package?
* Does the token and user have permission in that Workspace to use Application Package licensing?
# Overview
Discover the power and flexibility of DeGirum PySDK, engineered to streamline AI development and deliver consistent performance across diverse hardware platforms.
DeGirum PySDK is a Python library designed to make AI development and deployment simple and efficient. With PySDK, we aim to provide APIs with a very low barrier to entry for AI model inferencing while ensuring maximum performance. It is engineered for optimal pipelining and hardware acceleration across CPUs, GPUs, NPUs, and AI accelerators.
Our goal is to provide a consistent API so your code can target multiple supported runtimes and devices without changing application logic. PySDK integrates with DeGirum AI Hub, eliminating the need for manual model downloads. Whether you're running AI locally, on an AI server, or using hardware hosted in our AI Hub, the same PySDK code works across all environments.
## Why Developers Use PySDK
# Quickstart
See a live example of PySDK, demonstrating live inferencing and rapid deployment.
[](https://colab.research.google.com/github/DeGirum/PySDKExamples/blob/main/examples/google_colab/pysdk_hello_world.ipynb) to see how easy creating applications can be with PySDK. To run our Google Colab example, you do not need to install anything — you only need to register for an account on our AI Hub to access all of our AI Hub's features. In Google Colab, we demonstrate how to create an AI Hub token, access the AI Hub with PySDK, and run AI Hub inferencing on an image.
If you would like to see more Google Colab examples, go to our [Google Colab repository](https://github.com/DeGirum/PySDKExamples/tree/main/examples/google_colab) hosted on GitHub.
## Example Code
You can also use your own devices to run inference from models hosted on the AI Hub.
This code runs an inference locally using a detection model from the AI Hub, then outputs inference results.
{% code overflow="wrap" %}
```python
import degirum as dg
# Declaring variables
# Set your model name, inference host address, model zoo, and AI Hub token.
your_model_name = "yolov8n_relu6_coco--640x640_quant_tflite_multidevice_1"
your_host_address = "@local" # Can be "@cloud", host:port, or "@local"
your_model_zoo = "degirum/public"
# Specify a path to your image file
your_image = 'https://raw.githubusercontent.com/DeGirum/PySDKExamples/main/images/Cat.jpg'
# Loading a model
model = dg.load_model(
model_name = your_model_name,
inference_host_address = your_host_address,
zoo_url = your_model_zoo,
# optional parameters, such as output_confidence_threshold=0.5
)
# Perform AI model inference on your image
inference_result = model(your_image)
# Print the inference result
print(inference_result)
```
{% endcode %}
Example output:
{% code overflow="wrap" %}
```
- bbox: [241.37208024326583, 113.8805459421775, 897.3911454167619, 704.7711576639625]
category_id: 15
label: cat
score: 0.8781571984291077
```
{% endcode %}
To learn more about PySDK, such as installing and running it locally, proceed to pages like [PySDK Installation](/pysdk/installation).
# Installation
Follow comprehensive installation guides for PySDK, covering virtual environments, Docker images, and verification.
## Quick Install
Recommended: install in a virtual environment (venv/conda) to avoid dependency conflicts.
Note: DeGirum PySDK is installed from DeGirum’s package index at [https://pkg.degirum.com](https://pkg.degirum.com/) (this is the preferred install source).
```bash
pip install -i https://pkg.degirum.com degirum
```
Detailed, platform-specific installation steps (OS, accelerators/runtimes, troubleshooting) are provided in the [#installation-options](#installation-options "mention") section.
{% hint style="danger" %}
Starting with PySDK version 1.0.0, PySDK is governed by new [EULA](https://docs.degirum.com/pysdk/eula). Please read the new EULA before installing/upgrading.
{% endhint %}
### Token installation and license
Starting from ver. 1.0.0, the usage of premium runtime plugins requires license. The license is obtained automatically by PySDK from DeGirum AI Hub if you have AI Hub token installed on your system. Once requested, the plugin license is stored locally and automatically renewed on expiration. Default expiration period is 10 days. The license is node-locked.
The Free plan allows you to use PySDK premium runtimes on **one host**. If you need to use PySDK premium runtimes on more than one host, you need to upgrade your AI Hub workspace to [Professional or Enterprise plans](https://degirum.com/pricing).
To install existing AI Hub token, you run `degirum` CLI command: `degirum token install ` where `` is the AI Hub token string which you generate on [AI Hub](https://docs.degirum.com/ai-hub/workspaces/workspace-tokens).
To create new token and install it, you run `degirum` CLI command: `degirum token create`. If you run this command on a system having graphical desktop, it will open token generation page in your default browser for you. Otherwise it will print URL which you need to paste in any browser.
To upgrade your plan, follow [these instructions](https://docs.degirum.com/ai-hub/workspace-plans#upgrade-your-workspace-plan).
The premium plugins include:
* Akida (Brainchip)
* Axelera
* DeepX
* Hailo
* MemryX
* ONNX
* OpenVINO (Intel)
* Renesas
* RKNN (RockChip)
* TensorRT (NVIDIA)
Free plugins include DeGirum N2X Orca and Google TFLite.
## Supported Environments
PySDK currently supports these operating systems, CPU architectures, and Python versions.
Operating System
Supported CPU Architectures
Supported Python Versions
Ubuntu Linux 20.04, 22.04, 24.04
x86-64
3.9 ... 3.13
Ubuntu Linux 20.04, 22.04, 24.04
ARM AArch64
3.9 ... 3.13
Raspberry Pi OS (64 bit)
ARM AArch64
3.9 ... 3.13
Windows 10/11
x86-64
3.9 ... 3.13
macOS
ARM AArch64
3.9
## Supported Hardware
Below is a summary of what hardware we support, including the runtime, devices, and model binary formats.
| Vendor | Runtime | Devices | Model Binary Format |
| ---------- | -------- | ---------------------- | ------------------- |
| Hailo | HAILORT | HAILO8, HAILO8L | .hef |
| Axelera AI | AXELERA | Metis | .bin, .json, & .c |
| DEEPX | DEEPX | M1A | .dxnn |
| Intel | OPENVINO | CPU, GPU, NPU | .onnx, .bin & .xml |
| Renesas | RENESAS | RZ-V2N | .so |
| Rockchip | RKNN | RK3588, RK3568, RK3566 | .rknn |
| MemryX | MEMRYX | MX3 | .dfp |
| BrainChip | AKIDA | NSoC\_v2, AKD1500 | .bin |
| Google | TFLITE | EDGETPU | .tflite |
| DeGirum | N2X | ORCA1 | .n2x |
| NVIDIA | TENSORRT | DLA, GPU | .onnx |
| AMD | ONNX | VITIS\_NPU | .onnx |
## Installation Options
We recommend creating a Python virtual environment for PySDK and other DeGirum Python packages.
Open a command prompt or terminal, then follow the steps for your system below.
### Windows
If Python is not installed, disable the Windows Python App Execution Alias before installing Python and PySDK. Ensure the [Visual C++ Redistributable for Visual Studio 2015 or later](https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170) is installed.
Disabling the Windows Python App Execution Alias and Installing Python on Windows
1. **Locate the Alias:** By default, Windows will overwrite `python` with Python from the Microsoft Store when entered in a terminal. Search for App Execution Alias in Windows Settings and disable both `python.exe` and `python3.exe` aliases.
Windows Python App Execution Aliases forpython.exe and python3.exe .
2. **Install a Python version supported by PySDK:** After disabling the App Execution Alias, install Python from the [official Python website](https://www.python.org/downloads/) or with `winget`. When installing with [winget](https://learn.microsoft.com/en-us/windows/msix/app-installer/install-update-app-installer), include both `Python.Python.` and `Python.Launcher`.
**Installing PySDK in a Python Virtual Environment on Windows**
Ensure a supported Python version is installed, then watch the video or follow the steps below:
{% embed url="" %}
Windows Installation
{% endembed %}
{% stepper %}
{% step %}
**Create a Python Virtual Environment**
Use this command to create a virtual environment with the default Python installation:
{% code overflow="wrap" %}
```powershell
cd ~
python -m venv degirum-windows
```
{% endcode %}
Alternatively, with Python Launcher (explicit 3.12):
{% code overflow="wrap" %}
```powershell
py -3.12 -m venv degirum-windows
```
{% endcode %}
{% endstep %}
{% step %}
**Activate the Python Virtual Environment**
After creating the environment with `venv`, navigate to the `Scripts` folder in your virtual environment.
{% code overflow="wrap" %}
```powershell
cd ~\degirum-windows\Scripts
```
{% endcode %}
Once in that folder, run the activation command.
On cmd.exe:
{% code overflow="wrap" %}
```powershell
activate
```
{% endcode %}
On pwsh.exe:
{% code overflow="wrap" %}
```powershell
.\Activate.ps1
```
{% endcode %}
After activation, the virtual environment name appears before the terminal prompt.
PowerShell terminal with an active degirum-windows virtual environment
{% endstep %}
{% step %}
**Install PySDK with `pip install`**
With the environment active, use `pip` to install PySDK and optionally `degirum-tools`. The PySDK package is `degirum`, while `degirum-tools` helps you build AI applications with PySDK.
To install only PySDK:
{% code overflow="wrap" %}
```powershell
pip install -i https://pkg.degirum.com degirum
```
{% endcode %}
To install both PySDK and `degirum-tools`:
{% code overflow="wrap" %}
```powershell
pip install -i https://pkg.degirum.com degirum degirum-tools
```
{% endcode %}
{% endstep %}
{% step %}
**Verify PySDK Installation**
Run this command in the active virtual environment:
{% code overflow="wrap" %}
```python
degirum sys-info
```
{% endcode %}
Example output:
{% code overflow="wrap" %}
```powershell
(degirum-windows) PS C:\degirum\degirum-windows\Scripts> degirum sys-info
Devices:
N2X/CPU:
- '@Index': 0
TFLITE/CPU:
- '@Index': 0
- '@Index': 1
- '@Index': 2
- '@Index': 3
Software Version: 1.0.0
```
{% endcode %}
A list of detected devices confirms that PySDK is installed.
{% endstep %}
{% endstepper %}
### Linux
Ensure a Python version supported by PySDK is installed.
Installing Python on Linux
Commands vary by distribution. Use these examples as a guide.
For Ubuntu 24.04:
Update package lists and upgrade installed packages. Then, install Python and the `venv` module.
{% code overflow="wrap" %}
```bash
sudo apt update
sudo apt upgrade -y
sudo apt install -y python3 python3-venv
```
{% endcode %}
For Raspberry Pi OS:
You should only need to update package lists and upgrade installed packages. Python 3 is preinstalled on 64-bit Raspberry Pi OS.
{% code overflow="wrap" %}
```bash
sudo apt update
sudo apt upgrade -y
```
{% endcode %}
{% stepper %}
{% step %}
**Create a Python Virtual Environment**
Use this command to create a virtual environment:
{% code overflow="wrap" %}
```bash
cd ~
python3 -m venv degirum
```
{% endcode %}
Alternatively, to target Python 3.12 explicitly:
{% code overflow="wrap" %}
```bash
python3.12 -m venv degirum
```
{% endcode %}
{% endstep %}
{% step %}
**Activate the Python Virtual Environment**
After creating the environment with `venv`, activate it:
{% code overflow="wrap" %}
```bash
source ~/degirum/bin/activate
```
{% endcode %}
After activation, the virtual environment name appears before the terminal prompt.
Terminal with an active degirumvirtual environment
{% endstep %}
{% step %}
**Install PySDK with `pip install`**
With the environment active, use `pip` to install PySDK and optionally `degirum-tools`. The PySDK package is `degirum`, while `degirum-tools` helps you build AI applications with PySDK.
To install only PySDK:
{% code overflow="wrap" %}
```bash
pip install -i https://pkg.degirum.com degirum
```
{% endcode %}
To install both PySDK and `degirum-tools`:
{% code overflow="wrap" %}
```bash
pip install -i https://pkg.degirum.com degirum degirum-tools
```
{% endcode %}
{% endstep %}
{% step %}
**Verify PySDK Installation**
Run this command in the active virtual environment:
{% code overflow="wrap" %}
```python
degirum sys-info
```
{% endcode %}
Example output:
{% code overflow="wrap" %}
```bash
(degirum) root@dl4328:~# degirum sys-info
Devices:
N2X/CPU:
- '@Index': 0
TFLITE/CPU:
- '@Index': 0
- '@Index': 1
- '@Index': 2
- '@Index': 3
Software Version: 1.0.0
```
{% endcode %}
A list of detected devices confirms that PySDK is installed.
{% endstep %}
{% endstepper %}
### Docker Images
We also provide Docker images for [DeGirum AI server](https://hub.docker.com/r/degirum/aiserver) and [PySDK AI client](https://hub.docker.com/r/degirum/aiclient) installations.
Follow [this GitHub link to DeGirum Docker repo](https://github.com/DeGirum/docker-degirum) for details on how to run these images and for corresponding source Dockerfiles.
## Runtimes and Drivers
We support multiple hardware options and runtime environments. PySDK comes with support for N2X and TFLite runtimes. Follow [this link](/pysdk/runtimes-and-drivers) for more details.
### Limiting Runtime Plugin Loading
PySDK searches for runtime plugins at startup. To restrict which plugins load,\
set the `DG_PLUGINS_ALLOWED` environment variable to a list of plugin prefixes\
separated by any non alphanumeric character. For example:
{% code overflow="wrap" %}
```bash
export DG_PLUGINS_ALLOWED=n2x_runtime_agent;onnx_runtime_agent
```
{% endcode %}
Only plugins with prefixes in the list load when `DG_PLUGINS_ALLOWED` is set.\
If the variable is not defined, all available plugins load as before.
Supported plugin prefixes include `hailort_runtime_agent`, `openvino_runtime_agent`, `axelera_runtime_agent`, `memryxrt_runtime_agent`, `akida_runtime_agent`, `tflite_runtime_agent`, `n2x_runtime_agent`, `rknn_runtime_agent`, `onnx_runtime_agent`, and `tensorrt_runtime_agent`.
## Verification
After installing runtimes and device drivers, verify PySDK on Windows 11 or Ubuntu 24.04 by running:
{% code overflow="wrap" %}
```bash
degirum sys-info
```
{% endcode %}
Example output (PowerShell, Windows):
{% code overflow="wrap" %}
```powershell
(degirum-windows) PS C:\degirum\degirum-windows\Scripts> degirum sys-info
Devices:
N2X/CPU:
- '@Index': 0
TFLITE/CPU:
- '@Index': 0
- '@Index': 1
- '@Index': 2
- '@Index': 3
Software Version: 1.0.0
```
{% endcode %}
Example output (Linux):
{% code overflow="wrap" %}
```bash
(degirum) root@dl4328:~# degirum sys-info
Devices:
N2X/CPU:
- '@Index': 0
TFLITE/CPU:
- '@Index': 0
- '@Index': 1
- '@Index': 2
- '@Index': 3
Software Version: 1.0.0
```
{% endcode %}
## Troubleshooting
If you encounter errors such as:
{% code overflow="wrap" %}
```bash
ERROR: Could not find a version that satisfies the requirement degirum
ERROR: No matching distribution found for degirum
```
{% endcode %}
Create a new virtual environment and reinstall PySDK.
You can also try upgrading `pip` with this command:
{% code overflow="wrap" %}
```bash
python3 -m pip install -U pip
```
{% endcode %}
On Windows, confirm that the [Visual C++ Redistributable for Visual Studio 2015 or later](https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170) is installed.
# Token Installation and Management
Create, install, and manage DeGirum Workspace tokens for PySDK, including the one-token-per-device rule, auto-renewal behavior, and secure practices for small setups and large fleets.
This guide explains how to create, install, and manage DeGirum Workspace tokens for PySDK deployments, from a handful of developer machines to large fleets. It also clarifies key operational behaviors such as auto-renewal and the single-token-per-device rule, which affect best practices.
## What a Workspace token is used for
A **Workspace token** represents your Workspace permissions and is required for PySDK operation. It is used for:
* **AI Hub access and inference**: Authenticates and authorizes actions based on Workspace permissions (e.g., cloud inference and access to Workspace resources).
* **PySDK license generation**: PySDK uses the token to obtain the license required to run.
* **Application package licenses**: The token is also used to generate and validate licenses for DeGirum application packages such as `degirum-face` and `degirum-speech`.
{% hint style="info" %}
For PySDK deployments, install a Workspace token, not a personal token. A personal token cannot generate PySDK licenses or application package licenses.
{% endhint %}
## Workspace plans and permissions matter
Each Workspace is on a specific plan (see [Workspace Plans](/ai-hub/workspace-plans) for details), and the token’s capabilities depend on:
* the Workspace’s plan and entitlements, and
* the permissions granted to that Workspace.
Install a token from the Workspace that has the permissions and plan needed for the workloads you intend to run.
{% hint style="info" %}
**Only one token can be installed on a device.**
A device can have only one Workspace token installed at a time (per user environment). This is important when:
* you belong to multiple Workspaces, or
* you switch between Workspaces with different permissions.
{% endhint %}
Before installing a token, confirm you are using the token for the correct Workspace (the one with the permissions and plan you need). If you install a token for a different Workspace later, it replaces the previous token on that device.
## Use device-scoped tokens
DeGirum recommends using one token per device (or per user plus device) because it provides a strong balance of security, traceability, and operational control, especially at scale.
### Auto-renewal: security and leak detection
Workspace tokens are often time-limited. Once a token is installed on a device, PySDK can auto-renew it as needed to keep the system running.
If a token is copied off a device and used elsewhere:
* The other party may be able to use and renew that token.
* In practice, renewal becomes an early warning system. If someone else renews the token, your device’s next renewal attempt may fail, alerting you to a potential leak or misuse.
Because tokens are time-limited, leakage is naturally bounded unless the token continues to be renewed.
{% hint style="success" %}
Auto-renewal can make token misuse more detectable, and device-scoped tokens help keep the blast radius small.
{% endhint %}
## Token security best practices
* **Use** **one token per device** (or per user plus device).
* **Install tokens once on each device** and let PySDK handle renewal.
* **Set expirations and rotate tokens** as part of lifecycle operations.
* **Do not hardcode tokens in code** or commit them to repos.
### Never-expiring tokens (discouraged)
AI Hub can generate never-expiring tokens, and technically they can be shared. However, DeGirum strongly discourages using never-expiring shared tokens, especially across multiple devices, because:
* they increase the blast radius of a leak,
* they reduce traceability (it is harder to know which device used it),
* they make incident response disruptive (one revoke can break many systems at once).
Use never-expiring tokens only when you have a strong reason and compensating controls (e.g., strict secrets management, tight network controls, and a clear rotation and revocation plan).
## Deployment paths
### 1 to 10 devices: AI Hub UI to install via CLI
This is best for individuals, small teams, and labs.
{% stepper %}
{% step %}
**Create a token in AI Hub (UI)**
1. Open **AI Hub → Workspace → Workspace Tokens**
2. Click **Generate Token**
3. Set:
* **Description** (e.g., `John-MacBook`, `Lab-Workstation-01`)
* **Expiration** (recommended)
4. Copy the token and store it securely
{% hint style="success" %}
Use descriptions that identify the owner and device. This makes audits and cleanup simpler later.
{% endhint %}
{% endstep %}
{% step %}
**Install the token on the device**
On the target device:
```bash
degirum token install
```
Optional verification:
```bash
degirum token status
```
{% hint style="warning" %}
Only one token can be installed on a device. Installing a new Workspace token replaces the existing one.
{% endhint %}
{% endstep %}
{% step %}
**Use PySDK without embedding tokens**
Once installed, your applications should typically rely on the installed token so you do not have to pass `token=...` through scripts and configs.
{% endstep %}
{% endstepper %}
{% columns %}
{% column %}
**Pros**
* Fast onboarding
* Easy troubleshooting (`degirum token status`)
* Keeps secrets out of code
{% endcolumn %}
{% column %}
**Cons**
* Requires copy and paste for initial setup
* If you switch Workspaces often, you will need to reinstall the correct token (because only one can be installed at a time)
* Rotation requires re-install on affected machines
{% endcolumn %}
{% endcolumns %}
### 10 to 10,000+ devices: fleet provisioning and lifecycle management
This is best for fleets, production rollouts, and customer deployments.
At scale, the goal is:
* zero copy and paste
* one token per device
* central governance
* safe rotation and revocation
#### Option 1 (recommended): provisioning-time install (IT-managed fleets)
{% stepper %}
{% step %}
**Define a token naming convention**
In AI Hub, generate tokens using a consistent description scheme, for example:
* `team-env-assetid` → `vision-prod-LAP1234`
* `customer-site-deviceid` → `acme-warehouse-GW-008`
{% endstep %}
{% step %}
**Distribute tokens securely**
DeGirum recommends using your existing secure delivery channels:
* MDM (Intune, Jamf)
* provisioning scripts (Ansible, Salt, cloud-init)
* secrets managers (Vault, AWS Secrets Manager, Azure Key Vault)
* manufacturing or staging pipelines
{% hint style="warning" %}
Avoid email, shared documents, and embedding tokens in images or repos.
{% endhint %}
{% endstep %}
{% step %}
**Install during provisioning**
On each device:
```bash
degirum token install
```
Optional validation:
```bash
degirum token status
```
{% endstep %}
{% step %}
**Lifecycle management (rotate and revoke)**
* Lost or decommissioned device: revoke or delete the token in AI Hub
* Reprovision device: issue a new token and reinstall
* Optional local cleanup if the device is reachable:
```bash
degirum token clear
```
{% endstep %}
{% endstepper %}
{% columns %}
{% column %}
**Pros**
* Fully automatable
* Best auditability and smallest blast radius
* Operationally clean rotation and revocation
{% endcolumn %}
{% column %}
**Cons**
* Requires integrating token distribution into your provisioning workflow
* Rotation is best handled with automation (recommended at scale)
{% endcolumn %}
{% endcolumns %}
#### Option 2: self-service enrollment (developer-managed fleets)
If devices are developer-managed, self-service often gives the best user experience:
```bash
degirum token create
```
The CLI guides the user through authentication and installs the token locally. This avoids IT distributing secrets.
{% columns %}
{% column %}
**Pros**
* Best end-user experience
* Minimal secret handling by IT
* Great for onboarding many developers quickly
{% endcolumn %}
{% column %}
**Cons**
* Not ideal for locked-down production devices
* Requires user action during setup
* Users must ensure they enroll into the correct Workspace (only one token can be installed at a time)
{% endcolumn %}
{% endcolumns %}
## Recommended and discouraged practices
### Recommended
* One token per device (or per user plus device)
* Install via CLI and keep tokens out of code and config
* Use expirations and rotate as policy requires
* Use descriptive names for traceability
* Ensure the installed token corresponds to the Workspace with the right plan and permissions (since only one token can be installed)
### Discouraged
* A single never-expiring token shared across many devices
* Tokens embedded in repos, container images, or plaintext shared documents
## Troubleshooting
### “License/token missing” or “renewal failed”
{% stepper %}
{% step %}
**Check if a token is installed**
```bash
degirum token status
```
Confirm you installed a Workspace token (not a personal token). Personal tokens cannot generate PySDK or application package licenses.
{% endstep %}
{% step %}
**If missing (or wrong Workspace), install the correct token**
```
degirum token install --token
```
{% endstep %}
{% step %}
**If renewal fails unexpectedly**
Confirm the token hasn’t expired or been revoked in AI Hub.
{% hint style="danger" %}
If you suspect compromise, revoke the token in AI Hub and issue a new one.
{% endhint %}
{% endstep %}
{% endstepper %}
## FAQ
### Why can’t I keep tokens for multiple Workspaces installed at once?
PySDK uses a single installed token for licensing and Workspace permissions in that user environment. If you need to switch Workspaces, install the token for the Workspace you want to use on that device.
### Which packages use token-based licensing?
PySDK uses Workspace tokens for licensing, and DeGirum Application Packages such as `degirum-face` and `degirum-speech` also rely on token-based license generation and validation.
# Runtimes and Drivers
This page provides an overview of the runtimes and drivers supported by PySDK.
PySDK comes with support for N2X and TFLite runtimes.
In addition to N2X and TFLite runtimes, we support Hailo, DEEPX, Intel (OpenVINO), Axelera AI, EdgeCortix, MemryX, BrainChip, and Rockchip. Refer to these pages for more details, including runtime and driver setup instructions.
## Supported Runtime Versions
{% hint style="info" %}
This table is based on PySDK 0.20.0.
{% endhint %}
| Vendor | Runtime | Versions |
| ---------- | -------- | ----------------------------------------- |
| Hailo | HAILORT | 4.23.0/4.22.0/4.21.0/4.20.1/4.20.0/4.19.0 |
| DEEPX | DEEPX | 2.9.5 |
| Intel | OPENVINO | 2025.3.0, 2024.6.0, 2023.3.0 |
| Axelera AI | AXELERA | 1.4.1 |
| MemryX | MEMRYX | 2.0 |
| BrainChip | AKIDA | 2.11.0 |
| Rockchip | RKNN | 2.3.0 |
You may access runtime and driver installation instructions per-vendor here:
# Hailo
This page details the installation procedures for the Hailo runtime in PySDK. It outlines the steps for installing on Raspberry Pi and Ubuntu.
PySDK supports HailoRT versions 4.23.0, 4.22.0, 4.21.0, 4.20.1, 4.20.0, and 4.19.0.
## Installation on Raspberry Pi
If you are using a Hailo-8 or Hailo-8L AI accelerator with Raspberry Pi 5, please refer to [Hailo Raspberry Pi 5 installation guide](https://github.com/hailo-ai/hailo-rpi5-examples/blob/main/doc/install-raspberry-pi5.md).
## Installation on Ubuntu
Download the appropriate version of HailoRT based on the platform architecture.\
The machine's architecture can be found with the following command:
{% code overflow="wrap" %}
```bash
dpkg --print-architecture
```
{% endcode %}
Find the appropriate package on Hailo's Software Downloads Page.
| Package Name (Software Downloads Page) | HailoRT Package | CPU Architecture |
| ---------------------------------------- | -------------------------- | ---------------- |
| HailoRT – Ubuntu package (deb) for amd64 | hailort\_4.23.0\_amd64.deb | amd64 |
| HailoRT – Ubuntu package (deb) for arm64 | hailort\_4.23.0\_arm64.deb | arm64 |
| HailoRT – Ubuntu package (deb) for armel | hailort\_4.23.0\_armel.deb | armel |
| HailoRT – Ubuntu package (deb) for armhf | hailort\_4.23.0\_armv4.deb | armv4 |
{% hint style="info" %}
PySDK supports Hailo Runtime versions 4.23.0, 4.22.0, 4.21.0, 4.20.1, 4.20.0, and 4.19.0.
{% endhint %}
The Hailo's Software Downloads Page accessible via [this link](https://hailo.ai/developer-zone/software-downloads).
{% hint style="info" %}
For PySDK integration with Hailo hardware, software distributed by Hailo is required. To access the downloadable packages, a Hailo account is required.
{% endhint %}
After downloading the appropriate package, run the following command to install HailoRT:
{% code overflow="wrap" %}
```bash
sudo dpkg --install hailort_4.23.0_.deb
```
{% endcode %}
Download the **HailoRT – PCIe driver (deb)**\
(It can be quickly found by typing "PCIe driver" into the "Search for a package" window)\
Run the following command to install the driver:
{% code overflow="wrap" %}
```bash
sudo dpkg --install hailort-pcie-driver_4.23.0_all.deb
```
{% endcode %}
{% hint style="info" %}
PC restart is required after driver installation.
{% endhint %}
After the PC restart, you can verify the installation by running the following command:
{% code overflow="wrap" %}
```bash
hailortcli fw-control identify
```
{% endcode %}
# OpenVINO
This page details the installation procedures for the OpenVINO Runtime supported by PySDK.
PySDK supports OpenVINO™ versions 2023.3.0, 2024.6.0, and 2025.3.0. Version 2025.3.0 is recommended.
## Linux Installation
On Linux, installation via APT Package Manager or via .tgz archive is supported.
Currently, NPU requires that OpenVINO™ be installed via .tgz archive.
* **To install OpenVINO™ via APT Package Manager**: Follow the instructions at [OpenVINO APT Installation Guide](https://docs.openvino.ai/2024/get-started/install-openvino/install-openvino-apt.html).
* **To install OpenVINO™ via .tgz archive**: Follow the instructions at [OpenVINO Archive Installation Guide](https://docs.openvino.ai/2024/get-started/install-openvino/install-openvino-archive-linux.html).
After installation via .tgz archive, whenever you open a new terminal, run the following command to configure the environment:
{% code overflow="wrap" %}
```bash
source /setupvars.sh
```
{% endcode %}
When using PySDK in a Python interpreter or a development environment, ensure the interpreter is started from a terminal where this command has been executed.
## Windows Installation
On Windows, installation via .zip archive is supported.
* **To install OpenVINO™ via .zip archive**: Follow the instructions at [OpenVINO Windows Installation Guide](https://docs.openvino.ai/2024/get-started/install-openvino/install-openvino-archive-windows.html).
After installation, whenever you open a new terminal, run the following command to configure the environment:
{% code overflow="wrap" %}
```bash
\setupvars.bat
```
{% endcode %}
Ensure you use the Command Prompt (not PowerShell) to execute this script.
## GPU Drivers for OpenVINO™
To install Intel GPU drivers, follow the instructions at [OpenVINO GPU Driver Guide](https://docs.openvino.ai/2024/get-started/configurations/configurations-intel-gpu.html).
Integrated Intel GPUs are available through the `OPENVINO/GPU` designator when no discrete GPU is present. Systems with both integrated and discrete GPUs continue to bind `OPENVINO/GPU` to the discrete device.
## NPU Drivers for OpenVINO™
To use NPU with OpenVINO™, ensure that the host system has the latest NPU drivers installed. Regularly check for updates to maintain compatibility.
* **For Linux NPU Drivers**: Follow the instructions at [Intel NPU Driver for Linux](https://github.com/intel/linux-npu-driver/releases/tag/v1.10.1).
* **For Windows NPU Drivers**: Download the driver from [Intel NPU Driver for Windows](https://www.intel.com/content/www/us/en/download/794734/intel-npu-driver-windows.html). Refer to the [Windows NPU Driver Instructions](https://downloadmirror.intel.com/850247/NPU_Win_Release_Notes_v32.0.100.3967.pdf) for detailed setup guidance.
Sometimes, NPUs are disabled in the UEFI settings by default. Ensure the NPU is enabled in your system’s UEFI configuration.
## CPU Support for OpenVINO™
OpenVINO™ CPU runtime is supported on Intel and AMD x86-64 hosts. Use the `OPENVINO/CPU` designator to target CPU inference. No extra drivers are required beyond the OpenVINO™ runtime itself.
# Axelera AI
This page details installation procedures for the Axelera runtime in PySDK.
{% hint style="info" %}
Power cycle the system with the Axelera accelerator to reinitialize the device.
{% endhint %}
## Installing the Axelera Runtime and Driver
The following instructions will only install the necessary drivers to use the Axelera AI chip, and will not install the full version of Axelera AI's Voyager SDK. If you wish to install the full SDK, please read their [installation instructions](https://github.com/axelera-ai-hub/voyager-sdk/blob/release%2Fv1.4/docs%2Ftutorials%2Finstall.md).
{% stepper %}
{% step %}
Create the keyring directory if it doesn't exist.
{% code overflow="wrap" %}
```bash
sudo mkdir -p "$(dirname "/etc/apt/keyrings/axelera.gpg")"
```
{% endcode %}
{% endstep %}
{% step %}
Download and store the GPG key.
{% code overflow="wrap" %}
```bash
curl -fsSL "https://software.axelera.ai/artifactory/api/security/keypair/axelera/public" | gpg --dearmor | sudo tee "/etc/apt/keyrings/axelera.gpg" > /dev/null
```
{% endcode %}
{% endstep %}
{% step %}
Ensure correct permissions for the key.
{% code overflow="wrap" %}
```bash
sudo chmod 644 "/etc/apt/keyrings/axelera.gpg"
```
{% endcode %}
{% endstep %}
{% step %}
Add the repository source.
{% code overflow="wrap" %}
```bash
echo "deb [signed-by=/etc/apt/keyrings/axelera.gpg] https://software.axelera.ai/artifactory/axelera-apt-source/ stable main" | sudo tee "/etc/apt/sources.list.d/axelera.list" > /dev/null
```
{% endcode %}
{% endstep %}
{% step %}
Update APT.
{% code overflow="wrap" %}
```bash
sudo apt update
```
{% endcode %}
{% endstep %}
{% step %}
Install the packages with specific versions.
{% code overflow="wrap" %}
```bash
sudo apt install -y axelera-runtime-1.4.0 axelera-device-1.4.0 metis-dkms=1.2.2 axelera-riscv-gnu-newlib-toolchain-409b951ba662-7
```
{% endcode %}
{% endstep %}
{% step %}
Add Axelera runtime libraries to ldconfig
{% code overflow="wrap" %}
```bash
echo "/opt/axelera/runtime-1.4.0-1/lib" | sudo tee /etc/ld.so.conf.d/axelera.conf > /dev/null
sudo ldconfig
```
{% endcode %}
{% endstep %}
{% step %}
Perform a hard reset (power cycle) of your system. A regular restart will prevent the Metis driver from loading correctly.
{% endstep %}
{% endstepper %}
# MemryX
This page provides information on installing the MemryX Runtime for MemryX AI accelerators.
## Installation on Linux or Windows
If you are using [MemryX AI accelerator](https://memryx.com/products/), you need to install MemryX Runtime.
{% hint style="info" %}
PySDK 0.19.2 adds support for MemryX Runtime 2.0. Install the 2.0 release from MemryX.
{% endhint %}
Please refer to [MemryX installation guide](https://developer.memryx.com/get_started/install.html):
{% hint style="info" %}
Linux kernel v5.6.0 or higher is required for MemryX driver installation.
{% endhint %}
[MemryX Linux Runtime Installation](https://developer.memryx.com/get_started/install_driver.html)
[MemryX Windows Runtime Installation](https://developer.memryx.com/get_started/install_windows.html)
{% hint style="info" %}
PC restart is required after driver installation.
{% endhint %}
# BrainChip
This page provides step-by-step instructions for installing the Akida (BrainChip) runtime in PySDK.
### Akida (Brainchip) Runtime in PySDK
PySDK supports version **2.11.0** of Akida Runtime.
#### Supported Platforms
**Linux**
* **Architecture**: x86-64 or ARM64/AArch64
* **Python Versions**: 3.9–3.11
***
### Installation
Ensure your pip is up to date and install the Akida Runtime package:
{% code overflow="wrap" %}
```bash
pip install --upgrade pip
pip install akida==2.11.0
```
{% endcode %}
You can verify the installation by running the following command:
{% code overflow="wrap" %}
```bash
akida version
```
{% endcode %}
You can check available Brainchip devices by running the following command:
{% code overflow="wrap" %}
```bash
akida devices
```
{% endcode %}
{% hint style="danger" %}
REQUIRED: Add Akida Runtime to LD\_LIBRARY\_PATH
{% endhint %}
For PySDK to locate Akida Runtime, LD\_LIBRARY\_PATH must include the Akida Runtime library path.
**Temporary Setup**
To set `LD_LIBRARY_PATH` for the current terminal session, run the following command:
{% code overflow="wrap" %}
```bash
export LD_LIBRARY_PATH="$(python3 -c "import akida, os; print(os.path.dirname(akida.__file__))"):$LD_LIBRARY_PATH"
```
{% endcode %}
**Permanent Setup**
To make this change persistent, we find the Akida Runtime path and then add it to `~/.bashrc`. Run the following commands:
{% code overflow="wrap" %}
```bash
AKIDA_PATH=$(python3 -c 'import akida, os; print(os.path.dirname(akida.__file__))')
echo 'export LD_LIBRARY_PATH="'$AKIDA_PATH':$LD_LIBRARY_PATH"' >> ~/.bashrc
source ~/.bashrc
```
{% endcode %}
#### Troubleshooting
If no Brainchip device is detected in `degirum sys-info` output, please see the permanent setup instructions here.
If no Brainchip device is detected in `akida devices` output, you may need to (re)install the Akida PCIe driver.
### Driver Installation
Driver Installation is done through a script provided by Brainchip in their GitHub repository:
{% code overflow="wrap" %}
```bash
git clone https://github.com/Brainchip-Inc/akida_dw_edma
cd akida_dw_edma
sudo ./install.sh
```
{% endcode %}
After installation, check the device list again:
{% code overflow="wrap" %}
```bash
akida devices
```
{% endcode %}
# Rockchip
This page details the installation procedures for Rockchip devices.
PySDK supports version 2.0.0+ of the RKNN runtime on Rockchip systems.
You can check for an existing installation of RKNN Runtime by running the following command:
{% code overflow="wrap" %}
```bash
strings /usr/lib/librknnrt.so | grep librknnrt
```
{% endcode %}
Example output:
{% code overflow="wrap" %}
```bash
librknnrt version: 2.0.0b0 (35a6907d79@2024-03-24T10:31:14)
```
{% endcode %}
To verify that the Rockchip NPU driver is properly loaded and functioning on your system, you can check the system logs for NPU-related entries. Run the following command:
{% code overflow="wrap" %}
```bash
dmesg | grep rknpu
```
{% endcode %}
On some Pi systems, the Rockchip NPU may be disabled in the system configuration panel (e.g. raspi-config). In such cases, enable the NPU and reboot the system.
To install the RKNN Runtime, clone the RKNN repository from [this link](https://github.com/airockchip/rknn-toolkit2/tree/master). Then, simply copy the `librknnrt.so` file to the `/usr/lib/` directory:
{% code overflow="wrap" %}
```bash
git clone https://github.com/airockchip/rknn-toolkit2.git
cd rknn-toolkit2
sudo cp ./rknpu2/runtime/Linux/librknn_api/aarch64/librknnrt.so /usr/lib/
```
{% endcode %}
# ONNX
This page provides step-by-step instructions for installing the ONNX runtime in PySDK.
### ONNX Runtime in PySDK
PySDK supports version 1.20.1 of ONNX Runtime, on Linux, Windows, and Mac.
To install the ONNX runtime, download an 1.20.1 archive for your system [here](https://github.com/microsoft/onnxruntime/releases/tag/v1.20.1). Do not use the "-training-" archives. Then, extract it into an appropriate directory:
* On Windows, extract the archive to `C:\Program Files`, `C:\Program Files (x86)`, or `C:\ProgramData`.
* On Linux, extract the archive to `/usr/local/`.
* On Mac, extract the archive to `/usr/local/`.
{% hint style="warning" %}
Do not rename the `onnxruntime---1.20.1` directory after you extract it.
{% endhint %}
### AMD Ryzen™ AI in PySDK
PySDK supports AMD Ryzen™ AI version 1.2.
To install AMD Ryzen™ AI, click [this link](https://ryzenai.docs.amd.com/en/latest/inst.html) to read installation instructions.
The installation includes: (1) an NPU driver and (2) a "RyzenAI Software MSI installer."
{% hint style="warning" %}
* Make sure to launch the driver `npu_sw_installer.exe` from the terminal while in admin mode.
* Don't forget to append your `C:\path\to\miniconda3\Scripts` to the `Path` System environment variable before launching the installer.
* Restart the terminal (if using VS Code, restart all open windows) after installation.
{% endhint %}
Run `conda activate ` to enter the environment created by the installation wizard.
{% hint style="info" %}
You may optionally set set Ryzen™ AI environment variables. PySDK initializes the defaults for your processor type automatically. For more information, see the notes below.
{% endhint %}
At this point, you should be ready to run inference on models prepared for the Ryzen NPU from the DeGirum AI Hub Model Zoo.
#### About environment variables
Ryzen™ AI requires certain environment variables to be set depending on the processor configuration:
* Phoenix (PHX): AMD Ryzen™ 7940HS, 7840HS, 7640HS, 7840U, 7640U.
* Hawk (HPT): AMD Ryzen™ 8640U, 8640HS, 8645H, 8840U, 8840HS, 8845H, 8945H.
* Strix (STX): AMD Ryzen™ Ryzen AI 9 HX370, Ryzen AI 9 365.
#### About your processor
To find your processor configuration, go to **Settings -> System -> About**.
If you want to manually specify the environment variables, then either in Windows CMD, Powershell, or inside a Python script run the following:
* If your processor is Phoenix or Hawk (PHX/HPT):
* CMD:
{% code overflow="wrap" %}
```bash
set XLNX_VART_FIRMWARE=%RYZEN_AI_INSTALLATION_PATH%voe-4.0-win_amd64/xclbins/phoenix/1x4.xclbin
set XLNX_TARGET_NAME=AMD_AIE2_Nx4_Overlay
```
{% endcode %}
* Powershell:
{% code overflow="wrap" %}
```powershell
$env:XLNX_VART_FIRMWARE="$env:RYZEN_AI_INSTALLATION_PATH"+"voe-4.0-win_amd64/xclbins/phoenix/1x4.xclbin"
$env:XLNX_TARGET_NAME="AMD_AIE2_Nx4_Overlay"
```
{% endcode %}
* Python:
{% code overflow="wrap" %}
```python
import os
os.environ['XLNX_VART_FIRMWARE'] = os.environ['RYZEN_AI_INSTALLATION_PATH'] + 'voe-4.0-win_amd64/xclbins/phoenix/1x4.xclbin'
os.environ['XLNX_TARGET_NAME'] = "AMD_AIE2_Nx4_Overlay"
```
{% endcode %}
* If your processor is Strix (STX):
* CMD:
{% code overflow="wrap" %}
```bash
set XLNX_VART_FIRMWARE=%RYZEN_AI_INSTALLATION_PATH%voe-4.0-win_amd64/xclbins/strix/AMD_AIE2P_Nx4_Overlay.xclbin
set XLNX_TARGET_NAME=AMD_AIE2P_Nx4_Overlay
```
{% endcode %}
* Powershell:
{% code overflow="wrap" %}
```powershell
$env:XLNX_VART_FIRMWARE="$env:RYZEN_AI_INSTALLATION_PATH"+"voe-4.0-win_amd64/xclbins/strix/AMD_AIE2P_Nx4_Overlay.xclbin"
$env:XLNX_TARGET_NAME="AMD_AIE2P_Nx4_Overlay"
```
{% endcode %}
* Python:
{% code overflow="wrap" %}
```python
import os
os.environ['XLNX_VART_FIRMWARE'] = os.environ['RYZEN_AI_INSTALLATION_PATH'] + 'voe-4.0-win_amd64/xclbins/strix/AMD_AIE2P_Nx4_Overlay.xclbin'
os.environ['XLNX_TARGET_NAME'] = "AMD_AIE2P_Nx4_Overlay"
```
{% endcode %}
#### Changing environment with Python
When setting environment variables with Python, put the code lines before `import degirum` for PySDK to see the changes.
**These variables have to be set every time you open a fresh terminal to run inference.** In case you want to set them permanently, go to `Edit the system environment varibles` in Windows search and add (or edit) the two variables through the GUI.
{% hint style="warning" %}
Incorrect environment variables may lead to a crash. PySDK detects when the value of `env:XLNX_VART_FIRMWARE` doesn't match the CPU type and automatically sets it to the one appropriate for your device to prevent a system crash. However, be careful when manually redefining the environment variables.
{% endhint %}
#### About NPU Configurations:
The instructions above correspond the "Standard Configuration" of the NPU. An additional "Benchmark Configuration" is supported by Ryzen™ AI. The setup process is similar (requires different environment variable values) and is described in detail on the [Ryzen™ AI Runtime Setup Page.](https://ryzenai.docs.amd.com/en/latest/runtime_setup.html)
{% hint style="info" %}
Models will be recompiled when run with a different NPU configuration. PySDK detects when an inference on a model is launched in a configuration different from the one it was compiled for (if the cache exists) and recompiles it at runtime to prevent a crash. So expect some initial delay when trying a different NPU configuration.
{% endhint %}
#### About supported model format
DeGirum AI Hub Model Zoo supplies INT8 symmetrically quantized models required by the Ryzen™ AI NPU. Models come with precompiled caches to bypass the sometimes lengthy compilation process and enable immediate inference. If recompilation is needed or if cached models cause errors, remove the `_cache` subdirectory from the downloaded model directory. Inference will then proceed with just-in-time compilation automatically.
# TensorRT
PySDK supports the NVIDIA TensorRT runtime on Linux, Windows, and NVIDIA Jetson hardware. This page walks through installation on each platform.
## Supported Versions
PySDK is validated with the following setups:
1. AMD64 host, CUDA GPU compute capability 8.9, CUDA 12.4, TensorRT 10.6.
2. AMD64 host, CUDA GPU compute capability 12.0, CUDA 12.9, TensorRT 10.13.
3. ARM64 host, JetPack 6.2.
{% hint style="info" %}
CUDA GPU compute capability is determined by the [GPU installed in the system](https://developer.nvidia.com/cuda-gpus). Each level of CUDA GPU compute capability requires a minimum CUDA version (see the "Max CC" column in [this compatibility table](https://stackoverflow.com/questions/28932864/which-compute-capability-is-supported-by-which-cuda-versions/28933055#28933055)). In practice, RTX 5000 series GPUs require CUDA versions greater than 12.6 and a matching TensorRT version.
{% endhint %}
## Linux Installation
### CUDA
We recommend the local Debian Installer method.
{% stepper %}
{% step %}
Make sure you have a compatible GPU and that its drivers are installed.
{% endstep %}
{% step %}
Download CUDA from the [NVIDIA CUDA Toolkit archive](https://developer.nvidia.com/cuda-toolkit-archive).
{% endstep %}
{% step %}
Follow NVIDIA's [CUDA installation guide for Ubuntu](https://docs.nvidia.com/cuda/cuda-quick-start-guide/index.html#ubuntu).
{% endstep %}
{% step %}
Update `PATH` and `LD_LIBRARY_PATH`, replacing `` with the version you installed:
{% code overflow="wrap" %}
```bash
export PATH=/usr/local/cuda-/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda-/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
```
{% endcode %}
{% endstep %}
{% step %}
Verify the installation:
{% code overflow="wrap" %}
```bash
nvcc --version
```
{% endcode %}
{% endstep %}
{% endstepper %}
As an alternative, the default system repository often includes a supported version of CUDA.
### TensorRT
We recommend the local Debian Installer method.
{% stepper %}
{% step %}
Follow NVIDIA's [TensorRT download and installation instructions](https://docs.nvidia.com/deeplearning/tensorrt/latest/installing-tensorrt/installing.html#downloading-tensorrt).
{% endstep %}
{% step %}
Install the full `tensorrt` package. cuDNN is not required.
{% endstep %}
{% step %}
The `.deb` installer handles `LD_LIBRARY_PATH` automatically.
{% endstep %}
{% endstepper %}
## Windows Installation
### CUDA
Follow NVIDIA's [CUDA installation guide for Windows](https://docs.nvidia.com/cuda/cuda-quick-start-guide/index.html#windows). Python wheels are not required.
### TensorRT
{% stepper %}
{% step %}
Follow steps 1–4 of NVIDIA's [TensorRT zip-file installation guide](https://docs.nvidia.com/deeplearning/tensorrt/latest/installing-tensorrt/installing.html#zip-file-installation).
{% endstep %}
{% step %}
On Windows, TensorRT does not update `PATH` automatically. Make sure you complete step 4 from the guide above.
{% endstep %}
{% endstepper %}
## JetPack Installation
CUDA and TensorRT are included in JetPack. PySDK supports JetPack 6.2. See NVIDIA's [JetPack installation guide](https://docs.nvidia.com/sdk-manager/install-with-sdkm-jetson/index.html) for details.
# PySDK User Guide
Explore DeGirum's PySDK in depth. This guide covers the core concepts.
# Core Concepts
Explore the core components of PySDK—including the AI inference engine, AI model, and model zoo—to understand how they power modern edge AI applications.
{% embed url="" %}
## **Main Concepts in PySDK**
There are three main components in PySDK: the **AI inference engine**, the **AI model**, and the **AI model zoo**. Together, they form the core of the PySDK ecosystem, making it easy to add AI capabilities to any application.
{% stepper %}
{% step %}
**AI Inference Engine**
The component responsible for performing predictions by running AI models on hardware.
{% endstep %}
{% step %}
**AI Model**
The actual trained model used to make predictions, such as detecting objects or recognizing faces.
{% endstep %}
{% step %}
**AI Model Zoo**
A storage location where a collection of AI models is kept and accessed by the inference engine.
{% endstep %}
{% endstepper %}
### What is a Client Application?
Any program you create or use that leverages PySDK to perform AI tasks is a client application. This application could be a Python script, a web service, or any other software that communicates with PySDK to send inputs (like images or video frames) to the AI inference engine and receive predictions (such as detected objects or classifications).
### How the AI Inference Engine Operates Across Environments
The AI inference engine is responsible for running AI models on hardware, but it can be deployed and accessed in various environments to suit different application needs. PySDK supports three key types of inference setups:
* **AI Hub Inference**: When the AI inference engine runs on hardware hosted and managed by the DeGirum AI Hub.
* **AI Server Inference**: When the AI inference engine is controlled by a local or networked AI server.
* **Local Inference**: When the AI inference engine directly communicates with local AI hardware on the same machine as the client application.
## Types of AI Inference Supported by PySDK
Let’s explore the different types of inference setups supported by PySDK in more detail:
{% stepper %}
{% step %}
**AI Hub Inference**
In this setup, the DeGirum AI Hub Application Server manages the inference engine and connects to the DeGirum Device Farm, a collection of cloud-hosted computing nodes with diverse hardware configurations like CPUs, NPUs, and AI accelerators. The client application communicates with the server over a network using HTTP or SocketIO protocols.
**When to Use AI Hub Inference**
* Ideal for rapid prototyping and development when you want to test different models and hardware configurations.
* Useful when local hardware is unavailable or when you need flexibility in exploring various options without purchasing or configuring hardware.
**Key Advantage**
Similar to services like AWS Rekognition or Clarifai, but with greater flexibility, allowing users to choose hardware configurations and deploy custom models optimized for specific applications.
{% endstep %}
{% step %}
**AI Server Inference**
The DeGirum AI Server manages inference locally or across a network, acting as an intermediary between the client application and the AI hardware. The client application communicates with the AI server using HTTP or ASIO protocols, allowing multiple applications or machines to access shared AI hardware.
**When to Use AI Server Inference**
* Suitable for distributed environments where multiple applications or machines need shared access to the same AI hardware.
* Ideal when you want to separate application logic from hardware management and maintain centralized control over resources.
**Key Advantage**
Similar to NVIDIA Triton or OpenVINO Model Server, but more flexible, enabling easy scaling by allowing multiple applications to share hardware. Ideal for data centers, edge deployments, or lab setups.
{% endstep %}
{% step %}
**Local Inference**
In a local inference setup, both the client application and the AI inference engine operate on the same machine, eliminating the need for network communication between separate client and server components. The client directly interacts with the hardware using PySDK through efficient, low-latency function calls.
**When to Use Local Inference**
* Ideal for edge devices and standalone AI systems where minimizing latency and reducing external dependencies is critical.
* Simplifies deployment and maintenance when the application and AI hardware are on the same system.
**Key Advantage**
Similar to runtimes like ONNX Runtime and TensorFlow Lite, but PySDK’s API enables seamless deployment across different hardware configurations (CPUs, NPUs, AI accelerators) without modifying application code.
{% hint style="info" %}
While local inference supports multiple applications sharing the AI hardware on the same machine for some hardware options, AI server inference is recommended for distributed setups where multiple machines or applications need shared access to the same hardware over a network.
{% endhint %}
{% endstep %}
{% endstepper %}
### Summary of Communication Protocols
Inference Type
Server-Client Protocol
Description
AI Hub
Yes (HTTP+SocketIO)
Communication over the Internet with the Application Server. Best for rapid prototyping without hardware setup.
AI Server
Yes (HTTP/ASIO)
Communication between the client and the AI server using HTTP or ASIO. The server can run on the same machine (localhost) or a different machine on the local network. Ideal for multiple machines sharing the same AI hardware.
Local
No
Direct function calls to the hardware using PySDK. Supports multiple applications sharing the same hardware on the same machine.
## **AI Models**
An **AI model** is the core component responsible for making predictions, such as detecting objects, recognizing faces, or performing classifications. In PySDK, an AI model is defined by a set of files that include model configurations, binaries, and optional supporting files for labels and postprocessing.
### **Key Components of an AI Model**
{% stepper %}
{% step %}
**Model JSON File (`.json`)**
Contains the configuration and metadata required for loading and running the model.
Specifies key parameters such as the postprocessing logic needed for interpreting inference results.
For more details, see our page on [Model JSON Structure](/pysdk/user-guide-pysdk/model-json-structure).
{% endstep %}
{% step %}
**Model Binary Files**
Stores the model's weights and architecture in a format optimized for the target hardware and runtime. Required files depend on the model type and the hardware backend.
For more details, see [Supported Hardware](/pysdk/installation#supported-hardware).
{% endstep %}
{% step %}
**Optional Label File (`.json`)**
Label file to map output class indices to readable labels, as specified in the model JSON file.
{% endstep %}
{% step %}
**Optional Python Postprocessing File (`postprocess.py`)**
Defines custom postprocessing logic, if needed, as specified in the model JSON file.
For example, object detection models may require additional steps such as decoding bounding boxes or applying Non-Maximum Suppression (NMS).
{% endstep %}
{% endstepper %}
## **AI Model Zoos Supported by PySDK**
An AI model zoo is a collection of AI models. Model zoos simplify model management, centralize storage, and provide a consistent way to load models.
PySDK supports both local and AI Hub-based model zoos to accommodate diverse development and deployment needs.
{% stepper %}
{% step %}
**Local Model Zoos**
* Models are stored in a directory on the local computer.
* For local inferencing, models are located on the same computer as the client application.
* For AI server inference, the models are stored on the computer running the AI server software.
{% endstep %}
{% step %}
**AI Hub Model Zoos**
* Models are stored on the AI Hub.
* AI Hub provides a centralized location where models can be uploaded, organized, and shared across different environments, similar to platforms like Hugging Face model repositories.
* AI Hub model zoos act solely as storage and retrieval systems, ensuring that models are easily available without requiring manual downloads or local management. This allows for quick updates and streamlines deployment across various setups.
* While the AI Hub Model Zoo offers a web-based GUI for browsing and managing models, it also allows programmatic access through PySDK, making it easy for developers to load models directly into their applications.
{% endstep %}
{% endstepper %}
### **Supported Combinations of AI Inference and Model Zoo Types**
PySDK provides flexible options for combining AI inference types and model zoos, supporting almost all combinations to suit various deployment needs. Below are the supported configurations:
Inference Type
Model Zoo Type
Description
AI Hub
AI Hub
The client application connects to the Application Server accessing models stored in the AI Hub model zoo.
AI Server
AI Hub
The AI server downloads the models from the model zoo on the AI Hub and provides inference to the client application.
AI Server
Local Folder
The AI server accesses models stored locally and provides inference to the client application over a network.
Local
AI Hub
The client application, running on the same machine as the AI hardware, downloads models from a model zoo on the AI Hub.
Local
Local Folder
The client application directly accesses models stored locally on the machine.
Local
Local File
The client application loads a specific model directly from a .json configuration file.
If you'll host an AI server or perform inference with a local server, read the [AI Server Setup](/pysdk/user-guide-pysdk/setting-up-an-ai-server) page. You'll learn to setup a model zoo locally to prepare to run inferences.
If you plan to use the AI Hub for inference, go to [Loading an AI Model](/pysdk/user-guide-pysdk/loading-an-ai-model) to learn to load models. The models are already managed for you and you can go straight to running inferences.
{% hint style="warning" %}
AI Hub inference with a local model zoo is not supported. The AI Hub Server requires models to be hosted on the AI Hub for remote access.
{% endhint %}
# Organizing Models
Learn how AI Hub models and model zoos are organized. This page covers JSON model file naming conventions and model zoo directory structure. Local model zoos may follow these conventions.
## **JSON Model File Naming Conventions**
Each JSON model file in a model zoo follows a structured naming convention to maintain consistency and clarity:
{% code overflow="wrap" %}
```bash
--[_][_][_][_][_].json
```
{% endcode %}
{% hint style="info" %}
**`--`** and **`_`** are used as field separators.
**`<>`** denotes a field with variable contents.
**`[]`** denotes optional fields.
{% endhint %}
### **Field Descriptions**
| Field | Description | Possible Values |
| ------------ | --------------------------------------------------- | --------------------------------------------------------------------------------- |
| model family | The name of the model family | See [Model Family Names](#model-family-names) for model family naming conventions |
| resolution | Input tensor size (optional) | `x` |
| density | Model density (optional) | `pruned`, `dense` |
| precision | Model calculation precision (optional) | `quant`, `float` |
| agent | DeGirum runtime agent type for inference (optional) | `Agent type` or `multi` |
| hardware | Hardware for model inference (optional) | `Device type` or `multi` |
| version | Version of the compiled model | `` |
### **Model Family Names**
Use the following conventions when naming model families:
* **Do not use dashes (`-`)**. Use underscores (`_`) instead.
* **Do not use capital letters**. All names should be in lowercase.
* **Include the backbone network as a suffix** when applicable. For example:
* `posenet_mobilenet_v1`
* **Include the network version as a suffix** when applicable. For example:
* `mobilenet_v1`
* **Different model sizes** (such as small or large) should be defined as a suffix when applicable. For example:
* `yolo_v5s` (for a small version)
These conventions ensure models are easily identifiable and consistent across AI Hub and local model zoos.
## Model Zoo Directory Structure
Place each model in its own subdirectory with all required files.
Model zoos on the AI Hub follow this structure by default, and local model zoos can use the same approach.
#### Example: Model Directory Structure
{% code overflow="wrap" %}
```
/
├── mobilenet_v2_imagenet--224x224_quant_n2x_orca1_1/
│ ├── mobilenet_v2_imagenet--224x224_quant_n2x_orca1_1.json
│ ├── mobilenet_v2_imagenet--224x224_quant_n2x_orca1_1.n2x
│ └── labels.json (if required)
├── yolov8n_coco--640x640_quant_openvino_1/
│ ├── yolov8n_coco--640x640_quant_openvino_1.json
│ ├── yolov8n_coco--640x640_quant_openvino_1.xml
│ ├── yolov8n_coco--640x640_quant_openvino_1.bin
│ └── postprocess.py (if required)
└── yolov8n_relu6_coco_pose--640x640_quant_tflite_edgetpu_1/
├── yolov8n_relu6_coco_pose--640x640_quant_tflite_edgetpu_1.json
├── yolov8n_relu6_coco_pose--640x640_quant_tflite_edgetpu_1.tflite
└── labels.json (if required)
```
{% endcode %}
Each subdirectory corresponds to a single model and contains all the necessary files to define, load, and execute that model.
# Setting Up an AI Server
Read this page if you'll host an AI server or perform inference with a local server.
## Starting an AI Server
Use PySDK to configure and launch an AI server. The server runs on your host and processes inference requests from remote clients.
You can start the AI server in several ways:
* running AI server from a terminal directly on the host OS.
* running AI server as a Linux system service.
* running AI server with a Docker container.
### Terminal
To run the PySDK AI server from a terminal, perform the following steps:
{% stepper %}
{% step %}
**Install PySDK**
Any device with PySDK installed can run an AI server. Refer to [PySDK Installation](/pysdk/installation) for details on how to install PySDK.
{% endstep %}
{% step %}
**Create a directory for the local model zoo**
You'll need to create a directory to hold your models.
{% code overflow="wrap" %}
```bash
mkdir ~/degirum-model-zoo
```
{% endcode %}
{% endstep %}
{% step %}
**Download or copy models to the local model zoo**
After creating the model zoo directory, you'll need to populate it with models
To easily populate the model zoo, you can use the PySDK [CLI tool ](/pysdk/user-guide-pysdk/command-line-interface#download-model-zoo)for downloading models from the DeGirum AI Hub.
{% code overflow="wrap" %}
```bash
cd ~/degirum-model-zoo
degirum download-zoo
```
{% endcode %}
{% endstep %}
{% step %}
**Start the AI server**
Launch the server with the following command:
{% code overflow="wrap" %}
```bash
cd ~/degirum-model-zoo
degirum server
```
{% endcode %}
The server runs until you press `ENTER` in the terminal. By default, it listens on TCP port 8778. To specify a different port, use the `--port` argument:
{% code overflow="wrap" %}
```bash
degirum server --port
```
{% endcode %}
{% endstep %}
{% endstepper %}
### Linux Service
To automatically start the server on boot, configure it as a Linux service:
{% stepper %}
{% step %}
**Complete the terminal setup steps**
Follow all steps in [Starting AI Server from Terminal](#terminal) except for launching the server.
{% endstep %}
{% step %}
**Create a systemd service configuration file**
Create a file named `degirum.service` in the `/etc/systemd/system` directory. Use the following template:
{% code overflow="wrap" %}
```ini
[Unit]
Description=DeGirum AI Service
[Service]
WorkingDirectory=/home//
ExecStart= -m degirum.server --zoo /home//zoo
Restart=always
RestartSec=10
SyslogIdentifier=degirum-ai-server
User=
[Install]
WantedBy=multi-user.target
```
{% endcode %}
{% endstep %}
{% step %}
**Start the service**
Start the service using `systemctl`:
{% code overflow="wrap" %}
```bash
sudo systemctl start degirum.service
```
{% endcode %}
{% endstep %}
{% step %}
**Check the service status**
Check the service status using `systemctl`:
{% code overflow="wrap" %}
```bash
sudo systemctl status degirum.service
```
{% endcode %}
{% endstep %}
{% step %}
**Enable the service on startup**
Use `systemctl` to automatically enable the degirum service on startup:
{% code overflow="wrap" %}
```bash
sudo systemctl enable degirum.service
```
{% endcode %}
{% endstep %}
{% endstepper %}
### Docker Container
To run the AI server as a Docker container, follow these steps:
{% stepper %}
{% step %}
**Ensure Docker is installed**
Refer to the [official Docker documentation](https://docs.docker.com/engine/install/) for installation instructions
{% endstep %}
{% step %}
**Prepare the local model zoo**
If hosting models locally, create and populate a model zoo directory:
{% code overflow="wrap" %}
```bash
mkdir -p ~/degirum-model-zoo
cd ~/degirum-model-zoo
degirum download-zoo
```
{% endcode %}
{% endstep %}
{% step %}
**Run the Docker container**
When hosting models locally:
{% code overflow="wrap" %}
```bash
docker run --name aiserver -d -p 8778:8778 -v ~/degirum-model-zoo:/zoo --privileged degirum/aiserver:latest
```
{% endcode %}
When serving models only from AI Hub:
{% code overflow="wrap" %}
```bash
docker run --name aiserver -d -p 8778:8778 --privileged degirum/aiserver:latest
```
{% endcode %}
{% endstep %}
{% endstepper %}
## Rescanning Model Zoos
If you started your AI server in a terminal or as a Linux service, you can tell the AI server to rescan the local model zoo directory by executing the following command on the same host: `degirum server rescan-zoo`
If you started your AI server in the Docker container, then you should rescan the model zoo directory by restarting the container: `docker restart aiserver`
# Loading an AI Model
This is an end-to-end guide for loading a model. You'll start with connecting to an inference engine and model zoo, learn about filtering model lists, then loading a model.
## Connect to an Inference Engine and Model Zoo
The `degirum.connect()` function is the starting point for interacting with PySDK. It establishes a connection with the appropriate AI inference engine and model zoo based on the configuration you provide.
{% code overflow="wrap" %}
```python
import degirum
degirum.connect(
inference_host_address = "@local",
zoo_url = "workspace/zoo",
token = ""
)
```
{% endcode %}
When you call `degirum.connect()`, you will:
* Specify an inference host to run AI models
* Specify a model zoo from which AI models can be loaded
* Authenticate with the AI Hub using a token
`degirum.connect()` creates and returns a `ZooManager` object. This object enables:
* Searching for models available in the connected model zoo.
* Loading AI models and creating appropriate AI model handling objects for inference.
* Accessing model parameters to customize inference behavior.
### degirum.connect()
`degirum.connect()` takes three parameters: `inference_host_address`, `zoo_url`, and `token`. These parameters define where the inference will be run, what zoo will be used, and the token used for AI Hub authentication if an AI Hub model zoo will be used.
#### Zoo URL Parsing Behavior
The `zoo_url` parameter is handled the following ways:
* **AI Hub Inference** (`inference_host_address="@cloud"`)
* `zoo_url` may be given as `https://hub.degirum.com/workspace/zoo` or simply `workspace/zoo`.
* **Local Inference** (`inference_host_address="@local"`)
* When `zoo_url` starts with `http://` or `https://` or contains exactly one slash (e.g. `workspace/zoo`), it is treated as an AI Hub zoo.
* If the `zoo_url` is instead formatted differently, it will be handled like a local path. Prefix the path with `file://` to be explicit that it is a local path. If the path does not exist, `degirum.connect()` raises `"incorrect local model zoo URL: path does not exist"`. The path can point to either a directory or a model .json file.
* **AI Server Inference** (hostname or `host:port`)
* An empty `zoo_url` or a value starting with `aiserver://` selects the AI server's local zoo.
* Any other value must be a valid AI Hub zoo URL; otherwise, the error `"incorrect cloud model zoo URL"` is raised.
{% hint style="info" %}
The `zoo_url` parameter is handled differentely for PySDK versions including and prior to 0.16.2. If you need help configuring the `zoo_url` parameter, contact the DeGirum team.
{% endhint %}
{% code overflow="wrap" %}
```python
# Function Signature: degirum.connect()
degirum.connect(inference_host_address, zoo_url=None, token=None)
```
{% endcode %}
The following table lists all possible combinations of inference and zoo types with `degirum.connect():`
## Retrieve Supported Devices, Filter Models, then Load Models
### ZooManager.supported\_device\_types()
The `ZooManager.supported_device_types()` method returns a list of runtime and device combinations (in `"RUNTIME/DEVICE"` format) that the connected inference engine supports.
**Example:**
{% code overflow="wrap" %}
```python
import degirum as dg
# Set your inference host address, model zoo, and token in these variables.
your_host_address = "@cloud" # Can be "@cloud", host:port, or "@local"
your_model_zoo = "degirum/public"
your_token = ""
# Connect to DeGirum Application Server and an AI Hub model zoo
inference_manager = dg.connect(
inference_host_address = your_host_address,
zoo_url = your_model_zoo,
token = your_token
)
supported_types = inference_manager.supported_device_types()
print(supported_types)
```
{% endcode %}
Example output:
{% code overflow="wrap" %}
```
['N2X/ORCA1', 'TFLITE/EDGETPU', 'OPENVINO/CPU']
```
{% endcode %}
In this example, the inference engine returns a list of supported device types on the application server. In thise case, it's the application server hosted on `@cloud`.
### **ZooManager.list\_models()**
After obtaining a `ZooManager` object, you can use the `ZooManager.list_models()` method to retrieve and filter the list of available AI models.
{% code overflow="wrap" %}
```python
# Method Signature: ZooManager.list_models()
degirum.zoo_manager.ZooManager.list_models(*args, **kwargs)
```
{% endcode %}
This method:
* Filters models based on various criteria such as model family, runtime, device type, precision, postprocessor type, and more.
* Returns a list of model names that can be used later when loading models for inference.
#### **Use Cases**
* Exploring available model families (e.g., mobilenet, YOLO).
* Filtering models based on target hardware.
* Selecting models for specific precision or density.
#### Example Usage
{% code overflow="wrap" %}
```python
import degirum as dg
# Set your inference host address, model zoo, and token in these variables.
your_host_address = "@cloud" # Can be "@cloud", host:port, or "@local"
your_model_zoo = "degirum/public"
your_token = ""
# Connect to DeGirum Application Server and an AI Hub model zoo
inference_manager = dg.connect(
inference_host_address = your_host_address,
zoo_url = your_model_zoo,
token = your_token
)
model_list = inference_manager.list_models(device_type=["OPENVINO/CPU"])
print(model_list)
```
{% endcode %}
Example output:
{% code overflow="wrap" %}
```
['clip_RN50--224x224_quant_openvino_cpu_3', 'clip_rn50_image_encoder--224x224_float_openvino_cpu_1', 'clip_rn50_text_encoder--1x77_float_openvino_cpu_1']
```
{% endcode %}
In this example, the inference engine returns a list of models that run using the OpenVINO runtime on a CPU.
#### **Available Filtering Parameters and Their Sources**
The `ZooManager.list_models()` method filters models based on information retrieved from the model name string and also the model JSON fields.
For models named based on our recommended [model naming conventions](/pysdk/user-guide-pysdk/organizing-models#json-model-file-naming-conventions), the model name string store the `model_family`, `precision`, and `pruned` parameters.
The [model JSON file](/pysdk/user-guide-pysdk/model-json-structure) specifies the `RuntimeAgent`, `DeviceType`, and `SupportedDeviceTypes` fields.
#### **Combine list\_models() with supported\_device\_types() to Find Supported Models**
To find only the models that are compatible with the current inference engine, you can use the `supported_device_types()`method as a filter for `list_models()`.
**Example:**
{% code overflow="wrap" %}
```python
import degirum as dg
# Set your inference host address, model zoo, and token in these variables.
your_host_address = "@cloud" # Can be "@cloud", host:port, or "@local"
your_model_zoo = "degirum/public"
your_token = ""
# Connect to DeGirum Application Server and an AI Hub model zoo
inference_manager = dg.connect(
inference_host_address = your_host_address,
zoo_url = your_model_zoo,
token = your_token
)
# Retrieve supported device types
supported_types = inference_manager.supported_device_types()
# List models that match any supported runtime/device combination
supported_models = inference_manager.list_models(device_type=list(supported_types))
print("Models supported by the current inference engine:")
for model in supported_models:
print(model)
```
{% endcode %}
Example output:
{% code overflow="wrap" %}
```
Models supported by the current inference engine:
clip--224x224_float_tensorrt_gpu_1
clip_RN50--224x224_float_n2x_orca1_2
clip_RN50--224x224_float_tensorrt_gpu_1
```
{% endcode %}
### ZooManager.load\_model()
Once you have obtained supported models from filtering `list_models() with supported_device_types()` methods, you can load all of the resulting models for inference using the `degirum.zoo_manager.ZooManager.load_model()` method.
#### **Basic Usage**
To load a model, pass the model name string as the `model_name` argument to `load_model()`.
**Example:**
{% code overflow="wrap" %}
```python
import degirum as dg
# Set your inference host address, model zoo, and token in these variables.
your_host_address = "@cloud" # Can be "@cloud", host:port, or "@local"
your_model_zoo = "degirum/public"
your_token = ""
# Connect to DeGirum Application Server and an AI Hub model zoo
inference_manager = dg.connect(
inference_host_address = your_host_address,
zoo_url = your_model_zoo,
token = your_token
)
# Retrieve supported device types
supported_types = inference_manager.supported_device_types()
# List models that match any supported runtime/device combination
supported_models = inference_manager.list_models(device_type=list(supported_types))
# Load the first model from the list
model = inference_manager.load_model(model_name=supported_models[0])
# Print the model object
print(model)
```
{% endcode %}
Example output:
{% code overflow="wrap" %}
```
```
{% endcode %}
If a model with the specified name is found, the method returns a `degirum.model.Model` object that you can use to run inference.
If the model is not found, an exception will be raised.
#### **Passing Model Properties as Arguments**
You can pass additional model properties as keyword arguments to customize the behavior of the loaded model. These properties are directly assigned to the model object.
**Example:**
{% code overflow="wrap" %}
```python
import degirum as dg
# Set your inference host address, model zoo, and token in these variables.
your_host_address = "@cloud" # Can be "@cloud", host:port, or "@local"
your_model_zoo = "degirum/public"
your_token = ""
# Connect to DeGirum Application Server and an AI Hub model zoo
inference_manager = dg.connect(
inference_host_address = your_host_address,
zoo_url = your_model_zoo,
token = your_token
)
# Retrieve supported device types
supported_types = inference_manager.supported_device_types()
# List models that match any supported runtime/device combination
supported_models = inference_manager.list_models(device_type=list(supported_types))
# Load the first model from the list
model = inference_manager.load_model(
model_name=supported_models[0],
output_confidence_threshold=0.5,
input_pad_method="letterbox"
)
print(model)
```
{% endcode %}
Example output:
{% code overflow="wrap" %}
```
```
{% endcode %}
In this example:
* `output_confidence_threshold=0.5` sets a confidence threshold for inference results.
* `input_pad_method="letterbox"` specifies the padding method to maintain the input aspect ratio.
## Convenience Functions
### **degirum.get\_supported\_devices()**
You can retrieve supported device types using the `degirum.get_supported_devices()` function. This function combines the arguments of both `degirum.connect()` and `degirum.zoo_manager.ZooManager.get_supported_devices()`, allowing you to list supported devices with a single call.
#### Function Signature:
{% code overflow="wrap" %}
```python
degirum.get_supported_devices(inference_host_address, zoo_url='', token='')
```
{% endcode %}
#### Example:
{% code overflow="wrap" %}
```python
import degirum as dg
# Set your inference host address, model zoo, and token in these variables.
your_host_address = "@cloud" # Can be "@cloud", host:port, or "@local"
your_model_zoo = "degirum/public"
your_token = ""
# Retrieve supported device types
supported_devices = dg.get_supported_devices(
inference_host_address = your_host_address,
zoo_url = your_model_zoo,
token = your_token
)
print(supported_devices)
```
{% endcode %}
Example output:
{% code overflow="wrap" %}
```
dict_keys(['AKIDA/NSOC_V2', 'AXELERA/METIS', 'DEEPX/M1A', 'DUMMY/DUMMY', 'HAILORT/HAILO8', 'HAILORT/HAILO8L', 'MEMRYX/MX3', 'N2X/CPU', 'N2X/ORCA1', 'ONNX/CPU', 'OPENVINO/CPU', 'OPENVINO/GPU', 'OPENVINO/NPU', 'RKNN/RK3566', 'RKNN/RK3568', 'RKNN/RK3588', 'TENSORRT/DLA', 'TENSORRT/GPU', 'TFLITE/CPU', 'TFLITE/EDGETPU'])
```
{% endcode %}
### **degirum.list\_models()**
You can retrieve the list of models using the `degirum.list_models()` function. This function combines the arguments of both `degirum.connect()` and `degirum.zoo_manager.ZooManager.list_models()`, allowing you list models with a single call.
#### Function Signature:
{% code overflow="wrap" %}
```python
degirum.list_models(inference_host_address, zoo_url, token=None, **kwargs)
```
{% endcode %}
**Example:**
{% code overflow="wrap" %}
```python
import degirum as dg
# Set your inference host address, model zoo, and token in these variables.
your_host_address = "@cloud" # Can be "@cloud", host:port, or "@local"
your_model_zoo = "degirum/public"
your_token = ""
# List models from the AI Hub model zoo with specific filtering criteria
model_list = dg.list_models(
inference_host_address = your_host_address,
zoo_url = your_model_zoo,
token = your_token,
device_type=["OPENVINO/CPU"]
)
print(model_list)
```
{% endcode %}
Example output:
{% code overflow="wrap" %}
```
{'clip_RN50--224x224_quant_openvino_cpu_3': , 'clip_rn50_image_encoder--224x224_float_openvino_cpu_1': }
```
{% endcode %}
In this example, the method connects to the specified model zoo and returns a list of models that run with the OpenVINO runtime on a CPU.
### **degirum.load\_model()**
For convenience, you can directly load a model without explicitly obtaining a `ZooManager` object with `degirum.connect()`. The `degirum.load_model()` function combines the arguments of `degirum.connect()` and `ZooManager.load_model()`, allowing you to load models with a single call.
**Function Signature:**
{% code overflow="wrap" %}
```python
degirum.load_model(model_name, inference_host_address, zoo_url=None, token=None, **kwargs)
```
{% endcode %}
**Example:**
{% code overflow="wrap" %}
```python
import degirum as dg
# Set your inference host address, model zoo, and token in these variables.
your_host_address = "@cloud" # Can be "@cloud", host:port, or "@local"
your_model_zoo = "degirum/public"
your_token = ""
# Retrieve supported device types
supported_devices = dg.get_supported_devices(
inference_host_address = your_host_address,
zoo_url = your_model_zoo,
token = your_token,
)
# List models from the AI Hub model zoo with specific filtering criteria
model_list = dg.list_models(
inference_host_address = your_host_address,
zoo_url = your_model_zoo,
token = your_token,
device_type=list(supported_devices)
)
# Load the first model from the list with some optional parameters
model = dg.load_model(
model_name = list(model_list)[0],
inference_host_address = your_host_address,
zoo_url = your_model_zoo,
token = your_token,
overlay_show_probabilities = True,
output_confidence_threshold = 0.5
)
print(model)
```
{% endcode %}
Example output:
{% code overflow="wrap" %}
```
```
{% endcode %}
In this example, the code gets supported devices, lists models, then loads a model based on the filtered list with a specified parameter.
#### Minimum Code Example
Once you understand the steps above, you can streamline the code into a few lines.
{% code overflow="wrap" %}
```python
import degirum as dg
# Loading a model
model = dg.load_model(
model_name = "yolov8n_relu6_coco--640x640_quant_rknn_rk3588_1",
inference_host_address = "@cloud",
zoo_url = "degirum/public",
token = your_token
# optional parameters, such as overlay_show_probabilities = True
)
print(model)
```
{% endcode %}
# Running AI Model Inference
This is a walkthrough for running predictions. You'll learn about input data types, understanding the results, and finally processing inputs in batches for efficiency.
Once you have loaded an AI model and obtained a model handle, you can start running inferences. The [degirum.model.Model](https://docs.degirum.com/pysdk/user-guide-pysdk/pages/GfBG9YuP1z73SsOm25XD#degirum.model.model) class provides two methods for performing AI inference:
* [degirum.model.Model.predict()](https://docs.degirum.com/pysdk/user-guide-pysdk/pages/GfBG9YuP1z73SsOm25XD#degirum.model.model.predict): Runs prediction on a single data frame.
* [degirum.model.Model.predict\_batch()](https://docs.degirum.com/pysdk/user-guide-pysdk/pages/GfBG9YuP1z73SsOm25XD#degirum.model.model.predict_batch): Runs prediction on a batch of frames.
### Model.predict()
The `predict()` method takes a single input data frame and returns an inference result object. You can also call this method by calling `degirum.model.Model.__call__`. This is an alias for `Model()`. See [Single Frame Inference](#single-frame-inference) for more information.
{% hint style="info" %}
Try not to run `predict()` on videos, webcam feeds, and streams. Instead, use [predict\_batch()](#model.predict_batch).
{% endhint %}
{% code overflow="wrap" %}
```python
# Method Signature: Model.predict()
degirum.model.Model.predict(data)
```
{% endcode %}
**Example:**
{% code overflow="wrap" %}
```python
import degirum as dg
# Declaring variables
# Set your model, inference host address, model zoo, and token in these variables.
your_model_name = "model-name"
your_host_address = "@cloud" # Can be "@cloud", host:port, or "@local"
your_model_zoo = "degirum/public"
your_token = ""
# Specify the image you will run inference on
your_image = "path/image.jpg"
# Loading a model
model = dg.load_model(
model_name = your_model_name,
inference_host_address = your_host_address,
zoo_url = your_model_zoo,
token = your_token
# optional parameters, such as overlay_show_probabilities = True
)
# Run a prediction and assign it to result
result = model(your_image)
# Print the prediction result
print(result)
```
{% endcode %}
Example output:
{% code overflow="wrap" %}
```
- bbox: [240.37627136118888, 101.09044216232718, 898.4129315085123, 698.4668477562271]
category_id: 15
label: cat
score: 0.86873459815979
```
{% endcode %}
### Model.predict\_batch()
The `predict_batch()` method accepts an iterator of data frames, such as a list or a stream, and returns a generator. It processes the iterator in a pipeline to maximize throughput, making it more efficient than calling `predict()` repeatedly in a loop. This approach is ideal for processing a list of images or a video stream. See the [Batch Inference](#batch-inference) section for more information.
{% code overflow="wrap" %}
```python
# Method Signature: Model.predict_batch()
degirum.model.Model.predict_batch(data)
```
{% endcode %}
## Supported Input Data Types
PySDK models can handle images and raw tensors as data types.
The input you pass to `predict()` depends on the number of inputs the model has. If the model has one input, then you pass only one object to `predict()`.
{% hint style="warning" %}
The model may have multiple inputs. In this case, the data you pass to `predict()` is a list of objects: one object per corresponding input.
{% endhint %}
#### Check Input Data Type of Your Model
You can check what input type your model expects by inspecting the `model.model_info.InputType` property.
{% code overflow="wrap" %}
```python
import degirum as dg
# Declaring variables
# Set your model, inference host address, model zoo, and token in these variables.
your_model_name = "model-name"
your_host_address = "@cloud" # Can be "@cloud", host:port, or "@local"
your_model_zoo = "degirum/public"
your_token = ""
# Loading a model
model = dg.load_model(
model_name = your_model_name,
inference_host_address = your_host_address,
zoo_url = your_model_zoo,
token = your_token
# optional parameters, such as overlay_show_probabilities = True
)
# Print input data supported by your model.
print(model.model_info.InputType)
```
{% endcode %}
Example output:
{% code overflow="wrap" %}
```
['Image']
```
{% endcode %}
In this example, we check the input data type of our model.
The `InputType` property of the `ModelParams` class returned by the [degirum.model.Model.model\_info](https://docs.degirum.com/pysdk/user-guide-pysdk/pages/GfBG9YuP1z73SsOm25XD#degirum.model.model.model_info) property describes the number and the type of inputs of the model (see [Model Info](#model-info) section for details about model info properties).
The `model_info.InputType` property returns a list of input types (one entry per model input). The length of this list tells you how many separate inputs the model expects. For instance, a model that takes two images will have two entries in this list.
### Images
If your model expects image inputs (`InputType == "Image"`), you can supply the input frame in any of the following formats:
* Path to an image file.
* HTTP URL to an image.
* NumPy array.
* PIL `Image` object.
* Raw `bytes` of image data.
PySDK automatically converts these inputs into the format required by the model’s neural network according to the model’s preprocessor settings in its JSON configuration. For more details, see the [preprocessor parameters](/pysdk/user-guide-pysdk/model-json-structure#preprocessing-parameters).
### Tensors
If your model expects raw tensor inputs (`InputType == "Tensor"`), you should provide a multi-dimensional NumPy array with the appropriate shape and data type.
The array’s dimensions must match the model’s expected input shape, which you can find in the model info (`model.model_info.InputShape`). The data type of the array’s elements should match the model’s expected raw data type (`model.model_info.InputRawDataType`).
### Audio
If your model expects audio inputs (`InputType == "Audio"`), provide a one-dimensional NumPy array containing audio waveform samples. The waveform length and sampling rate must match `model.model_info.InputWaveformSize` and `model.model_info.InputSamplingRate`.
{% hint style="info" %}
Whisper encoder models in PySDK support both `NHWC` and `NCHW` feature layouts. The audio preprocessor inspects `InputTensorLayout` and the non-unit dimensions of `model.model_info.InputShape` so it can reorder mel bins and time frames automatically for the layout your model requires.
{% endhint %}
## Single Frame Inference
When you want to process one frame, use `predict()`.
{% code overflow="wrap" %}
```python
import degirum as dg
import cv2
# Declaring variables
# Set your model, inference host address, model zoo, and token in these variables.
your_model_name = "model-name"
your_host_address = "@cloud" # Can be "@cloud", host:port, or "@local"
your_model_zoo = "degirum/public"
your_token = ""
# Specify the image you will run inference on
your_image = "path/image.jpg"
# Loading a model
model = dg.load_model(
model_name = your_model_name,
inference_host_address = your_host_address,
zoo_url = your_model_zoo,
token = your_token
# optional parameters, such as overlay_show_probabilities = True
)
# Run a prediction and assign it to result
result = model(your_image)
# Print the prediction result
print(result)
```
{% endcode %}
Example output:
{% code overflow="wrap" %}
```
- bbox: [242.46119793154423, 110.32875074982861, 898.4762465007998, 698.5679996357975]
category_id: 15
label: cat
score: 0.86873459815979
```
{% endcode %}
## Batch Inference
When you have multiple frames to process, use `predict_batch()`. The `predict_batch()` method runs predictions on an iterable list of frames. The predictions run in a pipeline to maximize throughput, making it more efficient than calling `predict()` in a loop.
The `predict_batch()` method accepts a single parameter: an iterator object, for example, a list. Populate this iterator with the same types of data you pass to `predict()`, such as image paths, image URLs, NumPy arrays, or PIL `Image` objects.
`predict_batch()` returns a generator of results. You can loop over these results just as you would iterate through successive `predict()` calls.
In addition to raw frames, the iterator can yield two-element tuples. The first element must be the frame, and the second element can contain any metadata for that frame. This metadata is passed through the inference pipeline and becomes available via the `info` property of the corresponding result. This makes it easy to attach per-frame context such as timestamps or frame numbers.
Because `predict_batch()` returns a generator, simply calling the method does not immediately run inference. Frames are processed only when you iterate over the returned generator (for example, in a `for` loop).
#### Example: Iterating over predict\_batch results
{% code overflow="wrap" %}
```python
for result in model.predict_batch(['image1.jpg','image2.jpg']):
print(result)
```
{% endcode %}
#### **Example: Attaching frame metadata**
{% code overflow="wrap" %}
```python
def frame_source():
for idx, path in enumerate(['image1.jpg', 'image2.jpg']):
yield path, {'frame_index': idx}
for result in model.predict_batch(frame_source()):
print(result.info)
```
{% endcode %}
#### Example: Using `predict_batch()` on a video file
This example uses `predict_batch()` to process a video file. The `frame_source` generator yields frames from the video, and the model produces predictions for each frame. The results are overlaid on the frame (`result.image_overlay`) and displayed with OpenCV.
{% code overflow="wrap" %}
```python
import degirum as dg
import cv2
# Declaring variables
# Set your model, inference host address, model zoo, and token in these variables.
your_model_name = "model-name"
your_host_address = "@cloud" # Can be "@cloud", host:port, or "@local"
your_model_zoo = "degirum/public"
your_token = ""
# Specify the video you will run inference on
your_video = "path/video.mp4"
# Loading a model
model = dg.load_model(
model_name = your_model_name,
inference_host_address = your_host_address,
zoo_url = your_model_zoo,
token = your_token
# optional parameters, such as overlay_show_probabilities = True
)
# Open the video file
stream = cv2.VideoCapture(your_video)
# Define generator function to produce video frames
def frame_source(stream):
while True:
ret, frame = stream.read()
if not ret:
break # end of file
yield frame
# Run predict_batch() on frames from the video file
for result in model.predict_batch(frame_source(stream)):
# Print raw results for each frame
print(result)
# Release stream
stream.release()
```
{% endcode %}
Example output:
{% code overflow="wrap" %}
```
- bbox: [293.7637429237366, 230.0834002494812, 474.48908519744873, 621.3431906700134]
category_id: 0
label: person
score: 0.7896590232849121
```
{% endcode %}
In the example above, the results are continuously printed to the terminal until the video is complete.
## Inference Results
When you call `predict()`, you receive an inference result object derived from the [degirum.postprocessor.InferenceResults](https://docs.degirum.com/pysdk/user-guide-pysdk/pages/lMeFJKx8um7RZqppYeCI#degirum.postprocessor.inferenceresults) class. Likewise, `predict_batch()` returns a generator that yields inference result objects. These result classes, known as postprocessors, vary by AI model type—classification, object detection, pose detection, segmentation, and so on. From your perspective, they provide the same functionality.
`InferenceResults` objects contain the following data:
* [degirum.postprocessor.InferenceResults.image](https://docs.degirum.com/pysdk/user-guide-pysdk/pages/lMeFJKx8um7RZqppYeCI#degirum.postprocessor.inferenceresults.image): Original input image as a NumPy array or PIL image.
* [degirum.postprocessor.InferenceResults.image\_overlay](https://docs.degirum.com/pysdk/user-guide-pysdk/pages/lMeFJKx8um7RZqppYeCI#degirum.postprocessor.inferenceresults.image_overlay): Original image with inference results drawn on top. The overlay is model-dependent:
* Classification models: class labels with probabilities are *printed below* the original image.
* Object detection models: bounding boxes are *printed on* the original image.
* Hand and pose detection models: keypoints and keypoint connections are *printed on* the original image.
* Segmentation models: segments are *printed on* the original image.
* [degirum.postprocessor.InferenceResults.results](https://docs.degirum.com/pysdk/user-guide-pysdk/pages/lMeFJKx8um7RZqppYeCI#degirum.postprocessor.inferenceresults.results): Keeps a list of inference results in dictionary form. Follow the property link for detailed explanation of all result formats.
* [degirum.postprocessor.InferenceResults.image\_model](https://docs.degirum.com/pysdk/user-guide-pysdk/pages/lMeFJKx8um7RZqppYeCI#degirum.postprocessor.inferenceresults.image_model): Preprocessed image tensor that was fed into the model (in binary form). Populated only if you enable [Model.save\_model\_image](https://docs.degirum.com/pysdk/user-guide-pysdk/pages/GfBG9YuP1z73SsOm25XD#degirum.model.model.save_model_image) before performing predictions.
The results property is what you will typically use in your code. This property contains the core prediction data. Note that if the model outputs coordinates (e.g., bounding boxes), these have been converted back to the coordinates of the original image for your convenience.
#### Example: Combine predict\_batch() with image\_overlay to show prediction results on original video
This example will open your video, run inference on it, ad display the video with inference results annotated over it.
{% code overflow="wrap" %}
```python
import degirum as dg
import cv2
# Declaring variables
# Set your model, inference host address, model zoo, and token in these variables.
your_model_name = "model-name"
your_host_address = "@cloud" # Can be "@cloud", host:port, or "@local"
your_model_zoo = "degirum/public"
your_token = ""
# Specify the video you will run inference on
your_video = "path/video.mp4"
# Loading a model
model = dg.load_model(
model_name = your_model_name,
inference_host_address = your_host_address,
zoo_url = your_model_zoo,
token = your_token
# optional parameters, such as overlay_show_probabilities = True
)
# Open the video file
stream = cv2.VideoCapture(your_video)
# Define generator function to produce video frames
def frame_source(stream):
while True:
ret, frame = stream.read()
if not ret:
break # end of file
yield frame
# Process the video frames in a batch and display the overlay
for result in model.predict_batch(frame_source(stream)):
# Retrieve the overlay; if it's callable, call it
overlay = result.image_overlay() if callable(result.image_overlay) else result.image_overlay
# Display the overlay image in a window
cv2.imshow("Inference Overlay", overlay)
# Wait 1ms; press 'q' to quit early
if cv2.waitKey(1) & 0xFF == ord('q'):
break
# Release the video stream and close all OpenCV windows
stream.release()
cv2.destroyAllWindows()
```
{% endcode %}
# Model JSON Structure
This page outlines model JSON structure and parameters.
JSON Configuration File Parameters
## JSON Overview
All models in model zoos are paired with JSON configuration files that describe the model type, its intended function, the runtime environment it is compiled for, and its preprocessing and postprocessing settings.
These parameters fall into five sections:
* [General](#general-parameters): Basic information to identify the model.
* [DEVICE](#target-device-parameters): Environment the model expects.
* [MODEL\_PARAMETERS](#model-parameters): Settings for how the model operates.
* [PRE\_PROCESS](#preprocessing-parameters): Preprocessing settings for model inputs.
* [POST\_PROCESS](#postprocessing-parameters): Postprocessing settings for model outputs.
{% hint style="warning" %}
Incorrectly setting these parameters may decrease precision and performance.
{% endhint %}
### Example JSON Configuration
Here is a sample JSON configuration. Include or omit parameters as your model requires:
{% code overflow="wrap" %}
```json
{
"ConfigVersion": ,
"Checksum": "",
"DEVICE": [
{
"DeviceType": "",
"RuntimeAgent": "",
"SupportedDeviceTypes": ""
}
],
"PRE_PROCESS": [
{
"InputN": ,
"InputH": ,
"InputW": ,
"InputC": ,
"InputQuantEn":
}
],
"MODEL_PARAMETERS": [
{
"ModelPath": ""
}
],
"POST_PROCESS": [
{
"OutputPostprocessType": "",
"OutputNumClasses": ,
"LabelsPath": ""
}
]
}
```
{% endcode %}
***
## General Parameters
General parameters at the top of the JSON file specify its version and the model binary's checksum.
Parameter
Type
Mandatory
ConfigVersion
int
yes
Checksum
string
yes
* **ConfigVersion**\
The version of the JSON configuration file. The current JSON config version is 11. It is verified against the minimum compatible and current framework software versions. If the version is not within the acceptable range, a version-check runtime exception is generated during model loading. At this moment, PySDK is backwards-compatible with ConfigVersions introduced in prior PySDK releases.
* Version 11 was introduced in PySDK 0.17.0.
* Version 10 was introduced in PySDK 0.14.3.
* Version 9 was introduced in PySDK 0.12.0.
* **Checksum**\
The checksum of the model binary file.
***
## Device Parameters
**Section name in JSON:** `DEVICE`
This section includes three parameters: **DeviceType**, **RuntimeAgent**, and **SupportedDeviceTypes**. Refer to the [Supported Hardware](/pysdk/installation#supported-hardware) section for details on device compatibility.
Parameter
Type
Mandatory
DeviceType
string
yes
RuntimeAgent
string
yes
SupportedDeviceTypes
string
yes
* **DeviceType**\
The type of device on which the model will run.
* **RuntimeAgent**\
The runtime agent responsible for executing the model.
* **SupportedDeviceTypes**\
Lists the device types that are supported by the model. Refer to the Supported Hardware documentation for details on device compatibility.
* **FrameQueueDepth**\
Maximum size of client frame queue. Needs to be set to 20 for Axelera models with double\_buffer=true.
***
## Model Parameters
**Section name in JSON:** `MODEL_PARAMETERS`
These parameters control how the model operates.
Parameter
Type
Mandatory
Models
ModelPath
string
yes
All
* **ModelPath**\
The path to a model file.
***
## Preprocessing Parameters
**Section name in JSON:** `PRE_PROCESS`
These parameters control settings used to prepare and transform input data before it is fed into the model, ensuring proper formatting and normalization. This section may contain multiple elements (one per input tensor in multi-input networks).
### Input Configuration
Fundamental properties of the input data, including its type, dimensions, and layout.
Parameter
Type
Mandatory
Default
Input Type
InputN
int
yes
(none)
All
InputH
int
yes
(none)
All
InputW
int
yes
(none)
All
InputC
int
yes
(none)
All
InputType
string
No
"Image"
All
InputShape
int array
No
(none)
All
InputRawDataType
string
No
"DG_UINT8"
All
InputTensorLayout
string
No
"NHWC"
Image
* **InputN**\
The batch size for the input data tensor.
* **InputH**\
The height of the input data tensor.
* **InputW**\
The width of the input data tensor.
* **InputC**\
The number of channels in the input data tensor.
* **InputType**\
The model input type. The dimension order is defined by **InputTensorLayout**. This can be set to:
* `Image`
* `Tensor`
* **InputShape**\
The shape of the input data tensor in the format`[, , , ]`. You may specify the shape with this parameter or with the InputN, InputH, InputW, and InputC parameters.
* **InputRawDataType**\
The data type of raw binary tensor elements (how the preprocessor treats client data). This is a runtime parameter that can be changed on the fly. This can be set to:
* `DG_UINT8` (unsigned 8-bit integer)
* `DG_FLT` (32-bit floating point),
* `DG_INT16` (signed 16-bit integer).
* **InputTensorLayout**\
The dimensional layout of the raw binary tensor for inputs of raw image type and raw tensor type. This can be set to:
* `auto`
* `NHWC`
* `NCHW`
***
### Image Format & Manipulation
Governs the image input format, color space, resizing, padding, cropping, and slicing operations. These parameters are needed only when **InputType** is `Image`.
Parameter
Type
Mandatory
Default
ImageBackend
string
No
"auto"
InputResizeMethod
string
No
"bilinear"
InputPadMethod
string
No
"letterbox"
InputCropPercentage
double
No
1.0
InputImgFmt
string
No
"JPEG"
InputColorSpace
string
No
"RGB"
* **ImageBackend**\
The Python package used for image processing. When this is set to `auto`, the OpenCV backend will be tried first. This can be set to:
* `auto`
* `pil`
* `opencv`
* **InputResizeMethod**\
The interpolation algorithm used for image resizing. This can be set to:
* `nearest`
* `bilinear`
* `area`
* `bicubic`
* `lanczos`
* **InputPadMethod**\
Specifies how the input image is padded or cropped during resizing. This can be set to:
* `stretch`
* `letterbox`
* `crop-first`
* `crop-last`
* **InputCropPercentage**\
The crop percentage when **InputPadMethod** is set to `crop-first` or `crop-last`.
* **InputImgFmt**\
The image format for image inputs. Data type is defined by **InputRawDataType**. This can be set to:
* `JPEG`
* `RAW`
* **InputColorSpace**\
The color space required for image inputs. This can be set to `RGB` or `BGR`. If **InputImgFmt** is `JPEG`, the preprocessor automatically handles color conversion; if `RAW`, the raw tensor must be arranged accordingly.
***
### Normalization
Defines how input data is normalized, including scale factors and per-channel adjustments, to ensure consistency across inputs.
Parameter
Type
Mandatory
Default
Models
InputScaleEn
bool
No
false
Image
InputScaleCoeff
double
No
1./255.
Image
InputNormMean
float array
No
[]
Image
InputNormStd
float array
No
[]
Image
* **InputScaleEn**\
Specifies whether global data normalization is applied.
* **InputScaleCoeff**\
The scale factor used for global data normalization when **InputScaleEn** is `true`.
* **InputNormMean**\
The mean values for per-channel normalization of image inputs (e.g., `[0.485, 0.456, 0.406]`).
* **InputNormStd**\
The standard deviation values for per-channel normalization of image inputs (e.g., `[0.229, 0.224, 0.225]`).
***
### Quantization
Settings for converting input data into quantized formats to optimize processing efficiency and model performance.
Parameter
Type
Mandatory
Default
Models
InputQuantEn
bool
No
false
All
InputQuantOffset
float
No
0
All
InputQuantScale
float
No
1
All
* **InputQuantEn**\
Enables input quantization for image and raw tensor types, determining whether the model input is treated as `uint8` or `float32`.
* **InputQuantOffset**\
The quantization zero offset for image and raw tensor inputs.
* **InputQuantScale**\
The quantization scale. When quantization is enabled, data is scaled before quantization using the provided formula.
***
## Postprocessing Parameters
**Section name in JSON:** `POST_PROCESS`
These parameters transform model outputs into final, interpretable results.
### General Behavior
General settings for the output postprocessing algorithm and how output tensors are managed.
Parameter
Type
Mandatory
Default
PythonFile
string
No
(none)
LabelsPath
string
No
""
OutputPostprocessType
string
No
None
* **OutputPostprocessType**\
The type of output postprocessing algorithm. This can be set to:
* `Classification`
* `Detection`
* `DetectionDamoYolo`
* `DetectionYolo`
* `DetectionYoloPlates`
* `DetectionYoloV8`
* `DetectionYoloV8OBB`
* `DetectionYoloHailo`
* `FaceDetection`
* `HandDetection`
* `PoseDetection`
* `PoseDetectionYoloV8`
* `Segmentation`
* `SegmentationYoloV8`
* `Dequantization`
* `Null`
* `None`
* **PythonFile**\
The name of a Python file that contains server-side postprocessing code.
* **LabelsPath**\
The path to a label dictionary file.
* **OutputPostprocessType**\
The type of output post-processing algorithm.
***
### Thresholds & Alignment
Thresholds and alignment adjustments used during postprocessing to filter and refine model outputs.
Parameter
Type
Mandatory
Default
OutputConfThreshold
double
No
0.3
OutputNMSThreshold
double
No
0.6
OutputClassIDAdjustment
int
No
0
* **OutputConfThreshold**\
The confidence threshold below which results are filtered out.
* **OutputNMSThreshold**\
The threshold for the Non-Max Suppression (NMS) algorithm.
* **OutputClassIDAdjustment**\
The adjustment for the index of the first non-background class.
***
### Classification-Specific
Parameters tailored for classification tasks, such as enabling softmax and selecting the number of top classes.
Parameter
Type
Mandatory
Default
OutputSoftmaxEn
bool
No
false
OutputTopK
size_t
No
0
* **OutputSoftmaxEn**\
Specifies whether softmax is enabled during post-processing.
* **OutputTopK**\
The number of classes to include in the classification result. If set to zero, all classes above **OutputConfThreshold** are reported.
***
### Detection-Specific
Parameters tailored for object detection, including detection limits, scaling coefficients, and non-max suppression parameters.
Parameter
Type
Mandatory
Default
XScale
double
conditional
1
YScale
double
conditional
1
HScale
double
conditional
1
WScale
double
conditional
1
OutputNumClasses
int
No
20
MaxDetectionsPerClass
int
No
100
MaxClassesPerDetection
int
No
1
UseRegularNMS
bool
No
true
MaxDetections
int
No
100
PoseThreshold
double
No
0.8
NMSRadius
double
No
10
Stride
int
No
16
* **XScale**\
The X scale coefficient used to convert box center coordinates to an anchor-based coordinate system.
* **YScale**\
The Y scale coefficient used to convert box center coordinates to an anchor-based coordinate system.
* **HScale**\
The height scale coefficient used to convert box size coordinates to an anchor-based coordinate system.
* **WScale**\
The width scale coefficient used to convert box size coordinates to an anchor-based coordinate system.
* **OutputNumClasses**\
The number of output classes for detection models.
* **MaxDetectionsPerClass**\
The maximum number of object detection results to report per class.
* **MaxClassesPerDetection**\
The maximum number of classes to report for each detection.
* **UseRegularNMS**\
Specifies whether to use a regular (non-batched) NMS algorithm for object detection.
* **MaxDetections**\
The maximum number of object detection results to report.
* **PoseThreshold**\
The pose score threshold below which low-confidence poses are filtered out.
* **NMSRadius**\
The NMS radius for pose detection—a keypoint candidate is rejected if it lies within this pixel range of a previously detected instance.
* **Stride**\
The stride scale coefficient used for pose detection.
# Command Line Interface
Learn how to use the PySDK command line interface to manage AI models, control your AI server, and streamline model downloads.
During PySDK installation, the `degirum` executable [console script](https://setuptools.pypa.io/en/latest/userguide/entry_point.html#console-scripts) is added to the system path. This script provides a command-line interface (CLI) for PySDK management tasks and extends functionality through [entry points](https://setuptools.pypa.io/en/latest/userguide/entry_point.html#entry-points).
The PySDK CLI supports the following commands:
| Command | Description |
| ----------------------------------- | --------------------------------------------- |
| [download-zoo](#download-model-zoo) | Download AI models from the cloud model zoo |
| [install-runtime](#install-runtime) | Install runtime libraries for AI accelerators |
| [server](#server-control) | Control operation of the AI server |
| [sys-info](#system-info) | Get system information dump |
| [token](#manage-ai-hub-tokens) | Manage AI Hub tokens |
| [trace](#manage-tracing) | Manage AI server tracing |
| version | Print PySDK version |
Invoke the console script with one of the commands above, followed by its parameters.
{% code overflow="wrap" %}
```
degirum
```
{% endcode %}
## Download Model Zoo
Command: **download-zoo**
Using this command you can download ML models from the cloud model zoo of your choice specified by URL. The command has the following parameters:
| Parameter | Description | Possible Values | Default |
| ---------------- | ---------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | --------------------------- |
| `--path` | Local filesystem path to store models downloaded from a model zoo repo | Valid local directory path | Current directory |
| `--url` | Cloud model zoo URL | `"https://hub.degirum.com/[]"` | `"https://hub.degirum.com"` |
| `--token` | Cloud API access token | Valid token obtained at hub.degirum.com | Empty |
| `--model_family` | Model family name filter: model name substring or regular expression | Any | Empty |
| `--device` | Target inference device filter | `ORCA, CPU, GPU, EDGETPU, MYRIAD, DLA, DLA_FALLBACK, NPU, RK3588, RK3566, RK3568, NXP_VX, NXP_ETHOSU, ARMNN, VITIS_NPU` | Empty |
| `--runtime` | Runtime agent type filter | `N2X, TFLITE, TENSORRT, OPENVINO, ONNX, RKNN` | Empty |
| `--precision` | Model calculation precision filter | `QUANT, FLOAT` | None |
| `--pruned` | Model density filter | `PRUNED, DENSE` | None |
The URL parameter uses the form `"https://hub.degirum.com/"`, where `` is `/`. To find this suffix, go to the AI Hub, select the desired zoo, and click the copy button next to its name.
Filter parameters work the same way as in [degirum.zoo\_manager.ZooManager.list\_models](https://docs.degirum.com/pysdk/user-guide-pysdk/pages/YhwNF6qcgxroBmccSwg8#degirum.zoo_manager.zoomanager.list_models) and let you download only models that satisfy the filter conditions.
Once models are downloaded into the directory specified by `--path` parameter, you may use this directory as the model zoo to be served by AI server (see [Server Control Command](#server-control-command) section).
**Example:**
Download models for ORCA device type from DeGirum Public cloud model zoo into `./my-zoo` directory.
{% code overflow="wrap" %}
```
degirum download-zoo --path ./my-zoo --token --device ORCA
```
{% endcode %}
Here `` is your cloud API access token, which you can generate on the AI Hub.
## Install Runtime
Command: **install-runtime**
Use this command to install third-party AI accelerator runtime libraries on Debian-based systems. The command has the following parameters:
| Parameter | Description | Possible Values | Default |
| ----------------- | --------------------------------------------------------- | -------------------------------------- | ---------------- |
| `--list` | List available runtimes and their versions | N/A | Disabled |
| `plugin_name` | Runtime name to install | One of the runtimes listed by `--list` | Required |
| `plugin_versions` | Runtime version(s) to install; omit to install the latest | Specific version number(s) or `ALL` | Latest available |
You can provide multiple `plugin_versions` to install several versions at once.
**Examples:**
List available runtimes and versions:
{% code overflow="wrap" %}
```
degirum install-runtime --list
```
{% endcode %}
Install the latest ONNX runtime:
{% code overflow="wrap" %}
```
degirum install-runtime onnx
```
{% endcode %}
Install a specific version of OpenVINO runtime:
{% code overflow="wrap" %}
```
degirum install-runtime openvino 2025.3.0
```
{% endcode %}
## Server Control
**server**
Use this command to start the AI server, shut it down, or ask it to rescan its local model zoo.
> You can control only an AI server running on the same host as this command. Remote control is disabled for security reasons.
This command has the following subcommands, which are passed just after the command:
| Sub-command | Description |
| ------------ | ----------------------------------------- |
| `start` | Start AI server |
| `rescan-zoo` | Request AI server to rescan its model zoo |
| `shutdown` | Request AI server to shutdown |
| `cache-dump` | Dump AI server inference agent cache info |
The command has the following parameters:
| Parameter | Applicable To | Description | Possible Values | Default |
| ------------ | ------------- | -------------------------------------------------------------------------------------- | ---------------------- | ----------------- |
| `--zoo` | `start` | Local model zoo directory to serve models from (applicable only to `start` subcommand) | Any valid path | Current directory |
| `--quiet` | `start` | Do not display any output (applicable only to `start` subcommand) | N/A | Disabled |
| `--port` | `start` | TCP port to bind AI server to | 1...65535 | 8778 |
| `--protocol` | `start` | AI server protocol to use | `asio`, `http`, `both` | `asio` |
Starting with PySDK 0.10.0, the AI server supports two protocols: `asio` and `http`. `asio` is DeGirum's custom socket-based protocol used in earlier versions. The new `http` protocol relies on REST HTTP and WebSockets, so you can use the AI server from any language that supports these standards. Browser-based JavaScript, for example, requires the `http` protocol because it lacks native socket support.
The `asio` protocol is selected by default. Use `--protocol http` to enable the `http` protocol or `--protocol both` to enable both. When both are enabled, the AI server listens on two consecutive ports: the first for `asio` and the second for `http`.
**Examples:**
Start AI server to serve models from `./my-zoo` directory, bind it to default port, and use `asio` protocol:
{% code overflow="wrap" %}
```
degirum server start --zoo ./my-zoo
```
{% endcode %}
Start AI server to serve models from `./my-zoo` directory, use `asio` protocol on port 12345, and use `http` protocol on port 12346:
{% code overflow="wrap" %}
```
degirum server start --zoo ./my-zoo --port 12345 --protocol both
```
{% endcode %}
## System Info
Command: **sys-info**
This command displays system information for the local host or a remote AI server.
The command has the following parameters:
| Parameter | Description | Possible Values | Default |
| --------- | ------------------------------------------------------------------- | ------------------------------------ | ------- |
| `--host` | Remote AI server hostname or IP address; omit to query local system | Valid hostname, IP address, or empty | Empty |
**Example:**
Query system info from remote AI server at IP address `192.168.0.101`:
{% code overflow="wrap" %}
```
degirum sys-info --host 192.168.0.101
```
{% endcode %}
## Manage AI Hub Tokens
Command: **token**
Use this command to manage AI Hub access tokens. When a token is managed by this command, then that token will be automatically provided to any function call that uses a token. This way, you will not need to assign `token` to anything in your PySDK code.
{% hint style="info" %}
The DEGIRUM\_CLOUD\_TOKEN environment variable is not set by this command.
{% endhint %}
> Tokens are stored in a JSON file in your user data directory (`%APPDATA%\\DeGirum` on Windows or `~/.local/share/DeGirum` on Linux and macOS). When a token is installed, PySDK uses it automatically.
This command has the following subcommands, which are passed just after the command:
| Sub-command | Description |
| ----------- | ------------------------------------------------------------------------- |
| `status` | Show information about the installed token |
| `install` | Save a provided token to local storage |
| `clear` | Remove the installed token from storage; does not forcibly expire the key |
The command has the following parameters:
| Parameter | Applicable To | Description | Possible Values | Default |
| ------------- | --------------- | -------------------------------- | --------------------------------------- | ------------------------- |
| `--token` | `install` | Token string to install | Valid token obtained at hub.degirum.com | Empty |
| `--cloud_url` | All subcommands | Cloud server URL to operate with | `https://hub.degirum.com` or custom | `https://hub.degirum.com` |
| `--local` | All subcommands | Do not contact the cloud | N/A | Disabled |
**Examples:**
Install an existing token:
{% code overflow="wrap" %}
```
degirum token install --token
```
{% endcode %}
Example output:
{% code overflow="wrap" %}
```
Token is successfully installed on your system
```
{% endcode %}
Check the status of the installed token:
{% code overflow="wrap" %}
```
degirum token status
```
{% endcode %}
Example output:
{% code overflow="wrap" %}
```
token: dg_***************************************
$schema: https://hub.degirum.com/schemas/GetTokenInfoOutputBody.json
created_at: 'YYYY-MM-DDTHH:MM:SSZ'
description: Demo token
value: dg_***************************************
expiration: '0001-01-01T00:00:00Z'
user:
space:
```
{% endcode %}
Clear the currently installed token:
{% code overflow="wrap" %}
```
degirum token clear
```
{% endcode %}
Example output:
{% code overflow="wrap" %}
```
Token is successfully cleared from your system
```
{% endcode %}
## Manage Tracing
Command: **trace**
Use this command to manage the AI server tracing feature.
> The tracing feature is primarily for debugging and profiling. It is mainly intended for DeGirum customer support and isn't typically used directly by end users.
This command has the following subcommands, which are passed just after the command:
| Sub-command | Description |
| ----------- | --------------------------------------- |
| `list` | List all available trace groups |
| `configure` | Configure trace levels for trace groups |
| `read` | Read trace data to file |
The command has the following parameters:
| Parameter | Applicable To | Description | Possible Values | Default |
| ------------ | --------------- | ----------------------------------------------------------- | ------------------------------------------------------------------- | ----------- |
| `--host` | All subcommands | Remote AI server hostname or IP address | Valid hostname or IP address | `localhost` |
| `--file` | `read` | Filename to save trace data into; omit to print to console | Valid local filename | Empty |
| `--filesize` | `read` | Maximum trace data size to read | Any integer number | `10000000` |
| `--basic` | `configure` | Set `Basic` trace level for a given list of trace groups | One or multiple trace group names as returned by `list` sub-command | Empty |
| `--detailed` | `configure` | Set `Detailed` trace level for a given list of trace groups | One or multiple trace group names as returned by `list` sub-command | Empty |
| `--full` | `configure` | Set `Full` trace level for a given list of trace groups | One or multiple trace group names as returned by `list` sub-command | Empty |
**Examples:**
Query AI server at `192.168.0.101` address for the list of available trace groups and print it to console:
{% code overflow="wrap" %}
```
degirum trace list --host 192.168.0.101
```
{% endcode %}
Configure tracing for the AI server on `localhost` by setting trace levels for specific groups:
{% code overflow="wrap" %}
```
degirum trace configure --basic CoreTaskServer --detailed OrcaDMA OrcaRPC --full CoreRuntime
```
{% endcode %}
Read trace data from the AI server on `localhost` and save it to `./my-trace-1.txt`
{% code overflow="wrap" %}
```
degirum trace read --file ./my-trace-1.txt
```
{% endcode %}
# API Reference Guide
This page serves as an index for the PySDK API Reference Guide, providing access to detailed documentation of the classes, methods, and properties available in PySDK.
# PySDK Package
PySDK API Reference Guide. PySDK entry points: connect(), LOCAL/CLOUD designators.
{% hint style="info" %}
This API Reference is based on PySDK 0.20.0.
{% endhint %}
## \_LOCAL
`degirum.LOCAL = ZooManager._LOCAL`
`module-attribute`
Local inference designator: use it as a first argument of [degirum.connect](#degirum.connect) function to specify inference on local AI hardware
## \_CLOUD
`degirum.CLOUD = ZooManager._CLOUD`
`module-attribute`
Cloud inference designator: use it as a first argument of [degirum.connect](#degirum.connect) function to specify cloud-based inference
## connect(inference\_host\_address, ...)
`degirum.connect(inference_host_address, zoo_url='', token='')`
Connect to the AI inference host and model zoo of your choice.
This is the main PySDK entry point: you start your work with PySDK by calling this function.
The following use cases are supported:
1. You want to perform **cloud inferences** and take models from some **cloud model zoo**.
2. You want to perform inferences on some **AI server** and take models from some **cloud model zoo**.
3. You want to perform inferences on some **AI server** and take models from its **local model zoo**.
4. You want to perform inferences on **local AI hardware** and take models from some **cloud model zoo**.
5. You want to perform inferences on **local AI hardware** and take models from the **local model zoo** directory on your local drive.
6. You want to perform inferences on **local AI hardware** and use **particular model** from your local drive.
Parameters:
| Name | Type | Description | Default |
| ------------------------ | ----- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
| `inference_host_address` | `str` |
Inference engine designator; it defines which inference engine to use.
For AI Server-based inference it can be either the hostname or IP address of the AI Server host, optionally followed by the port number in the form :port.
For DeGirum Cloud Platform-based inference it is the string "@cloud" or degirum.CLOUD constant.
For local inference it is the string "@local" or degirum.LOCAL constant.
| *required* |
| `zoo_url` | `str` |
Model zoo URL string which defines the model zoo to operate with.
For a cloud model zoo, it is specified in the following format: \\[/\]. The \ part is the cloud platform root URL, typically . The optional \ part is the cloud zoo URL suffix in the form \/\. You can confirm zoo URL suffix by visiting your cloud user account and opening the model zoo management page. If \ is not specified, then DeGirum public model zoo degirum/public is used.
For AI Server-based inferences, you may omit both zoo\_url and token parameters. In this case locally-deployed model zoo of the AI Server will be used.
For local AI hardware inferences you specify zoo\_url parameter as either a path to a local model zoo directory, or a path to model's .json configuration file. The token parameter is not needed in this case.
| `''` |
| `token` | `str` |
Cloud API access token used to access the cloud zoo.
To obtain this token you need to open a user account on DeGirum cloud platform. Please login to your account and go to the token generation page to generate an API access token.
| `''` |
Returns:
| Type | Description |
| ----------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
| [`ZooManager`](/pysdk/user-guide-pysdk/api-ref/zoo-manager) | An instance of Model Zoo manager object configured to work with AI inference host and model zoo of your choice. |
Once you created Model Zoo manager object, you may use the following methods:
* [degirum.zoo\_manager.ZooManager.list\_models](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/pages/YhwNF6qcgxroBmccSwg8#degirum.zoo_manager.zoomanager.list_models) to list and search models available in the model zoo.
* [degirum.zoo\_manager.ZooManager.load\_model](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/pages/YhwNF6qcgxroBmccSwg8#degirum.zoo_manager.zoomanager.load_model) to create [degirum.model.Model](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/pages/GfBG9YuP1z73SsOm25XD#degirum.model.model) model handling object to be used for AI inferences.
* [degirum.zoo\_manager.ZooManager.model\_info](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/pages/YhwNF6qcgxroBmccSwg8#degirum.zoo_manager.zoomanager.model_info) to request model parameters.
## load\_model(model\_name, ...)
`degirum.load_model(model_name, inference_host_address, zoo_url='', token='', **kwargs)`
Load a model from the model zoo for the inference.
Parameters:
| Name | Type | Description | Default |
| ------------------------ | ----- | ------------------------------------------------------------------------------------------------------ | ---------- |
| `model_name` | `str` | Model name to load from the model zoo. | *required* |
| `inference_host_address` | `str` | Inference engine designator; it defines which inference engine to use. | *required* |
| `zoo_url` | `str` | Model zoo URL string which defines the model zoo to operate with. | `''` |
| `token` | `str` | Cloud API access token used to access the cloud zoo. | `''` |
| `**kwargs` | `any` | you may pass arbitrary model properties to be assigned to the model object in a form of property=value | `{}` |
Note
For detailed description of `zoo_url`, `inference_host_address`, and `token` parameters refer to [degirum.connect](#degirum.connect) function.
Returns (degirum.model.Model): An instance of [degirum.model.Model](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/pages/GfBG9YuP1z73SsOm25XD#degirum.model.model) model handling object to be used for AI inferences.
## list\_models(inference\_host\_address, ...)
`degirum.list_models(inference_host_address, zoo_url, token='', **kwargs)`
List models in the model zoo matching to specified filter values.
Parameters:
| Name | Type | Description | Default |
| ------------------------ | ----- | ---------------------------------------------------------------------- | ---------- |
| `inference_host_address` | `str` | Inference engine designator; it defines which inference engine to use. | *required* |
| `zoo_url` | `str` | Model zoo URL string which defines the model zoo to operate with. | *required* |
| `token` | `str` | Cloud API access token used to access the cloud zoo. | `''` |
| `**kwargs` | `any` | filter parameters to narrow down the list of models. | `{}` |
Note
For detailed description of `zoo_url`, `inference_host_address`, and `token` parameters refer to [degirum.connect](#degirum.connect) function. For detailed description of `kwargs` parameters refer to [degirum.ZooManager.list\_models](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/pages/YhwNF6qcgxroBmccSwg8#degirum.zoo_manager.zoomanager.list_models) method.
Returns:
| Type | Description |
| ------------------------------------------ | --------------------------------------------------------------- |
| `Union[List[str], Dict[str, ModelParams]]` | A dictionary with model names as keys and model info as values. |
## get\_supported\_devices(inference\_host\_address, ...)
`degirum.get_supported_devices(inference_host_address, zoo_url='', token='')`
Get runtime/device type names, which are available in the inference engine.
Parameters:
| Name | Type | Description | Default |
| ------------------------ | ----- | ---------------------------------------------------------------------- | ---------- |
| `inference_host_address` | `str` | Inference engine designator; it defines which inference engine to use. | *required* |
| `zoo_url` | `str` | not used anymore, kept for backward compatibility. | `''` |
| `token` | `str` | not used anymore, kept for backward compatibility. | `''` |
{% hint style="info" %}
For detailed description of `inference_host_address` parameter refer to [degirum.connect](#degirum.connect) function.
{% endhint %}
Returns:
| Type | Description |
| ----------- | ---------------------------------------------------------------------------------------- |
| `List[str]` | list of runtime/device type names; each element is a string in a format "RUNTIME/DEVICE" |
## enable\_default\_logger(level=logging.DEBUG)
`degirum.enable_default_logger(level=logging.DEBUG)`
Helper function for adding a StreamHandler to the package logger. Removes any existing handlers. Useful for debugging.
Parameters:
| Name | Type | Description | Default |
| ------- | ----- | ------------------------------------------------------------------------------ | ------- |
| `level` | `int` | Logging level as defined in logging python package. defaults to logging.DEBUG. | `DEBUG` |
Returns:
| Type | Description |
| --------------- | ------------------------------------------- |
| `StreamHandler` | Returns an instance of added StreamHandler. |
# Model Module
PySDK API Reference Guide. Core Model class.
{% hint style="info" %}
This API Reference is based on PySDK 0.20.0.
{% endhint %}
## degirum.model.Model
Bases: `ABC`
Model class. Handles whole inference lifecycle for a single model: input data preprocessing, inference, and postprocessing.
{% hint style="info" %}
You never construct model objects yourself -- instead you call [degirum.zoo\_manager.ZooManager.load\_model](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/pages/YhwNF6qcgxroBmccSwg8#degirum.zoo_manager.zoomanager.load_model) method to create [degirum.model.Model](#degirum.model.model) instances for you.
{% endhint %}
## custom\_postprocessor
`degirum.model.Model.custom_postprocessor`
`property` `writable`
Custom postprocessor class. When not None, the object of this class is returned as inference result. Such custom postprocessor classes must be inherited from [InferenceResults](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/pages/lMeFJKx8um7RZqppYeCI#degirum.postprocessor._inferenceresults.inferenceresults) class.
## device\_type
`degirum.model.Model.device_type`
`property` `writable`
The type of the device to be used for model inference in a format `/`
Setter accepts either a string which specifies single device in a format `/` or it can be a list of such strings. In this case the first supported device type from the list will be selected.
Supported device types can be obtained by [degirum.model.Model.supported\_device\_types](#degirum.model.model.supported_device_types) property.
Getter returns currently selected device type.
## devices\_available
`degirum.model.Model.devices_available`
`abstractmethod` `property`
The list of inference device indices which can be used for model inference.
## devices\_selected
`degirum.model.Model.devices_selected`
`property` `writable`
The list of inference device indices selected for model inference.
## eager\_batch\_size
`degirum.model.Model.eager_batch_size`
`property` `writable`
The size of the batch (number of consecutive frames before this model is switched to another model during batch predict) to be used by device scheduler when inferencing this model.
## extra\_device\_params
`degirum.model.Model.extra_device_params`
`property` `writable`
A dictionary of extra parameters to pass to the inference device runtime.
This can be used to control device-specific features not covered by standard model parameters.
For example, on Hailo devices, you can set the batch size by passing:
```python
model.extra_device_params["HAILO_BATCH_SIZE"] = 8
# or
model.extra_device_params.HAILO_BATCH_SIZE = 8
# or
model.extra_device_params = {"HAILO_BATCH_SIZE": 8}
```
## frame\_queue\_depth
`degirum.model.Model.frame_queue_depth`
`property` `writable`
The depth of the model prediction queue. When the queue size reaches this value, the next prediction call will block until there will be space in the queue.
## image\_backend
`degirum.model.Model.image_backend`
`property` `writable`
Graphical library (*backend*) to use for graphical tasks -- one of `'pil'`, `'opencv'`, `'auto'`
`'auto'` means try OpenCV first, if not installed, try PIL.
## inference\_results\_type
`degirum.model.Model.inference_results_type`
`property` `writable`
Inference result type. Specifies the type of inference results to be returned by the model inference. When empty, it is deduced from the model output\_postprocess\_type.
## inference\_timeout\_s
`degirum.model.Model.inference_timeout_s`
`property` `writable`
The maximum time in seconds to wait for inference result from the model.
## input\_crop\_percentage
`degirum.model.Model.input_crop_percentage`
`property` `writable`
Percentage of image to crop around. Valid range: `[0..1]`.
## input\_image\_format
`degirum.model.Model.input_image_format`
`property` `writable`
Defines the image format for model inputs of image type -- one of `'JPEG'` or `'RAW'`.
## input\_letterbox\_fill\_color
`degirum.model.Model.input_letterbox_fill_color`
`property` `writable`
Image fill color in case of `'letterbox'` padding (see [degirum.model.Model.input\_pad\_method](#degirum.model.model.input_pad_method) property for details).
3-element RGB tuple.
## input\_numpy\_colorspace
`degirum.model.Model.input_numpy_colorspace`
`property` `writable`
Input image colorspace -- one of `'RGB'`, `'BGR'`, or `'auto'`.
This parameter is used **only** to identify colorspace for NumPy arrays.
`'auto'` translates to `'BGR'` for `opencv` backend, and to `'RGB'` for `pil` backend.
## input\_pad\_method
`degirum.model.Model.input_pad_method`
`property` `writable`
Input image pad method -- one of `'stretch'`, `'letterbox'`, `'crop-first'`, or `'crop-last'`.
* In case of `'stretch'`, the input image will be resized to the model input size **without** preserving aspect ratio.
* In case of `'letterbox'`, the input image will be resized to the model input size preserving aspect ratio.
* In case of `'crop-first'`, the input image will be cropped to input\_crop\_percentage around the center and then resized.
* In the case of 'crop-last', if the model's input dimensions are square, the image is resized with its smaller side matching the model dimension, preserving the aspect ratio. If the dimensions are rectangular, the image is resized and stretched to fit the model's input dimensions. After resizing, the image is cropped to the model's input dimensions and aspect ratio based on the 'input\_crop\_percentage' property.
The voids will be filled with solid color specified by `input_letterbox_fill_color` property. In all cases [degirum.model.Model.input\_resize\_method](#degirum.model.model.input_resize_method) property specifies the algorithm for resizing.
## input\_resize\_method
`degirum.model.Model.input_resize_method`
`property` `writable`
Input image resize method -- one of `'nearest'`, `'bilinear'`, `'area'`, `'bicubic'`, or `'lanczos'`.
## input\_shape
`degirum.model.Model.input_shape`
`property` `writable`
Input tensor shapes. List of tensor shapes per input.
Each element of that list is another list containing tensor dimensions, slowest dimension first:
* if InputShape model parameter is specified, its value is used.
* otherwise, all *defined* InputN/H/W/C model parameters are used as \[InputN, InputH, InputW, InputC] list.
## label\_dictionary
`degirum.model.Model.label_dictionary`
`abstractmethod` `property`
Get model class label dictionary.
Each dictionary element is key-value pair, where the key is the class ID and the value is the class label string.
## measure\_time
`degirum.model.Model.measure_time`
`property` `writable`
Flag to enable measuring and collecting inference time statistics.
Call [degirum.model.Model.time\_stats](#degirum.model.model.time_stats) to query accumulated inference time statistics.
## model\_info
`degirum.model.Model.model_info`
`property`
Return model information object to provide read-only access to model parameters.
New deep copy is created each time.
## non\_blocking\_batch\_predict
`degirum.model.Model.non_blocking_batch_predict`
`property` `writable`
Flag to control the behavior of the generator object returned by `predict_batch()` method.
* When the flag is set to `True`, the generator accepts `None` from the inference input data iterator object (passed as `data` parameter): If `None` is returned, the model predict step is skipped for this iteration. Also, when no inference results are available in the result queue at this iteration, the generator yields `None` result.
* When the flag is set to `False` (default value), the generator does not allow `None` to be returned from the inference input data iterator object: If `None` is returned, an exception is raised. Also, when no inference results are available in the result queue at this iteration, the generator continues to the next iteration of the input data iterator.
* Setting this flag to `True` allows using `predict_batch()` generator in a non-blocking manner, assuming the design of input data iterator object is also non-blocking, i.e., returning `None` when no data is available instead of waiting for the data. Every next element request from the generator will not block the execution waiting for either input data or inference results, returning `None` when no results are available.
## output\_class\_set
`degirum.model.Model.output_class_set`
`property` `writable`
Labels filter: list of class labels/category IDs to be included in inference results.
{% hint style="info" %}
You can use [degirum.model.Model.label\_dictionary](#degirum.model.model.label_dictionary) property to obtain a list of model classes.
{% endhint %}
## output\_confidence\_threshold
`degirum.model.Model.output_confidence_threshold`
`property` `writable`
Confidence threshold used in inference result post-processing.
Valid range: `[0..1]`.
Only objects with scores higher than this threshold are reported.
{% hint style="info" %}
For classification models if [degirum.model.Model.output\_top\_k](#degirum.model.model.output_top_k) parameter is set to non-zero value, then it supersedes this threshold -- [degirum.model.Model.output\_top\_k](#degirum.model.model.output_top_k) highest score classes are always reported.
{% endhint %}
## output\_max\_classes\_per\_detection
`degirum.model.Model.output_max_classes_per_detection`
`property` `writable`
Max Detections Per Class number used in inference result post-processing, and specifies the maximum number of highest probability classes per anchor to be processed during the non-max suppression process for fast algorithm.
Applicable only for detection models.
## output\_max\_detections
`degirum.model.Model.output_max_detections`
`property` `writable`
Max Detection number used in inference result post-processing, and specifies the total maximum objects of number to be detected.
Applicable only for detection models.
## output\_max\_detections\_per\_class
`degirum.model.Model.output_max_detections_per_class`
`property` `writable`
Max Detections Per Class number used in inference result post-processing, and specifies the maximum number of objects to keep during per class non-max suppression process for regular algorithm.
Applicable only for detection models.
## output\_nms\_threshold
`degirum.model.Model.output_nms_threshold`
`property` `writable`
Non-Max Suppression (NMS) threshold used in inference result post-processing.
Valid range: `[0..1]`.
Applicable only for models which utilize NMS algorithm.
## output\_pose\_threshold
`degirum.model.Model.output_pose_threshold`
`property` `writable`
Pose detection threshold used in inference result post-processing.
Valid range: `[0..1]`.
Applicable only for pose detection models.
## output\_postprocess\_type
`degirum.model.Model.output_postprocess_type`
`property` `writable`
Inference result post-processing type.
You may set it to `'None'` to bypass post-processing.
## output\_top\_k
`degirum.model.Model.output_top_k`
`property` `writable`
The number of classes with highest scores to report for classification models.
When set to `0`, then report all classes with scores greater than [degirum.model.Model.output\_confidence\_threshold](#degirum.model.model.output_confidence_threshold).
## output\_use\_regular\_nms
`degirum.model.Model.output_use_regular_nms`
`property` `writable`
Use Regular NMS value used in inference result post-processing and specifies the algorithm to use for detection postprocessing.
If value is `True`, regular Non-Max suppression algorithm is used -- NMS is calculated for each class separately and after that all results are merged.
If value is `False`, fast Non-Max suppression algorithm is used -- NMS is calculated for all classes simultaneously.
## overlay\_alpha
`degirum.model.Model.overlay_alpha`
`property` `writable`
Alpha-blend weight for inference results drawing on overlay image.
`float` number in range `[0..1]`.
See [InferenceResults.image\_overlay](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/pages/lMeFJKx8um7RZqppYeCI#degirum.postprocessor.inferenceresults.image_overlay) for more details.
## overlay\_blur
`degirum.model.Model.overlay_blur`
`property` `writable`
Overlay blur option.
`None` for no blur, `"all"` to blur all objects, a class label or list of class labels to blur specific objects.
## overlay\_color
`degirum.model.Model.overlay_color`
`property` `writable`
Color for inference results drawing on overlay image.
3-element RGB tuple or list of 3-element RGB tuples.
The `overlay_color` property is used to define the color to draw overlay details. In the case of a single RGB tuple, the corresponding color is used to draw all the overlay data: points, boxes, labels, segments, etc. In the case of a list of RGB tuples the behavior depends on the model type:
* For classification models different colors from the list are used to draw labels of different classes.
* For detection models different colors are used to draw labels *and boxes* of different classes.
* For pose detection models different colors are used to draw keypoints of different persons.
* For segmentation models different colors are used to highlight segments of different classes.
If the list size is less than the number of classes of the model, then `overlay_color` values are used cyclically, for example, for three-element list it will be `overlay_color[0]`, then `overlay_color[1]`, `overlay_color[2]`, and again `overlay_color[0]`.
The default value of `overlay_color` is a single RBG tuple of yellow color for all model types except segmentation models. For segmentation models it is the list of RGB tuples with the list size equal to the number of model classes. You can use [degirum.model.Model.label\_dictionary](#degirum.model.model.label_dictionary) property to obtain a list of model classes. Each color is automatically assigned to look pretty and different from other colors in the list.
## overlay\_font\_scale
`degirum.model.Model.overlay_font_scale`
`property` `writable`
Font scale for inference results drawing on overlay image.
`float` positive number.
See [InferenceResults.image\_overlay](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/pages/lMeFJKx8um7RZqppYeCI#degirum.postprocessor.inferenceresults.image_overlay) for more details.
## overlay\_line\_width
`degirum.model.Model.overlay_line_width`
`property` `writable`
Line width for inference results drawing on overlay image.
See [InferenceResults.image\_overlay](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/pages/lMeFJKx8um7RZqppYeCI#degirum.postprocessor.inferenceresults.image_overlay) for more details.
## overlay\_show\_labels
`degirum.model.Model.overlay_show_labels`
`property` `writable`
Flag to enable/disable drawing class labels on overlay image.
See [InferenceResults.image\_overlay](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/pages/lMeFJKx8um7RZqppYeCI#degirum.postprocessor.inferenceresults.image_overlay) for more details.
## overlay\_show\_probabilities
`degirum.model.Model.overlay_show_probabilities`
`property` `writable`
Flag to enable/disable drawing class probabilities on overlay image.
See [InferenceResults.image\_overlay](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/pages/lMeFJKx8um7RZqppYeCI#degirum.postprocessor.inferenceresults.image_overlay) for more details.
## save\_model\_image
`degirum.model.Model.save_model_image`
`property` `writable`
Flag to enable/disable saving of model input image in inference results.
Model input image is the image converted to AI model input specifications as raw binary array.
## supported\_device\_types
`degirum.model.Model.supported_device_types`
`property`
The list of supported device types in format `/` for this model.
## \_\_call\_\_(data)
`degirum.model.Model.__call__(data)`
Perform whole inference lifecycle: input data preprocessing, inference and postprocessing.
Same as [degirum.model.Model.predict](#degirum.model.model.predict).
## \_\_enter\_\_
`degirum.model.Model.__enter__()`
Context manager enter handler.
## \_\_exit\_\_(exc\_type, ...)
`degirum.model.Model.__exit__(exc_type, exc_val, exc_tb)`
Context manager exit handler.
## \_\_init\_\_(model\_name, ...)
`degirum.model.Model.__init__(model_name, model_params, supported_device_types)`
Constructor.
{% hint style="info" %}
You never construct model objects yourself -- instead you call [degirum.zoo\_manager.ZooManager.load\_model](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/pages/YhwNF6qcgxroBmccSwg8#degirum.zoo_manager.zoomanager.load_model) method to create [degirum.model.Model](#degirum.model.model) instances for you.
{% endhint %}
## get\_inference\_results\_class
`degirum.model.Model.get_inference_results_class()`
Get inference results class, deduced from model parameters
## get\_inference\_results\_type
`degirum.model.Model.get_inference_results_type()`
Get inference results type string, deduced from model parameters
## predict(data)
`degirum.model.Model.predict(data)`
Perform whole inference lifecycle: input data preprocessing, inference, and postprocessing.
Parameters:
| Name | Type | Description | Default |
| ------ | ----- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
| `data` | `any` |
Inference input data. Input data type depends on the model.
If the model expects image data, then the input data is either:
Input image path string.
NumPy 3D array of pixels in a form HWC. where color dimension is native to selected graphical backend (RGB for 'pil' and BGR for 'opencv' backend)
PIL.Image object (only for 'pil' backend).
If the model expects audio data, then the input data is NumPy 1D array with audio data samples.
If the model expects raw tensor data, then the input data is NumPy multidimensional array with shape matching model input.
In case of multi-input model a list of elements of the supported data type is expected.
| *required* |
Returns:
| Type | Description |
| ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| InferenceResults | Inference result object, which allows you to access inference results as a dictionary or as an overlay image if it is supported by the model. For your convenience, all image coordinates in case of detection models are converted from model coordinates to original image coordinates. |
## predict\_batch(data)
`degirum.model.Model.predict_batch(data)`
Perform whole inference lifecycle for all objects in given iterator object (for example, `list`).
Such iterator object should return the same object types which regular [degirum.model.Model.predict](#degirum.model.model.predict) method accepts.
Parameters:
| Name | Type | Description | Default |
| ------ | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
| `data` | `iterator` |
Inference input data iterator object such as list or generator function.
Each element returned by this iterator can be one of the following:
A single input data object, in case of single-input model.
A list of input data objects, in case of multi-input model.
A tuple containing a pair of input data object or a list of input data objects as a first element and frame info object as a second element of the tuple.
The input data object type depends on the model.
If the model expects image data, then the input data object is either:
Input image path string.
NumPy 3D array of pixels in a form HWC, where color dimension is native to selected graphical backend (RGB for 'pil' and BGR for 'opencv' backend).
PIL.Image object (only for 'pil' backend).
If the model expects audio data, then the input data object is NumPy 1D array with audio data samples.
If the model expects raw tensor data, then the input data object is NumPy multidimensional array with shape matching model input.
The frame info object is passed to the inference result object unchanged and can be accessed via info property of the inference result object.
| *required* |
Returns:
| Type | Description |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Iterator[` [`InferenceResults`](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/pages/lMeFJKx8um7RZqppYeCI#degirum.postprocessor.inferenceresults)`]` | Generator object which iterates over inference result objects. This allows you directly using the result of [degirum.model.Model.predict\_batch](#degirum.model.Model.predict_batch) in `for` loops. |
Example
{% code overflow="wrap" %}
```
for result in model.predict_batch(['image1.jpg','image2.jpg']):
print(result)
```
{% endcode %}
## predict\_dir(path, ...)
`degirum.model.Model.predict_dir(path, *, recursive=False, extensions=['.jpg', '.jpeg', '.png', '.bmp'])`
Perform whole inference lifecycle for all files from specified directory matching given file extensions.
Supports only single-input models.
Parameters:
| Name | Type | Description | Default |
| ------------ | ----------- | --------------------------------------------------------------------------------------- | ----------------------------------- |
| `path` | `str` | Directory name containing files to be processed. | *required* |
| `recursive` | `bool` | True to recursively walk through all subdirectories in a directory. Default is `False`. | `False` |
| `extensions` | `list[str]` | Single string or list of strings containing file extension(s) to process. | `['.jpg', '.jpeg', '.png', '.bmp']` |
Returns:
| Type | Description |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Iterator[` [`InferenceResults`](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/pages/lMeFJKx8um7RZqppYeCI#degirum.postprocessor.inferenceresults)`]` | Generator object to iterate over inference result objects. This allows you directly using the result of [degirum.model.Model.predict\_dir](#degirum.model.Model.predict_dir) in `for` loops. |
Example
{% code overflow="wrap" %}
```
for result in model.predict_dir('./some_path'):
print(result)
```
{% endcode %}
## reset\_time\_stats
`degirum.model.Model.reset_time_stats()`
Reset inference time statistics.
[degirum.model.Model.time\_stats](#degirum.model.model.time_stats) method will return empty dictionary after this call.
## time\_stats
`degirum.model.Model.time_stats()`
Query inference time statistics.
Returns:
| Type | Description |
| ------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `dict` |
Dictionary containing time statistic objects.
A key in that dictionary is a string description of a particular inference step.
Each statistic object keeps min, max, and average values in milliseconds, accumulated over all inferences performed on this model since the model creation of last call of statistic reset method degirum.model.Model.reset\_time\_stats.
|
# Zoo Manager Module
PySDK API Reference Guide. Load, list and authenticate against local, server or cloud zoos.
{% hint style="info" %}
This API Reference is based on PySDK 0.20.0.
{% endhint %}
## degirum.zoo\_manager.ZooManager
Class that manages a model zoo.
A *model zoo* in terminology of PySDK is a collection of AI models and simultaneously an ML inference engine type and location.
Depending on the deployment location, there are several types of model zoos supported by PySDK:
* **Local** model zoo: Deployed on the local file system of the PySDK installation host. Inferences are performed on the same host using AI accelerators installed on that host.
* AI **server** model zoo: Deployed on remote host with DeGirum AI server running on that host. Inferences are performed by DeGirum AI server on that remote host.
* **Cloud** Platform model zoo: Deployed on DeGirum Cloud Platform. Inferences are performed by DeGirum Cloud Platform servers.
The type of the model zoo is defined by the URL string which you pass as `zoo_url` parameter into the constructor.
Zoo manager provides the following functionality:
* List and search models available in the connected model zoo.
* Create AI model handling objects to perform AI inferences.
* Request various AI model parameters.
## \_\_init\_\_(inference\_host\_address, ...)
`degirum.zoo_manager.ZooManager.__init__(inference_host_address, zoo_url='', token='')`
Constructor.
Note
Typically, you never construct `ZooManager` objects yourself -- instead you call [degirum.connect](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/pages/nDBtWYNXU5TsNZwgXw5G#degirum.connect) function to create `ZooManager` instances for you.
For the description of arguments see [degirum.connect](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/pages/nDBtWYNXU5TsNZwgXw5G#degirum.connect)
## list\_models(\*args, ...)
`degirum.zoo_manager.ZooManager.list_models(*args, **kwargs)`
Get a list of names of AI models available in the connected model zoo which match specified filtering criteria.
Other Parameters:
| Name | Type | Description |
| ------------------ | ----- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `model_family` | `str` |
Model family name filter.
When you pass a string, it will be used as search substring in the model name. For example, "yolo", "mobilenet".
You may also pass re.Pattern object. In this case it will do regular expression pattern search.
|
| `runtime` | `str` | Runtime agent type -- string or list of strings of runtime agent types. |
| `device` | `str` | Target inference device -- string or list of strings of device names. |
| `device_type` | `str` | Target inference device(s) -- string or list of strings of full device type names in "RUNTIME/DEVICE" format. |
| `precision` | `str` |
Model calculation precision - string or list of strings of model precision labels.
Possible labels: "quant", "float".
|
| `pruned` | `str` |
Model density -- string or list of strings of model density labels.
Possible labels: "dense", "pruned".
|
| `postprocess_type` | `str` |
Model output postprocess type -- string or list of strings of postprocess type labels.
For example: "Classification", "Detection", "Segmentation".
|
Returns:
| Type | Description |
| ------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Union[List[str], Dict[str, ModelParams]]` | The list of model name strings matching specified filtering criteria. Use a string from that list as a parameter of [degirum.zoo\_manager.ZooManager.load\_model](#degirum.zoo_manager.ZooManager.load_model) method. |
Example
Find all models of `"yolo"` family capable to run either on CPU or on DeGirum Orca AI accelerator from all registered model zoos:
{% code overflow="wrap" %}
```
yolo_model_list = zoo_manager.list_models("yolo", device=["cpu", "orca"])
```
{% endcode %}
## load\_model(model\_name, ...)
`degirum.zoo_manager.ZooManager.load_model(model_name, **kwargs)`
Create and return the model handling object for given model name.
Parameters:
| Name | Type | Description | Default |
| ------------ | ----- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
| `model_name` | `str` | Model name string identifying the model to load. It should exactly match the model name as it is returned by [degirum.zoo\_manager.ZooManager.list\_models](#degirum.zoo_manager.ZooManager.list_models) method. | *required* |
| `**kwargs` | `any` | you may pass arbitrary model properties to be assigned to the model object in a form of property=value | `{}` |
Returns:
| Type | Description |
| ----------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [`Model`](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/pages/GfBG9YuP1z73SsOm25XD#degirum.model.Model) |
Model handling object. Using this object you perform AI inferences on this model and also configure various model properties, which define how to do input image preprocessing and inference result post-processing:
Call degirum.model.Model.predict method to perform AI inference of a single frame. Inference result object is returned.
|
## model\_info(model\_name)
`degirum.zoo_manager.ZooManager.model_info(model_name)`
Request model parameters for given model name.
Parameters:
| Name | Type | Description | Default |
| ------------ | ----- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
| `model_name` | `str` | Model name string. It should exactly match the model name as it is returned by [degirum.zoo\_manager.ZooManager.list\_models](#degirum.zoo_manager.ZooManager.list_models) method. | *required* |
Returns:
| Type | Description |
| ------------- | ------------------------------------------------------------------------------- |
| `ModelParams` | Model parameter object which provides read-only access to all model parameters. |
{% hint style="info" %}
You cannot modify actual model parameters -- any changes of model parameter object returned by this method are not applied to the real model. Use properties of model handling objects returned by [degirum.zoo\_manager.ZooManager.load\_model](#degirum.zoo_manager.zoomanager.load_model) method to change parameters of that particular model instance on the fly.
{% endhint %}
## supported\_device\_types
`degirum.zoo_manager.ZooManager.supported_device_types()`
Get runtime/device type names, which are available in the inference system.
Returns:
| Type | Description |
| ------ | ---------------------------------------------------------------------------------------- |
| `list` | list of runtime/device type names; each element is a string in a format "RUNTIME/DEVICE" |
## system\_info(update=False)
`degirum.zoo_manager.ZooManager.system_info(update=False)`
Return host system information dictionary
Parameters:
| Name | Type | Description | Default |
| -------- | ------ | ---------------------------------------------------------- | ------- |
| `update` | `bool` | force update system information, otherwise take from cache | `False` |
Returns:
| Type | Description |
| ------ | ------------------------------------------------------------------------------------------------------------------------------------------ |
| `dict` | host system information dictionary. Format: `{"Devices": {"/": {}, ...}, ["Software Version": ""]}` |
# Postprocessor Module
PySDK API Reference Guide. InferenceResults containers.
{% hint style="info" %}
This API Reference is based on PySDK 0.20.0.
{% endhint %}
## degirum.postprocessor.InferenceResults
Inference results container class.
This class is a base class for a set of classes designed to handle inference results of particular model types such as classification, detection etc.
{% hint style="info" %}
You never construct model objects yourself. Objects of those classes are returned by various predict methods of [degirum.model.Model](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/pages/GfBG9YuP1z73SsOm25XD#degirum.model.model) class.
{% endhint %}
## image
`degirum.postprocessor.InferenceResults.image`
`property`
Original image.
Returned image object type is defined by the selected graphical backend (see [degirum.model.Model.image\_backend](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/pages/GfBG9YuP1z73SsOm25XD#degirum.model.model.image_backend)).
## image\_model
`degirum.postprocessor.InferenceResults.image_model`
`property`
Model input image data: image converted to AI model input specifications.
Image type is raw binary array.
## image\_overlay
`degirum.postprocessor.InferenceResults.image_overlay`
`property`
Image with AI inference results drawn on a top of original image.
Drawing details depend on the inference result type:
* For classification models the list of class labels with probabilities is printed below the original image.
* For object detection models bounding boxes of detected object are drawn on the original image.
* For pose detection models detected keypoints and keypoint connections are drawn on the original image.
* For segmentation models detected segments are drawn on the original image.
Returned image object type is defined by the selected graphical backend (see [degirum.model.Model.image\_backend](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/pages/GfBG9YuP1z73SsOm25XD#degirum.model.model.image_backend)).
## info
`degirum.postprocessor.InferenceResults.info`
`property`
Input data frame information object.
## overlay\_alpha
`degirum.postprocessor.InferenceResults.overlay_alpha`
`property` `writable`
Alpha-blend weight for overlay details.
## overlay\_blur
`degirum.postprocessor.InferenceResults.overlay_blur`
`property` `writable`
Overlay blur option. None for no blur, "all" to blur all objects, a class label or list of class labels to blur specific objects.
## overlay\_color
`degirum.postprocessor.InferenceResults.overlay_color`
`property` `writable`
Color for inference results drawing on overlay image.
3-element RGB tuple or list of 3-element RGB tuples.
## overlay\_fill\_color
`degirum.postprocessor.InferenceResults.overlay_fill_color`
`property` `writable`
Image fill color in case of image padding.
3-element RGB tuple.
## overlay\_font\_scale
`degirum.postprocessor.InferenceResults.overlay_font_scale`
`property` `writable`
Font scale to use for overlay text.
## overlay\_line\_width
`degirum.postprocessor.InferenceResults.overlay_line_width`
`property` `writable`
Line width in pixels for inference results drawing on overlay image.
## overlay\_show\_labels
`degirum.postprocessor.InferenceResults.overlay_show_labels`
`property` `writable`
Specifies if class labels should be drawn on overlay image.
## overlay\_show\_probabilities
`degirum.postprocessor.InferenceResults.overlay_show_probabilities`
`property` `writable`
Specifies if class probabilities should be drawn on overlay image.
## results
`degirum.postprocessor.InferenceResults.results`
`property`
Inference results list.
Each element of the list is a dictionary containing information about one inference result. The dictionary contents depends on the AI model.
**For classification models** each inference result dictionary contains the following keys:
* `category_id`: class numeric ID.
* `label`: class label string.
* `score`: class probability.
Example
{% code overflow="wrap" %}
```
[
{'category_id': 0, 'label': 'cat', 'score': 0.99},
{'category_id': 1, 'label': 'dog', 'score': 0.01}
]
```
{% endcode %}
**For multi-label classification models** each inference result dictionary contains the following keys:
* `classifier`: object class string.
* `results`: list of class labels and its scores. Scores are optional.
The `results` list element is a dictionary with the following keys:
* `label`: class label string.
* `score`: optional class label probability.
Example
{% code overflow="wrap" %}
```
[
{
'classifier': 'vehicle color',
'results': [
{'label': 'red', 'score': 0.99},
{'label': 'blue', 'score': 0.01}
]
},
{
'classifier': 'vehicle type',
'results': [
{'label': 'car', 'score': 0.99},
{'label': 'truck', 'score': 0.01}
]
}
]
```
{% endcode %}
**For object detection models** each inference result dictionary may contain the following keys:
* `category_id`: detected object class numeric ID.
* `label`: detected object class label string.
* `score`: detected object probability.
* `bbox`: detected object bounding box list `[xtop, ytop, xbot, ybot]`.
* `landmarks`: optional list of keypoints or landmarks. It is the list of dictionaries, one per each keypoint/landmark.
* `mask`: optinal dictionary of run-length encoded (RLE) object segmentation mask array representation.
* `angle`: optional angle (in radians) for rotating bounding box around its center. This is used in the case of oriented bounding boxes.
The `landmarks` list is defined for special cases like pose detection of face points detection results. Each `landmarks` list element is a dictionary with the following keys:
* `category_id`: keypoint numeric ID.
* `label`: keypoint label string.
* `score`: keypoint detection probability.
* `landmark`: keypoint coordinate list `[x,y,visibility,a,b,...]`.
* `connect`: optional list of IDs of connected keypoints.
The `mask` dictionary is defined for the special case of object segmentation results, with the following keys:
* `x_min`: x-coordinate in the model input image at which the top-left corner of the box enclosing this mask should be placed.
* `y_min`: y-coordinate in the model input image at which the top-left corner of the box enclosing this mask should be placed.
* `height`: height of segmentation mask array
* `width`: width of segmentation mask array
* `data`: string representation of a buffer of unsigned 32-bit integers carrying the RLE segmentation mask array.
The object detection keys (`bbox`, `score`, `label`, and `category_id`) must be either all present or all absent. In the former case the result format is suitable to represent pure object detection results. In the later case, one of the following keys must be present:
* the `landmarks` key
* the `mask` key
The following statements are then true:
* If the `landmarks` key is present, the result format is suitable to represent pure landmark detection results, such as pose detection.
* If the `mask` key is present, the result format is suitable to represent pure segmentation results. If, optionally, the `category_id` key is also present, the result format is suitable to represent semantic segmentation results.
When both object detection keys and the `landmarks` key are present, the result format is suitable to represent mixed model results, when the model detects not only object bounding boxes, but also keypoints/landmarks within the bounding box.
When both object detection keys and the `mask` key are present, the result format is suitable to represent mixed model results, when the model detects not only object bounding boxes, but also segmentation masks within the bounding box (i.e. instance segmentation).
Example of pure object detection results:
Example
{% code overflow="wrap" %}
```
[
{'category_id': 0, 'label': 'cat', 'score': 0.99, 'bbox': [10, 20, 100, 200]},
{'category_id': 1, 'label': 'dog', 'score': 0.01, 'bbox': [200, 100, 300, 400]}
]
```
{% endcode %}
Example of oriented object detection results:
Example
{% code overflow="wrap" %}
```
[
{'category_id': 0, 'label': 'car', 'score': 0.99, 'bbox': [10, 20, 100, 200], 'angle': 0.79}
]
```
{% endcode %}
Example of landmark object detection results:
Example
{% code overflow="wrap" %}
```
[
{
'landmarks': [
{'category_id': 0, 'label': 'Nose', 'score': 0.99, 'landmark': [10, 20]},
{'category_id': 1, 'label': 'LeftEye', 'score': 0.98, 'landmark': [15, 25]},
{'category_id': 2, 'label': 'RightEye', 'score': 0.97, 'landmark': [18, 28]}
]
}
]
```
{% endcode %}
Example of segmented object detection results:
Example
{% code overflow="wrap" %}
```
[
{
'mask': {'x_min': 1, 'y_min': 1, 'height': 2, 'width': 2, 'data': 'AAAAAAEAAAAAAAAAAQAAAAIAAAABAAAA'}
}
]
```
{% endcode %}
**For hand palm detection** models each inference result dictionary contains the following keys:
* `score`: probability of detected hand.
* `handedness`: probability of right hand.
* `landmarks`: list of dictionaries, one per each hand keypoint.
Each `landmarks` list element is a dictionary with the following keys:
* `label`: classified object class label.
* `category_id`: classified object class index.
* `landmark`: landmark point coordinate list `[x, y, z]`.
* `world_landmark`: metric world landmark point coordinate list `[x, y, z]`.
* `connect`: list of adjacent landmarks indexes.
Example
{% code overflow="wrap" %}
```
[
{
'score': 0.99,
'handedness': 0.98,
'landmarks': [
{
'label': 'Wrist',
'category_id': 0,
'landmark': [10, 20, 30],
'world_landmark': [10, 20, 30],
'connect': [1]
},
{
'label': 'Thumb',
'category_id': 1,
'landmark': [15, 25, 35],
'world_landmark': [15, 25, 35],
'connect': [0]
}
]
}
]
```
{% endcode %}
**For segmentation models** inference result is a single-element list. That single element is a dictionary, containing single key `data`. The value of this key is 2D numpy array of integers, where each integer value represents a class ID of the corresponding pixel. The class IDs are defined by the model label dictionary.
Example
{% code overflow="wrap" %}
```
[
{
'data': numpy.array([
[0, 0, 0, 1, 1, 1],
[0, 0, 0, 1, 1, 1],
[0, 0, 0, 1, 1, 1],
[2, 2, 2, 3, 3, 3],
[2, 2, 2, 3, 3, 3],
[2, 2, 2, 3, 3, 3],
])
}
]
```
{% endcode %}
## generate\_colors
`degirum.postprocessor.InferenceResults.generate_colors()`
`staticmethod`
Generate a list of unique RGB color tuples.
## generate\_overlay\_color(num\_classes, ...)
`degirum.postprocessor.InferenceResults.generate_overlay_color(num_classes, label_dict)`
`staticmethod`
Overlay colors generator.
Parameters:
| Name | Type | Description | Default |
| ------------- | ------ | --------------------------- | ---------- |
| `num_classes` | `int` | number of class categories. | *required* |
| `label_dict` | `dict` | Model labels dictionary. | *required* |
Returns:
| Type | Description |
| -------------------- | -------------------------------------- |
| `Union[list, tuple]` | Overlay color tuple or list of tuples. |
## supported\_types
`degirum.postprocessor.InferenceResults.supported_types()`
`staticmethod`
List supported result types for this postprocessor.
## degirum.postprocessor.\_InferenceResults.InferenceResults
Inference results container class.
This class is a base class for a set of classes designed to handle inference results of particular model types such as classification, detection etc.
{% hint style="info" %}
You never construct model objects yourself. Objects of those classes are returned by various predict methods of [degirum.model.Model](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/pages/GfBG9YuP1z73SsOm25XD#degirum.model.model) class.
{% endhint %}
## image
`degirum.postprocessor._InferenceResults.InferenceResults.image`
`property`
Original image.
Returned image object type is defined by the selected graphical backend (see [degirum.model.Model.image\_backend](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/pages/GfBG9YuP1z73SsOm25XD#degirum.model.model.image_backend)).
## image\_model
`degirum.postprocessor._InferenceResults.InferenceResults.image_model`
`property`
Model input image data: image converted to AI model input specifications.
Image type is raw binary array.
## image\_overlay
`degirum.postprocessor._InferenceResults.InferenceResults.image_overlay`
`property`
Image with AI inference results drawn on a top of original image.
Drawing details depend on the inference result type:
* For classification models the list of class labels with probabilities is printed below the original image.
* For object detection models bounding boxes of detected object are drawn on the original image.
* For pose detection models detected keypoints and keypoint connections are drawn on the original image.
* For segmentation models detected segments are drawn on the original image.
Returned image object type is defined by the selected graphical backend (see [degirum.model.Model.image\_backend](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/pages/GfBG9YuP1z73SsOm25XD#degirum.model.model.image_backend)).
## info
`degirum.postprocessor._InferenceResults.InferenceResults.info`
`property`
Input data frame information object.
## overlay\_alpha
`degirum.postprocessor._InferenceResults.InferenceResults.overlay_alpha`
`property` `writable`
Alpha-blend weight for overlay details.
## overlay\_blur
`degirum.postprocessor._InferenceResults.InferenceResults.overlay_blur`
`property` `writable`
Overlay blur option. None for no blur, "all" to blur all objects, a class label or list of class labels to blur specific objects.
## overlay\_color
`degirum.postprocessor._InferenceResults.InferenceResults.overlay_color`
`property` `writable`
Color for inference results drawing on overlay image.
3-element RGB tuple or list of 3-element RGB tuples.
## overlay\_fill\_color
`degirum.postprocessor._InferenceResults.InferenceResults.overlay_fill_color`
`property` `writable`
Image fill color in case of image padding.
3-element RGB tuple.
## overlay\_font\_scale
`degirum.postprocessor._InferenceResults.InferenceResults.overlay_font_scale`
`property` `writable`
Font scale to use for overlay text.
## overlay\_line\_width
`degirum.postprocessor._InferenceResults.InferenceResults.overlay_line_width`
`property` `writable`
Line width in pixels for inference results drawing on overlay image.
## overlay\_show\_labels
`degirum.postprocessor._InferenceResults.InferenceResults.overlay_show_labels`
`property` `writable`
Specifies if class labels should be drawn on overlay image.
## overlay\_show\_probabilities
`degirum.postprocessor._InferenceResults.InferenceResults.overlay_show_probabilities`
`property` `writable`
Specifies if class probabilities should be drawn on overlay image.
## results
`degirum.postprocessor._InferenceResults.InferenceResults.results`
`property`
Inference results list.
Each element of the list is a dictionary containing information about one inference result. The dictionary contents depends on the AI model.
**For classification models** each inference result dictionary contains the following keys:
* `category_id`: class numeric ID.
* `label`: class label string.
* `score`: class probability.
Example
{% code overflow="wrap" %}
```
[
{'category_id': 0, 'label': 'cat', 'score': 0.99},
{'category_id': 1, 'label': 'dog', 'score': 0.01}
]
```
{% endcode %}
**For multi-label classification models** each inference result dictionary contains the following keys:
* `classifier`: object class string.
* `results`: list of class labels and its scores. Scores are optional.
The `results` list element is a dictionary with the following keys:
* `label`: class label string.
* `score`: optional class label probability.
Example
{% code overflow="wrap" %}
```
[
{
'classifier': 'vehicle color',
'results': [
{'label': 'red', 'score': 0.99},
{'label': 'blue', 'score': 0.01}
]
},
{
'classifier': 'vehicle type',
'results': [
{'label': 'car', 'score': 0.99},
{'label': 'truck', 'score': 0.01}
]
}
]
```
{% endcode %}
**For object detection models** each inference result dictionary may contain the following keys:
* `category_id`: detected object class numeric ID.
* `label`: detected object class label string.
* `score`: detected object probability.
* `bbox`: detected object bounding box list `[xtop, ytop, xbot, ybot]`.
* `landmarks`: optional list of keypoints or landmarks. It is the list of dictionaries, one per each keypoint/landmark.
* `mask`: optinal dictionary of run-length encoded (RLE) object segmentation mask array representation.
* `angle`: optional angle (in radians) for rotating bounding box around its center. This is used in the case of oriented bounding boxes.
The `landmarks` list is defined for special cases like pose detection of face points detection results. Each `landmarks` list element is a dictionary with the following keys:
* `category_id`: keypoint numeric ID.
* `label`: keypoint label string.
* `score`: keypoint detection probability.
* `landmark`: keypoint coordinate list `[x,y,visibility,a,b,...]`.
* `connect`: optional list of IDs of connected keypoints.
The `mask` dictionary is defined for the special case of object segmentation results, with the following keys:
* `x_min`: x-coordinate in the model input image at which the top-left corner of the box enclosing this mask should be placed.
* `y_min`: y-coordinate in the model input image at which the top-left corner of the box enclosing this mask should be placed.
* `height`: height of segmentation mask array
* `width`: width of segmentation mask array
* `data`: string representation of a buffer of unsigned 32-bit integers carrying the RLE segmentation mask array.
The object detection keys (`bbox`, `score`, `label`, and `category_id`) must be either all present or all absent. In the former case the result format is suitable to represent pure object detection results. In the later case, one of the following keys must be present:
* the `landmarks` key
* the `mask` key
The following statements are then true:
* If the `landmarks` key is present, the result format is suitable to represent pure landmark detection results, such as pose detection.
* If the `mask` key is present, the result format is suitable to represent pure segmentation results. If, optionally, the `category_id` key is also present, the result format is suitable to represent semantic segmentation results.
When both object detection keys and the `landmarks` key are present, the result format is suitable to represent mixed model results, when the model detects not only object bounding boxes, but also keypoints/landmarks within the bounding box.
When both object detection keys and the `mask` key are present, the result format is suitable to represent mixed model results, when the model detects not only object bounding boxes, but also segmentation masks within the bounding box (i.e. instance segmentation).
Example of pure object detection results:
Example
{% code overflow="wrap" %}
```
[
{'category_id': 0, 'label': 'cat', 'score': 0.99, 'bbox': [10, 20, 100, 200]},
{'category_id': 1, 'label': 'dog', 'score': 0.01, 'bbox': [200, 100, 300, 400]}
]
```
{% endcode %}
Example of oriented object detection results:
Example
{% code overflow="wrap" %}
```
[
{'category_id': 0, 'label': 'car', 'score': 0.99, 'bbox': [10, 20, 100, 200], 'angle': 0.79}
]
```
{% endcode %}
Example of landmark object detection results:
Example
{% code overflow="wrap" %}
```
[
{
'landmarks': [
{'category_id': 0, 'label': 'Nose', 'score': 0.99, 'landmark': [10, 20]},
{'category_id': 1, 'label': 'LeftEye', 'score': 0.98, 'landmark': [15, 25]},
{'category_id': 2, 'label': 'RightEye', 'score': 0.97, 'landmark': [18, 28]}
]
}
]
```
{% endcode %}
Example of segmented object detection results:
Example
{% code overflow="wrap" %}
```
[
{
'mask': {'x_min': 1, 'y_min': 1, 'height': 2, 'width': 2, 'data': 'AAAAAAEAAAAAAAAAAQAAAAIAAAABAAAA'}
}
]
```
{% endcode %}
**For hand palm detection** models each inference result dictionary contains the following keys:
* `score`: probability of detected hand.
* `handedness`: probability of right hand.
* `landmarks`: list of dictionaries, one per each hand keypoint.
Each `landmarks` list element is a dictionary with the following keys:
* `label`: classified object class label.
* `category_id`: classified object class index.
* `landmark`: landmark point coordinate list `[x, y, z]`.
* `world_landmark`: metric world landmark point coordinate list `[x, y, z]`.
* `connect`: list of adjacent landmarks indexes.
Example
{% code overflow="wrap" %}
```
[
{
'score': 0.99,
'handedness': 0.98,
'landmarks': [
{
'label': 'Wrist',
'category_id': 0,
'landmark': [10, 20, 30],
'world_landmark': [10, 20, 30],
'connect': [1]
},
{
'label': 'Thumb',
'category_id': 1,
'landmark': [15, 25, 35],
'world_landmark': [15, 25, 35],
'connect': [0]
}
]
}
]
```
{% endcode %}
**For segmentation models** inference result is a single-element list. That single element is a dictionary, containing single key `data`. The value of this key is 2D numpy array of integers, where each integer value represents a class ID of the corresponding pixel. The class IDs are defined by the model label dictionary.
Example
{% code overflow="wrap" %}
```
[
{
'data': numpy.array([
[0, 0, 0, 1, 1, 1],
[0, 0, 0, 1, 1, 1],
[0, 0, 0, 1, 1, 1],
[2, 2, 2, 3, 3, 3],
[2, 2, 2, 3, 3, 3],
[2, 2, 2, 3, 3, 3],
])
}
]
```
{% endcode %}
## \_\_init\_\_(\*, ...)
`degirum.postprocessor._InferenceResults.InferenceResults.__init__(*, inference_results, conversion, input_image=None, model_image=None, draw_color=(255, 255, 128), line_width=3, show_labels=True, show_probabilities=False, alpha='auto', font_scale=1.0, fill_color=(0, 0, 0), blur=None, frame_info=None, label_dictionary={}, input_shape=None)`
Constructor.
{% hint style="info" %}
You never construct [InferenceResults](#degirum.postprocessor._inferenceresults.inferenceresults) objects yourself -- the ancestors of this class are returned as results of AI inferences from [degirum.model.Model.predict](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/pages/GfBG9YuP1z73SsOm25XD#degirum.model.model.predict), [degirum.model.Model.predict\_batch](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/pages/GfBG9YuP1z73SsOm25XD#degirum.model.model.predict_batch), and [degirum.model.Model.predict\_dir](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/pages/GfBG9YuP1z73SsOm25XD#degirum.model.model.predict_dir) methods.
{% endhint %}
Parameters:
| Name | Type | Description | Default |
| -------------------- | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------- |
| `inference_results` | `list` | Inference results data. | *required* |
| `conversion` | `Callable` | Coordinate conversion function accepting two arguments `(x,y)` and returning two-element tuple. This function should convert model-based coordinates to input image coordinates. | *required* |
| `input_image` | `any` | Original input data. | `None` |
| `model_image` | `any` | Input data converted per AI model input specifications. | `None` |
| `draw_color` | `tuple` | Color for inference results drawing on overlay image. | `(255, 255, 128)` |
| `line_width` | `int` | Line width in pixels for inference results drawing on overlay image. | `3` |
| `show_labels` | `bool` | True to draw class labels on overlay image. | `True` |
| `show_probabilities` | `bool` | True to draw class probabilities on overlay image. | `False` |
| `alpha` | `Union[float, str]` | Alpha-blend weight for overlay details. | `'auto'` |
| `font_scale` | `float` | Font scale to use for overlay text. | `1.0` |
| `fill_color` | `tuple` | RGB color tuple to use for filling if any form of padding is used. | `(0, 0, 0)` |
| `blur` | `Union[str, list, None]` | Optional blur parameter to apply to the overlay image. If None, no blur is applied. If "all" all objects are blurred. If a class label or a list of class labels is provided, only objects with those labels are blurred. | `None` |
| `frame_info` | `any` | Input data frame information object. | `None` |
| `label_dictionary` | `dict[str, str]` | Model label dictionary. | `{}` |
| `input_shape(list)` | | Model input shape. Mandatory for image-type postprocessing. | *required* |
## \_\_str\_\_
`degirum.postprocessor._InferenceResults.InferenceResults.__str__()`
Conversion to string
## generate\_colors
`degirum.postprocessor._InferenceResults.InferenceResults.generate_colors()`
`staticmethod`
Generate a list of unique RGB color tuples.
## generate\_overlay\_color(num\_classes, ...)
`degirum.postprocessor._InferenceResults.InferenceResults.generate_overlay_color(num_classes, label_dict)`
`staticmethod`
Overlay colors generator.
Parameters:
| Name | Type | Description | Default |
| ------------- | ------ | --------------------------- | ---------- |
| `num_classes` | `int` | number of class categories. | *required* |
| `label_dict` | `dict` | Model labels dictionary. | *required* |
Returns:
| Type | Description |
| -------------------- | -------------------------------------- |
| `Union[list, tuple]` | Overlay color tuple or list of tuples. |
## supported\_types
`degirum.postprocessor._InferenceResults.InferenceResults.supported_types()`
`staticmethod`
List supported result types for this postprocessor.
## degirum.postprocessor.\_DetectionResults.DetectionResults
Bases: [InferenceResults](#degirum.postprocessor._inferenceresults.inferenceresults)
InferenceResult class implementation for detection results type
## image\_overlay
`degirum.postprocessor._DetectionResults.DetectionResults.image_overlay`
`property`
Image with AI inference results drawn. Image type is defined by the selected graphical backend.
## \_\_str\_\_
`degirum.postprocessor._DetectionResults.DetectionResults.__str__()`
Convert inference results to string
## generate\_overlay\_color(num\_classes, ...)
`degirum.postprocessor._DetectionResults.DetectionResults.generate_overlay_color(num_classes, label_dict)`
`staticmethod`
Overlay colors generator.
Parameters:
| Name | Type | Description | Default |
| ------------- | ------ | --------------------------- | ---------- |
| `num_classes` | `int` | number of class categories. | *required* |
| `label_dict` | `dict` | Model labels dictionary. | *required* |
Returns:
| Type | Description |
| ------ | --------------------------------------------------- |
| `list` | general overlay color data for segmentation results |
## supported\_types
`degirum.postprocessor._DetectionResults.DetectionResults.supported_types()`
`staticmethod`
List supported result types for this postprocessor.
## degirum.postprocessor.\_ClassificationResults.ClassificationResults
Bases: [InferenceResults](#degirum.postprocessor._inferenceresults.inferenceresults)
InferenceResult class implementation for classification results type
## image\_overlay
`degirum.postprocessor._ClassificationResults.ClassificationResults.image_overlay`
`property`
Image with AI inference results drawn. Image type is defined by the selected graphical backend. Each time this property is accessed, new overlay image object is created and all overlay details are redrawn according to the current settings of overlay\_\*\*\* properties.
## overlay\_show\_labels\_below
`degirum.postprocessor._ClassificationResults.ClassificationResults.overlay_show_labels_below`
`property` `writable`
Specifies if overlay labels should be drawn below the image or on image itself
## \_\_str\_\_
`degirum.postprocessor._ClassificationResults.ClassificationResults.__str__()`
Convert inference results to string
## supported\_types
`degirum.postprocessor._ClassificationResults.ClassificationResults.supported_types()`
`staticmethod`
List supported result types for this postprocessor.
## degirum.postprocessor.\_MultiLabelClassificationResults.MultiLabelClassificationResults
Bases: [InferenceResults](#degirum.postprocessor._inferenceresults.inferenceresults)
InferenceResult class implementation for multi-label classification results type
## image\_overlay
`degirum.postprocessor._MultiLabelClassificationResults.MultiLabelClassificationResults.image_overlay`
`property`
Image with AI inference results drawn. Image type is defined by the selected graphical backend. Each time this property is accessed, new overlay image object is created and all overlay details are redrawn according to the current settings of overlay\_\*\*\* properties.
## overlay\_show\_labels\_below
`degirum.postprocessor._MultiLabelClassificationResults.MultiLabelClassificationResults.overlay_show_labels_below`
`property` `writable`
Specifies if overlay labels should be drawn below the image or on image itself
## \_\_str\_\_
`degirum.postprocessor._MultiLabelClassificationResults.MultiLabelClassificationResults.__str__()`
Convert inference results to string
## supported\_types
`degirum.postprocessor._MultiLabelClassificationResults.MultiLabelClassificationResults.supported_types()`
`staticmethod`
List supported result types for this postprocessor.
## degirum.postprocessor.\_SegmentationResults.SegmentationResults
Bases: [InferenceResults](#degirum.postprocessor._inferenceresults.inferenceresults)
InferenceResult class implementation for segmentation results type
## image\_overlay
`degirum.postprocessor._SegmentationResults.SegmentationResults.image_overlay`
`property`
Image with AI inference results drawn. Image type is defined by the selected graphical backend.
## \_\_str\_\_
`degirum.postprocessor._SegmentationResults.SegmentationResults.__str__()`
Convert inference results to string
## generate\_overlay\_color(num\_classes, ...)
`degirum.postprocessor._SegmentationResults.SegmentationResults.generate_overlay_color(num_classes, label_dict)`
`staticmethod`
Overlay colors generator.
Parameters:
| Name | Type | Description | Default |
| ------------- | ------ | --------------------------- | ---------- |
| `num_classes` | `int` | number of class categories. | *required* |
| `label_dict` | `dict` | Model labels dictionary. | *required* |
Returns:
| Type | Description |
| ------ | --------------------------------------------------- |
| `list` | general overlay color data for segmentation results |
## supported\_types
`degirum.postprocessor._SegmentationResults.SegmentationResults.supported_types()`
`staticmethod`
List supported result types for this postprocessor.
## degirum.postprocessor.\_Hand\_DetectionResults.Hand\_DetectionResults
Bases: [InferenceResults](#degirum.postprocessor._inferenceresults.inferenceresults)
InferenceResult class implementation for pose detection results type
## image\_overlay
`degirum.postprocessor._Hand_DetectionResults.Hand_DetectionResults.image_overlay`
`property`
Image with AI inference results drawn. Image type is defined by the selected graphical backend.
## \_\_str\_\_
`degirum.postprocessor._Hand_DetectionResults.Hand_DetectionResults.__str__()`
Convert inference results to string
## supported\_types
`degirum.postprocessor._Hand_DetectionResults.Hand_DetectionResults.supported_types()`
`staticmethod`
List supported result types for this postprocessor.
# AI Server Module
PySDK API Reference Guide. CLI launcher for the DeGirum AI server.
{% hint style="info" %}
This API Reference is based on PySDK 0.20.0.
{% endhint %}
DeGirum AI server launcher and model downloader.
{% hint style="info" %}
The functionality of this module is now exposed via PySDK CLI.
{% endhint %}
The purpose of this module is to start DeGirum AI server:
{% code overflow="wrap" %}
```
python -m degirum.server --zoo [--port ]
```
{% endcode %}
Other Parameters:
| Name | Type | Description |
| -------- | ----- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--zoo` | `str` |
Path to a local model zoo directory, containing AI models you want your AI server to serve.
One possible way to fill local model zoo directory is to download models from a model zoo repo using download\_models() function and provide the path to the local zoo directory as --zoo parameter.
|
| `--port` | `int` |
TCP port to bind server to.
Default is 8778.
|
* When AI server is started, it runs indefinitely.
* If you started the server from a terminal, you may press `Enter` to shut it down.
* If you started the server headless (for example, as a service), then to shut down the server you need to kill the Python process which runs the server.
The module also exposes `download_models()` function which can be used to prepare local model zoo directory to be served by AI server:
* You first download models from the model zoo repo of your choice into some local directory of your choice by calling [degirum.server.download\_models](#degirum.server.download_models) function.
* Then you start the AI server providing the path to that local directory as `--zoo` parameter.
## degirum.server.download\_models(path, \*, url=ZooManager.\_default\_cloud\_url, token='', \*\*kwargs)
Download all models from a model zoo repo specified by the URL.
Parameters:
| Name | Type | Description | Default |
| ------- | ----- | ------------------------------------------------------------------------------------------------------- | -------------------- |
| `path` | `str` | Local filesystem path to store models downloaded from a model zoo repo. | *required* |
| `url` | `str` |
Zoo repo URL.
If not specified, then DeGirum public model zoo URL will be used.
When you pass a string, it will be used as search substring in the model name. For example, "yolo", "mobilenet".
You may also pass re.Pattern. In this case it will do regular expression pattern search.
|
| `device` | `str` |
Target inference device -- string or list of strings of device names.
If passed, only models targeting the specified device(s) will be downloaded.
|
| `precision` | `str` |
Model calculation precision -- string or list of strings of model precision labels.
Possible labels: "quant", "float".
|
| `pruned` | `str` |
Model density -- string or list of strings of model density labels.
Possible labels: "dense", "pruned".
|
| `runtime` | `str` |
Runtime agent type -- string or list of strings of runtime agent types.
Possible types: "n2x", "tflite", "tensorrt", "openvino".
|
# Miscellaneous Modules
PySDK API Reference Guide. Console logging, verbosity control and helper exceptions.
{% hint style="info" %}
This API Reference is based on PySDK 0.20.0.
{% endhint %}
## degirum.log.DGLog
Console logging class with programmable verbosity.
## print(message)
`degirum.log.DGLog.print(message)`
`staticmethod`
Print message to log according to current verbosity level.
Parameters:
| Name | Type | Description | Default |
| --------- | ----- | ------------------------ | ---------- |
| `message` | `str` | Message string to print. | *required* |
## set\_verbose\_state(state)
`degirum.log.DGLog.set_verbose_state(state)`
`staticmethod`
Set log verbosity state.
Parameters:
| Name | Type | Description | Default |
| ------- | ------ | ------------------------------------------------------------------------------ | ---------- |
| `state` | `bool` | If `True`, then log prints messages to console, otherwise no messages printed. | *required* |
## degirum.log.async\_log\_wrap(f=None, \*, log\_level=logging.DEBUG)
{% code overflow="wrap" %}
```
async_log_wrap(f: Callable[P, Awaitable[R]]) -> Callable[P, Awaitable[R]]
```
{% endcode %}
{% code overflow="wrap" %}
```
async_log_wrap(*, log_level: int = logging.DEBUG) -> Callable[[Callable[P, Awaitable[R]]], Callable[P, Awaitable[R]]]
```
{% endcode %}
Decorator to log async function entry and exit with execution time.
Parameters:
| Name | Type | Description | Default |
| ----------- | ---------- | --------------------------------- | ------- |
| `f` | `Callable` | Async function to log. | `None` |
| `log_level` | `int` | Logging level of the log entries. | `DEBUG` |
## degirum.log.log\_wrap(f=None, \*, log\_level=logging.DEBUG)
{% code overflow="wrap" %}
```
log_wrap(f: Callable[P, R]) -> Callable[P, R]
```
{% endcode %}
{% code overflow="wrap" %}
```
log_wrap(*, log_level: int = logging.DEBUG) -> Callable[[Callable[P, R]], Callable[P, R]]
```
{% endcode %}
Decorator to log function entry and exit with execution time.
Parameters:
| Name | Type | Description | Default |
| ----------- | ---------- | --------------------------------- | ------- |
| `f` | `Callable` | Sync function to log | `None` |
| `log_level` | `int` | Logging level of the log entries. | `DEBUG` |
## degirum.exceptions.DegirumException
Bases: `Exception`
Base type for all DeGirum exceptions.
## degirum.exceptions.validate\_color\_tuple(color)
Validate if color has acceptable representation.
Parameters:
| Name | Type | Description | Default |
| ------- | ----- | ------------------------- | ---------- |
| `color` | `Any` | Color object to validate. | *required* |
Raises:
| Type | Description |
| ---------------------------------------------------------- | ------------------------------------------------------------------------- |
| [`DegirumException`](#degirum.exceptions.DegirumException) | if color is not a three-element tuple and each element is integer number. |
Returns:
| Type | Description |
| ------- | ---------------------------------- |
| `tuple` | color sequence converted to tuple. |
# Older PySDK User Guides
Index of older PySDK user guides.
# Release Notes
This page features release notes for releases of PySDK. You may download PySDK versions listed here from PyPI.org.
### Version 1.4.0 (8/11/2026)
**New Features and Modifications**
PySDK end-of-life release: license checking has been removed and all plugins are made available to all users. This is the final public release of this software.
### Version 1.3.1 (6/2/2026)
**New Features and Modifications**
RockChip RKNN runtime agent performance has been improved by introducing asynchronous pipelined inference and model context caching.
Across streaming AI inference workloads using models from the DeGirum public zoo, the average performance gain is 3.2x (+216%).
When the required model context is already cached, model switching latency is reduced from tens of milliseconds to effectively zero, enabling concurrent multi-model inference with almost no switching overhead.
**Bug Fixes**
On x86-64 systems without AVX2 support, launching AI Server with MemryX runtime installed causes the process to terminate with `Illegal instruction (core dumped)` error message.
***
### Version 1.3.0 (4/24/2026)
**New Features and Modifications**
1. Added support for MemryX runtime version 2.2.
2. MemryX runtime now operates in MXA Manager mode. The MXA Manager service (or `mxa_manager` executable) must be up and running to use PySDK with the MemryX accelerator.
3. Reintroduced support for Ubuntu 20.04 on ARM64 platforms. This also enables using PySDK with Yocto-built Linux distributions whose glibc is version 2.31 or later.
4. Hardware-side batching has been added to the TensorRT plugin. This feature can improve performance on powerful hardware by grouping multiple frames into a single inference executed on the device. In some cases, sufficiently powerful hardware can process a batch of frames nearly as quickly as a single frame, resulting in higher overall throughput.
To use hardware-side batching, the model’s ONNX input must support a dynamic batch dimension. This means the first dimension (batch size) should be set to -1. For example:
* Non-batched input shape: \[1, 3, 512, 512]
* Batched input shape: \[-1, 3, 512, 512]
You must also specify the batch size for execution. This can be done in one of two ways:
* In the model’s JSON configuration: add the DeviceBatch parameter under the ExtraDeviceParams section within DEVICE.
* At runtime: set the batch size programmatically using `model.extra_device_params.DeviceBatch = `
5. OpenVINO runtime agent performance is improved due to implementation of asynchronous pipelined inference and improved model storage and request management.
6. OpenVINO runtime agent: added support for additional OpenVINO tensor element types: `uint16`, `int16`, and `double`.
7. OpenVINO runtime agent: the following extra device parameters are supported:
* `OPENVINO_ENABLE_CPU_PINNING`: enables or disables CPU pinning. Disabling it can help overall throughput when several models or workloads run in parallel on the same host.
* `OPENVINO_INFERENCE_NUM_THREADS`: sets the maximum number of CPU threads OpenVINO may use for inference work. This is useful when limiting per-model CPU consumption in multi-model pipelines.
* `OPENVINO_NUM_STREAMS`: sets the number of parallel execution streams OpenVINO uses. More streams can improve throughput for some workloads, while fewer streams can reduce contention in shared systems.
* `OPENVINO_DENORMALS_OPTIMIZATION`: enables denormal flushing on CPU. This can improve performance in some cases, but it may slightly affect numerical accuracy. To assign such parameter, use `model.extra_device_params.KEY = value` syntax, for example, `model.extra_device_params.OPENVINO_INFERENCE_NUM_THREADS = 2`.
These parameters are particularly useful in multi-model pipeline scenarios, where limiting threads or streams per model can improve total system throughput rather than maximizing a single model in isolation. The defaults are optimal for performance in a single model scenario. For more information on each parameter, please refer to the following documents:
*
*
* .
8. Added Windows support for `degirum install-runtime` command, allowing ONNX Runtime and OpenVINO installation via degirum CLI. For example, to install OpenVINO runtime, execute the following command: `degirum install-runtime openvino`. If Windows reports access denied error, re-run the command from an Administrator PowerShell or Administrator Command Prompt. The installers may need to write into shared locations such as `C:\ProgramData` or `C:\Program Files`.
9. You can now integrate custom or third-party inference engines directly into the DeGirum PySDK using Python.
**New Public API**
The custom engine integration involves three components that work together:
* **`ExtZooAccessorBase`** — manages the model zoo and creates model instances.
* **`ExtModelBase`** — wraps a single model and manages its lifecycle via `_predict_handler`.
* **Runtime class** (yours) — performs the actual inference in its `predict()` method.
You register the whole stack under an `@`-prefixed name using `register_zoo_accessor`, then use that name as `inference_host_address` throughout PySDK.
**Function** `degirum.register_zoo_accessor(designator, accessor_class)`
Registers a custom inference engine under a short @-prefixed name. Once registered, use that name everywhere you would normally pass `inference_host_address`. Any call to `degirum.connect(designator, ...)` or `degirum.load_model(..., designator, ...)` will route inference through your custom accessor class instead of the built-in cloud or local engines. This lets you plug third-party inference runtimes into the standard PySDK workflow without modifying the SDK itself. To unregister your custom inference engine, call `degirum.register_zoo_accessor(designator, None)`.
| Parameter | Description |
| ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `designator` | A string starting with `@` (e.g. `"@myengine"`) that identifies the custom engine. Pass this string as `inference_host_address` to `degirum.connect()` or `degirum.load_model()`. |
| `accessor_class` | A class derived from `degirum.ExtZooAccessorBase` (see below). Pass `None` to unregister the designator. |
**Class** `degirum.zoo_manager.ExtZooAccessorBase`
Subclass this class to implement your custom inference engine. You only need to implement a subclass constructor with the following signature: `__init__(self, url: str, token: str = "")`. It should call the base class constructor with proper arguments as described below.
**Constructor** `ExtZooAccessorBase(url, token="", *, devices, model_class, assets_mgr=LocalZooAssets)`
| Parameter | Description |
| ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `url` | Zoo URL or local path (as passed from `degirum.connect()` as `zoo_url` argument). |
| `token` | Optional authentication token (as passed from `degirum.connect()` as `token` argument). |
| `devices` | Mapping of `"RUNTIME/DEVICE"` runtime/device designator strings to the number of available devices of that type (e.g. `{"ONNX/CPU": 1}`). |
| `model_class` | Your `ExtModelBase` subclass used to create model instances. See below. |
| `assets_mgr` | Optional custom asset-manager class (defaults to `LocalZooAssets`, suitable to handle local model zoos: collections of model assets in a local directory). |
**Class** `degirum._ext_zoo_accessor.ExtModelBase`
Subclass this class to implement the model inference logic.
**Abstract method to implement:** `_predict_handler(self)` — a context manager that sets up and tears down the inference runtime.
Implementations should:
* Lazily create a **runtime object** and assign it to `self._runtime` on the first call, or when `self._model_parameters.dirty` is set.
* Pass `self._model_parameters` and `self._result_callback` to the runtime constructor.
* On teardown, set `self._runtime.callback` to `None` to break reference cycles.
The runtime object must expose a `predict(frame_data, frame_info)` method (called by the base `Model` pipeline for each input frame) and a `callback` attribute (to which `self._result_callback` is assigned). See the **Runtime class** section below.
Typical implementation of `_predict_handler`:
```
@contextmanager
def _predict_handler(self):
if self._runtime is None or self._model_parameters.dirty:
self._runtime = MyRuntime(self._model_parameters, self._result_callback)
try:
yield
finally:
self._runtime.callback = None # break reference cycle
```
**Runtime class**
The runtime class, assigned to `self._runtime` in `_predict_handler`, is where inference is actually performed. It must implement a constructor and a `predict` method, and expose a `callback` attribute.
**Constructor** `__init__(self, model_params, callback)`
| Parameter | Description |
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `model_params` | Parameters object read from the model's JSON configuration file. Use it to query the settings needed for inference. |
| `callback` | The `_result_callback(result)` method of the parent model class. Call it to asynchronously report inference results. You must also assign it to the `self.callback` attribute. |
You will also need to create preprocessor and postprocessor objects to use them in the `predict()` method.
Typical implementation (based on ONNX runtime):
```
def __init__(self, model_params, callback):
from degirum.CoreClient import Preprocess, Postprocess
self.callback = callback
self._preprocess = Preprocess(model_params)
self._postprocess = Postprocess(model_params)
self._session = ort.InferenceSession(model_params.ModelPath)
```
**Attribute** `callback`
Assign the `callback` constructor argument to this attribute in the constructor: `self.callback = callback`. Use it to report inference results. The parent model object also uses this attribute to break the circular reference between itself and the runtime object when inference completes.
**Method** `predict(self, frame_data, frame_info: str)`
This method is the main method of the class, where the inference is performed.
| Parameter | Description |
| ------------ | ------------------------------------------------------------------- |
| `frame_data` | A list of `memoryview` input tensors, one tensor per model input. |
| `frame_info` | Frame info string as passed from the top level. Typically not used. |
It receives a list of raw frames (one per model input), already resized according to model parameters. Each frame is a `memoryview` object wrapping a numpy array of input tensor data.
Processing steps:
1. **Preprocess** — use the core-level preprocessor object (created in the constructor) to convert input tensor data to the raw binary format the model expects.
2. **Run inference** — execute your custom, runtime-specific inference code.
3. **Postprocess** — convert the raw output tensors to human-readable JSON. If your model's output tensors are compatible with one of the PySDK core-level postprocessors, you can call `self._postprocess.forward()` for this step.
4. **Deliver the result** — call `self.callback` with the JSON result.
Typical implementation (based on ONNX runtime):
```
def predict(self, frame_data, frame_info: str):
# apply core-level preprocessing to the input frame data
preprocessed = self._preprocess.forward(frame_data)
# run inference
outputs = self._session.run(
None,
{
inp.name: arr
for inp, arr in zip(self._session.get_inputs(), preprocessed)
},
)
# apply core-level postprocessing to the inference outputs
result = self._postprocess.forward(outputs)
# deliver result via callback
self.callback(result)
```
If your inference runtime supports asynchronous execution, `predict()` can be implemented in a non-blocking way: start the inference and return immediately. Once inference completes, perform the postprocessing step and deliver the result via `self.callback`. How you detect completion depends on the runtime — it may use callbacks, polling, events, or any other mechanism the runtime provides.
**Bug Fixes**
1. OpenVINO runtime agent: an error `"Wrong value f64 for property key INFERENCE_PRECISION_HINT"` is reported when performing inference on Windows systems. This is due to ABI mismatch between internal type enum for versions 2025.3 and 2023.3.
2. OpenVINO runtime agent: on systems with more than one OPENVINO/GPU device inference may run on a device other than selected in `model.devices_selected` list.
***
### Version 1.2.1 (3/19/2026)
**New Features and Modifications**
1. DEEPX runtime version 3.2.0 is supported. Corresponding driver version should be 2.1.0-2.
**Bug Fixes**
1. "Python postprocessor: configure\_worker: wrong worker\_id" error occurs intermittently when running models with Python postprocessor.
2. `degirum download-zoo` CLI command does not honor token installed by `degirum token install` or `degirum token create`.
***
### Version 1.2.0 (3/6/2026)
**New Features and Modifications**
1. `update_if_newer` property is added to `degirum.model._ClientModel` class. This class is returned by `degirum.load_model` function when `@local` inference is requested. This property affects how cloud models are handled by local model cache in case of `@local` inference. When the cloud model is requested for the inference and this property is set to `True`, the local model cache always queries the cloud zoo for the model checksum and downloads the model from the cloud zoo to the local cache if checksums mismatch. When set to `False` (this is default setting) the cloud model is downloaded once and the model checksum is never queried. Please note that in previous releases model checksums are always queried, so the default behavior is changed.
**Bug Fixes**
1. When `input_shape` model property is set to the same value, model parameters get invalidated causing model reload. Model parameters invalidation and model reload must happen only when a model parameter is changed to some new value.
2. DeepX runtime agent: when a model compiled with older version of DeepX model compiler is loaded for inference, inference hangs in DeepX runtime. Now the model version is validated against supported versions and the error message is generated in case of not supported version.
3. Fixed bugs in Audio pre-processor for YAMNET models.
***
### Version 1.1.0 (2/17/2026)
**New Features and Modifications**
1. ONNX runtime plugin is redesigned to improve inference performance.
2. DEEPX runtime version 3.1.0 is supported.
3. `--token` parameter is added to `degirum server start` command to be able to install token prior to server launch to enable proper license activation.
4. `degirum machine-id` CLI command is added to print host machine-id
5. `degirum token create` command is modified to generate meaningful token description like `Created by PySDK for @ ()`
6. `degirum download-llm` CLI command is added to download and install LLM models from DeGirum model zoo for OpenVino GenAI plugin.
**Bug Fixes**
1. Automatic renewal of expired token is now performed prior to the PySDK license verification. The absence of such automatic renewal caused license verification error in case when installed token has finite duration and is expired.
2. Proper import of `pyseccomp` is implemented in Python post-processor engine: plain `import pyseccomp` fails on some systems causing `ModuleNotFoundError: No module named 'pyseccomp'` error.
### Version 1.0.0 (1/26/2026)
**This is the first production release of PySDK.**
> :warning: IMPORTANT:
We have updated the PySDK release distribution process. Pre-production releases (prior to ver. 1.0.0) were published on PyPI (pypi.org). Production releases are distributed through the [DeGirum Package Service](https://pkg.degirum.com) **and** pypi.org.
To install the production version of PySDK from DeGirum Package Service, use the following command:
```
pip install -i https://pkg.degirum.com degirum
```
> :warning: IMPORTANT:
Starting from ver. 1.0.0, the usage of premium runtime plugins requires license. The license is obtained automatically by PySDK from DeGirum AI Hub if you have AI Hub token installed on your system. Once requested, the plugin license is stored locally and automatically renewed on expiration. Default expiration period is 10 days. The license is node-locked.
The Free plan allows you to use PySDK premium runtimes on **one host**. If you need to use PySDK premium runtimes on more than one host, you need to upgrade your AI Hub workspace to [Professional or Enterprise plans](https://degirum.com/pricing).
To install existing AI Hub token, you run degirum CLI command: `degirum token install ` where `` is the AI Hub token string which you generate on [AI Hub](https://docs.degirum.com/ai-hub/workspaces/workspace-tokens).
To create new token and install it, you run degirum CLI command: `degirum token create`. If you run this command on a system having graphical desktop, it will open token generation page in your default browser for you. Otherwise it will print URL which you need to paste in any browser.
To upgrade your plan follow [these instructions](https://docs.degirum.com/ai-hub/workspace-plans#upgrade-your-workspace-plan).
The premium plugins include:
* Akida (Brainchip)
* Axelera
* DeepX
* EdgeCortix
* Hailo
*
* ONNX
* OpenVINO (Intel)
* Renesas
* RKNN (RockChip)
* TensorRT (nVidia)
Free plugins include DeGirum N2X Orca and Google TFLite.
**New Features and Modifications**
1. YOLO26 models are initially supported by PySDK.
2. Axelera runtime version 1.5.3 is supported. Please note that models compiled by Axelera compiler ver. prior to 1.5.x are not compatible with 1.5.x.
3. Double-buffering and DMA buffers are supported for Axelera runtime. This is performance optimization.
4. `degirum token install` CLI command now accepts token string without `--token` keyword.
Before: `degirum token install --token `
Now: `degirum token install `
**Bug Fixes**
1. `PythonFile` model parameter is not assigned properly on model loading: the value from model JSON file always overwrites the value assigned in run-time.
***
### Version 0.20.0 (12/16/2025)
**New Features and Modifications**
1. Minimum supported Ubuntu version for ARM64 platforms is now 22.04 (bumped from 20.04). For x86-64 platforms it is still 20.04.
2. Python 3.13 is supported.
3. Renesas AI accelerators are initially supported for Linux OS. The runtime/device designator for these devices is `"RENESAS/RZ-V2*"` where `*` is the particular device model (L, M, MA, H, and N). The supported runtime version is 2.5.1.
4. `model.predict_batch()` latency is improved. Previous versions yield only one ready result for each consumed input frame. This led to accumulated latency which never reduced. With new change the code yields results until no more ready results are available.
5. Audio pre-processor quality and performance for Whisper models is improved.
6. Cloud model zoo cache is now shared between all processes which use PySDK on a given system. In previous versions for `@local` inferences temporary non-persistent cloud zoo cache was created for each process.
7. `PythonFile` model parameter is now can be modified in runtime. For example, to temporary disable Python post-processor assign `model._model_parameters.PythonFile = ""`
8. Startup time of Python postprocessor worker processes is significantly improved due to parallelization.
9. TensorRT runtime: added JIT compilation of pre-quantized ONNX models.
10. TensorRT runtime: implemented loading .engine files supplied by the model JSON. This bypasses the JIT entirely.
11. TensorRT runtime: new `extra_device_parameters` are supported for TensorRT runtime models: `UseFP16` and `UseINT8`, You may set them by assigning `model.extra_device_params.UseFP16` and `model.extra_device_params.UseINT8` respectively.
12. Token creation procedure via `degirum token create` is improved for headless systems (systems without GUI browser).
13. Empty zoo\_url for `@local` inference type is now treated as default public cloud zoo.
14. `degirum.get_supported_devices()`: `zoo_url` and `token` parameters are deprecated as extraneous - you may now omit them.
**Bug Fixes**
1. Timing information was stripped from inference results only **after** creation of inference result object, which exposed that timing data to custom post-processor constructors. This led to single fake result containing only timing information to appear when no actual results are reported by the model. Now it is stripped from inference results before that.
2. YOLO detection result processors incorrectly quantized confidence threshold when output score tensor quantization parameters produce out-of range results. Now clamping is applied to prevent wrap over.
3. Proper detection of device count is implemented for Windows platform. In previous versions it was just hardcoded to one device.
***
### Version 0.19.2 (11/01/2025)
> ATTENTION: this release contains critical bug fixes. We strongly recommend to upgrade to this release from 0.19.0 and 0.19.1 versions.
**Bug Fixes**
1. **Critical bug fix:** YOLOv8 pose detection and object detection+segmentation models produce incorrect results.
2. **Critical bug fix:** when Numpy package version 2.0 or newer is installed, then all models reporting raw tensors like ReID embedding models, pure segmentation models, or models with "None" postprocessor type produce incorrect results. This bug is due to incompatibility of pybind11 library ver. 2.10.3 using in PuSDK build and Numpy ver 2.0 and above.
3. When Hailo runtime ver. 4.20.1 is installed and Hailo multiprocess service is **not** running, then PySDK produces HAILO\_INVALID\_OPERATION error.
4. When running inferences of models for runtime, switching models causes segmentation fault.
***
### Version 0.19.1 (10/28/2025)
> ATTENTION: this release has critical bugs. Please upgrade to newer release 0.19.2 or above!
**New Features and Modifications**
1. Improved performance of Detection post-processors, especially on ARM hosts.
2. Audio preprocessor improvements: now supports different tensor layouts and input dimensions for Whisper preprocessing.
3. HailoRT versions 4.23.0 and 4.20.1 are supported.
4. MemryX runtime version 2.0 is supported.
**Bug Fixes**
1. Implemented a timeout bypass mechanism for the Axelera runtime plugin to prevent indefinite stalling when the Axelera API fails.
2. Incorrect inference results appear intermittently when performing streaming predictions using TensorRT plugin due to race condition in TensorRT plugin implementation.
***
### Version 0.19.0 (10/13/2025)
> ATTENTION: this release has critical bugs. Please upgrade to newer release 0.19.2 or above!
**New Features and Modifications**
1. [EdgeCortix](https://www.edgecortix.com/en/) AI accelerators are initially supported for Linux OS. The runtime/device designator for these devices is `"EDGECORTIX/SAKURA2"`.
2. Axelera runtime version 1.4.1 is supported.
3. Error handling in Axelera runtime plugin is improved:
* Waits with infinite timeouts are replaced with finite.
* Errors in inference stream no longer disable device operations.
4. PySDK image pre-processor throughput is significantly improved by reducing the amount of double-buffering along the processing pipeline. This improvement is more significant for slower hosts.
5. Introduced a new model parameter: `ExtraDeviceParams`. This parameter is used for hardware-specific configurations.
* Inside the Model JSON, it is a JSON object under the key `ExtraDeviceParams` in `DEVICE` section.
* Is is exposed as `degirum.model.Model.extra_device_params` property in PySDK for setting these parameters.
* Use `model.extra_device_params.KEY = VALUE` syntax for setting these parameters.
6. Hailo Runtime Agent no longer uses `degirum.model.Model.eager_batch_size` property for internal Hailo batching control. Instead, now it uses `HAILO_BATCH_SIZE` inside the model's `ExtraDeviceParams`. Now, to set a model's batch size for Hailo devices, use `model.extra_device_params.HAILO_BATCH_SIZE = `
7. `numpy` package maximum version limitation in PySDK package requirements is relaxed to `< 3.0`.
8. `degirum.model.Model.model.output_class_set` and `degirum.model.Model.model.overlay_blur` property setters now accept list, set, or string arguments interchangeably. Property getters now return lists.
9. Now `degirum.get_supported_devices()` function requires only one argument: `inference_host_address`. There is no need to pass `zoo_url` and `token` parameters anymore but they left for backward-compatibility.
10. `degirum.model.Model.model.devices_selected` property is now updated when `degirum.model.Model.model.device_type` property changes so the selected device list remains valid for newly selected device type. Particularly, when `devices_selected` list becomes empty after device type change, it is automatically assigned with `devices_available` value.
**Bug Fixes**
1. `"Tensor"` preprocessor type now allows data types other than `DG_FLT` or `DG_UINT8` to pass through unmodified.
2. ONNX runtime agent now correctly supports conversion for all input tensor data types, not only `DG_FLT` or `DG_UINT8`.
3. Segmentation results renderer fails on bounding boxes with zero area in mask resize.
***
### Version 0.18.3 (09/19/2025)
**New Features and Modifications**
1. Axelera runtime version 1.4 is supported.
2. OpenVINO GenAI runtime agent is initially supported. This runtime agent allows running inferences of LLM models in PySDK using OpenVINO GenAI runtime.
3. Integrated Intel GPU is now available as inference device for OPENVINO runtime, when it is the only GPU in the system (if you have both discrete and integrated GPUs, the integrated GPU is still not available). The device designator is "OPENVINO/GPU" as for discrete GPU.
4. Timeouts are implemented for all AI server commands. Before that modification, some commands, like system information request, may wait indefinitely for AI server response when AI server process already exited for whatever reason.
5. The `degirum install-runtime` CLI commands have been updated to better support operation in root environments.
6. The performance of rendering of segmentation results is greatly improved. The memory consumption to store segmentation masks is also reduced.
7. Tensor Preprocessor: quantization/dequantization functionality is implemented. When input tensor data type (as specified by `InputRawDataType` model parameter) does not match model input data type (as deduced from `InputQuantEn` model parameter), either quantization or dequantization of the input tensor happens. When `InputRawDataType` is DG\_UINT8 and `InputQuantEn` is false, then dequantization is performed. When `InputRawDataType` is DG\_FLT and `InputQuantEn` is true, then quantization is performed. Quantization/dequantization parameters are specified by `InputQuantScale` and `InputQuantOffset` model parameters.
**Bug Fixes**
1. Cloud inference started from a separate thread may produce the error `"got Future attached to a different loop"`.
2. The `degirum server cache-dump` command now always outputs valid JSON.
3. Floating point overflow in bounding box coordinate decoding computation is fixed in YOLOv8 Detection post-processor.
***
### Version 0.18.2 (08/12/2025)
**New Features and Modifications**
Axelera runtime is now supported on ARM64 platforms.
***
### Version 0.18.1 (08/11/2025)
**New Features and Modifications**
1. New post-processor types are supported:
* `Null` post-processor. This post-processor effectively disables all post-processing, which is useful for model bechmarking. The post-processor tag is `"Null"`.
* `Dequantization` post-processor. This post-processor performs dequantization of all integer-type output tensor results (compare it with `None` post-processor, which just returns all output tensor results as-is). The post-processor tag is `"Dequantization"`.
2. Verbosity of `degirum token` CLI commands is improved. Now all commands produce some confirmation message on completion.
**Bug Fixes**
1. When cloud API token with unlimited lifetime is installed in PySDK using `degirum token install` CLI command, the following `dg.connect()` call raises error `"Invalid isoformat string"`.
***
### Version 0.18.0 (08/05/2025)
**New Features and Modifications**
1. `degirum token` command tree has been added to PySDK CLI. Using these commands you can manage cloud access API tokens in PySDK. You can install existing token into PySDK internal storage, you can create and install new token, you can query the status of the currently installed token, you can renew currently installed token, when it is expired, and you can delete currently installed token from the PySDK internal storage. The PySDK internal storage is arranged as JSON file in user-specific directory: %APPDATA%\DeGirum for Windows and \~/.local/share/DeGirum for Linux/MacOS. The following commands are supported:
* `degirum token status`: prints the status of the currently installed token. If Internet connection is available, it queries the most current token status from the DeGirum AI Hub otherwise it prints latest known token status.
* `degirum token install --token `: installs the provided token into the PySDK internal storage. You need to obtain the token string from DeGirum AI Hub Web GUI.
* `degirum token create`: opens automatic token creation URL in your default browser (when available) or prints this URL into console. Using provided URL you need to authenticate in DeGirum AI Hub. Meanwhile, PySDK CLI will wait until you open provided URL and authenticate in DeGirum AI Hub. Once authenticated, the new token with 2-week expiration will be created automatically, and PySDK CLI will pick it up and install into PySDK internal storage.
* `degirum token renew`: renews currently installed token. Renewal happens even if the token is already expired. Renewed token is then installed into the PySDK internal storage replacing the currently installed token.
* `degirum token clear`: removes the currently installed token from the PySDK internal storage. It does not delete the token from the AI Hub.
2. `degirum.connect` function now tries to use currently installed token, if no token is provided as `token` argument (see `degirum token` CLI description above). It will try to renew this token automatically if it is about to be expired or already expired. This feature is intended to simplify the token handling in PySDK, when you install the time-limited token once, using PySDK CLI, and then use it indefinitely on this system from all your scripts.
3. `degirum install-runtime` command tree has been added to PySDK CLI. Using these commands you can install third-party AI accelerator runtime libraries on your system greatly simplifying system housekeeping. The following commands are supported:
* `degirum install-runtime --list`: list all runtimes available for installation. Using this command you may determine arguments for the following command.
* `degirum install-runtime `: install particular version of particular runtime. You may specify `ALL` instead of the version number to install all supported versions of that runtime.
* The following runtimes are supported for Linux Ubuntu/Debian OS in this release (more runtimes will be supported in the future releases):
| Designator | Runtime | Versions |
| ---------- | ----------------------- | ---------------------------- |
| `akida` | Brainchip Akida runtime | 2.11.0 |
| `axelera` | Axelera Voyager runtime | 1.3.3 |
| `memryx` | MemryX runtime | 1.2.3-1 |
| `onnx` | ONNX runtime | 1.20.1 |
| `openvino` | Intel OpenVINO runtime | 2024.6.0, 2024.2.0, 2023.3.0 |
| `rknn` | Rockchip RKNN runtime | 2.3.0 |
4. HAILO runtime agent now supports HAILORT runtime version 4.22.
5. Axelera runtime agent now supports Voyager SDK runtime version 1.3.3.
6. Preprocessing functionality for Whisper models is implemented for audio preprocessor. The `InputType` model parameter value for such models is `Audio`. The audio preprocessor distinguishes Whisper models by combination of `InputFrameSize` equal to 400 and `InputFrameHopStepSize` equal to 160.
7. Now `degirum.connect()` checks the cloud token and cloud zoo validity for non-public zoo access. If you pass an empty or incorrect token, you will get an error `"Unable to connect to server hub.degirum.com: your cloud API access token is not valid"`. If you pass correct token, but the cloud zoo does not exist or not accessible to you, the you will get an error `"Cloud model zoo '/' either does not exist or you do not have access to it."`
8. When calling `degirum.connect(degirum.LOCAL)` function to connect for local inference with an empty string passed as `zoo_url` argument, the error `"ZooManager: incorrect local model zoo URL"` is raised. In previous releases empty `zoo_url` argument led to local zoo selection in the current working directory causing a lot of confusion.
**Bug Fixes**
1. Multiple bug fixes of intermittent failures of ORCA USB accelerator inferences.
***
### Version 0.17.3 (07/15/2025)
**Bug Fixes**
Automatic reconnect on critical error functionality in the cloud inference protocol has been broken due to recent changes in this protocol made in 0.17.2 release.
***
### Version 0.17.2 (07/14/2025)
**New Features and Modifications**
1. [Axelera](https://axelera.ai) AI accelerators are initially supported for Linux OS. The runtime/device designator for these devices is `"AXELERA/METIS"`.
2. DEEPX runtime inference performance is improved by avoiding extra memory copy of inference result tensors.
**Bug Fixes**
1. Frequent websocket disconnects happen during streaming cloud inferences in PySDK, significantly reducing streaming frame rate. This bug was due to the race condition in the websocket-client package used by python-socketio synchronous client. Now PySDK uses python-socketio asynchronous client, which does not use websocket-client package.
***
### Version 0.17.1 (06/25/2025)
**New Features and Modifications**
1. ONNX runtime agent now supports ONNX runtime version 1.20.1.
**Bug Fixes**
1. Hailo models with hardware accelerated NMS layer did not work with PySDK and HailoRT ver. 4.21. Now this bug is fixed.
2. Once the model is loaded by Hailo agent, it remembers model batch size and any attempt to change the batch size of already loaded model has no effect.
***
### Version 0.17.0 (06/20/2025)
**Known Issues**
> Hailo models with hardware accelerated NMS layer do not work with PySDK and HailoRT ver. 4.21. We recommend to use HailoRT ver. 4.20 for such models until the PySDK fix will be released in the next PySDK version.
**New Features and Modifications**
1. Batching is supported for HailoRT runtime. You control the batch size via `degirum.model.Model.eager_batch_size` property. For single-context models (models which fully fit into accelerator memory) the Hailo runtime batch size is set to auto to improve performance, because such models do not benefit from batching. For multi-context models which do benefit from batching the Hailo runtime batch size is set equal to `degirum.model.Model.eager_batch_size`.
2. DEEPX runtime version 2.9.5 is supported. This version supports production version of DEEPX M.2 accelerator card.
3. Multi-device support was enabled for DEEPX runtime. Now PySDK can operate with multiple DEEPX M.2 accelerator cards.
4. Models with built-in NMS layer are supported for HailoRT runtime.
5. Maximum supported model parameters configuration version is increased from 10 to 11.
6. The post-processor for DamoYolo models is implemented. The post-processor tag is `"DetectionDamoYolo"`.
7. Custom landmark shapes (other than 2) are supported in PoseDetectionYoloV8 post-processor. Now `landmark` key in detection results may contain coordinate list with more than 2 elements, for example `[x,y,score]`.
8. AI annotation renderers now support zero-size line width. You set it via `degirum.model.Model.overlay_line_width` property. When it is set to zero, no lines are drawn.
9. The URL parsing logic of `degirum.connect()` is made more reliable. Now in the case of local inference or AI server inference the zoo URL is first checked is it a cloud zoo URL. The URL is considered as cloud zoo URL if it starts with `http://` or `https://` scheme or it contains exactly one slash like `workspace/zoo`. Only if provided URL does not look like cloud zoo URL it is then checked is it local zoo URL.
* In the case of local inference it is checked is it a valid local path; and if it is not, then the error message `"incorrect local model zoo URL: path does not exist"` is raised. You may use explicit `file://` scheme for local paths.
* In the case of AI server inference it is considered as local model zoo URL if it is empty string or it starts with `aiserver://` scheme. Otherwise the error `"incorrect cloud model zoo URL"` is raised.
10. Runtime plugin loading whitelist is implemented. The list of runtime plugins allowed to load can be specified in DG\_PLUGINS\_ALLOWED environment variable using the following syntax: plugin prefixes separated by any non alphanumeric separator. For example: `DG_PLUGINS_ALLOWED=n2x_runtime_agent;onnx_runtime_agent`. When DG\_PLUGINS\_ALLOWED is not defined, all plugins will be loaded as before.
**Bug Fixes**
1. When there is no connection to the cloud zoo, the model from AI server model zoo cache can be loaded without token authorization.
***
### Version 0.16.2 (05/09/2025)
**New Features and Modifications**
1. C++ SDK example has been modified to support inference of models from cloud zoos.
2. HAILO runtime agent now supports HAILORT runtime version 4.21.
3. MEMRYX runtime agent is supported on Windows 10/11 OS.
4. TENSORRT runtime agent is supported on Windows 10/11 OS.
5. Dequantization post-processor is implemented. This post-processor performs dequantization of model output tensors according to model dequantization settings for those tensors. The post-processor tag is `"Dequantization"`.
6. New `InferenceResultsType` model parameter is added. This model parameter specifies objects of which PySDK result class (a class derived from `InferenceResults` base class) will be returned as inference results. When the `InferenceResultsType` model parameter is not defined then the `OutputPostprocessType` model parameter is used instead, as before. Please note that in previous releases the `OutputPostprocessType` model parameter was used for both selecting which Core-level post-processor is applied to the model output tensors and objects of what PySDK result class to return as inference results. Having separate parameter to select PySDK result class allows flexibility of matching different Core-level post-processors with different PySDK result classes. For example, "Dequantization" Core-level post-processor can be used with "Classification" PySDK result class.
7. `degirum.inference_results_type` model property is added to control `InferenceResultsType` model parameter in run-time.
8. `degirum.get_inference_results_class` method is added to return PySDK result class which objects are returned as inference results for this
9. `degirum postprocessor.register_postprocessor` function is added. It registers your own PySDK result class for a specific inference result type (as defined by `InferenceResultsType` model parameter, see above). This function accepts two arguments: inference result type string and PySDK result class. PySDK result class must inherit `degirum postprocessor.InferenceResults` base class. Once your class is registered, objects of this class will be returned as inference results for models with `InferenceResultsType` model parameter equal to the inference result type string you passed when calling `register_postprocessor` function. This allows developing new custom PySDK result classes and seamlessly integrating them into PySDK.
**Bug Fixes**
1. Various error messages appear when trying to do inference of more than 32 different models using HAILORT runtime agent on a single Hailo accelerator device.
2. Inference multiplexing on all available devices was not performed for the TFLITE and OPENVINO runtime agents: only single device with index 0 was always used. This bug was introduced in ver. 0.16.0.
***
### Version 0.16.1 (04/29/2025)
**New Features and Modifications**
MemryX runtime agent now supports MemryX runtime version 1.2.
**Bug Fixes**
TensorRT runtime agent failed to run inferences on DLA devices producing `"Cuda failure: invalid device ordinal"` error.
***
### Version 0.16.0 (04/17/2025)
**New Features and Modifications**
1. [DEEPX](https://deepx.ai) AI accelerators are initially supported for Linux OS. The runtime/device designator for these devices is `"DEEPX/M1A"`.
> NOTE: due to current limitations of DEEPX runtime the number of supported devices is limited to one.
2. Error handling is improved for Hailo runtime agent: now all errors reported by HailoRT runtime are treated as critical.
3. TensorRT runtime agent performance is improved due to implementation of asynchronous pipelined inference.
4. Hailo runtime agent is supported on Windows 10/11 OS.
5. `postprocess_type` keyword argument is added to `degirum.zoo_manager.ZooManager.list_models` method. It allows filtering models by post-processor type. Possible values for this argument are: `"Classification"`, `"Detection"`, `"Segmentation"` etc. They correspond to `OutputPostprocessType` model parameter values.
**Bug Fixes**
1. Oriented bounding box post-processor incorrectly interpreted rotation angle tensors for certain OBB models, which led to the error message `"Execution failed. Condition ' == ' is not met"`.
# EULA v1.0 and later
Review the DeGirum PySDK End-User License Agreement (EULA) for PySDK v1.0+ to understand the terms governing your use of DeGirum PySDK and any Application Packages designed to operate with it.
**IMPORTANT** - PLEASE READ THE TERMS AND CONDITIONS OF THIS LICENSE AGREEMENT CAREFULLY BEFORE USING THIS SOFTWARE.
This End-User License Agreement ("EULA") is a legal agreement between you, the "Licensee", (either an individual or a single entity) and DeGirum Corporation, a Delaware corporation (the "DeGirum Corporation"), governing your use of the DeGirum PySDK software ("Software"). By installing, copying, or otherwise using the Software, you agree to be bound by the terms of this EULA. If you do not agree to the terms of this EULA, do not install or use the Software.
**Software scope.** For purposes of this EULA, “Software” means the DeGirum PySDK software and any DeGirum-provided application packages, sample applications, extensions, plug-ins, or other components that are built using, distributed with, or designed to operate with PySDK (collectively, “Application Packages”), unless DeGirum provides different terms with a specific component.
**Version applicability.** This EULA applies to DeGirum PySDK version 1.0 and later. If you are using DeGirum PySDK version 0.20.0 or earlier, your use is governed by the [prior PySDK EULA](/pysdk/eula-v0.20.0-and-earlier) applicable to those versions.
## 1. GRANT OF LICENSE
Subject to full and ongoing compliance with this Agreement and payment of all applicable fees, DeGirum Corporation grants Licensee a non-exclusive, non-transferable, non-sublicensable, revocable, limited license to use the Software solely for internal development purposes and integration into Licensee’s products, as expressly authorized herein.
## 2. RESTRICTIONS
You agree not to, and you will not permit others to:
1. Sell, lease, rent, transfer, assign, sublicense, or otherwise distribute the Software without prior written consent from DeGirum Corporation;
2. Reverse engineer, decompile, disassemble, or attempt to drive the source code of the Software, except to the extent that expressly permitted by applicable law;
3. Use the Software for competitive analysis, benchmarking, or development and commercialization of competing products;
4. Use the Software in any manner that could damage, disable, overburden, impair or interfere with DeGirum Corporation’s services or interfere with any third party's use and enjoyment of the Software;
5. Use the Software for any unlawful purpose or in violation of any applicable laws or regulations;
6. Attempt to gain unauthorized access to the Software or its related system or networks; or
7. Export or re-export the Software in violation of any applicable U.S. or foreign export laws or regulations.
DeGirum Corporation reserves the right to monitor compliance with these restrictions and may terminate the license immediately upon any violation. DeGirum Corporation may also seek injunctive relief and pursue all available legal remedies for any breach of these restrictions.
## 3. OWNERSHIP
All rights, title, and interest in and to the Software, including all intellectual property rights therein, all copies, modifications, enhancements, updates, derivative works, and related materials are and will remain the exclusive property of DeGirum Corporation. Any rights not expressly granted to you are reserved by DeGirum Corporation. You agree not to challenge, contest, or otherwise impair DeGirum Corporation's ownership of the Software or the validity or enforceability of DeGirum Corporation's intellectual property rights related to the Software. Furthermore, you acknowledge that any feedback, suggestions, or ideas you provide regarding the Software may be used by DeGirum Corporation without any obligation to compensate you, and you hereby assign all rights in such feedback to DeGirum Corporation.
## 4. TERMINATION
This EULA is effective until terminated. Your rights under this EULA will terminate automatically without notice from DeGirum Corporation if you fail to comply with any term(s) of this EULA. Upon termination of this EULA for any reason:
1. all rights and licenses granted to you under this EULA shall immediately terminate;
2. you must immediately cease all use of the Software;
3. you must promptly destroy all copies, full or partial, of the Software in your possession or control.
Section 2, 3, 5, 6, 7, 8, 9, 10, 11 and 12 herein shall survive any termination of this EULA.
## 5. DATA ACCURACY & DISCLAIMER
The Software may transmit or process data. DeGirum makes no warranty as to the accuracy, completeness, timeliness, or reliability of any such data, and shall have no liability for delays, errors, omissions, interruptions, or losses of data.
## 6. THIRD-PARTY SOFTWARE
Any third-party software accessible through or included with the Software is provided “AS IS” and subject solely to the applicable third-party license. DeGirum disclaims all responsibility and liability for third-party products.
## 7. DATA, CLOUD SERVICES, AND CUSTOMER CONTENT
### 7.1 DeGirum Services
Certain features of the Software may interact with DeGirum-hosted services, including the DeGirum Hub available at hub.degirum.com (collectively, the “DeGirum Services”). By using the DeGirum Services, you authorize DeGirum Corporation to receive, access, host, process, transmit, and store data as described in this Section 7.
### 7.2 Customer Data stored in DeGirum databases
In connection with account creation, authentication, billing, support, and operation of the DeGirum Services, DeGirum Corporation may collect and store the following information in its databases (collectively, “Customer Data”):
* User email address.
* User Auth0 ID (Auth0 is DeGirum’s authentication provider).
* User ChargeBee ID (ChargeBee is DeGirum’s subscription and billing administration provider).
* Image size for each image passed for inference.
### 7.3 Customer Content stored in AWS S3
In connection with the DeGirum Services, DeGirum Corporation may store certain customer-provided files and artifacts in cloud storage, including Amazon Web Services (AWS) S3 (“Customer Content”), such as:
* Customer models (stored in AWS S3 buckets).
* For each unsuccessful compilation request: the model checkpoint file and all images passed for compilation (stored in AWS S3 buckets).
### 7.4 Purpose of processing
DeGirum Corporation will process Customer Data and Customer Content for the purposes of: (a) providing and operating the DeGirum Services and Software functionality; (b) authenticating users and preventing fraud or abuse; (c) administering subscriptions and billing; (d) providing customer support, troubleshooting, and service communications; and (e) maintaining, securing, and improving the DeGirum Services and Software (including debugging, performance monitoring, and reliability).
### 7.5 Third-party service providers
DeGirum Corporation may use third-party service providers to support the DeGirum Services, including Auth0 (authentication), ChargeBee (subscription and billing administration), and Amazon Web Services (AWS) (hosting and storage). These providers may process Customer Data and/or Customer Content on DeGirum Corporation’s behalf solely to provide their services.
### 7.6 Customer responsibilities; rights in Customer Content
You represent and warrant that you have all rights, permissions, and lawful basis necessary to provide Customer Content to DeGirum Corporation and to permit the processing described in this EULA (including any rights or consents needed to submit images or other materials that may contain personal information). As between you and DeGirum Corporation, you retain ownership of your Customer Content. You grant DeGirum Corporation a limited, worldwide, non-exclusive, royalty-free license to host, store, copy, transmit, and process Customer Content solely as necessary to provide, secure, support, and improve the DeGirum Services and Software as described in this EULA.
### 7.7 Security; retention; deletion
DeGirum Corporation will use reasonable administrative, technical, and organizational measures designed to protect Customer Data and Customer Content against unauthorized access, loss, misuse, or alteration.
Unsuccessful compilation retention: DeGirum Corporation will retain model checkpoint files and images submitted for compilation associated with an unsuccessful compilation request for up to thirty (30) days for troubleshooting, support, and service integrity purposes, unless a shorter period is required by law or you request earlier deletion where applicable.
DeGirum Corporation may retain certain information as required by law or as reasonably necessary to protect its rights, prevent fraud or abuse, maintain system integrity, and comply with backup, audit, and dispute-resolution requirements.
### 7.8 License verification & renewal
#### 7.8.1 License verification
The Software includes license enforcement features that verify license validity while the Software is in use. Verification is performed using a combination of (a) local checks and (b) periodic communications with our license services.
#### 7.8.2 Information used
To perform license verification and renewal, the Software may use and/or transmit the following:
1. a license token or entitlement identifier;
2. a machine identifier (device ID) used to associate the license with a specific device; and
3. the license renewal date and related license status metadata (e.g., active/expired).
“Machine identifier” means an identifier derived from device and/or operating system characteristics or generated by the Software and stored locally.
#### 7.8.3 Verification method & frequency
Local verification: During normal operation, the Software may validate license status using locally stored license information (for example, a local license file).
Periodic online verification and renewal: Approximately once every ten (10) days while the Software is in use, the Software will contact our servers (including AI Hub license services, as applicable) to verify and renew license status. Upon successful verification, the locally stored license information may be updated to reflect a refreshed renewal date.
#### 7.8.4 Consequences of failed verification
Because the Software does not provide an offline grace period, if the periodic online verification or renewal cannot be completed when required (including due to lack of connectivity or blocked communications), the Software may:
1. suspend operation,
2. restrict functionality,
3. require re-authentication or re-activation, and/or
4. treat the license as expired until verification succeeds.
#### 7.8.5 Limited purpose; no content collection
Information collected or transmitted under this section is used solely for license verification, renewal, fraud prevention, and related compliance purposes. The license verification process is not intended to collect the contents of your files or other customer data processed by the Software.
Additional details about how DeGirum collects and uses personal information are described in DeGirum Corporation’s [Privacy Policy](https://degirum.com/privacy-policy?utm_source=docs.degirum.com\&utm_medium=docs\&utm_campaign=pysdk-eula\&utm_content=sec-7-8-5).
### 7.9 Privacy policy
DeGirum Corporation’s collection and use of personal information may also be described in DeGirum Corporation’s [Privacy Policy](https://degirum.com/privacy-policy?utm_source=docs.degirum.com\&utm_medium=docs\&utm_campaign=pysdk-eula\&utm_content=sec-7-9). In the event of a conflict between this EULA and the Privacy Policy, this EULA controls for licensing and permitted use of the Software, and the Privacy Policy controls for privacy disclosures.
## 8. DISCLAIMER OF WARRANTIES
THE SOFTWARE IS PROVIDED "AS IS" AND "AS AVAILABLE" WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. DEGIRUM COPORATION DOES NOT WARRANT THAT THE SOFTWARE WILL MEET YOUR REQUIREMENTS OR THAT THE OPERATION OF THE SOFTWARE WILL BE UNINTERRUPTED OR ERROR-FREE. YOU ACKNOWLEDGE AND AGREE THAT USE OF THE SOFTWARE IS AT YOUR OWN RISK.
## 9. LIMITATION OF LIABILITY
IN NO EVENT SHALL DEGIRUM CORPORATION BE LIABLE FOR ANY SPECIAL, INCIDENTAL, INDIRECT, EXEMPLARY OR CONSEQUENTIAL DAMAGES WHATSOEVER (INCLUDING, BUT NOT LIMITED TO, DIRECT, INDIRECT, INCIDENTAL, CONSEQUENTIAL, SPECIAL, EXEMPLARY, OR PUNITIVE DAMAGES, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES, LOSS OF USE, DAMAGES FOR LOSS OF PROFITS, BUSINESS INTERRUPTION, LOSS OF INFORMATION OR DATA, OR ANY OTHER PECUNIARY LOSS) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING FROM OR RELATING TO THE SOFTWARE OR THIS AGREEMENT, EVEN IF DEGIRUM CORPORATION HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE SOFTWARE IS BORNE BY YOU. SHOULD THE SOFTWARE PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR, OR CORRECTION.
## 10. GOVERNING LAW
This EULA shall be governed by and construed in accordance with the laws of California, without regard to its conflict of law principles.
## 11. ENTIRE AGREEMENT
This EULA constitutes the entire agreement between you and DeGirum Corporation with respect to the Software and supersedes all prior or contemporaneous understandings regarding such subject matter. DeGirum Corporation reserves the right, at its sole discretion, to modify, amend, or replace this EULA and the license granted hereunder. Any such amendments will be effective upon notice to you or upon posting of the amended EULA on DeGirum Corporation’s website. By continuing to access or use the Software after any revisions become effective, you agree to be bound by the revised terms. If you do not agree to the new terms, you are no longer authorized to use the Software.
## 12. ARBITRATION
Any dispute arising from or relating to this Agreement shall be resolved by binding arbitration in Santa Clara County, California, administered by the American Arbitration Association, and judgment may be entered on the award in any court of competent jurisdiction.
DeGirum Corporation\
275 Saratoga Ave Suite 220, Santa Clara, CA 95050\
[https://degirum.com](https://degirum.com/?utm_source=docs.degirum.com\&utm_medium=footer\&utm_campaign=pysdk-eula\&utm_content=contact-section)\
+1-650-660-4619
By using the Software, you acknowledge that you have read this EULA, understand it, and agree to be bound by its terms and conditions.
# EULA v0.20.0 and earlier
Review the DeGirum PySDK End-User License Agreement for PySDK v0.20.0 and earlier to understand the terms governing your use of DeGirum PySDK.
**IMPORTANT** - PLEASE READ THE TERMS AND CONDITIONS OF THIS LICENSE AGREEMENT CAREFULLY BEFORE USING THIS SOFTWARE.
This End-User License Agreement ("EULA") is a legal agreement between you (either an individual or a single entity) and DeGirum Corporation, a Delaware corporation (the “DeGirum Corporation”), governing your use of the DeGirum PySDK software ("Software"). By installing, copying, or otherwise using the Software, you agree to be bound by the terms of this EULA. If you do not agree to the terms of this EULA, do not install or use the Software.
**Version applicability.** This EULA applies to DeGirum PySDK version 0.20.0 and earlier. If you are using DeGirum PySDK version 0.10 or later, your use is governed by the [PySDK EULA](/pysdk/eula) applicable to those versions.
## 1. GRANT OF LICENSE
### 1.1 Non-Commercial Use
DeGirum Corporation grants you a limited, non-exclusive, non-transferable license to use the Software free of charge for personal, educational, or other non-commercial purposes. Non commercial use is defined as use where you do not receive any direct or indirect compensation, fee, or revenue for the use of the Software. This license is revocable at any time at DeGirum Corporation’s sole discretion. Any use beyond the scope of this license, including but not limited to commercial use, is strictly prohibited without explicit written permission from DeGirum Corporation.
### 1.2 Commercial Use
For any commercial use of the Software, you must obtain a commercial license from DeGirum Corporation. Commercial use includes, but is not limited to:
(a) Using the Software for business purposes,
(b) Using the Software in a commercial environment, or
(c) Any activity intended for commercial advantage or monetary compensation.
Please contact DeGirum Corporation for licensing terms and pricing for commercial use.
## 2. RESTRICTIONS
You agree not to, and you will not permit others to:
(a) Sell, lease, rent, transfer, assign, sublicense, or otherwise distribute the Software without prior written consent from DeGirum Corporation.
(b) Reverse engineer, decompile, disassemble, or attempt to drive the source code of the Software, except to the extent that expressly permitted by applicable law.
(c) Modify, alter, or create derivative works based on the Software without prior written authorization from DeGirum Corporation.
(d) Use the Software in any manner that could damage, disable, overburden, impair or interfere with DeGirum Corporation’s services or interfere with any third party's use and enjoyment of the Software.
(e) Use the Software for any unlawful purpose or in violation of any applicable laws or regulations.
(f) Attempt to gain unauthorized access to the Software or its related system or networks.
DeGirum Corporation reserves the right to monitor compliance with these restrictions and may terminate the license immediately upon any violation. DeGirum Corporation may also seek injunctive relief and pursue all available legal remedies for any breach of these restrictions.
## 3. OWNERSHIP
All rights, title, and interest in and to the Software, including all intellectual property rights therein, are and will remain the exclusive property of DeGirum Corporation. Any rights not expressly granted to you are reserved by DeGirum Corporation. You agree not to challenge, contest, or otherwise impair DeGirum Corporation's ownership of the Software or the validity or enforceability of DeGirum Corporation's intellectual property rights related to the Software. Furthermore, you acknowledge that any feedback, suggestions, or ideas you provide regarding the Software may be used by DeGirum Corporation without any obligation to compensate you, and you hereby assign all rights in such feedback to DeGirum Corporation.
## 4. TERMINATION
This EULA is effective until terminated. Your rights under this EULA will terminate automatically without notice from DeGirum Corporation if you fail to comply with any term(s) of this EULA. Upon termination of this EULA for any reason:
(a) all rights and licenses granted to you under this EULA shall immediately terminate;
(b) you must immediately cease all use of the Software;
(c) you must promptly destroy all copies, full or partial, of the Software in your possession or control.
Section 3, 5 and 6 herein shall survive any termination of this EULA.
## 5. DISCLAIMER OF WARRANTIES
THE SOFTWARE IS PROVIDED "AS IS" AND "AS AVAILABLE" WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT. DEGIRUM COPORATION DOES NOT WARRANT THAT THE SOFTWARE WILL MEET YOUR REQUIREMENTS OR THAT THE OPERATION OF THE SOFTWARE WILL BE UNINTERRUPTED OR ERROR-FREE. YOU ACKNOWLEDGE AND AGREE THAT USE OF THE SOFTWARE IS AT YOUR OWN RISK.
## 6. LIMITATION OF LIABILITY
In no event shall DeGirum Corporation be liable for any special, incidental, indirect, exemplary or consequential damages whatsoever (including, but not limited to, procurement of substitute goods or services, loss of use, damages for loss of profits, business interruption, loss of information or data, or any other pecuniary loss) however caused and on any theory of liability, whether in contract, strict liability, or tort (including negligence or otherwise) arising in any way out of the use of or inability to use the Software, even if DeGirum Corporation has been advised of the possibility of such damages. The entire risk as to the quality and performance of the Software is borne by you. Should the Software prove defective, you assume the cost of all necessary servicing, repair, or correction.
## 7. GOVERNING LAW
This EULA shall be governed by and construed in accordance with the laws of the jurisdiction in which DeGirum Corporation is headquartered, without regard to its conflict of law principles.
## 8. ENTIRE AGREEMENT
This EULA constitutes the entire agreement between you and DeGirum Corporation with respect to the Software and supersedes all prior or contemporaneous understandings regarding such subject matter. DeGirum Corporation reserves the right, at its sole discretion, to modify, amend, or replace this EULA and the license granted hereunder. Any such amendments will be effective upon notice to you or upon posting of the amended EULA on DeGirum Corporation’s website. By continuing to access or use the Software after any revisions become effective, you agree to be bound by the revised terms. If you do not agree to the new terms, you are no longer authorized to use the Software.
DeGirum Corporation\
275 Saratoga Ave Suite 220, Santa Clara, CA 95050\
+1-650-660-4619
By using the Software, you acknowledge that you have read this EULA, understand it, and agree to be bound by its terms and conditions.
# Overview
We provide the DeGirum Tools Python package to aid development of AI applications with PySDK. In this group, we'll outline main concepts of DeGirum Tools and provide the API Reference Guide.
{% hint style="info" %}
This overview was written for DeGirum Tools version 0.24.1.
{% endhint %}
## Core Concepts
DeGirum Tools extends PySDK with a kit for building multi-threaded, low-latency media pipelines.\
Where PySDK focuses on running a single model well, DeGirum Tools focuses on everything around it: video ingest, pre- and post-processing, multi-model fusion, result annotation, stream routing, and more.
In one sentence:
> DeGirum Tools is a flow-based mini-framework that lets you prototype complex AI applications in a few dozen lines of Python.
### Model Registry
Use the [Model Registry](/degirum-tools/model_registry) when you need reproducible model picks across hardware. Describe each model once in YAML, filter by `task`, `hardware`, or metadata, and load it through the generated `ModelSpec` helpers. Registry entries can also capture connection defaults so you can keep reusing the same `inference_manager` created via `degirum.connect`.
### Inference Support Utilities
The [inference\_support](/degirum-tools/inference_support) helpers smooth the edges between PySDK and your application. Inference Support utilities include:
* [attach\_analyzers()](/degirum-tools/inference_support#attach_analyzers) – layer result analyzers without wrapping your model manually.
* [predict\_stream()](/degirum-tools/inference_support#predict_stream) / [annotate\_video()](/degirum-tools/inference_support#annotate_video) – quick video loops when a full gizmo graph is overkill.
* [model\_time\_profile()](/degirum-tools/inference_support#model_time_profile) – benchmark a model in fewer than 10 lines of code.
* [warmup\_device()](/degirum-tools/inference_support#warmup_device) / [warmup\_model()](/degirum-tools/inference_support#warmup_model) – preheat hardware so the first real inference arrives at full speed.
### Compound Models
[Compound models](/degirum-tools/compound_models) wrap two PySDK models into a single `predict()` / `predict_batch()` interface. Some of the compound model classes provided by DeGirum Tools include:
| Class | What it Does |
| --------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------- |
| [CombiningCompoundModel](/degirum-tools/compound_models#combiningcompoundmodel) | Runs two models in parallel on the same image and concatenates results. |
| [CroppingAndClassifyingCompoundModel](/degirum-tools/compound_models#croppingandclassifyingcompoundmodel) | Detector → crops → classifier (adds labels back). |
| [CroppingAndDetectingCompoundModel](/degirum-tools/compound_models#croppinganddetectingcompoundmodel) | Detector → crops → refined detector (with optional NMS). |
Use compound models exactly how you would use normal models:
```python
compound = CroppingAndClassifyingCompoundModel(detector, classifier)
for res in compound.predict_batch(my_images):
...
```
In addition to compound models, you may encounter PseudoModels.
A PseudoModel is a ModelLike object that behaves like a PySDK model but does not actually run inference. Instead, it generates bounding-box results according to predefined rules, such as dividing an image into a grid (TileExtractorPseudoModel) or returning fixed ROIs with optional motion filtering (RegionExtractionPseudoModel).
These pseudo‑models serve as drop‑in “detectors” within compound-model pipelines, allowing ROI extraction and tiling without relying on a real detection network, while still following the standard predict()/predict\_batch() interface.
### Streams
The flow behind DeGirum Tools is supported by the Streams subsystem. There are three constituent Python submodules: [streams.py](#streams), [streams\_base.py](/degirum-tools/streams/streams_base), and [streams\_gizmos.py](/degirum-tools/streams/streams_gizmos). In this subsystem, the two most important concepts in streams are gizmos and compositions.
#### Gizmos
A [Gizmo](/degirum-tools/streams/streams_gizmos) is a worker that:
{% stepper %}
{% step %}
Consumes from one or more input streams.
{% endstep %}
{% step %}
Runs its custom `run()` loop (decode, resize, infer, etc.).
{% endstep %}
{% step %}
Pushes new `StreamData` to any number of output streams. `StreamData` is described in more detail in [streams.py](#streams).
{% endstep %}
{% endstepper %}
Because every gizmo lives in its own thread, pipelines scale across CPU cores with minimal user code.
Gizmo families built into DeGirum Tools include:
| Family | Example Classes | Typical Use |
| ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- |
| Video IO |
| Camera/file capture; iterate images from paths/arrays/PIL; live preview; archival |
| Transform | [ResizingGizmo](/degirum-tools/streams/streams_gizmos#resizinggizmo) | Pre-process frames (letterbox, crop, pad) |
| AI Inference | [AiSimpleGizmo](/degirum-tools/streams/streams_gizmos#aisimplegizmo), [AiObjectDetectionCroppingGizmo](/degirum-tools/streams/streams_gizmos#aiobjectdetectioncroppinggizmo) | Run models, cascade detectors & classifiers |
| Post-fusion | [CropCombiningGizmo](/degirum-tools/streams/streams_gizmos#cropcombininggizmo), [AiResultCombiningGizmo](/degirum-tools/streams/streams_gizmos#airesultcombininggizmo) | Merge multi-crop or multi-model outputs |
| Utility | [SinkGizmo](/degirum-tools/streams/streams_gizmos#sinkgizmo), [FPSStabilizingGizmo](/degirum-tools/streams/streams_gizmos#fpsstabilizinggizmo) | Collect results in the main thread; stabilize frame rate |
Gizmos pass data around by using the [Stream](/degirum-tools/streams/streams_base#stream) class. A stream is an iterable queue that moves `StreamData` objects between threads. Each queue may be bounded (with optional drop policy) to prevent bottlenecks, and it automatically propagates a poison pill sentinel to shut the pipeline down cleanly.
#### Stream Validation and Tag Requirements
Starting with DeGirum Tools 0.20.0, gizmos declare upstream metadata requirements via a `require_tags(inp)` method. When you build a `Composition`, it validates that each gizmo’s requirements are satisfied by the connected upstream stages and raises an error when they are not.
Examples:
* Cropping/combining and analyzer gizmos require inference-related tags from upstream inference stages.
* FPS stabilizing gizmos require video tags from upstream sources.
This check helps catch wiring mistakes early when you assemble larger graphs.
#### Compositions
A [Composition](/degirum-tools/streams/streams_base#composition) collects any connected gizmos and controls their life-cycle:
* `start()` – spawn threads
* `stop()` – signal abort & join
* `wait()` – block until completion
* `get_bottlenecks()` – diagnose dropped-frame hotspots
Use it as a context-manager so everything shuts down even on exceptions.
For activity monitoring, see [Watchdog](/degirum-tools/streams/streams_base#watchdog), which tracks tick frequency and timing to detect stalls.
### Analyzers
Analyzers provide advanced processing of inference results with specialized functionality. The available analyzers include:
| Analyzer | Description |
| ----------------------------------------------------------- | ------------------------------------------------------------------ |
| [Zone Counter](/degirum-tools/analyzers/zone_count) | Tracks objects entering and exiting defined zones |
| [Object Selector](/degirum-tools/analyzers/object_selector) | Filters and selects specific objects based on criteria |
| [Object Tracker](/degirum-tools/analyzers/object_tracker) | Tracks objects across frames with customizable tracking parameters |
| [Line Counter](/degirum-tools/analyzers/line_count) | Counts objects crossing defined lines in the scene |
| [Event Detector](/degirum-tools/analyzers/event_detector) | Detects and processes specific events in the video stream |
| [Notifier](/degirum-tools/analyzers/notifier) | Sends notifications for detected events and conditions |
| [Clip Saver](/degirum-tools/analyzers/clip_saver) | Saves video clips of detected events |
An [Analyzer](/degirum-tools/analyzers) subclass provides two key capabilities:
* `analyze(result)` – Process and modify inference results by adding custom fields or performing calculations
* `annotate(result, image)` – Draw overlays on the image (bounding boxes, text labels, etc.)
Analyzers can be attached to any model or compound model:
```python
import degirum_tools
import numpy as np
# Create a line counter that counts people crossing a line
LINES: List[Tuple[int, int, int, int]] = [
(634, 539, 950, 539),
]
window_name = "Line Counter"
# Create an ObjectTracker to track people
tracker = degirum_tools.ObjectTracker(
trail_depth=20, anchor_point=degirum_tools.AnchorPoint.BOTTOM_CENTER
)
# Create a LineCounter to detect when people cross pre-defined lines
counter = degirum_tools.LineCounter(lines)
# Attach ObjectTracker and LineCounter to the model
degirum_tools.attach_analyzers(model, [tracker, counter])
# Run predictions and print the line counts
for result in degirum_tools.predict_stream(model, video_source):
if hasattr(result, "line_counts"):
print([lc.to_dict() for lc in result.line_counts])
```
When used inside a gizmo pipeline, analyzers can filter or decorate results in-flight. They can also accumulate state across frames for multi-frame analysis, with cleanup handled in the `finalize()` method.
### Support Modules
DeGirum Tools includes other [Support Modules](/degirum-tools/support):
* [Math Support](/degirum-tools/support/math_support): geometry, NMS, tiling, and k-means helpers.
* [UI Support](/degirum-tools/support/ui_support): display, FPS meter, timers, and image stacks.
* [Video Support](/degirum-tools/support/video_support): open or create streams (e.g., `create_video_stream`), detect devices/RTSP sources (`detect_rtsp_cameras`).
* Additional modules: audio I/O, evaluation framework, object storage helpers.
### Environment Variables
See [Environment Variables](/degirum-tools/environment-variables) for configuration keys used by DeGirum Tools and helpers.
### Remote Assets
Use the lightweight [Remote Assets](/degirum-tools/remote-assets) catalog to reference sample images and videos from PySDK Examples without downloading files. Access them via module attributes (for example, `remote_assets.cat`) or enumerate names with `list_images()` and `list_videos()`.
# Model Registry
DeGirum Tools API Reference Guide. YAML registry for selecting models by task, hardware, and runtime defaults.
{% hint style="info" %}
This API Reference is based on DeGirum Tools version 1.2.0.
{% endhint %}
### Model Registry Overview
Use this module when you want to use a working model without memorizing model details. `ModelRegistry` acts as a guided menu of DeGirum models, while `ModelSpec` captures the handful of settings required to load one of those choices. Together they help pick a model with confidence and launch inference in just a few lines of code.
Key Features
* Capture one model request with `ModelSpec` so it can be shared, reused, or versioned alongside your project.
* Browse curated registries to surface models by goal, compatible hardware, or descriptive metadata.
* Layer simple filters to shrink the catalog to only the options that fit your scenario.
* Keep common defaults (host, zoo, properties) in a single place so every run follows the same playbook.
Typical Usage
1. Point `ModelRegistry` at the YAML file or hosted URL that lists the models available to your team.
2. Narrow the registry with helpers such as `for_task` and `for_hardware` to focus on the models that match your intent.
3. Choose a remaining `ModelSpec` (or let ranking helpers do it) to represent the model you plan to run.
4. Call `ModelSpec.load_model()` to launch the model and start running inference.
Example:
{% code overflow="wrap" %}
```python
from degirum_tools import Display, ModelRegistry, ModelSpec, remote_assets
registry = ModelRegistry(
config_file="https://assets.degirum.com/registry/models.yaml",
)
model_spec = (
registry
.for_task("coco_detection")
.top_model_spec()
)
model = model_spec.load_model()
inference_result = model(remote_assets.three_persons)
print(inference_result)
with Display("Model Registry Demo") as output_display:
output_display.show_image(inference_result.image_overlay)
```
{% endcode %}
Integration Notes
* Registry files can live in source control or be hosted online; point the constructor at whichever location you maintain.
* `ModelSpec.load_model()` opens the connection with `degirum.connect` on your behalf, so you do not have to manage sessions manually.
* Override defaults as needed when experimenting, without editing the shared registry catalog.
Key Functions
* `ModelSpec.load_model()` turns a saved specification into a ready-to-run model.
* `ModelSpec.ensure_local()` downloads the assets you need for offline use and returns a spec that targets `@local`.
* `ModelRegistry.all_model_specs()` lists the prepared specs that match the filters you have applied.
* `ModelRegistry.best_model_spec()` helps you pick a model based on ranking fields such as accuracy scores.
Configuration Options
* Registry defaults set shared values such as the inference host, zoo URL, tokens, and model properties.
* Metadata filters accept literal values or simple callables when you want to match custom fields in your catalog.
## Classes
## ModelSpec
`ModelSpec`
`dataclass`
Serializable description of a single model load request.
Attributes:
| Name | Type | Description |
| ------------------------ | ---------------- | ------------------------------------------------------------------------------------------------------------------- |
| `model_name` | `str` | Exact model identifier expected by the zoo. Ignored when `model_url` is provided. |
| `zoo_url` | `str` | Zoo location that hosts this model. When left empty, registry defaults or explicit overrides must supply the value. |
| `model_url` | `str` | Direct URL to the model file. When set, overrides `model_name` and `zoo_url`. |
| `inference_host_address` | `str` | Target inference host in PySDK locator format (for example `@cloud` or `@local`). |
| `token` | `str \| None` | Optional authentication token for the zoo. |
| `model_properties` | `dict[str, Any]` | Keyword arguments forwarded to `dg.ZooManager.load_model`. |
| `metadata` | `dict \| None` | Free-form informational payload, typically copied from the registry entry. |
Examples:
{% code overflow="wrap" %}
```python
spec = ModelSpec(
model_name="",
zoo_url="",
inference_host_address="@local",
)
spec.load_model()
spec = ModelSpec(model_url="")
spec.load_model()
```
{% endcode %}
### ModelSpec Methods
#### \_\_post\_init\_\_
`__post_init__()`
Validate required fields, support `model_url`, and normalize values.
Raises:
| Type | Description |
| ------------ | ---------------------------------------------------- |
| `ValueError` | If mandatory attributes are missing or inconsistent. |
#### download\_model(destination=None, ...)
`download_model(destination=None, *, cloud_sync=False)`
Download the model file to a local directory.
Parameters:
| Name | Type | Description | Default |
| ------------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- |
| `destination` | `str \| Path \| None` | Destination directory or path. If a directory is provided, the model is saved into a subdirectory named after the model. When `None`, the default application data directory is used. | `None` |
| `cloud_sync` | `bool` | When `True`, checks the cloud zoo for an updated model version and downloads it if the local copy is missing or out of date. | `False` |
Returns:
| Name | Type | Description |
| ------------ | ------ | ---------------------------------------------- |
| `local_path` | `Path` | Path to the downloaded model assets directory. |
#### ensure\_local(cloud\_sync=False)
`ensure_local(cloud_sync=False)`
Ensures the model is present locally; downloads if needed. Returns a **new** ModelSpec with zoo\_url set to local path and inference\_host\_address to '@local'.
Parameters:
| Name | Type | Description | Default |
| ------------ | ------ | ---------------------------------------------------------------------------------------------------------------------------- | ------- |
| `cloud_sync` | `bool` | When `True`, checks the cloud zoo for an updated model version and downloads it if the local copy is missing or out of date. | `False` |
Returns:
| Name | Type | Description |
| ------------ | ----------- | --------------------------------------------------- |
| `local_spec` | `ModelSpec` | New specification pointing to the local model copy. |
#### load\_model(zoo=None)
`load_model(zoo=None)`
Resolve the specification into a ready-to-use model instance.
Parameters:
| Name | Type | Description | Default |
| ----- | -------------------- | --------------------------------------------------------------------------------------------- | ------- |
| `zoo` | `ZooManager \| None` | Optional pre-connected inference manager. When `None`, `zoo_connect` is called automatically. | `None` |
Returns:
| Name | Type | Description |
| ------- | ------- | ----------------------------------- |
| `model` | `Model` | Loaded model returned by the PySDK. |
#### zoo\_connect
`zoo_connect()`
Create a connection to the configured model zoo.
Returns:
| Name | Type | Description |
| ------------------- | ------------ | ----------------------------------------------------------------------- |
| `inference_manager` | `ZooManager` | Reusable inference manager suitable for repeated calls to `load_model`. |
## ModelRegistry
`ModelRegistry`
Queryable collection of `ModelSpec` entries.
Registry data is sourced from structured YAML files that describe the available models, their target tasks, and compatible hardware. Instances remain immutable, and filtering methods return new copies so intermediate views can be chained without side effects.
Examples:
{% code overflow="wrap" %}
```python
registry = ModelRegistry(
config_file="https://assets.degirum.com/registry/models.yaml",
)
filtered = registry.for_task("coco_detection")
first_spec = filtered.top_model_spec()
print(first_spec.model_name)
```
{% endcode %}
### ModelRegistry Methods
#### \_\_init\_\_(\*, ...)
`__init__(*, config=None, config_file=None)`
Create a registry from a configuration dictionary or YAML file.
Parameters:
| Name | Type | Description | Default |
| ------------- | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- |
| `config` | `dict[str, dict] \| None` | Parsed registry configuration. If provided, `config_file` is ignored. | `None` |
| `config_file` | `Path \| str \| None` | Path or URL to a YAML registry document. Defaults to `models.yaml` located alongside this module. The latest DeGirum-managed catalog is published at . | `None` |
Raises:
| Type | Description |
| ----------------- | -------------------------------------------------------------------- |
| `RuntimeError` | If a remote registry cannot be retrieved. |
| `ValidationError` | When the configuration does not satisfy `ModelRegistry.schema_text`. |
#### all\_model\_specs(\*, ...)
`all_model_specs(*, inference_host_address=None, zoo_url=None, token=None, model_properties=None)`
Create `ModelSpec` objects for every model in the registry.
Parameters:
| Name | Type | Description | Default |
| ------------------------ | -------------- | --------------------------------------------------------------------------------------------------- | ------- |
| `inference_host_address` | `str \| None` | Destination inference host. If omitted, the registry defaults are used. | `None` |
| `zoo_url` | `str \| None` | Override for the zoo URL. A common use case is redirecting all specs to a local zoo during testing. | `None` |
| `token` | `str \| None` | Authentication token to apply to each spec. | `None` |
| `model_properties` | `dict \| None` | Keyword arguments merged into the default properties and applied to every resulting spec. | `None` |
Returns:
| Name | Type | Description |
| ------- | ----------------- | -------------------------------------------------- |
| `specs` | `list[ModelSpec]` | Specifications matching the current filtered view. |
#### best\_model\_spec(key, ...)
`best_model_spec(key, compare='max', **kwargs)`
Select the model with the best numeric metadata value.
Parameters:
| Name | Type | Description | Default |
| ---------- | ---------------- | --------------------------------------------------------------------------- | ---------- |
| `key` | `str` | Metadata field to inspect. | *required* |
| `compare` | `str` | Either `"max"` (default) for the largest value or `"min"` for the smallest. | `'max'` |
| `**kwargs` | `dict[str, Any]` | Overrides forwarded to `all_model_specs`. | `{}` |
Returns:
| Name | Type | Description |
| ----------- | ----------- | ---------------------------------------------------- |
| `best_spec` | `ModelSpec` | Model whose metadata matches the requested criteria. |
Raises:
| Type | Description |
| -------------- | ------------------------------------------------------------------------- |
| `RuntimeError` | If the registry contains no models. |
| `ValueError` | If none of the models expose the requested key or provide numeric values. |
#### for\_alias(alias)
`for_alias(alias)`
Filter models by alias.
Parameters:
| Name | Type | Description | Default |
| ------- | ----- | -------------------------------- | ---------- |
| `alias` | `str` | Registry alias to match exactly. | *required* |
Returns:
| Name | Type | Description |
| ---------- | --------------- | -------------------------------------------------- |
| `registry` | `ModelRegistry` | Filtered registry containing only matching models. |
#### for\_hardware(hardware)
`for_hardware(hardware)`
Filter models to those compatible with a specific hardware target.
Parameters:
| Name | Type | Description | Default |
| ---------- | ----- | ------------------------------------------------------------------------ | ---------- |
| `hardware` | `str` | Hardware identifier in `RUNTIME/DEVICE` format, for example `N2X/ORCA1`. | *required* |
Returns:
| Name | Type | Description |
| ---------- | --------------- | -------------------------------------------------- |
| `registry` | `ModelRegistry` | Filtered registry containing only matching models. |
#### for\_meta(meta)
`for_meta(meta)`
Filter models by metadata key/value pairs.
Parameters:
| Name | Type | Description | Default |
| ------ | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- |
| `meta` | `dict[str, Any]` | Dictionary describing metadata criteria. Values may be callables that receive a metadata dictionary and return `True` when the model should be included. Non-callable values must match exactly. | *required* |
Returns:
| Name | Type | Description |
| ---------- | --------------- | -------------------------------------------------- |
| `registry` | `ModelRegistry` | Filtered registry containing only matching models. |
#### for\_task(task)
`for_task(task)`
Filter models by the task label declared in the registry.
Parameters:
| Name | Type | Description | Default |
| ------ | ----- | ----------------------------------------------------------- | ---------- |
| `task` | `str` | Task identifier such as `face_detection` or `segmentation`. | *required* |
Returns:
| Name | Type | Description |
| ---------- | --------------- | -------------------------------------------------- |
| `registry` | `ModelRegistry` | Filtered registry containing only matching models. |
#### get\_aliases
`get_aliases()`
Return every unique alias present in the registry.
Returns:
| Name | Type | Description |
| --------- | ----------- | ----------------------- |
| `aliases` | `list[str]` | Sorted list of aliases. |
#### get\_hardware
`get_hardware()`
Return every unique hardware target present in the registry.
Returns:
| Name | Type | Description |
| ---------- | ----------- | ------------------------------------ |
| `hardware` | `list[str]` | Sorted list of hardware identifiers. |
#### get\_tasks
`get_tasks()`
Return every unique task label present in the registry.
Returns:
| Name | Type | Description |
| ------- | ----------- | -------------------------- |
| `tasks` | `list[str]` | Sorted list of task names. |
#### top\_model\_spec(\*\*kwargs)
`top_model_spec(**kwargs)`
Return the first model listed in the current registry view.
Parameters:
| Name | Type | Description | Default |
| ---------- | ---------------- | ----------------------------------------- | ------- |
| `**kwargs` | `dict[str, Any]` | Overrides forwarded to `all_model_specs`. | `{}` |
Returns:
| Name | Type | Description |
| ---------- | ----------- | ------------------------------------- |
| `top_spec` | `ModelSpec` | Specification for the top-most entry. |
Raises:
| Type | Description |
| -------------- | ---------------------------------- |
| `RuntimeError` | If the filtered registry is empty. |
#### with\_defaults(\*, ...)
`with_defaults(*, inference_host_address=None, zoo_url=None, token=None, model_properties=None)`
Return a copy of the registry with overridden default settings.
Parameters:
| Name | Type | Description | Default |
| ------------------------ | -------------- | ----------------------------------------------------------------------------- | ------- |
| `inference_host_address` | `str \| None` | Override for the default inference target. | `None` |
| `zoo_url` | `str \| None` | Fallback zoo URL applied when entries omit an explicit value. | `None` |
| `token` | `str \| None` | Token injected into downstream connections. | `None` |
| `model_properties` | `dict \| None` | Base keyword arguments merged into every `ModelSpec` emitted by the registry. | `None` |
Returns:
| Name | Type | Description |
| ---------- | --------------- | ------------------------------------------- |
| `registry` | `ModelRegistry` | New registry with updated defaults applied. |
# Inference Support
DeGirum Tools API Reference Guide. Utilities to connect, run, annotate, and profile inferences.
{% hint style="info" %}
This API Reference is based on DeGirum Tools version 1.2.0.
{% endhint %}
## Inference Support Overview
This module provides utility functions and classes for integrating DeGirum PySDK models into various inference scenarios, including:
* **Connecting** to different model zoo endpoints (cloud, AI server, or local accelerators).
* **Attaching** custom result analyzers to models or compound models, enabling additional data processing or custom overlay annotation on inference results.
* **Running** inferences on video sources or streams (local camera, file, RTSP, YouTube links, etc.) with optional real-time annotation and saving to output video files.
* **Measuring** model performance using a profiling function that times inference runs under various conditions.
### Key Concepts
1. **Model Zoo Connections**: Functions like `connect_model_zoo` provide a unified way to choose between different inference endpoints (cloud, server, local hardware).
2. **Analyzer Attachment**: By calling `attach_analyzers` or using specialized classes within the streaming toolkit, you can process each inference result through user-defined or library-provided analyzers (subclasses of `ResultAnalyzerBase`).
3. **Video Inference and Annotation**: Functions `predict_stream` and `annotate_video` demonstrate how to run inference on live or file-based video streams. They optionally include steps to create overlays (bounding boxes, labels, etc.) and even show a real-time display or write to an output video file.
4. **Model Time Profiling**: `model_time_profile` provides a convenient way to measure the performance (FPS, average inference time, etc.) of a given DeGirum PySDK model under test conditions.
## Basic Usage Example
{% code overflow="wrap" %}
```python
from degirum_tools import (
ModelSpec,
remote_assets,
attach_analyzers,
annotate_video,
model_time_profile,
ResultAnalyzerBase,
)
# Define a simple analyzer that draws text on each frame
class MyDummyAnalyzer(ResultAnalyzerBase):
def analyze(self, result):
# Optional: add custom logic here, e.g. track or filter detections
pass
def annotate(self, result, image):
"""
Draws a simple "Dummy Analyzer" label in the top-left corner of each frame.
"""
import cv2
cv2.putText(
image,
"Dummy Analyzer",
(10, 30),
cv2.FONT_HERSHEY_SIMPLEX,
1.0,
(0, 255, 0),
2
)
return image
# Describe the model once so the configuration is reusable.
model_spec = ModelSpec(
model_name="",
zoo_url="degirum/degirum",
inference_host_address="@cloud",
)
with model_spec.load_model() as model:
# Attach dummy analyzer
attach_analyzers(model, MyDummyAnalyzer())
# Annotate a video with detection results + dummy analyzer and set a path to save the video
annotate_video(
model,
video_source_id=remote_assets.store_short,
output_video_path="annotated_output.mp4",
show_progress=True, # Show a progress bar in console
visual_display=True, # Open an OpenCV window to display frames
show_ai_overlay=True, # Use model's overlay with bounding boxes
fps=None # Use the source's native frame rate
)
# Time-profile the model
profile = model_time_profile(model, iterations=100)
print("Time profiling results:")
print(f" Elapsed time: {profile.elapsed:.3f} s")
print(f" Observed FPS: {profile.observed_fps:.2f}")
print(f" Max possible FPS: {profile.max_possible_fps:.2f}")
```
{% endcode %}
## Functions
#### connect\_model\_zoo(inference\_option=CloudInference)
`connect_model_zoo(inference_option=CloudInference)`
Connect to a model zoo endpoint based on the specified inference option.
This function provides a convenient way to switch between
* Cloud-based inference (`CloudInference`),
* AI server on LAN/VPN (`AIServerInference`),
* Local hardware accelerator (`LocalHWInference`).
It uses environment variables (see `degirum_tools.environment`) to resolve the zoo address/URL and token as needed.
Parameters:
| Name | Type | Description | Default |
| ------------------ | ----- | ------------------------------------------------------------------- | ---------------- |
| `inference_option` | `int` | One of `CloudInference`, `AIServerInference`, or `LocalHWInference` | `CloudInference` |
Raises:
| Type | Description |
| ----------- | --------------------------------------------- |
| `Exception` | If an invalid `inference_option` is provided. |
Returns:
| Type | Description |
| ------------ | ------------------------------------------------------------------------------------ |
| `ZooManager` | dg.zoo\_manager.ZooManager: A model zoo manager connected to the requested endpoint. |
#### attach\_analyzers(model, ...)
`attach_analyzers(model, analyzers)`
Attach or detach analyzer(s) to a model or compound model.
For single or compound models, analyzers can augment the inference results with extra metadata and/or custom overlay. If attaching analyzers to a compound model (e.g., `compound_models.CompoundModelBase`), the analyzers are invoked at the final stage of each inference result.
Parameters:
| Name | Type | Description | Default |
| ----------- | ----------------------------------------------------------- | ----------------------------------------------------------------------------------------- | ---------- |
| `model` | `Union[Model, CompoundModelBase]` | The model or compound model to which analyzers will be attached. | *required* |
| `analyzers` | `Union[ResultAnalyzerBase, List[ResultAnalyzerBase], None]` | One or more analyzer objects. Passing None will detach any previously attached analyzers. | *required* |
Usage
attach\_analyzers(my\_model, MyAnalyzerSubclass())
Notes
* If `model` is a compound model, the call is forwarded to `model.attach_analyzers()`.
* If `model` is a standard PySDK model, this function subclasses the model's current result class with a new class that additionally calls each analyzer in turn for `analyze()` and `annotate()` steps. This subclass is assigned to `model._custom_postprocessor` property.
#### predict\_stream(model, ...)
`predict_stream(model, video_source_id, source_type=VideoSourceType.AUTO, *, fps=None, analyzers=None)`
Run a PySDK model on a live or file-based video source, yielding inference results.
This function is a generator that continuously
1. Reads frames from the specified video source.
2. Runs inference on each frame via `model.predict_batch`.
3. If analyzers are provided, each result is wrapped in a dynamic postprocessor that calls analyzers' `analyze()` and `annotate()` methods.
Parameters:
| Name | Type | Description | Default |
| ----------------- | ----------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
| `model` | `Model` | Model to run on each incoming frame. | *required* |
| `video_source_id` | `Union[int, str, Path, None]` | Identifier for the video source. Possible types include: - An integer camera index (e.g., 0 for default webcam). - A local file path or string/Path, e.g., "video.mp4". - A streaming URL (RTSP/YouTube link). - None if no source is available (not typical). | *required* |
| `source_type` | `Union[str, VideoSourceType]` | Video backend to use. Options: - VideoSourceType.AUTO or "auto": Automatically choose best backend - VideoSourceType.GSTREAMER or "gstream": Force GStreamer backend - VideoSourceType.OPENCV or "opencv": Force OpenCV backend | `AUTO` |
| `fps` | `Optional[float]` | If provided, caps the effective reading/processing rate to the given FPS. If the input source is slower, this has no effect. If faster, frames are decimated. | `None` |
| `analyzers` | `Union[ResultAnalyzerBase, List[ResultAnalyzerBase], None]` | One or more analyzers to apply to each inference result. If None, no additional analysis or annotation is performed beyond the model's standard postprocessing. | `None` |
Yields:
| Name | Type | Description |
| -------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [InferenceResults](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/postprocessor#degirum.postprocessor.inferenceresults) | [InferenceResults](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/postprocessor#degirum.postprocessor.inferenceresults) | The inference result for each processed frame. If analyzers are present, the result object is wrapped to allow custom annotation in its `.image_overlay`. |
Example:
{% code overflow="wrap" %}
```python
# Using enum (recommended)
for res in predict_stream(my_model, "my_video.mp4", VideoSourceType.GSTREAMER, fps=15, analyzers=MyAnalyzer()):
annotated_img = res.image_overlay # includes custom overlay
# do something with annotated_img
# Using string (backward compatible)
for res in predict_stream(my_model, "my_video.mp4", "gstream", fps=15, analyzers=MyAnalyzer()):
annotated_img = res.image_overlay # includes custom overlay
# do something with annotated_img
```
{% endcode %}
#### annotate\_video(model, ...)
`annotate_video(model, video_source_id, output_video_path, source_type=VideoSourceType.AUTO, *, show_progress=True, visual_display=True, show_ai_overlay=True, fps=None, analyzers=None)`
Run a model on a video source and save the annotated output to a video file.
This function
1. Opens the input video source.
2. Processes each frame with the specified model.
3. (Optionally) calls any analyzers to modify or annotate the inference result.
4. If `show_ai_overlay` is True, retrieves the `image_overlay` from the result (which includes bounding boxes, labels, etc.). Otherwise, uses the original frame.
5. Writes the annotated frame to `output_video_path`.
6. (Optionally) displays the annotated frames in a GUI window and shows progress.
Parameters:
| Name | Type | Description | Default |
| ------------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
| `model` | `Model` | The model to run on each frame of the video. | *required* |
| `video_source_id` | `Union[int, str, Path, None, VideoCapture]` | The video source, which can be: - A cv2.VideoCapture object already opened by `open_video_stream`, - An integer camera index (e.g., 0), - A file path or a URL (RTSP/YouTube). | *required* |
| `output_video_path` | `str` | Path to the output video file. The file is created or overwritten as needed. | *required* |
| `source_type` | `Union[str, VideoSourceType]` | Video backend to use. Options: - VideoSourceType.AUTO or "auto": Automatically choose best backend - VideoSourceType.GSTREAMER or "gstream": Force GStreamer backend - VideoSourceType.OPENCV or "opencv": Force OpenCV backend | `AUTO` |
| `show_progress` | `bool` | If True, displays a textual progress bar or frame counter (for local file streams). | `True` |
| `visual_display` | `bool` | If True, opens an OpenCV window to show the annotated frames in real time. | `True` |
| `show_ai_overlay` | `bool` | If True, uses the result's `image_overlay`. If False, uses the original unannotated frame. | `True` |
| `fps` | `Optional[float]` | If provided, caps the effective processing rate to the given FPS. Otherwise, uses the native FPS of the video source if known. | `None` |
| `analyzers` | `Union[ResultAnalyzerBase, List[ResultAnalyzerBase], None]` | One or more analyzers to apply. Each analyzer's `analyze_and_annotate()` is called on the result before writing the frame. | `None` |
Example:
{% code overflow="wrap" %}
```
# Using enum (recommended)
annotate_video(my_model, "input.mp4", "output.mp4", VideoSourceType.GSTREAMER, analyzers=[MyAnalyzer()])
# Using string (backward compatible)
annotate_video(my_model, "input.mp4", "output.mp4", "gstream", analyzers=[MyAnalyzer()])
```
{% endcode %}
#### model\_time\_profile(model, ...)
`model_time_profile(model, iterations=100, input_image_format='JPEG')`
Profile the inference performance of a DeGirum PySDK model by running a specified number of iterations on a synthetic (zero-pixel) image.
This function
1. Adjusts the model's settings to measure time (`model.measure_time = True`).
2. Warms up the model by performing one inference.
3. Resets time statistics and runs the specified number of iterations.
4. Restores original model parameters after profiling.
Parameters:
| Name | Type | Description | Default |
| ------------ | ------- | ------------------------------------------------------------------------ | ---------- |
| `model` | `Model` | A PySDK model to profile. Must accept images as input. | *required* |
| `iterations` | `int` | Number of inference iterations to run (excluding the warm-up iteration). | `100` |
Raises:
| Type | Description |
| --------------------- | ------------------------------------------------------- |
| `NotImplementedError` | If the model does not accept images (e.g., audio/text). |
Returns:
| Name | Type | Description |
| ------------------ | ------------------ | ------------------------------------------------------------------- |
| `ModelTimeProfile` | `ModelTimeProfile` | An object containing timing details, measured FPS, and other stats. |
Example:
{% code overflow="wrap" %}
```python
profile = model_time_profile(my_model, iterations=50)
print("Elapsed seconds:", profile.elapsed)
print("Observed FPS:", profile.observed_fps)
print("Core Inference Stats:", profile.time_stats["CoreInferenceDuration_ms"])
```
{% endcode %}
#### warmup\_device(model, ...)
`warmup_device(model, chosen_device)`
Warm up given model on the given device in devices\_available
#### warmup\_model(model)
`warmup_model(model)`
Warms up a model by performing one inference on a dummy input frame on each device selected.
This is useful for initializing the model internals and hardware, which can reduce latency on the first real inference.
Parameters:
| Name | Type | Description | Default |
| ------- | ------- | ------------------------------------------ | ---------- |
| `model` | `Model` | The DeGirum PySDK model object to warm up. | *required* |
## Classes
## ModelTimeProfile
`ModelTimeProfile`
`dataclass`
Container for model time profiling results.
Attributes:
| Name | Type | Description |
| ------------------ | ------- | ------------------------------------------------------------------------------------------------------------- |
| `elapsed` | `float` | Total elapsed time in seconds for the profiling run. |
| `iterations` | `int` | Number of inference iterations performed. |
| `observed_fps` | `float` | The measured frames per second (iterations / elapsed). |
| `max_possible_fps` | `float` | Estimated maximum possible FPS based on the model's core inference duration, ignoring overhead. |
| `parameters` | `dict` | A copy of the model's metadata or parameters for reference. |
| `time_stats` | `dict` | A dictionary containing detailed timing statistics from the model's built-in time measurement (if available). |
# Compound Models
DeGirum Tools API Reference Guide. Chain multiple models in cropping, combining or async flows.
{% hint style="info" %}
This API Reference is based on DeGirum Tools version 1.2.0.
{% endhint %}
## Compound Models Module Overview
This module provides a toolkit for creating compound models using the DeGirum PySDK.
A compound model orchestrates multiple underlying models into a pipeline to enable complex inference scenarios. Common examples include:
* Detecting objects and then classifying each detected object.
* Running coarse detection first, then applying a refined detection model on specific regions.
* Combining outputs from multiple independent models into a unified inference result.
Compound models run in a single thread and are intended primarily for simple usage scenarios. Compound models still provide efficient batch prediction pipelining using batch\_predict() in non-blocking mode. For more performant applications requiring better scalability and more flexible connections, we recommend using Gizmos, which in multiple threads.
### Key Concepts
* **Model Composition**: Compound models sequentially (or concurrently) invoke multiple models. Typically, results from the first model (e.g., bounding boxes from detection) feed into subsequent models (classification or refined detection).
* **Pipeline Workflow**: A typical workflow involves:
1. Run `model1` to identify regions of interest (ROIs).
2. Crop these ROIs and run them through `model2`.
3. Integrate or transform outputs from `model2` back into the original context.
* **Unified Model Interface**: All compound models follow the same interface as regular models in DeGirum SDK, providing `.predict()` for single frames and `.predict_batch()` for iterators of frames.
### Included Compound Models
* **CombiningCompoundModel**: Combines results from two models run concurrently on the same input.
* **CroppingCompoundModel**: Crops regions identified by `model1` and feeds them into `model2`.
* **CroppingAndClassifyingCompoundModel**: Specialized pipeline: object detection (`model1`) followed by classification (`model2`) of each detected object.
* **CroppingAndDetectingCompoundModel**: Pipeline for refined detection: initial coarse detection (`model1`) followed by detailed detection (`model2`) within each ROI.
* **RegionExtractionPseudoModel**: Extracts predefined regions of interest without actual inference, optionally filtering by motion detection.
### Basic Usage Examples
**Detection + Classification**:
{% code overflow="wrap" %}
```python
from degirum_tools import ModelSpec, remote_assets
from degirum_tools.compound_models import CroppingAndClassifyingCompoundModel
# Describe the individual models once
detector_spec = ModelSpec(
model_name="",
inference_host_address="@cloud", # Can be '@cloud', host:port, or '@local'
zoo_url="degirum/degirum",
)
classifier_spec = ModelSpec(
model_name="",
inference_host_address="@cloud",
zoo_url="degirum/degirum",
)
with detector_spec.load_model() as detector, classifier_spec.load_model() as classifier:
# Creating a compound model pipeline
compound_model = CroppingAndClassifyingCompoundModel(detector, classifier)
# Single frame inference using predict()
print("Using predict():")
single_result = compound_model(remote_assets.cat)
print(single_result)
# Batch inference using predict_batch()
print("Using predict_batch():")
for batch_result in compound_model.predict_batch(
[remote_assets.cat, remote_assets.two_cats]
):
print(batch_result)
```
{% endcode %}
**Detection + Detection**:
{% code overflow="wrap" %}
```python
from degirum_tools import ModelSpec, remote_assets
from degirum_tools.compound_models import CombiningCompoundModel
# Describe the detectors up front
detector1_spec = ModelSpec(
model_name="",
inference_host_address="@cloud", # Can be '@cloud', host:port, or '@local'
zoo_url="degirum/degirum",
)
detector2_spec = ModelSpec(
model_name="",
inference_host_address="@cloud",
zoo_url="degirum/degirum",
)
with detector1_spec.load_model() as detector1, detector2_spec.load_model() as detector2:
# Creating a compound model that merges results from both detectors
compound_detector = CombiningCompoundModel(detector1, detector2)
# Single frame inference using predict()
print("Using predict():")
single_result = compound_detector(remote_assets.cat)
print(single_result.results)
# Batch inference using predict_batch()
print("Using predict_batch():")
for batch_result in compound_detector.predict_batch(
[remote_assets.cat, remote_assets.two_cats]
):
print(batch_result.results)
```
{% endcode %}
See class-level documentation below for details on individual classes and additional configuration options.
## Classes
## ModelLike
`ModelLike`
Bases: `ABC`
A base class which provides a common interface for all models, similar to PySDK model class.
When calling `predict_batch(data)`, each item in `data` can be:
* A single frame (image/array/etc.), or
* A 2-element tuple in the form `(frame, frame_info)`.
The `frame_info` object (of any type) then appears in the final `InferenceResults.info` attribute, allowing you to carry custom metadata through the pipeline.
### ModelLike Methods
#### \_\_call\_\_(data)
`__call__(data)`
Perform a whole inference lifecycle on a single frame (callable alias to `predict()`).
Parameters:
| Name | Type | Description | Default |
| ------ | ----- | ------------------------------------------------------------------------------------ | ---------- |
| `data` | `any` | Inference input data, typically an image or array, or a tuple `(frame, frame_info)`. | *required* |
Returns:
| Type | Description |
| -------------------------- | ----------------------------------------------------------- |
| `InferenceResults or None` | The combined inference result object, or None if no result. |
#### predict(data)
`predict(data)`
Perform a whole inference lifecycle on a single frame.
Parameters:
| Name | Type | Description | Default |
| ------ | ----- | ------------------------------------------------------------------------------------ | ---------- |
| `data` | `any` | Inference input data, typically an image or array, or a tuple `(frame, frame_info)`. | *required* |
Returns:
| Type | Description |
| -------------------------- | ----------------------------------------------------------- |
| `InferenceResults or None` | The combined inference result object, or None if no result. |
#### predict\_batch(data)
`predict_batch(data)`
`abstractmethod`
Perform a whole inference lifecycle for all objects in the given iterator object (for example, `list`).
Each item in `data` can be a single frame (any type acceptable to the model) or a 2-element tuple `(frame, frame_info)`. In the latter case, `frame_info` is carried through and placed in `InferenceResults.info` for that frame.
Parameters:
| Name | Type | Description | Default |
| ------ | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
| `data` | `iterator` | Inference input data iterator object such as a list or a generator function. Each element returned by this iterator should be compatible with what regular PySDK models accept. | *required* |
Returns:
| Type | Description |
| ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------- |
| `Iterator[InferenceResults or None]` | A generator or iterator over the inference result objects (or None in non-blocking mode). This allows you to use the result in `for` loops. |
## FrameInfo
`FrameInfo`
Class to hold frame info.
By default, DeGirum PySDK allows you to pass any arbitrary object as 'frame info' alongside each frame in `predict_batch()`.
Attributes:
| Name | Type | Description |
| ------------ | ----- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `result1` | `any` | The result object produced by the first model in a compound pipeline. For instance, an [InferenceResults](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/postprocessor#degirum.postprocessor.inferenceresults) object. |
| `sub_result` | `int` | The index of a sub-result within `result1` (e.g., which bounding box led to this cropped image). |
## CompoundModelBase
`CompoundModelBase`
Bases: `ModelLike`
Compound model class which combines two models into one pipeline.
One model is considered *primary* (model1), and the other is *nested* (model2).
The primary model (`model1`) processes the input frames. Its results are then passed to the nested model (`model2`).
### Attributes
#### non\_blocking\_batch\_predict
`non_blocking_batch_predict`
`property` `writable`
Flag controlling whether `predict_batch()` operates in non-blocking mode for model1. In non-blocking mode, `predict_batch()` can yield `None` when no results are immediately available.
Returns:
| Type | Description |
| ------ | ------------------------------------------------------ |
| `bool` | True if non-blocking mode is enabled, False otherwise. |
### CompoundModelBase Methods
#### \_\_getattr\_\_(attr)
`__getattr__(attr)`
Fallback for getters of model-like attributes to the primary model (model1).
#### \_\_init\_\_(model1, ...)
`__init__(model1, model2)`
Constructor.
Parameters:
| Name | Type | Description | Default |
| -------- | ----------- | ----------------------------------------------------- | ---------- |
| `model1` | `ModelLike` | Model to be used for the first step of the pipeline. | *required* |
| `model2` | `ModelLike` | Model to be used for the second step of the pipeline. | *required* |
#### \_\_setattr\_\_(key, ...)
`__setattr__(key, value)`
Intercepts attempts to set attributes. If the attribute already exists on the instance, the class, or is being set inside `__init__`, the attribute is set normally. Otherwise, the attribute assignment is delegated to the primary model (`model1`) if defined. This prevents adding new attributes outside of `__init__`.
#### attach\_analyzers(analyzers)
`attach_analyzers(analyzers)`
Attach analyzers to a model.
Parameters:
| Name | Type | Description | Default |
| ----------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------ | ---------- |
| `analyzers` | `Union[ResultAnalyzerBase, list[ResultAnalyzerBase], None]` | A single analyzer, or a list of analyzer objects, or `None` to detach all analyzers. | *required* |
#### predict\_batch(data)
`predict_batch(data)`
Perform a whole inference lifecycle for all objects in the given iterator object (for example, `list`).
Works in a pipeline fashion
1. Pass input frames (or `(frame, frame_info)` tuples) to `model1`.
2. Use `queue_result1(result1)` to feed `model2`.
3. Collect `model2` results, transform them with `transform_result2(result2)`,
4. Yield the final output.
Parameters:
| Name | Type | Description | Default |
| ------ | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
| `data` | `iterator` | Inference input data iterator object such as a list or a generator function. Each element returned should be compatible with model inference requirements. | *required* |
Returns:
| Type | Description |
| ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `Iterator[InferenceResults or None]` | Generator object which iterates over the combined inference result objects (or None in non-blocking mode). This allows you to use the result in `for` loops. |
#### queue\_result1(result1)
`queue_result1(result1)`
`abstractmethod`
Process the result of the first model and put it into the queue.
Parameters:
| Name | Type | Description | Default |
| --------- | -------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------- | ---------- |
| `result1` | [InferenceResults](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/postprocessor#degirum.postprocessor.inferenceresults) | Prediction result of the first model. | *required* |
#### transform\_result2(result2)
`transform_result2(result2)`
`abstractmethod`
Transform (or integrate) the result of the second model.
Parameters:
| Name | Type | Description | Default |
| --------- | -------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------- | ---------- |
| `result2` | [InferenceResults](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/postprocessor#degirum.postprocessor.inferenceresults) | Prediction result of the second model. | *required* |
Returns:
| Type | Description |
| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| `InferenceResults or None` | Transformed/combined result to be returned by the compound model. If None, that means no result is produced at this iteration. |
## Nested Classes
### NonBlockingQueue
`NonBlockingQueue`
Bases: `Queue`
Specialized non-blocking queue which acts as an iterator to feed data to the nested model.
### NonBlockingQueue Methods
#### \_\_iter\_\_
`__iter__()`
Yield items from the queue until a `None` sentinel is reached.
Yields:
| Type | Description |
| ------------- | --------------------------------------------------------- |
| `any or None` | The item from the queue, or `None` if the queue is empty. |
## CombiningCompoundModel
`CombiningCompoundModel`
Bases: `CompoundModelBase`
Compound model class which executes two models in parallel on the same input data and merges their results.
Restriction: both models should produce the same type of inference results (e.g., both detection).
### CombiningCompoundModel Methods
#### queue\_result1(result1)
`queue_result1(result1)`
Queues the original image from `result1` and a new `FrameInfo` instance that references `result1`. This `(frame, frame_info)` tuple is then read by `model2`.
Parameters:
| Name | Type | Description | Default |
| --------- | -------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
| `result1` | [InferenceResults](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/postprocessor#degirum.postprocessor.inferenceresults) | Inference result from model1. We extract `result1.image` as the frame, and create a `FrameInfo` so we know which `result1` this frame corresponds to. | *required* |
#### transform\_result2(result2)
`transform_result2(result2)`
Merges results from `model2` into `result1` that was stored in `FrameInfo`.
This implementation appends the second model's inference results to the first model's result list.
Parameters:
| Name | Type | Description | Default |
| --------- | -------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- | ---------- |
| `result2` | [InferenceResults](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/postprocessor#degirum.postprocessor.inferenceresults) | Inference result of the second model, which has `info` attribute containing the `FrameInfo`. | *required* |
Returns:
| Type | Description |
| -------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------- |
| [InferenceResults](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/postprocessor#degirum.postprocessor.inferenceresults) | The merged inference results (model1 + model2). |
## CroppingCompoundModel
`CroppingCompoundModel`
Bases: `CompoundModelBase`
Compound model class which crops the original image according to results of the first model and then passes these cropped images to the second model.
Restriction: the first model should be of object detection type.
### CroppingCompoundModel Methods
#### \_\_init\_\_(model1, ...)
`__init__(model1, model2, crop_extent=0.0, crop_extent_option=CropExtentOptions.ASPECT_RATIO_NO_ADJUSTMENT)`
Constructor.
Parameters:
| Name | Type | Description | Default |
| -------------------- | ------------------- | ---------------------------------------------------------------- | ---------------------------- |
| `model1` | `ModelLike` | Object detection model that produces bounding boxes. | *required* |
| `model2` | `ModelLike` | Classification model that will process each cropped region. | *required* |
| `crop_extent` | `float` | Extent of cropping (in percent of bbox size) to expand the bbox. | `0.0` |
| `crop_extent_option` | `CropExtentOptions` | Method of applying extended crop to the input image for model2. | `ASPECT_RATIO_NO_ADJUSTMENT` |
#### queue\_result1(result1)
`queue_result1(result1)`
Put the original image into the queue, along with bounding boxes from the first model.
If no bounding boxes are detected, puts a small black image to keep the pipeline in sync.
Parameters:
| Name | Type | Description | Default |
| --------- | -------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------- | ---------- |
| `result1` | [InferenceResults](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/postprocessor#degirum.postprocessor.inferenceresults) | Prediction result of the first (object detection) model. | *required* |
## CroppingAndClassifyingCompoundModel
`CroppingAndClassifyingCompoundModel`
Bases: `CroppingCompoundModel`
Compound model class which
1. Runs an object detection (model1) to generate bounding boxes.
2. Crops each bounding box from the original image.
3. Runs a classification (model2) on each cropped image.
4. Patches the original detection results with the classification labels.
Restriction: first model must be object detection, second model must be classification.
### CroppingAndClassifyingCompoundModel Methods
#### \_\_init\_\_(model1, ...)
`__init__(model1, model2, crop_extent=0.0, crop_extent_option=CropExtentOptions.ASPECT_RATIO_NO_ADJUSTMENT)`
Constructor.
Parameters:
| Name | Type | Description | Default |
| -------------------- | ------------------- | --------------------------------------------------------- | ---------------------------- |
| `model1` | `ModelLike` | An object detection model producing bounding boxes. | *required* |
| `model2` | `ModelLike` | A classification model to classify each cropped region. | *required* |
| `crop_extent` | `float` | Extent of cropping (in percent of bbox size). | `0.0` |
| `crop_extent_option` | `CropExtentOptions` | Specifies how to adjust the bounding box before cropping. | `ASPECT_RATIO_NO_ADJUSTMENT` |
#### predict\_batch(data)
`predict_batch(data)`
Perform the full inference lifecycle for all objects in the given iterator (for example, `list`), but patch model1 bounding box labels with classification results from model2.
Parameters:
| Name | Type | Description | Default |
| ------ | ---------- | --------------------------------------------------------------------------------------------------------------------------- | ---------- |
| `data` | `iterator` | Iterator of input frames for model1. Each element returned by this iterator should be compatible with regular PySDK models. | *required* |
Returns:
| Type | Description |
| ---------------------------- | ------------------------------------------------------------------------------------------- |
| `Iterator[InferenceResults]` | Yields the detection results with patched classification labels after each frame completes. |
#### transform\_result2(result2)
`transform_result2(result2)`
Transform (patch) the classification result into the original detection results.
Parameters:
| Name | Type | Description | Default |
| --------- | -------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | ---------- |
| `result2` | [InferenceResults](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/postprocessor#degirum.postprocessor.inferenceresults) | Classification result of model2. | *required* |
Returns:
| Type | Description |
| -------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| `InferenceResults or None` | The detection result (from model1) patched with classification labels, or None if we haven't moved to a new frame yet. |
## CroppingAndDetectingCompoundModel
`CroppingAndDetectingCompoundModel`
Bases: `CroppingCompoundModel`
Compound model class which
1. Uses an object detection model (model1) to generate bounding boxes (ROIs).
2. Crops each bounding box from the original image.
3. Uses another object detection model (model2) to further detect objects in each cropped region.
4. Combines the results of the second model from all cropped regions, mapping coords back to the original image.
Optionally, you can add model1 detections to the final result and/or apply NMS.
When model1 results are added, each detection from model2 will have a `crop_index` field, indicating which bounding box from model1 it corresponds to.
Restriction
First model should be object detection or pseudo-detection model like `RegionExtractionPseudoModel`, second model should be object detection.
### CroppingAndDetectingCompoundModel Methods
#### \_\_init\_\_(model1, ...)
`__init__(model1, model2, *, crop_extent=0.0, crop_extent_option=CropExtentOptions.ASPECT_RATIO_NO_ADJUSTMENT, add_model1_results=False, nms_options=None)`
Constructor.
Parameters:
| Name | Type | Description | Default |
| -------------------- | ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------- |
| `model1` | `ModelLike` | Object detection model (or pseudo-detection). | *required* |
| `model2` | `ModelLike` | Object detection model. | *required* |
| `crop_extent` | `float` | Extent of cropping in percent of bbox size. | `0.0` |
| `crop_extent_option` | `CropExtentOptions` | Method of applying extended crop to the input image for model2. | `ASPECT_RATIO_NO_ADJUSTMENT` |
| `add_model1_results` | `bool` | If True, merges model1 detections into the final combined result. Each detection from model2 will have a `crop_index` field, indicating which bounding box from model1 it corresponds to. | `False` |
| `nms_options` | `Optional[NmsOptions]` | If provided, applies non-maximum suppression (NMS) to the combined result. | `None` |
#### predict\_batch(data)
`predict_batch(data)`
Perform the full inference lifecycle for all objects in the given iterator object (for example, `list`):
1. model1 detects or extracts bounding boxes (ROIs).
2. Each ROI is passed to model2 for detection.
3. model2 results for each ROI are merged and mapped back to original coordinates.
4. (Optional) NMS is applied and results from model1 can be included.
Parameters:
| Name | Type | Description | Default |
| ------ | ---------- | --------------------------------------------------------------------------------------------------------------------------- | ---------- |
| `data` | `iterator` | Iterator of input frames for model1. Each element returned by this iterator should be compatible with regular PySDK models. | *required* |
Returns:
| Type | Description |
| ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| `Iterator[InferenceResults]` | Generator object which iterates over final detection results with possibly merged bounding boxes, adjusted to original image coordinates. |
#### transform\_result2(result2)
`transform_result2(result2)`
Combine detection results from model2 for each bbox from model1, translating coordinates back to the original image space.
Parameters:
| Name | Type | Description | Default |
| --------- | -------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------- | ---------- |
| `result2` | [InferenceResults](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/postprocessor#degirum.postprocessor.inferenceresults) | Detection result of the second model. | *required* |
Returns:
| Type | Description |
| -------------------------- | --------------------------------------------------------------------------------------------- |
| `InferenceResults or None` | The final detection results for the previous frame if a new frame started, or None otherwise. |
## RegionExtractionPseudoModel
`RegionExtractionPseudoModel`
Bases: `ModelLike`
Pseudo-model class which extracts regions from a given image according to given ROI boxes.
### Attributes
#### custom\_postprocessor: Optional\[type]
`custom_postprocessor: Optional[type]`
`property` `writable`
Custom postprocessor class. Required for attaching analyzers to the pseudo-model.
When set, this replaces the default postprocessor with a user-defined postprocessor.
Returns:
| Type | Description |
| ---------------- | --------------------------------------------------------- |
| `Optional[type]` | The user-defined postprocessor class, or None if not set. |
#### non\_blocking\_batch\_predict
`non_blocking_batch_predict`
`property` `writable`
Controls non-blocking mode for `predict_batch()`.
Returns:
| Type | Description |
| ------ | ------------------------------------------------------ |
| `bool` | True if non-blocking mode is enabled; otherwise False. |
### RegionExtractionPseudoModel Methods
#### \_\_getattr\_\_(attr)
`__getattr__(attr)`
Fallback for getters of model-like attributes to `model2`.
#### \_\_init\_\_(roi\_list, ...)
`__init__(roi_list, model2, *, motion_detect=None)`
Constructor.
Parameters:
| Name | Type | Description | Default |
| --------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
| `roi_list` | `Union[list, ndarray]` | Can be: - list of ROI boxes in `[x1, y1, x2, y2]` format, - 2D NumPy array of shape (N, 4), - 3D NumPy array of shape (K, M, 4), which will be flattened. | *required* |
| `model2` | `Model` | The second model in the pipeline. | *required* |
| `motion_detect` | `Optional[MotionDetectOptions]` | \* When None, disabled motion detection. \* When not None, applies motion detection before extracting ROI boxes. Boxes without motion are skipped. | `None` |
#### predict\_batch(data)
`predict_batch(data)`
Perform a pseudo-inference that outputs bounding boxes defined in `roi_list`.
If motion detection is enabled, skip ROIs where motion is not detected.
Parameters:
| Name | Type | Description | Default |
| ------ | ---------- | -------------------------------------------------------------------------------------------------------------------------------- | ---------- |
| `data` | `iterator` | Iterator over the input images or frames. Each element returned by this iterator should be compatible with regular PySDK models. | *required* |
Returns:
| Type | Description |
| ------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------- |
| `Iterator[InferenceResults or None]` | Yields pseudo-inference results containing ROIs as bounding boxes, or yields None in non-blocking mode when no data is available. |
## NmsOptions
`NmsOptions`
`dataclass`
Options for non-maximum suppression (NMS) algorithm.
Attributes:
| Name | Type | Description |
| ---------------- | ----------------------- | ----------------------------------------------------------------------- |
| `threshold` | `float` | IoU or IoS threshold for box clustering (range \[0..1]). |
| `use_iou` | `bool` | If True, use IoU for box clustering, otherwise IoS. |
| `box_select` | `NmsBoxSelectionPolicy` | Box selection policy (e.g., keep the box with the highest probability). |
| `class_agnostic` | `bool` | If True, perform class-agnostic NMS. |
## MotionDetectOptions
`MotionDetectOptions`
`dataclass`
Options for motion detection algorithm.
Attributes:
| Name | Type | Description |
| ----------- | ------- | ------------------------------------------------------------------------------------------------------- |
| `threshold` | `float` | Threshold for motion detection \[0..1], representing fraction of changed pixels relative to frame size. |
| `look_back` | `int` | Number of frames to look back to detect motion. |
# Tile Compound Models
DeGirum Tools API Reference Guide. Process large images by tiling and combining model results.
{% hint style="info" %}
This API Reference is based on DeGirum Tools version 1.2.0.
{% endhint %}
## Tile Compound Models Module Overview
This module implements tiling-based compound models for object detection. It provides pseudo-models that extract image tiles and combine results from real detection models to efficiently process large images.
Key Features
* **Tile Extraction**: Generate local and global tiles with configurable overlap
* **Two-Stage Processing**: Run detection on each tile then merge results
* **Edge-Aware Fusion**: Optional fusion of detections near tile boundaries
* **Motion Filtering**: Skip tiles without motion to reduce computation
* **Result Management**: Translate box coordinates and apply NMS
Typical Usage
1. Create a `TileExtractorPseudoModel` with grid parameters
2. Wrap it in `TileModel` or a derived class along with a detection model
3. Iterate over `predict_batch` to obtain merged detection results
Integration Notes
* Designed for DeGirum PySDK models
* Works with compound model utilities such as cropping and NMS
* Supports customization of crop extent and overlap thresholds
* Compatible with local and cloud inference backends
Key Classes
* `TileExtractorPseudoModel`: Produces image tiles for downstream models
* `TileModel`: Runs detection on each tile and merges results
* `LocalGlobalTileModel`: Combines global and local tiles with size-based filtering
* `BoxFusionTileModel`: Fuses edge detections across tiles
Configuration Options
* Tile grid size and overlap
* Motion detection thresholds
* Edge fusion parameters
* NMS settings for final results
## Classes
## TileExtractorPseudoModel
`TileExtractorPseudoModel`
Bases: `ModelLike`
Extracts a grid of (optionally-overlapping) image tiles.
The class behaves like a DeGirum pseudo-model: instead of running inference it produces synthetic detection results whose bounding boxes correspond to tile coordinates. These results are then consumed by a second, real model in a two-stage compound pipeline.
Parameters:
| Name | Type | Description | Default |
| ----------------- | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
| `cols` | `int` | Number of columns in the tile grid. | *required* |
| `rows` | `int` | Number of rows in the tile grid. | *required* |
| `overlap_percent` | `float` | Desired overlap between neighbouring tiles, expressed as a fraction in \[0, 1]. | *required* |
| `model2` | `Model` | The downstream model that will receive each tile. | *required* |
| `global_tile` | `bool` | If True, emit an additional tile that represents the entire image (label "GLOBAL"). Mutually exclusive with tile\_mask and motion\_detect. Defaults to False. | `False` |
| `tile_mask` | `list[int] \| None` | Indices of tiles (0-based, row-major) to keep. Tiles not listed are skipped. Ignored when global\_tile is True. | `None` |
| `motion_detect` | `MotionDetectOptions \| None` | Enable per-tile motion filtering. Tiles in which less than threshold x area pixels changed during the last look\_back frames are suppressed. Ignored when global\_tile is True. | `None` |
Raises:
| Type | Description |
| ---------------- | --------------------------------------------- |
| `AssertionError` | If mutually-exclusive arguments are supplied. |
### TileExtractorPseudoModel Methods
#### \_\_init\_\_(cols, ...)
`__init__(cols, rows, overlap_percent, model2, *, global_tile=False, tile_mask=None, motion_detect=None)`
Constructor.
Parameters:
| Name | Type | Description | Default |
| ----------------- | ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
| `cols` | `int` | Number of columns to divide the image into. | *required* |
| `rows` | `int` | Number of rows to divide the image into. | *required* |
| `overlap_percent` | `float` | Desired overlap between neighbouring tiles. | *required* |
| `model2` | `Model` | Model which will be used as a second step of the compound model pipeline. | *required* |
| `global_tile` | `bool` | Indicates whether the global (whole) image should also be sent to model2. | `False` |
| `tile_mask` | `list[int] \| None` | Optional list of indices to keep during tile generation. Tile indices are counted starting from the top row to the bottom row, left to right. | `None` |
| `motion_detect` | `MotionDetectOptions \| None` | Motion detection options. When None, motion detection is disabled. When enabled, ROI boxes where motion is not detected will be skipped. | `None` |
#### predict\_batch(data)
`predict_batch(data)`
Perform whole inference lifecycle for all objects in given iterator object (for example, `list`).
Parameters:
| Name | Type | Description | Default |
| ------ | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
| `data` | `Iterable` | Inference input data iterator object such as list or generator function. Each element returned by this iterator should be compatible to that regular PySDK model accepts. | *required* |
Yields:
| Type | Description |
| ------------------ | -------------------------------------------------------------------------------------------- |
| `DetectionResults` | Combined inference result objects. This allows you directly using the result in `for` loops. |
## TileModel
`TileModel`
Bases: `CroppingAndDetectingCompoundModel`
Tiling wrapper that runs detection on every tile.
This compound model wires a TileExtractorPseudoModel (model 1) to a normal detection model (model 2). Each tile is cropped, passed to model 2, and the resulting boxes are translated back to the original image coordinates. Optionally, detections from model 1 can be merged or deduplicated via NMS.
Parameters:
| Name | Type | Description | Default |
| -------------------- | -------------------------- | ---------------------------------------------------------------------------------------------- | ---------------------------- |
| `model1` | `TileExtractorPseudoModel` | The tile generator. | *required* |
| `model2` | `Model` | A detection model compatible with model1's image backend. | *required* |
| `crop_extent` | `float` | Extra context (percent of box size) to include around every tile before passing it to model 2. | `0` |
| `crop_extent_option` | `CropExtentOptions` | How the extra context is applied. | `ASPECT_RATIO_NO_ADJUSTMENT` |
| `add_model1_results` | `bool` | If True, detections produced by model 1 are appended to the final result. | `False` |
| `nms_options` | `NmsOptions \| None` | Non-maximum suppression settings performed on the merged result. | `None` |
Attributes:
| Name | Type | Description |
| ------------------------- | ----- | ---------------------------------------------------------------------------------------------------- |
| `output_postprocess_type` | `str` | Mirrors model2.output\_postprocess\_type so downstream tooling recognises this as a detection model. |
Raises:
| Type | Description |
| ----------- | ------------------------------------------------ |
| `Exception` | If the two models use different image back-ends. |
### TileModel Methods
#### \_\_init\_\_(model1, ...)
`__init__(model1, model2, *, crop_extent=0, crop_extent_option=CropExtentOptions.ASPECT_RATIO_NO_ADJUSTMENT, add_model1_results=False, nms_options=None)`
Constructor.
Parameters:
| Name | Type | Description | Default |
| -------------------- | -------------------------- | ---------------------------------------------------------------- | ---------------------------- |
| `model1` | `TileExtractorPseudoModel` | Tile extractor pseudo-model. | *required* |
| `model2` | `Model` | PySDK object detection model. | *required* |
| `crop_extent` | `float` | Extent of cropping in percent of bbox size. | `0` |
| `crop_extent_option` | `CropExtentOptions` | Method of applying extending crop to the input image for model2. | `ASPECT_RATIO_NO_ADJUSTMENT` |
| `add_model1_results` | `bool` | True to add detections of model1 to the combined result. | `False` |
| `nms_options` | `NmsOptions \| None` | Non-maximum suppression (NMS) options. | `None` |
## LocalGlobalTileModel
`LocalGlobalTileModel`
Bases: `TileModel`
Runs a fine-/coarse tiling strategy with size-based result fusion.
Two kinds of tiles are produced:
* Local: the regular grid (fine resolution)
* Global: a single full-frame tile
After detection:
* Large objects (area >= large\_object\_threshold x image\_area) are kept only from the global tile.
* Small objects are kept only from the local tiles.
Parameters:
| Name | Type | Description | Default |
| ------------------------ | -------------------------- | ---------------------------------------------------------------------------------------------- | ---------------------------- |
| `model1` | `TileExtractorPseudoModel` | Must be configured with global\_tile=True. | *required* |
| `model2` | `Model` | Detection model run on each tile. | *required* |
| `large_object_threshold` | `float` | Area ratio separating "large" from "small" objects. Defaults to 0.01. | `0.01` |
| `crop_extent` | `float` | Extra context (percent of box size) to include around every tile before passing it to model 2. | `0` |
| `crop_extent_option` | `CropExtentOptions` | How the extra context is applied. | `ASPECT_RATIO_NO_ADJUSTMENT` |
| `add_model1_results` | `bool` | If True, detections produced by model 1 are appended to the final result. | `False` |
| `nms_options` | `NmsOptions \| None` | Non-maximum suppression settings performed on the merged result. | `None` |
Note
Unlike BoxFusionLocalGlobalTileModel, no edge fusion is applied; objects that overlap tile borders may be duplicated.
### LocalGlobalTileModel Methods
#### \_\_init\_\_(model1, ...)
`__init__(model1, model2, large_object_threshold=0.01, *, crop_extent=0, crop_extent_option=CropExtentOptions.ASPECT_RATIO_NO_ADJUSTMENT, add_model1_results=False, nms_options=None)`
Constructor.
Parameters:
| Name | Type | Description | Default |
| ------------------------ | -------------------------- | --------------------------------------------------------------------------------------------------------------------- | ---------------------------- |
| `model1` | `TileExtractorPseudoModel` | Tile extractor pseudo-model. | *required* |
| `model2` | `Model` | PySDK object detection model. | *required* |
| `large_object_threshold` | `float` | A threshold to determine if an object is considered large or not. This is relative to the area of the original image. | `0.01` |
| `crop_extent` | `float` | Extent of cropping in percent of bbox size. | `0` |
| `crop_extent_option` | `CropExtentOptions` | Method of applying extending crop to the input image for model2. | `ASPECT_RATIO_NO_ADJUSTMENT` |
| `add_model1_results` | `bool` | True to add detections of model1 to the combined result. | `False` |
| `nms_options` | `NmsOptions \| None` | Non-maximum suppression (NMS) options. | `None` |
#### transform\_result2(result2)
`transform_result2(result2)`
Transform result of the **second model**.
This implementation combines results of the **second model** over all bboxes detected by the first model, translating bbox coordinates to original image coordinates.
Parameters:
| Name | Type | Description | Default |
| --------- | ------------------ | ----------------------------------------- | ---------- |
| `result2` | `DetectionResults` | Detection result of the **second model**. | *required* |
Returns:
| Type | Description |
| -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `DetectionResults or None` | Combined results of the **second model** over all bboxes detected by the first model, where bbox coordinates are translated to original image coordinates. Returns None if no new frame is available. |
## BoxFusionTileModel
`BoxFusionTileModel`
Bases: `_EdgeMixin`, `TileModel`
TileModel with edge/overlap-aware bounding-box fusion.
After model 2 has produced detections for every tile, boxes whose centres fall within the user-defined edge band are compared with neighbours. If the 1-D IoU in either axis exceeds fusion\_threshold the boxes are merged (weighted-box fusion).
Parameters:
| Name | Type | Description | Default |
| -------------------- | -------------------------- | ---------------------------------------------------------------------------------------------- | ---------------------------- |
| `model1` | `TileExtractorPseudoModel` | Tile generator. | *required* |
| `model2` | `Model` | Detection model. | *required* |
| `edge_threshold` | `float` | Size of the edge band expressed as a fraction of tile width/height. Defaults to 0.02. | `0.02` |
| `fusion_threshold` | `float` | 1-D IoU needed to fuse two edge boxes. Defaults to 0.8. | `0.8` |
| `crop_extent` | `float` | Extra context (percent of box size) to include around every tile before passing it to model 2. | `0` |
| `crop_extent_option` | `CropExtentOptions` | How the extra context is applied. | `ASPECT_RATIO_NO_ADJUSTMENT` |
| `add_model1_results` | `bool` | If True, detections produced by model 1 are appended to the final result. | `False` |
| `nms_options` | `NmsOptions \| None` | Non-maximum suppression settings performed on the merged result. | `None` |
### BoxFusionTileModel Methods
#### \_\_init\_\_(model1, ...)
`__init__(model1, model2, edge_threshold=0.02, fusion_threshold=0.8, *, crop_extent=0, crop_extent_option=CropExtentOptions.ASPECT_RATIO_NO_ADJUSTMENT, add_model1_results=False, nms_options=None)`
Constructor.
Parameters:
| Name | Type | Description | Default |
| -------------------- | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------- |
| `model1` | `TileExtractorPseudoModel` | Tile extractor pseudo-model. | *required* |
| `model2` | `Model` | PySDK object detection model. | *required* |
| `edge_threshold` | `float` | A threshold to determine if an object is considered an edge detection or not. The edge\_threshold determines the amount of space next to the tiles edges where if a detection overlaps this space it is considered an edge detection. This edge space is relative (a percent) of the width/height of a tile. | `0.02` |
| `fusion_threshold` | `float` | A threshold to determine whether or not to fuse two edge detections. This corresponds to the 1D-IoU of two boxes, of either dimension. If the boxes overlap in both dimensions and one of the dimension's 1D-IoU is greater than the fusion\_threshold, the boxes are fused. | `0.8` |
| `crop_extent` | `float` | Extent of cropping in percent of bbox size. | `0` |
| `crop_extent_option` | `CropExtentOptions` | Method of applying extending crop to the input image for model2. | `ASPECT_RATIO_NO_ADJUSTMENT` |
| `add_model1_results` | `bool` | True to add detections of model1 to the combined result. | `False` |
| `nms_options` | `NmsOptions \| None` | Non-maximum suppression (NMS) options. | `None` |
#### transform\_result2(result2)
`transform_result2(result2)`
Transform result of the **second model**.
This implementation combines results of the **second model** over all bboxes detected by the first model, translating bbox coordinates to original image coordinates.
Parameters:
| Name | Type | Description | Default |
| --------- | ------------------ | ----------------------------------------- | ---------- |
| `result2` | `DetectionResults` | Detection result of the **second model**. | *required* |
Returns:
| Type | Description |
| -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `DetectionResults or None` | Combined results of the **second model** over all bboxes detected by the first model, where bbox coordinates are translated to original image coordinates. Returns None if no new frame is available. |
## BoxFusionLocalGlobalTileModel
`BoxFusionLocalGlobalTileModel`
Bases: `BoxFusionTileModel`
Local-/global-tiling plus edge fusion.
Combines the size-based filtering of LocalGlobalTileModel with the edge-aware fusion of BoxFusionTileModel.
Parameters:
| Name | Type | Description | Default |
| ------------------------ | -------------------------- | ---------------------------------------------------------------------------------------------- | ---------------------------- |
| `model1` | `TileExtractorPseudoModel` | Must output a global tile. | *required* |
| `model2` | `Model` | Detection model. | *required* |
| `large_object_threshold` | `float` | Area ratio that classifies a detection as "large". Defaults to 0.01. | `0.01` |
| `edge_threshold` | `float` | Width of the edge band as a fraction of tile dimensions. Defaults to 0.02. | `0.02` |
| `fusion_threshold` | `float` | 1-D IoU used by the fusion logic. Defaults to 0.8. | `0.8` |
| `crop_extent` | `float` | Extra context (percent of box size) to include around every tile before passing it to model 2. | `0` |
| `crop_extent_option` | `CropExtentOptions` | How the extra context is applied. | `ASPECT_RATIO_NO_ADJUSTMENT` |
| `add_model1_results` | `bool` | If True, detections produced by model 1 are appended to the final result. | `False` |
| `nms_options` | `NmsOptions \| None` | Non-maximum suppression settings performed on the merged result. | `None` |
### BoxFusionLocalGlobalTileModel Methods
#### \_\_init\_\_(model1, ...)
`__init__(model1, model2, large_object_threshold=0.01, edge_threshold=0.02, fusion_threshold=0.8, *, crop_extent=0, crop_extent_option=CropExtentOptions.ASPECT_RATIO_NO_ADJUSTMENT, add_model1_results=False, nms_options=None)`
Constructor.
Parameters:
| Name | Type | Description | Default |
| ------------------------ | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------- |
| `model1` | `TileExtractorPseudoModel` | Tile extractor pseudo-model. | *required* |
| `model2` | `Model` | PySDK object detection model. | *required* |
| `large_object_threshold` | `float` | A threshold to determine if an object is considered large or not. This is relative to the area of the original image. | `0.01` |
| `edge_threshold` | `float` | A threshold to determine if an object is considered an edge detection or not. The edge\_threshold determines the amount of space next to the tiles edges where if a detection overlaps this space it is considered an edge detection. This edge space is relative (a percent) of the width/height of a tile. | `0.02` |
| `fusion_threshold` | `float` | A threshold to determine whether or not to fuse two edge detections. This corresponds to the 1D-IoU of two boxes, of either dimension. If the boxes overlap in both dimensions and one of the dimension's 1D-IoU is greater than the fusion\_threshold, the boxes are fused. | `0.8` |
| `crop_extent` | `float` | Extent of cropping in percent of bbox size. | `0` |
| `crop_extent_option` | `CropExtentOptions` | Method of applying extending crop to the input image for model2. | `ASPECT_RATIO_NO_ADJUSTMENT` |
| `add_model1_results` | `bool` | True to add detections of model1 to the combined result. | `False` |
| `nms_options` | `NmsOptions \| None` | Non-maximum suppression (NMS) options. | `None` |
#### transform\_result2(result2)
`transform_result2(result2)`
Transform result of the **second model**.
This implementation combines results of the **second model** over all bboxes detected by the first model, translating bbox coordinates to original image coordinates.
Parameters:
| Name | Type | Description | Default |
| --------- | ------------------ | ----------------------------------------- | ---------- |
| `result2` | `DetectionResults` | Detection result of the **second model**. | *required* |
Returns:
| Type | Description |
| -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `DetectionResults or None` | Combined results of the **second model** over all bboxes detected by the first model, where bbox coordinates are translated to original image coordinates. Returns None if no new frame is available. |
# Streams
DeGirum Tools API Reference Guide. Streaming toolkit for building multi-threaded pipelines.
{% hint style="info" %}
This API Reference is based on DeGirum Tools version 1.2.0.
{% endhint %}
## Streaming Toolkit Overview
This module provides a streaming toolkit for building multi-threaded processing pipelines, where data (images, video frames, or arbitrary objects) flows through a series of *processing blocks* called gizmos. The toolkit allows you to:
* Acquire or generate data from one or more sources (e.g., camera feeds, video files).
* Process the data in a pipeline (possibly in parallel), chaining multiple gizmos together.
* Optionally display or save the processed data, or feed it into AI inference models.
* Orchestrate everything in a [Composition](/degirum-tools/streams/streams_base#composition), which manages the lifecycle (threads) of all connected gizmos.
### Core Concepts
1. **Stream**:
* Represents a queue of data items [StreamData](/degirum-tools/streams/streams_base#streamdata), such as frames from a camera or images from a directory.
* Gizmos push (`put`) data into Streams or read (`get`) data from them.
* Streams can optionally drop data (the oldest item) if they reach a specified maximum queue size, preventing pipeline bottlenecks.
2. **Gizmo**:
* A gizmo is a discrete processing node in the pipeline.
* Each gizmo runs in its own thread, pulling data from its input stream(s), processing it, and pushing results to its output stream(s).
* Example gizmos include:
* Video-sourcing gizmos that read frames from a webcam or file.
* AI inference gizmos that run a model on incoming frames.
* Video display or saving gizmos that show or store processed frames.
* Gizmos that perform transformations (resizing, cropping, analyzing) on data.
* Gizmos communicate via Streams. A gizmo output Stream can feed multiple downstream gizmos.
* Gizmos keep a list of input streams that they are connected to.
* Gizmos own their input streams.
3. **Composition**:
* A [Composition](/degirum-tools/streams/streams_base#composition) is a container that holds and manages multiple gizmos (and their Streams).
* Once gizmos are connected, you can call `composition.start()` to begin processing. Each gizmo `run()` method executes in a dedicated thread.
* Call `composition.stop()` to gracefully stop processing and wait for threads to finish.
4. **StreamData** and **StreamMeta**:
* Each item in the pipeline is encapsulated by a [StreamData](/degirum-tools/streams/streams_base#streamdata) object, which holds:
* `data`: The actual payload (e.g., an image array, a frame).
* `meta`: A [StreamMeta](/degirum-tools/streams/streams_base#streammeta) object that can hold extra metadata from each gizmo (e.g., a detection result, timestamps, bounding boxes, etc.).
* Gizmos can append to [StreamMeta](/degirum-tools/streams/streams_base#streammeta) so that metadata accumulates across the pipeline.
5. **Metadata Flow (StreamMeta)**:
* How [StreamMeta](/degirum-tools/streams/streams_base#streammeta) works:
* [StreamMeta](/degirum-tools/streams/streams_base#streammeta) itself is a container that can hold any number of "meta info" objects.
* Each meta info object is "tagged" with one or more string tags, such as `"dgt_video"`, `"dgt_inference"`, etc.
* You append new meta info by calling `meta.append(my_info, [list_of_tags])`.
* You can retrieve meta info objects by searching with `meta.find("tag")` (returns *all* matches) or `meta.find_last("tag")` (returns the *most recent* match).
* **Important**: A gizmo generally clones (`.clone()`) the incoming [StreamMeta](/degirum-tools/streams/streams_base#streammeta) before appending its own metadata to avoid upstream side effects.
* This design lets each gizmo add new metadata, while preserving what was provided by upstream gizmos.
* High-Level Example:
* A camera gizmo outputs frames with meta tagged `"dgt_video"` containing properties like FPS, width, height, etc.
* An AI inference gizmo downstream takes `StreamData(data=frame, meta=...)`, runs inference, then:
1. Clones the metadata container.
2. Appends its inference results under the `"dgt_inference"` tag.
* If *two* AI gizmos run in series, both will append metadata with the same `"dgt_inference"` tag. A later consumer can call `meta.find("dgt_inference")` to get both sets of results or `meta.find_last("dgt_inference")` to get the most recent result.
### Basic Usage Example
A simple pipeline might look like this:
{% code overflow="wrap" %}
```python
import time
import cv2
import degirum_tools.streams as streams
# Create gizmos. If you are on a laptop or have a webcam attached,
# VideoSourceGizmo(0) will typically select the default camera.
video_source = streams.VideoSourceGizmo(0)
video_display = streams.VideoDisplayGizmo("Camera Preview")
# Connect them
video_source >> video_display
# Build composition
composition = streams.Composition(video_source, video_display)
composition.start(wait=False) # Don't block main thread
try:
start_time = time.time()
while time.time() - start_time < 10: # Run for 10 seconds
cv2.waitKey(5) # Let OpenCV handle window events
finally:
composition.stop()
cv2.destroyAllWindows()
```
{% endcode %}
### Key Steps
1. **Create** your gizmos (e.g., `VideoSourceGizmo`, `VideoDisplayGizmo`, AI inference gizmos, etc.).
2. **Connect** them together using the `>>` operator (or `connect_to()` method) to form a processing graph. E.g.:
{% code overflow="wrap" %}
```
source >> processor >> sink
```
{% endcode %}
3. **Initialize** a [Composition](/degirum-tools/streams/streams_base#composition) with the top-level gizmo(s).
4. **Start** the [Composition](/degirum-tools/streams/streams_base#composition) to launch each gizmo in its own thread.
5. (Optional) **Wait** for the pipeline to finish or perform other tasks. You can query statuses, queue sizes, or get partial results in real time.
6. **Stop** the pipeline when done.
### Advanced Topics
* **Non-blocking vs Blocking**: Streams can drop items if configured (`allow_drop=True`) to handle real-time feeds.
* **Multiple Inputs or Outputs**: Some gizmos can have multiple input streams and/or broadcast results to multiple outputs.
* **Error Handling**: If any gizmo encounters an error, the [Composition](/degirum-tools/streams/streams_base#composition) can stop the whole pipeline, allowing you to handle exceptions centrally.
For practical code examples, see the `dgstreams_demo.ipynb` notebook in the PySDKExamples.
## Functions
#### load\_composition(description, ...)
`load_composition(description, global_context=None, local_context=None)`
Load a [Composition](/degirum-tools/streams/streams_base#composition) of gizmos and connections from a description.
The description can be provided as a JSON/YAML file path, a YAML string, or a Python dictionary conforming to the JSON schema defined in `composition_definition_schema`.
Composition Description Schema (YAML):
{% code overflow="wrap" %}
```yaml
type: object
additionalProperties: false
required: [gizmos, connections]
properties:
vars:
type: object
description: The collection of variables, keyed by variable name
additionalProperties: false
patternProperties:
"^[a-zA-Z_][a-zA-Z0-9_]*$":
oneOf:
- type: [string, number, boolean, array]
description: The variable value; can be $(expression) to evaluate
- type: object
description: The only object key is the class name to instantiate; value are the constructor parameters
additionalProperties: false
minProperties: 1
maxProperties: 1
patternProperties:
"^[a-zA-Z_][a-zA-Z0-9_.]*$":
type: object
description: The constructor parameters of the object
additionalProperties: true
gizmos:
type: object
description: The collection of gizmos, keyed by gizmo instance name
additionalProperties: false
patternProperties:
"^[a-zA-Z_][a-zA-Z0-9_]*$":
oneOf:
- type: string
description: The gizmo class name to instantiate (if no parameters are needed)
- type: object
description: The only object key is the class name to instantiate; value are the constructor parameters
additionalProperties: false
minProperties: 1
maxProperties: 1
patternProperties:
"^[a-zA-Z_][a-zA-Z0-9_.]*$":
type: object
description: The constructor parameters of the gizmo
additionalProperties: true
connections:
type: array
description: The list of connections between gizmos
items:
type: array
description: The connection between gizmos
items:
oneOf:
- type: string
- type: array
description: Gizmo with input index
prefixItems:
- type: string
- type: number
items: false
```
{% endcode %}
Parameters:
| Name | Type | Description | Default |
| ---------------- | ------------- | ------------------------------------------------------------------------------------------------------------- | ---------- |
| `description` | `str or dict` | A YAML string or dict describing the Composition, or a path to a .json/.yaml file containing the description. | *required* |
| `global_context` | `dict` | Global context for evaluating expressions in the description (variables, etc.). Defaults to None. | `None` |
| `local_context` | `dict` | Local context for evaluating expressions. Defaults to None. | `None` |
Returns:
| Name | Type | Description |
| ------------- | ------------- | ------------------------------------------------------------------------------------------------------------------ |
| `Composition` | `Composition` | A [Composition](/degirum-tools/streams/streams_base#composition) object representing the described gizmo pipeline. |
# Streams Base
DeGirum Tools API Reference Guide. Defines Stream, Gizmo and Composition core classes.
{% hint style="info" %}
This API Reference is based on DeGirum Tools version 1.2.0.
{% endhint %}
## Classes
## StreamMeta
`StreamMeta`
Stream metainfo class (metadata container).
**Overview**
* A [StreamMeta](/degirum-tools/streams#streammeta) instance is a container that holds a chronologically ordered list of metainfo objects (called "meta infos") produced by gizmos in a streaming pipeline.
* Each time a gizmo adds new metadata (e.g., inference results, resizing information), it is *appended* to the tail of this list.
* The gizmo may associate the appended metadata with one or more *tags*, so that downstream gizmos or the user can retrieve specific metadata objects by those tags.
**Appending and Tagging**
* To store new metadata, a gizmo calls `self.meta.append(meta_obj, tags)`, where `meta_obj` is the metadata to attach, and `tags` is a string or list of strings labeling that metadata (e.g., "tag\_inference", "tag\_resize").
* Internally, [StreamMeta](/degirum-tools/streams#streammeta) keeps track of a list of appended objects and a mapping of tags to the indices in that list.
**Retrieving Metadata**
* You can retrieve all metadata objects tagged with a certain tag via `find(tag)`, which returns a list of all matching objects in the order they were appended.
* You can retrieve only the most recently appended object with `find_last(tag)`.
* For example, an inference gizmo might attach an inference result with the tag `"tag_inference"`, so a downstream gizmo can do: `inference_result = stream_data.meta.find_last("tag_inference")`.
* If no metadata matches the requested tag, these methods return `[]` or `None`.
**Modifications and Cloning**
* **Important**: Never modify a received [StreamMeta](/degirum-tools/streams#streammeta) or its stored objects in-place, because it may create side effects for upstream components. Call `clone()` if you need to make changes. `clone()` creates a shallow copy of the metainfo list and a copy of the tag-index map.
* If you want to remove the most recent entry associated with a certain tag, call `remove_last(tag)` (occasionally useful in advanced pipeline scenarios).
**Typical Usage**
A typical processing pipeline might look like
1. A video source gizmo creates a new [StreamMeta](/degirum-tools/streams#streammeta), appends frame info under tag `"Video"`.
2. A resizing gizmo appends new dimension info under tag `"Resize"`.
3. An AI inference gizmo appends the inference result under tag `"Inference"`.
4. A display gizmo reads the final metadata to overlay bounding boxes, etc.
This incremental metadata accumulation is extremely flexible and allows each gizmo to contribute to a unified record of the data's journey.
**Example**:
{% code overflow="wrap" %}
```python
# In a gizmo, produce meta and append:
data.meta.append({"new_width": 640, "new_height": 480}, "Resize")
# In a downstream gizmo:
resize_info = data.meta.find_last("Resize")
if resize_info:
w, h = resize_info["new_width"], resize_info["new_height"]
```
{% endcode %}
**CAUTION**: Never modify the existing metadata objects in place. If you need to adapt previously stored metadata for your own use, first copy the data structure or call `clone()` on the [StreamMeta](/degirum-tools/streams#streammeta).
### StreamMeta Methods
#### \_\_init\_\_(meta=None, ...)
`__init__(meta=None, tags=[])`
Constructor.
Parameters:
| Name | Type | Description | Default |
| ------ | ----------------------- | ------------------------------------------------------------------ | ------- |
| `meta` | `Any` | Initial metainfo object. Defaults to None. | `None` |
| `tags` | `Union[str, List[str]]` | Tag or list of tags to associate with the initial metainfo object. | `[]` |
#### append(meta, ...)
`append(meta, tags=[])`
Append a metainfo object to this StreamMeta.
Parameters:
| Name | Type | Description | Default |
| ------ | ----------------------- | ---------------------------------------------------------- | ---------- |
| `meta` | `Any` | The metainfo object to append. | *required* |
| `tags` | `Union[str, List[str]]` | Tag or list of tags to associate with the metainfo object. | `[]` |
#### clone
`clone()`
Shallow clone this StreamMeta.
This creates a copy of the internal list and tags dictionary, but does not deep-copy the metainfo objects.
Returns:
| Name | Type | Description |
| ------------ | ------------ | ----------------------------- |
| `StreamMeta` | `StreamMeta` | A cloned StreamMeta instance. |
#### find(tag)
`find(tag)`
Find metainfo objects by tag.
Parameters:
| Name | Type | Description | Default |
| ----- | ----- | ---------------------- | ---------- |
| `tag` | `str` | The tag to search for. | *required* |
Returns:
| Type | Description |
| ----------- | ------------------------------------------------------------------------------- |
| `List[Any]` | List\[Any]: A list of metainfo objects that have the given tag (empty if none). |
#### find\_last(tag)
`find_last(tag)`
Find the last metainfo object with a given tag.
Parameters:
| Name | Type | Description | Default |
| ----- | ----- | ---------------------- | ---------- |
| `tag` | `str` | The tag to search for. | *required* |
Returns:
| Name | Type | Description |
| ----- | ---------- | ----------------------------------------------------------------------- |
| `Any` | `optional` | The last metainfo object associated with the tag, or None if not found. |
#### get(idx)
`get(idx)`
Get a metainfo object by index.
Parameters:
| Name | Type | Description | Default |
| ----- | ----- | --------------------------------------------- | ---------- |
| `idx` | `int` | The index of the metainfo object in the list. | *required* |
Returns:
| Name | Type | Description |
| ----- | ----- | ------------------------------------------- |
| `Any` | `Any` | The metainfo object at the specified index. |
#### remove\_last(tag)
`remove_last(tag)`
Remove the last metainfo object associated with a tag.
Parameters:
| Name | Type | Description | Default |
| ----- | ----- | ---------------------------------------------------------------- | ---------- |
| `tag` | `str` | The tag whose last associated metainfo object should be removed. | *required* |
## StreamData
`StreamData`
Single data element of the streaming pipeline.
### StreamData Methods
#### \_\_init\_\_(data, ...)
`__init__(data, meta=StreamMeta())`
Constructor.
Parameters:
| Name | Type | Description | Default |
| ------ | ------------ | -------------------------------------------------------------------------- | -------------- |
| `data` | `Any` | The data payload. | *required* |
| `meta` | `StreamMeta` | The metainfo associated with the data. Defaults to a new empty StreamMeta. | `StreamMeta()` |
#### append\_meta(meta, ...)
`append_meta(meta, tags=[])`
Append an additional metainfo object to this [StreamData](/degirum-tools/streams#streamdata)'s metadata.
Parameters:
| Name | Type | Description | Default |
| ------ | ----------- | ------------------------------------------------------------- | ---------- |
| `meta` | `Any` | The metainfo object to append. | *required* |
| `tags` | `List[str]` | Tags to associate with this metainfo object. Defaults to \[]. | `[]` |
## Stream
`Stream`
Bases: `Queue`
Queue-based iterable stream with optional item drop.
### Stream Methods
#### \_\_init\_\_(maxsize=0, ...)
`__init__(maxsize=0, allow_drop=False)`
Constructor.
Parameters:
| Name | Type | Description | Default |
| ------------ | ------ | -------------------------------------------------------------------------------------------- | ------- |
| `maxsize` | `int` | Maximum stream depth (queue size); use 0 for unlimited depth. Defaults to 0. | `0` |
| `allow_drop` | `bool` | If True, allow dropping the oldest item when the stream is full on put(). Defaults to False. | `False` |
Raises:
| Type | Description |
| ----------- | ------------------------------------------------------ |
| `Exception` | If maxsize is non-zero and less than `min_queue_size`. |
#### \_\_iter\_\_
`__iter__()`
Return an iterator over the stream's items.
#### close
`close()`
Close the stream by inserting a poison pill.
#### put(item, ...)
`put(item, block=True, timeout=None)`
Put an item into the stream, with optional dropping.
If the stream is full and `allow_drop` is True, the oldest item will be removed to make room.
Parameters:
| Name | Type | Description | Default |
| --------- | ------- | ------------------------------------------------------------------------------------------ | ---------- |
| `item` | `Any` | The item to put. | *required* |
| `block` | `bool` | Whether to block if the stream is full (ignored if dropping is enabled). Defaults to True. | `True` |
| `timeout` | `float` | Timeout in seconds for the blocking put. Defaults to None (no timeout). | `None` |
## Gizmo
`Gizmo`
Base class for all gizmos (streaming pipeline processing blocks).
Each gizmo owns zero or more input streams that deliver data for processing (data-generating gizmos have no input stream).
A gizmo can be connected to other gizmos to receive data from them. One gizmo can broadcast data to multiple others (a single gizmo output feeding multiple destinations).
A data element moving through the pipeline is a tuple `(data, meta)` where:
* `data` is the raw data (e.g., an image, a frame, or any object),
* `meta` is a [StreamMeta](/degirum-tools/streams#streammeta) object containing accumulated metadata.
Subclasses must implement the abstract `run()` method to define a gizmo processing loop. The `run()` method is launched in a separate thread by the [Composition](/degirum-tools/streams#composition) and should run until no more data is available or until an abort signal is set.
The `run()` implementation should:
* Periodically check the `_abort` flag (set via `abort()`) to see if it should terminate.
* Handle poison pills (`Stream._poison`) if they appear in the input streams, which signal "no more data."
Below is a minimal example similar to `ResizingGizmo`. This gizmo simply reads items from its single input, processes them, and sends results downstream until either `_abort` is set or the input stream is exhausted:
{% code overflow="wrap" %}
```python
def run(self):
# For a single-input gizmo, iterate over all data in input #0
for item in self.get_input(0):
# If we were asked to abort, break out immediately
if self._abort:
break
# item is a StreamData object: item.data is the image/frame, item.meta is the metadata
input_image = item.data
# 1) Do the resizing (your logic can use OpenCV, PIL, etc.)
resized_image = do_resize(input_image, width=640, height=480)
# 'do_resize' is just a placeholder; you'd implement your own resizing function.
# 2) Update the metadata
# - Clone the existing metadata first.
# - In Python all objects are passed by reference, so if you do not clone but try A >> B and A >> C, C will receive the meta object modified by B.
out_meta = item.meta.clone()
out_meta.append(
{
"frame_width": 640,
"frame_height": 480,
"method": "your_resize_method"
},
tags=self.get_tags()
)
# 3) Send the processed item downstream
self.send_result(StreamData(resized_image, out_meta))
```
{% endcode %}
Class Variables
key\_gizmo (str): Key name for gizmo name in timing metadata entries. Defaults to "gizmo". key\_timestamp (str): Key name for timestamp in timing metadata entries. Defaults to "timestamp".
Notes
* If your gizmo has multiple inputs, you can call `self.get_input(i)` for each input or iterate over `self.get_inputs()` if you need to merge or synchronize multiple streams.
* Always check `_abort` periodically inside your main loop if your gizmo could run for a long time or block on I/O.
* When done, you do not need to manually send poison pills; the [Composition](/degirum-tools/streams#composition) handles closing any downstream streams once each gizmo `run()` completes.
* If, instead of `self.get_input(0)`, you use `self.get_input(0).get()` or `.get_nowait()`, you must check if you receive a poison pill.
* In simple loops, `self.get_input(0)` will terminate the loop.
* In multi-input gizmos where simple nested for-loops aren't usable, get\_nowait() is typically used to read input streams.
* This way the gizmo code may query all inputs on a non-blocking manner and properly terminate loops.
### Gizmo Methods
#### \_\_getitem\_\_(index)
`__getitem__(index)`
Enable `gizmo[index]` syntax for specifying connections.
Returns a tuple `(self, input_stream)` which can be used on the right side of the `>>` operator for connecting gizmos (e.g., `source_gizmo >> target_gizmo[index]`).
Parameters:
| Name | Type | Description | Default |
| ------- | ----- | ------------------------------------- | ---------- |
| `index` | `int` | The input stream index on this gizmo. | *required* |
Returns:
| Type | Description |
| ----------------- | -------------------------------------------------------------------- |
| `(Gizmo, Stream)` | tuple: A tuple of (this gizmo, the Stream at the given input index). |
#### \_\_init\_\_(input\_stream\_sizes=\[])
`__init__(input_stream_sizes=[])`
Constructor.
Parameters:
| Name | Type | Description | Default |
| -------------------- | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- |
| `input_stream_sizes` | `List[tuple]` | List of (maxsize, allow\_drop) tuples for each input stream. Use an empty list for no inputs. Each tuple defines the input stream's depth (0 means unlimited) and whether dropping is allowed. | `[]` |
#### \_\_rshift\_\_(other\_gizmo)
`__rshift__(other_gizmo)`
Connect another gizmo to this gizmo using the `>>` operator.
This implements the right-shift operator, allowing syntax like `source >> target` or `source >> target[input_index]`.
Parameters:
| Name | Type | Description | Default |
| ------------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------ | ---------- |
| `other_gizmo` | `Gizmo or tuple` | Either a Gizmo to connect (assumes input 0), or a tuple `(gizmo, inp)` where `inp` is the input index or Stream of that gizmo. | *required* |
Returns:
| Name | Type | Description |
| ------- | ------- | ------------------------------------------------------------------ |
| `Gizmo` | `Gizmo` | The source gizmo (other\_gizmo), enabling chaining of connections. |
#### abort(abort=True)
`abort(abort=True)`
Set or clear the abort flag to stop the run loop.
Parameters:
| Name | Type | Description | Default |
| ------- | ------ | ----------------------------------------------------------------------- | ------- |
| `abort` | `bool` | True to request aborting the run loop, False to clear the abort signal. | `True` |
#### connect\_to(other\_gizmo, ...)
`connect_to(other_gizmo, inp=0)`
Connect an input stream of this gizmo to another gizmo's output.
Parameters:
| Name | Type | Description | Default |
| ------------- | --------------- | -------------------------------------------------------------------------------------------- | ---------- |
| `other_gizmo` | `Gizmo` | The source gizmo to connect from. | *required* |
| `inp` | `int or Stream` | The input index of this gizmo (or an input Stream) to use for the connection. Defaults to 0. | `0` |
Returns:
| Name | Type | Description |
| ------- | ------- | ------------------------------- |
| `Gizmo` | `Gizmo` | This gizmo (to allow chaining). |
#### get\_connected
`get_connected()`
Recursively gather all gizmos connected to this gizmo.
Returns:
| Name | Type | Description |
| ----- | ----- | --------------------------------------------------------------------------------- |
| `set` | `set` | A set of Gizmo objects that are connected (directly or indirectly) to this gizmo. |
#### get\_input(inp)
`get_input(inp)`
Get a specific input stream by index.
Parameters:
| Name | Type | Description | Default |
| ----- | ----- | -------------------------------------- | ---------- |
| `inp` | `int` | Index of the input stream to retrieve. | *required* |
Returns:
| Name | Type | Description |
| -------- | -------- | ------------------------------------ |
| `Stream` | `Stream` | The input stream at the given index. |
Raises:
| Type | Description |
| ----------- | -------------------------------------------- |
| `Exception` | If the requested input index does not exist. |
#### get\_inputs
`get_inputs()`
Get all input streams of this gizmo.
Returns:
| Type | Description |
| -------------- | -------------------------------------------- |
| `List[Stream]` | List\[Stream]: List of input stream objects. |
#### get\_tags
`get_tags()`
Get the list of meta tags for this gizmo.
Returns:
| Type | Description |
| ----------- | ------------------------------------------------------------------------------ |
| `List[str]` | List\[str]: Tags associated with this gizmo (by default, just its class name). |
#### require\_tags(inp)
`require_tags(inp)`
Get the list of meta tags this gizmo requires in upstream meta for a specific input.
Returns:
| Type | Description |
| ----------- | --------------------------------------------------------------------------------- |
| `List[str]` | List\[str]: Tags required by this gizmo in upstream meta for the specified input. |
#### run
`run()`
Run the gizmo's processing loop.
This method should retrieve data from input streams (if any), process it, and send results to outputs. Subclasses implement this method to define the gizmo's behavior.
Important guidelines for implementation
* Check `self._abort` periodically and exit the loop if it becomes True.
* If reading from an input stream via `get()` or `get_nowait()`, check for the poison pill (`Stream._poison`). If encountered, exit the loop.
* For example, a typical single-input loop could be:
{% code overflow="wrap" %}
```python
for data in self.get_input(0):
if self._abort:
break
result = self.process(data)
self.send_result(result)
```
{% endcode %}
There is no need to send a poison pill to outputs; the [Composition](/degirum-tools/streams#composition) will handle closing output streams.
#### send\_result(data)
`send_result(data)`
Send a result to all connected output streams.
Automatically adds timing metadata (gizmo name and timestamp) to each result.
Parameters:
| Name | Type | Description | Default |
| ------ | -------------------- | ------------------------------------------------------------------------------------------------------ | ---------- |
| `data` | `StreamData or None` | The data result to send. If None (or a poison pill) is provided, all connected outputs will be closed. | *required* |
## Composition
`Composition`
Orchestrates and runs a set of connected gizmos.
Usage
1. Add all gizmos to the composition using `add()` or by calling the composition instance.
2. Connect the gizmos together using `connect_to()` or the `>>` operator.
3. Start the execution by calling `start()`.
4. To stop the execution, call `stop()` (or use the composition as a context manager).
### Composition Methods
#### \_\_call\_\_(gizmo)
`__call__(gizmo)`
Add a gizmo to this composition (callable syntax).
Equivalent to calling `add(gizmo)`.
Parameters:
| Name | Type | Description | Default |
| ------- | ------- | ----------------- | ---------- |
| `gizmo` | `Gizmo` | The gizmo to add. | *required* |
Returns:
| Name | Type | Description |
| ------- | ------- | --------------- |
| `Gizmo` | `Gizmo` | The same gizmo. |
#### \_\_enter\_\_
`__enter__()`
Start the composition when entering a context (without waiting).
Returns:
| Name | Type | Description |
| ------------- | ------------- | ------------------------------------------------------------------- |
| `Composition` | `Composition` | The composition itself (so that context manager usage is possible). |
#### \_\_exit\_\_(exc\_type, ...)
`__exit__(exc_type, exc_val, exc_tb)`
On exiting a context, wait for all gizmos to finish (and raise any errors).
Automatically calls `wait()` to ensure all threads have completed.
#### \_\_init\_\_(\*gizmos)
`__init__(*gizmos)`
Initialize the composition with optional initial gizmos.
Parameters:
| Name | Type | Description | Default |
| --------- | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------- |
| `*gizmos` | `Gizmo or Iterator[Gizmo]` | Optional gizmos (or iterables of gizmos) to add initially. If a Gizmo is provided, all gizmos connected to it (including itself) are added. If an iterator of gizmos is provided, all those gizmos (and their connected gizmos) are added. | `()` |
#### add(gizmo)
`add(gizmo)`
Add a gizmo to this composition.
Parameters:
| Name | Type | Description | Default |
| ------- | ------- | ----------------- | ---------- |
| `gizmo` | `Gizmo` | The gizmo to add. | *required* |
Returns:
| Name | Type | Description |
| ------- | ------- | -------------------------------- |
| `Gizmo` | `Gizmo` | The same gizmo, for convenience. |
#### get\_bottlenecks
`get_bottlenecks()`
Get gizmos that experienced input queue bottlenecks in the last run.
For this to be meaningful, the composition must have been started with `detect_bottlenecks=True`.
Returns:
| Type | Description |
| ------------ | ------------------------------------------------------------------------------------------------------------------- |
| `List[dict]` | A list of dictionaries where each key is a gizmo name and the value is the number of frames dropped for that gizmo. |
#### get\_current\_queue\_sizes
`get_current_queue_sizes()`
Get current sizes of each gizmo's input queues. Can be used to analyze deadlocks.
Returns:
| Type | Description |
| ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `List[dict]` | A list of dictionaries where each key is a gizmo name and the value is a list containing the gizmo's result count followed by the size of each of its input queues. |
#### request\_stop
`request_stop()`
Signal all gizmos in this composition to stop (abort).
This sets each gizmo's abort flag, clears all remaining items from their input queues, and sends poison pills to unblock any waiting gets. This method does not wait for threads to finish; call `wait()` to join threads.
#### start(\*, ...)
`start(*, wait=True, detect_bottlenecks=False)`
Start the execution of all gizmos (each in its own thread): launch run() method of every registered gizmo.
Parameters:
| Name | Type | Description | Default |
| -------------------- | ------ | ----------------------------------------------------------------------------------------------------------------- | ------- |
| `wait` | `bool` | If True, wait until all gizmos have finished. Defaults to True. | `True` |
| `detect_bottlenecks` | `bool` | If True, enable frame dropping on all streams to detect bottlenecks (see `get_bottlenecks()`). Defaults to False. | `False` |
#### stop
`stop()`
Stop the composition by aborting all gizmos and waiting for all threads to finish.
#### wait
`wait()`
Wait for all gizmo threads in the composition to finish.
Raises:
| Type | Description |
| ----------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| `Exception` | If the composition has not been started, or if any gizmo raised an error during execution (the exception message will contain details). |
## Watchdog
`Watchdog`
Monitors activity rate and timing using tick events and a filtered TPS estimate.
Tracks the frequency of `tick()` calls and the time since the last one. The `check()` method evaluates whether the activity is recent enough and meets a minimum TPS (ticks per second) threshold, using a single-pole low-pass filter to smooth TPS estimation.
### Watchdog Methods
#### \_\_init\_\_(time\_limit, ...)
`__init__(time_limit, tps_threshold, smoothing=0.9)`
Initializes the Watchdog.
Parameters:
| Name | Type | Description | Default |
| --------------- | ------- | ------------------------------------------------------------- | ---------- |
| `time_limit` | `float` | Maximum allowed time (in seconds) since the last tick. | *required* |
| `tps_threshold` | `float` | Minimum required filtered ticks per second. | *required* |
| `smoothing` | `float` | Smoothing factor for the low-pass filter (0 < smoothing < 1). | `0.9` |
#### check
`check()`
Checks whether the watchdog is within the allowed timing and TPS threshold.
Returns:
| Type | Description |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Tuple[bool, float]` | Tuple\[bool, float]: A tuple containing: - bool: True if the watchdog is active (recent enough and meets TPS threshold), False otherwise. - float: The current TPS value. |
#### tick
`tick()`
Records the current timestamp and updates the filtered TPS estimate.
Should be called regularly to track system activity. Uses the time between ticks to calculate instantaneous TPS and applies a low-pass filter to smooth the estimate.
# Streams Gizmos
DeGirum Tools API Reference Guide. Reusable gizmos for video, inference, display, etc.
{% hint style="info" %}
This API Reference is based on DeGirum Tools version 1.2.0.
{% endhint %}
## Classes
## VideoSourceGizmo
`VideoSourceGizmo`
Bases: `Gizmo`
Video source gizmo with OpenCV and GStreamer support.
Captures frames from a video source (camera, video file, etc.) and outputs them as [StreamData](/degirum-tools/streams/streams_base#streamdata) into the pipeline. Supports both OpenCV and GStreamer backends for maximum compatibility and performance.
Example:
{% code overflow="wrap" %}
```
# Using enum (recommended)
source = VideoSourceGizmo("video.mp4", source_type=VideoSourceType.GSTREAMER)
# Using string (backward compatible)
source = VideoSourceGizmo("video.mp4", source_type="gstream")
# Auto-detect best backend (default)
source = VideoSourceGizmo("video.mp4")
```
{% endcode %}
### VideoSourceGizmo Methods
#### \_\_del\_\_
`__del__()`
Destructor to ensure video source is released.
#### \_\_init\_\_(video\_source=None, ...)
`__init__(video_source=None, *, source_type=VideoSourceType.AUTO, stop_composition_on_end=False, retry_on_error=False, fps_override=None, resolution_override=None)`
Constructor.
Parameters:
| Name | Type | Description | Default |
| ------------------------- | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- |
| `video_source` | `int or str` | A cv2.VideoCapture-compatible video source (device index as int, or file path/URL as str). Defaults to None. | `None` |
| `source_type` | `Union[str, VideoSourceType]` | Video backend to use. Options: - VideoSourceType.AUTO or "auto": Automatically choose best backend - VideoSourceType.GSTREAMER or "gstream": Force GStreamer backend - VideoSourceType.OPENCV or "opencv": Force OpenCV backend | `AUTO` |
| `stop_composition_on_end` | `bool` | If True, stop the [Composition](/degirum-tools/streams/streams_base#composition) when the video source is over. Defaults to False. | `False` |
| `retry_on_error` | `bool` | If True, retry opening the video source on error after some time. Defaults to False. | `False` |
| `fps_override` | `float` | If provided, overrides the FPS value reported by source (some IP cameras report 100FPS). Defaults to None. | `None` |
| `resolution_override` | `Tuple[int, int]` | If provided, overrides the resolution (width, height) reported by source. Defaults to None. | `None` |
#### get\_tags
`get_tags()`
Get list of tags assigned to this gizmo.
Returns:
| Type | Description |
| ----------- | ------------------------------------------------------------- |
| `List[str]` | List\[str]: Tags for this gizmo (its name and the video tag). |
#### run
`run()`
Run the video capture loop.
Continuously reads frames from the video source and sends each frame (with metadata) downstream until the source is exhausted or abort is signaled.
## IteratorSourceGizmo
`IteratorSourceGizmo`
Bases: `Gizmo`
Iterator-based source gizmo.
Takes an iterator that yields images in various formats (file paths, numpy arrays, or PIL images) and outputs them as StreamData into the pipeline, similar to VideoSourceGizmo but for static images.
### IteratorSourceGizmo Methods
#### \_\_init\_\_(iterator, ...)
`__init__(iterator, *, fps=0.0)`
Constructor.
Parameters:
| Name | Type | Description | Default |
| ---------- | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
| `iterator` | `Iterator` | An iterator that yields images. Each yielded item can be: - str: path to an image file - numpy.ndarray: image as numpy array - PIL.Image: PIL image object | *required* |
| `fps` | `float` | Optional FPS value to include in metadata (default 0.0, meaning not applicable) | `0.0` |
#### get\_tags
`get_tags()`
Get list of tags assigned to this gizmo.
Returns:
| Type | Description |
| ----------- | ------------------------------------------------------------- |
| `List[str]` | List\[str]: Tags for this gizmo (its name and the video tag). |
#### run
`run()`
Run the iterator source loop.
Iterates over the iterator, converts each item to a numpy array image, creates appropriate metadata (similar to VideoSourceGizmo), and sends the results downstream.
## FPSStabilizingGizmo
`FPSStabilizingGizmo`
Bases: `Gizmo`
FPS stabilizing gizmo that maintains stable frame rate by sending fake frames when input is empty.
When the input queue has frames available, they are sent through as-is. When the input queue is empty, fake frames are generated by gradually shading the last genuine frame to maintain the target FPS derived from video metadata.
### FPSStabilizingGizmo Methods
#### \_\_init\_\_(\*, ...)
`__init__(*, stream_depth=10, allow_drop=False)`
Constructor.
Parameters:
| Name | Type | Description | Default |
| -------------- | ------ | ----------------------------------------------------------------------------- | ------- |
| `stream_depth` | `int` | Depth of the input frame queue. Defaults to 10. | `10` |
| `allow_drop` | `bool` | If True, allow dropping frames if the input queue is full. Defaults to False. | `False` |
#### require\_tags(inp)
`require_tags(inp)`
Get the list of meta tags this gizmo requires in upstream meta for a specific input.
Returns:
| Type | Description |
| ---- | --------------------------------------------------------------------------------- |
| | List\[str]: Tags required by this gizmo in upstream meta for the specified input. |
#### run
`run()`
Run the FPS stabilizing loop.
Continuously reads frames from input queue and sends them downstream. When the input queue is empty, generates fake frames by gradually shading the last genuine frame to maintain stable FPS derived from video metadata.
## VideoDisplayGizmo
`VideoDisplayGizmo`
Bases: `Gizmo`
OpenCV-based video display gizmo.
Displays incoming frames in one or more OpenCV windows.
### VideoDisplayGizmo Methods
#### \_\_init\_\_(window\_titles='Display', ...)
`__init__(window_titles='Display', *, show_ai_overlay=False, show_fps=False, stream_depth=10, allow_drop=False, multiplex=False)`
Constructor.
Parameters:
| Name | Type | Description | Default |
| ----------------- | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- |
| `window_titles` | `str or List[str]` | Title or list of titles for the display window(s). If a list is provided, multiple windows are opened (one per title). Defaults to "Display". | `'Display'` |
| `show_ai_overlay` | `bool` | If True, overlay AI inference results on the displayed frame (when available). Defaults to False. | `False` |
| `show_fps` | `bool` | If True, show the FPS on the display window(s). Defaults to False. | `False` |
| `stream_depth` | `int` | Depth of the input frame queue. Defaults to 10. | `10` |
| `allow_drop` | `bool` | If True, allow dropping frames if the input queue is full. Defaults to False. | `False` |
| `multiplex` | `bool` | If True, use a single input stream and display frames in a round-robin across multiple windows; if False, each window corresponds to its own input stream. Defaults to False. | `False` |
Raises:
| Type | Description |
| ----------- | -------------------------------------------------------------------------------- |
| `Exception` | If multiplex is True while allow\_drop is also True (unsupported configuration). |
#### run
`run()`
Run the video display loop.
Fetches frames from the input stream(s) and shows them in the window(s) (with optional overlays and FPS display) until all inputs are exhausted or aborted.
## VideoSaverGizmo
`VideoSaverGizmo`
Bases: `Gizmo`
Video saving gizmo.
Writes incoming frames to an output video file.
### VideoSaverGizmo Methods
#### \_\_init\_\_(filename, ...)
`__init__(filename, *, show_ai_overlay=False, stream_depth=10, allow_drop=False)`
Constructor.
Parameters:
| Name | Type | Description | Default |
| ----------------- | ------ | -------------------------------------------------------------------------------------------------- | ---------- |
| `filename` | `str` | Path to the output video file. | *required* |
| `show_ai_overlay` | `bool` | If True, overlay AI inference results on frames before saving (when available). Defaults to False. | `False` |
| `stream_depth` | `int` | Depth of the input frame queue. Defaults to 10. | `10` |
| `allow_drop` | `bool` | If True, allow dropping frames if the input queue is full. Defaults to False. | `False` |
#### run
`run()`
Run the video saving loop.
Reads frames from the input stream and writes them to the output file until the stream is exhausted or aborted.
## VideoStreamerGizmo
`VideoStreamerGizmo`
Bases: `Gizmo`
Video streaming gizmo.
Streams incoming frames to RTSP/RTMP stream using ffmpeg. `MediaServer` must be running to accept the stream.
### VideoStreamerGizmo Methods
#### \_\_init\_\_(stream\_url, ...)
`__init__(stream_url, *, fps=0, show_ai_overlay=False, stream_depth=10, allow_drop=False)`
Constructor.
Parameters:
| Name | Type | Description | Default |
| ----------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- |
| `stream_url` | `str` | RTMP/RTSP URL to stream to (e.g., 'rtsp\://user:password\@hostname:port/stream'). Typically you use `MediaServer` class to start media server and then use its RTMP/RTSP URL like `rtsp://localhost:8554/mystream` | *required* |
| `fps` | `float` | Frames per second for the stream. Defaults to 0, meaning to deduce from upstream video source. | `0` |
| `show_ai_overlay` | `bool` | If True, overlay AI inference results on frames before saving (when available). Defaults to False. | `False` |
| `stream_depth` | `int` | Depth of the input frame queue. Defaults to 10. | `10` |
| `allow_drop` | `bool` | If True, allow dropping frames if the input queue is full. Defaults to False. | `False` |
#### run
`run()`
Run the video saving loop.
Reads frames from the input stream and writes them to the output file until the stream is exhausted or aborted.
## ResizingGizmo
`ResizingGizmo`
Bases: `Gizmo`
OpenCV-based image resizing/padding gizmo.
Resizes incoming images to a specified width and height, using the chosen padding or cropping method.
### ResizingGizmo Methods
#### \_\_init\_\_(w, ...)
`__init__(w, h, pad_method='letterbox', resize_method='bilinear', stream_depth=10, allow_drop=False)`
Constructor.
Parameters:
| Name | Type | Description | Default |
| --------------- | ------ | ------------------------------------------------------------------------------------------------------- | ------------- |
| `w` | `int` | Target width for output images. | *required* |
| `h` | `int` | Target height for output images. | *required* |
| `pad_method` | `str` | Padding method to use ("stretch", "letterbox", "crop-first", "crop-last"). Defaults to "letterbox". | `'letterbox'` |
| `resize_method` | `str` | Resampling method to use ("nearest", "bilinear", "area", "bicubic", "lanczos"). Defaults to "bilinear". | `'bilinear'` |
| `stream_depth` | `int` | Depth of the input frame queue. Defaults to 10. | `10` |
| `allow_drop` | `bool` | If True, allow dropping frames if the input queue is full. Defaults to False. | `False` |
#### get\_tags
`get_tags()`
Get list of tags assigned to this gizmo.
Returns:
| Type | Description |
| ----------- | -------------------------------------------------------------- |
| `List[str]` | List\[str]: Tags for this gizmo (its name and the resize tag). |
#### run
`run()`
Run the resizing loop.
Resizes each input image according to the configured width, height, padding, and resizing method, then sends the result with updated metadata downstream.
## AiGizmoBase
`AiGizmoBase`
Bases: `Gizmo`
Base class for AI model inference gizmos.
Handles loading the model and iterating over input data for inference in a background thread.
### AiGizmoBase Methods
#### \_\_init\_\_(model, ...)
`__init__(model, *, analyzers=None, non_blocking_batch_predict_timeout_s=0.01, stream_depth=10, allow_drop=False, inp_cnt=1, **kwargs)`
Constructor.
Parameters:
| Name | Type | Description | Default |
| -------------------------------------- | -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
| `model` | `Model or str` | A DeGirum model object or model name string to load. If a string is provided, the model will be loaded via `degirum.load_model()` using the given kwargs. | *required* |
| `analyzers` | `list` | List of analyzer objects to apply to inference results (e.g., EventDetector, EventNotifier instances). Defaults to None. | `None` |
| `non_blocking_batch_predict_timeout_s` | `float` | Input queue get timeout for non-blocking batch prediction mode. Defaults to 0.001 seconds. | `0.01` |
| `stream_depth` | `int` | Depth of the input stream queue. Defaults to 10. | `10` |
| `allow_drop` | `bool` | If True, allow dropping frames on input overflow. Defaults to False. | `False` |
| `inp_cnt` | `int` | Number of input streams (for models requiring multiple inputs). Defaults to 1. | `1` |
| `**kwargs` | `any` | Additional parameters to pass to `degirum.load_model()` when loading the model (if model is given as a name). | `{}` |
#### get\_tags
`get_tags()`
Get list of tags assigned to this gizmo.
Returns:
| Type | Description |
| ----------- | ----------------------------------------------------------------- |
| `List[str]` | List\[str]: Tags for this gizmo (its name and the inference tag). |
#### on\_result(result)
`on_result(result)`
`abstractmethod`
Handle a single inference result (to be implemented by subclasses).
Parameters:
| Name | Type | Description | Default |
| -------- | -------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- | ---------- |
| `result` | [InferenceResults](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/postprocessor#degirum.postprocessor.inferenceresults) | The inference result object from the model. | *required* |
#### run
`run()`
Run the model inference loop.
Internally feeds data from the input stream(s) into the model and yields results, invoking `on_result` for each inference result.
## AiSimpleGizmo
`AiSimpleGizmo`
Bases: `AiGizmoBase`
AI inference gizmo with no custom result processing.
Passes through input frames and attaches the raw inference results to each frame's metadata.
### AiSimpleGizmo Methods
#### on\_result(result)
`on_result(result)`
Append the inference result to the input frame's metadata and send it downstream.
Parameters:
| Name | Type | Description | Default |
| -------- | -------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- | ---------- |
| `result` | [InferenceResults](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/postprocessor#degirum.postprocessor.inferenceresults) | The inference result for the current frame. | *required* |
## AiObjectDetectionCroppingGizmo
`AiObjectDetectionCroppingGizmo`
Bases: `Gizmo`
Gizmo that crops detected objects from frames of an object detection model.
Each input frame with object detection results yields one or more cropped images as output.
Output
* **Image**: The cropped portion of the original image corresponding to a detected object.
* **Meta-info**: A dictionary containing:
* `original_result`: Reference to the original detection result ([InferenceResults](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/postprocessor#degirum.postprocessor.inferenceresults)) for the frame.
* `cropped_result`: The detection result entry for this specific crop.
* `cropped_index`: The index of this object in the original results list.
* `is_last_crop`: True if this crop is the last one for the frame.
Note
`cropped_index` and `is_last_crop` are only present if at least one object is detected in the frame.
The `validate_bbox()` method can be overridden in subclasses to filter out undesirable detections.
### AiObjectDetectionCroppingGizmo Methods
#### \_\_init\_\_(labels, ...)
`__init__(labels, *, send_original_on_no_objects=True, crop_extent=0.0, crop_extent_option=CropExtentOptions.ASPECT_RATIO_NO_ADJUSTMENT, crop_aspect_ratio=1.0, stream_depth=10, allow_drop=False)`
Constructor.
Parameters:
| Name | Type | Description | Default |
| ----------------------------- | ------------------- | --------------------------------------------------------------------------------------------------------------------------------- | ---------------------------- |
| `labels` | `List[str]` | List of class labels to process. Only objects whose class is in this list will be cropped. | *required* |
| `send_original_on_no_objects` | `bool` | If True, when no objects are detected in a frame, the original frame is sent through. Defaults to True. | `True` |
| `crop_extent` | `float` | Extra padding around the bounding box, as a percentage of the bbox size. Defaults to 0.0. | `0.0` |
| `crop_extent_option` | `CropExtentOptions` | Method for applying the crop extent (e.g., aspect ratio adjustment). Defaults to CropExtentOptions.ASPECT\_RATIO\_NO\_ADJUSTMENT. | `ASPECT_RATIO_NO_ADJUSTMENT` |
| `crop_aspect_ratio` | `float` | Desired aspect ratio (W/H) for the cropped images. Defaults to 1.0. | `1.0` |
| `stream_depth` | `int` | Depth of the input frame queue. Defaults to 10. | `10` |
| `allow_drop` | `bool` | If True, allow dropping frames on overflow. Defaults to False. | `False` |
#### get\_tags
`get_tags()`
Get list of tags assigned to this gizmo.
Returns:
| Type | Description |
| ----------- | ------------------------------------------------------------ |
| `List[str]` | List\[str]: Tags for this gizmo (its name and the crop tag). |
#### require\_tags(inp)
`require_tags(inp)`
Get the list of meta tags this gizmo requires in upstream meta for a specific input.
Returns:
| Type | Description |
| ----------- | --------------------------------------------------------------------------------- |
| `List[str]` | List\[str]: Tags required by this gizmo in upstream meta for the specified input. |
#### run
`run()`
Run the object cropping loop.
For each input frame, finds all detected objects (matching the specified labels and passing validation) and sends out a cropped image for each. If no objects are detected and `send_original_on_no_objects` is True, the original frame is forwarded.
#### validate\_bbox(result, ...)
`validate_bbox(result, idx)`
Decide whether a detected object should be cropped (can be overridden in subclasses).
Parameters:
| Name | Type | Description | Default |
| -------- | -------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------- | ---------- |
| `result` | [InferenceResults](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/postprocessor#degirum.postprocessor.inferenceresults) | The detection result for the frame. | *required* |
| `idx` | `int` | The index of the object in `result.results` to validate. | *required* |
Returns:
| Name | Type | Description |
| ------ | ------ | -------------------------------------------------------------------- |
| `bool` | `bool` | True if the object should be cropped; False if it should be skipped. |
## CropCombiningGizmo
`CropCombiningGizmo`
Bases: `Gizmo`
Gizmo to combine original frames with their after-crop results.
Expects N+1 inputs: one input stream of original frames (index 0), and N input streams of inference results from cropped images. This gizmo synchronizes and attaches the after-crop inference results back to each original frame's metadata.
### CropCombiningGizmo Methods
#### \_\_init\_\_(crop\_inputs\_num=1, ...)
`__init__(crop_inputs_num=1, *, stream_depth=10)`
Constructor.
Parameters:
| Name | Type | Description | Default |
| ----------------- | ----- | ----------------------------------------------------------------------------------------- | ------- |
| `crop_inputs_num` | `int` | Number of crop result input streams (excluding the original frame stream). Defaults to 1. | `1` |
| `stream_depth` | `int` | Depth for each crop input stream's queue. Defaults to 10. | `10` |
#### require\_tags(inp)
`require_tags(inp)`
Get the list of meta tags this gizmo requires in upstream meta for a specific input.
Returns:
| Type | Description |
| ----------- | --------------------------------------------------------------------------------- |
| `List[str]` | List\[str]: Tags required by this gizmo in upstream meta for the specified input. |
#### run
`run()`
Run the crop combining loop.
Synchronizes original frames with their corresponding after-crop result streams, merges the inference results from crops back into the original frame's metadata, and sends the updated frame downstream.
## AiResultCombiningGizmo
`AiResultCombiningGizmo`
Bases: `Gizmo`
Gizmo to combine inference results from multiple AI gizmos of the same type.
### AiResultCombiningGizmo Methods
#### \_\_init\_\_(inp\_cnt, ...)
`__init__(inp_cnt, *, stream_depth=10)`
Constructor.
Parameters:
| Name | Type | Description | Default |
| -------------- | ----- | --------------------------------------------------- | ---------- |
| `inp_cnt` | `int` | Number of input result streams to combine. | *required* |
| `stream_depth` | `int` | Depth of each input stream's queue. Defaults to 10. | `10` |
#### get\_tags
`get_tags()`
Get list of tags assigned to this gizmo.
Returns:
| Type | Description |
| ----------- | ----------------------------------------------------------------- |
| `List[str]` | List\[str]: Tags for this gizmo (its name and the inference tag). |
#### require\_tags(inp)
`require_tags(inp)`
Get the list of meta tags this gizmo requires in upstream meta for a specific input.
Returns:
| Type | Description |
| ----------- | --------------------------------------------------------------------------------- |
| `List[str]` | List\[str]: Tags required by this gizmo in upstream meta for the specified input. |
#### run
`run()`
Run the result combining loop.
Collects inference results from all input streams, merges their results into a single combined result, and sends it downstream.
## AiPreprocessGizmo
`AiPreprocessGizmo`
Bases: `Gizmo`
Preprocessing gizmo that applies a model's preprocessor to input images.
It generates preprocessed image data to be fed into the model.
Output
* **Data**: Preprocessed image bytes ready for model input.
* **Meta-info**: Dictionary including:
* `image_input`: The original input image.
* `converter`: A function to convert coordinates from model output back to the original image.
* `image_result`: The preprocessed image (present only if the model is configured to provide it).
Attributes:
| Name | Type | Description |
| ------------------ | ----- | ---------------------------------------------------- |
| `key_image_input` | `str` | Metadata key for the original input image. |
| `key_converter` | `str` | Metadata key for the coordinate conversion function. |
| `key_image_result` | `str` | Metadata key for the preprocessed image. |
### AiPreprocessGizmo Methods
#### \_\_init\_\_(model, ...)
`__init__(model, *, stream_depth=10, allow_drop=False)`
Constructor.
Parameters:
| Name | Type | Description | Default |
| -------------- | ------- | --------------------------------------------------------------- | ---------- |
| `model` | `Model` | The model object (PySDK model) whose preprocessor will be used. | *required* |
| `stream_depth` | `int` | Depth of the input frame queue. Defaults to 10. | `10` |
| `allow_drop` | `bool` | If True, allow dropping frames on overflow. Defaults to False. | `False` |
#### get\_tags
`get_tags()`
Get list of tags assigned to this gizmo.
Returns:
| Type | Description |
| ----------- | ------------------------------------------------------------------ |
| `List[str]` | List\[str]: Tags for this gizmo (its name and the preprocess tag). |
#### run
`run()`
Run the preprocessing loop.
Applies the model's preprocessor to each input frame and sends the resulting data (and updated meta-info) downstream.
## AiAnalyzerGizmo
`AiAnalyzerGizmo`
Bases: `Gizmo`
Gizmo to apply a chain of analyzers to an inference result, with optional filtering.
Each analyzer (e.g., EventDetector, EventNotifier) processes the inference result and may add events or notifications. If filters are provided, only results that contain at least one of the specified events/notifications are passed through.
### AiAnalyzerGizmo Methods
#### \_\_init\_\_(analyzers, ...)
`__init__(analyzers, *, filters=None, stream_depth=10, allow_drop=False)`
Constructor.
Parameters:
| Name | Type | Description | Default |
| -------------- | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
| `analyzers` | `List` | List of analyzer objects to apply (e.g., EventDetector, EventNotifier instances). | *required* |
| `filters` | `set` | A set of event names or notification names to filter results. Only results that have at least one of these events or notifications will be forwarded (others are dropped). Defaults to None (no filtering). | `None` |
| `stream_depth` | `int` | Depth of the input frame queue. Defaults to 10. | `10` |
| `allow_drop` | `bool` | If True, allow dropping frames on overflow. Defaults to False. | `False` |
#### get\_tags
`get_tags()`
Get list of tags assigned to this gizmo.
Returns:
| Type | Description |
| ----------- | ------------------------------------------------------------------------------------ |
| `List[str]` | List\[str]: Tags for this gizmo (its name, the inference tag, and the analyzer tag). |
#### require\_tags(inp)
`require_tags(inp)`
Get the list of meta tags this gizmo requires in upstream meta for a specific input.
Returns:
| Type | Description |
| ----------- | --------------------------------------------------------------------------------- |
| `List[str]` | List\[str]: Tags required by this gizmo in upstream meta for the specified input. |
#### run
`run()`
Run the analyzer processing loop.
For each input frame, clones its inference result and runs all analyzers on it (which may add events/notifications). If filters are specified, the result is dropped unless it contains at least one of the specified events or notifications. The possibly modified inference result is appended to the frame's metadata and sent downstream. After processing all frames, all analyzers are finalized.
## SinkGizmo
`SinkGizmo`
Bases: `Gizmo`
Sink gizmo that receives results and accumulates them in an internal queue.
This gizmo does not send data further down the pipeline. Instead, it stores all incoming results so they can be retrieved (for example, by iterating over the gizmo's output in the main thread).
### SinkGizmo Methods
#### \_\_call\_\_
`__call__()`
Retrieve the internal queue for iteration.
Returns:
| Name | Type | Description |
| -------- | -------- | -------------------------------------------------------------------------------------------- |
| `Stream` | `Stream` | The input Stream (queue) of this sink gizmo, which can be iterated to get collected results. |
#### \_\_init\_\_(\*, ...)
`__init__(*, stream_depth=10, allow_drop=False)`
Constructor.
Parameters:
| Name | Type | Description | Default |
| -------------- | ------ | -------------------------------------------------------------- | ------- |
| `stream_depth` | `int` | Depth of the input queue. Defaults to 10. | `10` |
| `allow_drop` | `bool` | If True, allow dropping frames on overflow. Defaults to False. | `False` |
#### run
`run()`
Run gizmo (no operation).
Immediately returns, as the sink simply collects incoming data without processing.
# Analyzers
DeGirum Tools API Reference Guide. Abstract base for result analyzers and overlays.
{% hint style="info" %}
This API Reference is based on DeGirum Tools version 1.2.0.
{% endhint %}
## Result Analyzer Base Module Overview
This module provides a base class (`ResultAnalyzerBase`) for performing custom post-processing and image annotation on DeGirum PySDK inference results. These analyzers can be used with compound models, streaming gizmos, and regular models to add advanced data processing and annotation steps to inference pipelines.
### Key Concepts
* **Analysis**: By overriding the `analyze()` method, child classes can read and augment the [InferenceResults](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/postprocessor#degirum.postprocessor.inferenceresults) (e.g., by adding extra keys to the internal `results` list).
* **Annotation**: By overriding the `annotate()` method, child classes can draw additional overlays on the original input image (e.g., bounding boxes, text labels, or any custom markings).
* **Integration**: Analyzers can be attached to a model or a compound model via the `attach_analyzers()` method, so their analysis and annotation is automatically applied to each inference result.
### Typical Usage Example
1. Create a custom analyzer subclass:
{% code overflow="wrap" %}
```python
from degirum_tools import ResultAnalyzerBase
class MyCustomAnalyzer(ResultAnalyzerBase):
def analyze(self, result):
# E.g., add custom fields to each detection
for r in result.results:
r["custom_info"] = "my_data"
def annotate(self, result, image):
# E.g., draw text or bounding boxes on the image
# Return the annotated image
return image
```
{% endcode %}
2. Attach it to a model or compound model:
{% code overflow="wrap" %}
```
model.attach_analyzers(MyCustomAnalyzer())
```
{% endcode %}
3. Run inference. Each [InferenceResults](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/postprocessor#degirum.postprocessor.inferenceresults) object will be passed through your analyzer, optionally modifying the result data and providing a new overlay.
## Functions
#### image\_overlay\_substitute(result, ...)
`image_overlay_substitute(result, analyzers)`
Substitute the `image_overlay` property of the given inference `result` object so that future calls to `result.image_overlay` automatically apply the analyzers' `annotate()` methods.
This method creates a new class that inherits from the original result class and overrides the `image_overlay` property. The new class does the following:
1. Calls the base class's `image_overlay` property to get the image annotated by original result class.
2. Applies each analyzer's `annotate()` method to the image, in order.
Parameters:
| Name | Type | Description | Default |
| ----------- | -------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- | ---------- |
| `result` | [InferenceResults](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/postprocessor#degirum.postprocessor.inferenceresults) | The inference result whose `image_overlay` property will be replaced. | *required* |
| `analyzers` | `List[ResultAnalyzerBase]` | A list of analyzer objects to apply for annotation. | *required* |
#### clone\_result(result)
`clone_result(result)`
Create a shallow clone of a DeGirum PySDK inference result object, duplicating the internal inference results list but reusing references to the original image.
This is useful when you want to create a separate copy of the result for further modifications without altering the original.
Parameters:
| Name | Type | Description | Default |
| -------- | -------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------- | ---------- |
| `result` | [InferenceResults](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/postprocessor#degirum.postprocessor.inferenceresults) | The inference result object to clone. | *required* |
Returns:
| Type | Description |
| -------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- |
| [InferenceResults](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/postprocessor#degirum.postprocessor.inferenceresults) | A cloned result object with `result._inference_results` deep-copied. |
## Classes
## ResultAnalyzerBase
`ResultAnalyzerBase`
Bases: `ABC`
Base class for result analyzers which can modify or extend the content of inference results and optionally annotate images with new data.
Subclasses should override
* `analyze(result)`: to augment or inspect the inference result.
* `annotate(result, image)`: to draw additional overlays or text onto the provided image.
### ResultAnalyzerBase Methods
#### \_\_del\_\_
`__del__()`
Called when the analyzer object is about to be destroyed.
Invokes `finalize()` to ensure any open resources are cleaned up.
#### analyze(result)
`analyze(result)`
`abstractmethod`
Analyze and optionally modify a DeGirum PySDK inference result.
This method should access and potentially modify `result.results` (the list of detections, classifications, or similar structures) to add any custom fields. These modifications will then appear in downstream processes or when the result is displayed/serialized.
Parameters:
| Name | Type | Description | Default |
| -------- | -------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | ---------- |
| `result` | [InferenceResults](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/postprocessor#degirum.postprocessor.inferenceresults) | The inference result object to analyze. Subclasses can read and/or modify the internal `results` list or other properties. | *required* |
#### analyze\_and\_annotate(result, ...)
`analyze_and_annotate(result, image)`
Helper method to perform both analysis and annotation in one step.
1. Calls `self.analyze(result)`.
2. Calls `self.annotate(result, image)`.
Parameters:
| Name | Type | Description | Default |
| -------- | -------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------- | ---------- |
| `result` | [InferenceResults](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/postprocessor#degirum.postprocessor.inferenceresults) | The inference result object to process. | *required* |
| `image` | `ndarray` | The image to annotate. | *required* |
Returns:
| Type | Description |
| --------- | -------------------------------------------------- |
| `ndarray` | numpy.ndarray: The annotated image after analysis. |
#### annotate(result, ...)
`annotate(result, image)`
Annotate an image with additional data derived from the analysis step.
Called after `analyze()` has been invoked on `result`. This method is typically used to draw bounding boxes, text, or other graphical elements representing the analysis data.
Parameters:
| Name | Type | Description | Default |
| -------- | -------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------- | ---------- |
| `result` | [InferenceResults](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/postprocessor#degirum.postprocessor.inferenceresults) | The (already analyzed) inference result object. | *required* |
| `image` | `ndarray` | The original (or base) image to annotate. | *required* |
Returns:
| Type | Description |
| --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ndarray` | numpy.ndarray: The annotated image. By default, this base implementation returns the image unchanged. Subclasses should override to perform custom drawing. |
#### finalize
`finalize()`
Perform any finalization or cleanup actions before the analyzer is discarded.
This can be useful for analyzers that accumulate state (e.g., for multi-frame analysis). By default, this does nothing.
# Zone Counter
DeGirum Tools API Reference Guide. Count objects present in polygonal zones.
{% hint style="info" %}
This API Reference is based on DeGirum Tools version 1.2.0.
{% endhint %}
## Zone Counter Analyzer Module Overview
This module provides a zone counting analyzer (`ZoneCounter`) with support for both traditional list-based zones and named zones (dictionary-based). It features clean separation of concerns and a user-friendly API.
Key Features
* **Named Zones**: Use descriptive names instead of numeric indices
* **Clean Architecture**: Separation of geometry (spatial) and state (temporal) logic
* **Zone Events**: Track entry, exit, occupied, and empty events per zone
* **All ZoneCounter Features**: Supports all triggering methods, tracking, timeouts, etc.
* **Better Maintainability**: Smaller, focused components that are easier to test and extend
Architecture
The implementation separates concerns into focused components:
* **\_ZoneGeometry**: Handles spatial logic - polygon shapes, masks, and triggering
* **\_ZoneState**: Handles temporal logic - object tracking, timeouts, occupancy state
* **ZoneCounter**: Orchestrates the components and generates events
Typical Usage
{% code overflow="wrap" %}
```python
from degirum_tools.analyzers import ZoneCounter
counter = ZoneCounter(
zones={
"entrance": entrance_polygon,
"parking_spot_1": spot1_polygon,
"exit_area": exit_polygon,
},
use_tracking=True,
timeout_frames=3, # Hysteresis threshold
enable_zone_events=True,
)
model.attach_analyzers(counter)
result = model(frame)
# Access named zone counts
print(result.zone_counts) # {"entrance": {...}, "parking_spot_1": {...}, ...}
# Access zone events
for event in result.zone_events:
print(f"{event['event_type']} in {event['zone_id']} at {event['timestamp']}")
```
{% endcode %}
Zone Events
When `enable_zone_events=True`, the analyzer generates four types of events:
* **zone\_entry**: Track becomes established in zone (after timeout\_frames + 1 consecutive in-zone detections)
* **zone\_exit**: Track exits zone (hysteresis counter reaches 0)
* **zone\_occupied**: Zone transitions from empty to occupied (when first track becomes established)
* **zone\_empty**: Zone transitions from occupied to empty (after all tracks exit)
Event structure:
{% code overflow="wrap" %}
```yaml
{
"event_type": str, # Event type
"zone_index": int, # Numeric zone index (0-based)
"zone_id": str, # Zone name/ID
"timestamp": float, # Unix timestamp
"track_id": int | None, # Track ID (entry/exit) or None (occupied/empty)
"object_label": str | None, # Object class (entry/exit) or None (occupied/empty)
"dwell_time": float | None, # Duration: in zone (exit), empty/occupied (transitions), None (entry)
"frame_index": int | None, # Frame index (if available)
}
```
{% endcode %}
Integration Notes
* Requires detection results with bounding boxes
* Zone events require `use_tracking=True`
* Compatible with ObjectTracker analyzer upstream
* Results structure matches ZoneCounter for easy migration
Configuration Options
* `zones`: Dictionary mapping zone names to polygons
* `class_list`: Optional list of class labels to count
* `triggering_position`: Anchor point or IoPA-based triggering
* `timeout_frames`: Hysteresis threshold for symmetric entry/exit smoothing
* `enable_zone_events`: Generate zone-level events
* `show_overlay`: Visual annotations
* `per_class_display`: Show per-class counts
Hysteresis Smoothing
The analyzer uses symmetric hysteresis for stable zone presence detection:
* **timeout\_frames**: Controls the hysteresis threshold. Objects need `timeout_frames + 1` consecutive in-zone detections to become "established" (counted). The internal counter increments when in zone, decrements when outside or missing, and is clamped to \[0, timeout\_frames + 1]. Objects exit when the counter reaches 0.
* **Symmetric behavior**: The same threshold applies for both entry and exit, providing smooth transitions without flickering. Brief departures don't reset the counter, and brief appearances don't immediately establish presence.
* **Default**: 0 (immediate entry/exit on first detection). Requires `use_tracking=True` if > 0.
## Classes
## ZoneCounter
`ZoneCounter`
Bases: `ResultAnalyzerBase`
Analyzer that counts objects inside user-defined polygonal zones.
This analyzer integrates with PySDK inference results to determine whether detected or tracked objects lie within user-defined polygon zones. It supports per-class counting, object tracking with symmetric hysteresis (timeout\_frames) for entry/exit smoothing, zone events, and interactive editing.
Backward compatible with old ZoneCounter API (list of zones) while supporting new dict-based named zones interface.
Attributes:
| Name | Type | Description |
| -------------------- | ---- | --------------------------------------------- |
| `key_in_zone` | | Key for zone presence flags in result objects |
| `key_frames_in_zone` | | Key for frame counts in result objects |
| `key_time_in_zone` | | Key for time-in-zone in result objects |
| `key_zone_events` | | Key for zone events list in result objects |
### ZoneCounter Methods
#### \_\_init\_\_(zones=None, ...)
`__init__(zones=None, *, count_polygons=None, class_list=None, per_class_display=False, triggering_position=AnchorPoint.BOTTOM_CENTER, bounding_box_scale=1.0, iopa_threshold=0.0, use_tracking=False, timeout_frames=0, enable_zone_events=False, window_name=None, show_overlay=True, show_inzone_counters=None, annotation_color=None, annotation_line_width=None)`
Initialize ZoneCounter.
Parameters:
| Name | Type | Description | Default |
| ----------------------- | --------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------- |
| `zones` | `Union[Dict[str, ndarray], List[ndarray], ndarray, None]` | Dict mapping zone names to polygons (new style) OR list of polygons (old style backward compat) | `None` |
| `class_list` | `Optional[List[str]]` | List of class labels to count (None = all classes) | `None` |
| `per_class_display` | `bool` | Show per-class counts separately | `False` |
| `triggering_position` | `Optional[Union[List[AnchorPoint], AnchorPoint]]` | Anchor point(s) or None for IoPA | `BOTTOM_CENTER` |
| `bounding_box_scale` | `float` | Scale factor for bboxes (greater than 0, up to 1) | `1.0` |
| `iopa_threshold` | `Union[float, List[float]]` | IoPA threshold (single value or list per zone) | `0.0` |
| `use_tracking` | `bool` | Enable object tracking | `False` |
| `timeout_frames` | `int` | Hysteresis threshold for zone presence (requires tracking if > 0). Objects need timeout\_frames + 1 consecutive in-zone detections to become established (counted), and the counter decrements when outside/missing. Objects exit when counter reaches 0. | `0` |
| `enable_zone_events` | `bool` | Generate zone-level events (requires tracking) | `False` |
| `window_name` | `Optional[str]` | OpenCV window name for interactive polygon editing (None = disabled) | `None` |
| `show_overlay` | `bool` | Draw zone overlays | `True` |
| `show_inzone_counters` | `Optional[str]` | Show presence counters ('time', 'frames', 'all', None) | `None` |
| `annotation_color` | `Optional[tuple]` | RGB color for overlays (None = auto) | `None` |
| `annotation_line_width` | `Optional[int]` | Line width for overlays (None = auto) | `None` |
#### analyze(result)
`analyze(result)`
Analyze inference result and update zone counts and events.
Parameters:
| Name | Type | Description | Default |
| -------- | ---- | --------------------------- | ---------- |
| `result` | | Inference result to analyze | *required* |
#### annotate(result, ...)
`annotate(result, image)`
Draw zone overlays and counts on the image.
Parameters:
| Name | Type | Description | Default |
| -------- | --------- | ------------------------------- | ---------- |
| `result` | | Inference result with zone data | *required* |
| `image` | `ndarray` | Image to annotate | *required* |
Returns:
| Type | Description |
| --------- | --------------- |
| `ndarray` | Annotated image |
#### window\_attach(win\_name)
`window_attach(win_name)`
Attach an OpenCV window for interactive polygon editing.
Parameters:
| Name | Type | Description | Default |
| ---------- | ----- | -------------------------------------- | ---------- |
| `win_name` | `str` | Name of the OpenCV window to attach to | *required* |
# Object Tracker
DeGirum Tools API Reference Guide. Track objects across frames.
{% hint style="info" %}
This API Reference is based on DeGirum Tools version 1.2.0.
{% endhint %}
## Object Tracker Analyzer Module Overview
Implements multi-object tracking using [BYTETrack algorithm](https://github.com/ifzhang/bytetrack).
Key Features
* **Persistent Object Identity**: Maintains consistent track IDs across frames
* **Class Filtering**: Optionally tracks only specified object classes
* **Track Lifecycle Management**: Handles track creation, updating, and removal
* **Trail Visualization**: Records and displays object movement history
* **Track Retention**: Configurable buffer for handling temporary object disappearances
* **Visual Overlay**: Displays track IDs and optional trails on frames
* **Integration Support**: Provides track IDs for downstream analyzers (e.g., zone counting, line crossing)
Typical Usage
1. Create an `ObjectTracker` instance with desired tracking parameters
2. Process each frame's detection results through the tracker
3. Access track IDs and trails from the augmented results
4. Optionally visualize tracking results using the annotate method
5. Use track IDs in downstream analyzers for advanced analytics
Integration Notes
* Requires detection results with bounding boxes and confidence scores
* Track IDs are added to detection results as `track_id` field
* Trail information is stored in `trails` and `trail_classes` dictionaries
* Works effectively with zone counting and line crossing analyzers
* Supports both frame-based and time-based track retention
Key Classes
* `STrack`: Internal class representing a single tracked object with state
* `ObjectTracker`: Main analyzer class that processes detections and maintains tracks
Configuration Options
* `class_list`: Filter tracking to specific object classes
* `track_thresh`: Confidence threshold for initiating new tracks
* `track_buffer`: Frames to retain tracks after object disappearance
* `match_thresh`: IoU threshold for matching detections to existing tracks
* `trail_depth`: Number of recent positions to keep for trail visualization
* `show_overlay`: Enable/disable visual annotations
* `annotation_color`: Customize overlay appearance
## Classes
## STrack
`STrack`
Represents a single tracked object in the multi-object tracking system.
Each STrack holds the object's bounding box state, unique track identifier, detection confidence score, and tracking status (e.g., new, tracked, lost, removed). A Kalman filter is used internally to predict and update the object's state across frames.
Tracks are created when new objects are detected, updated when detections are matched to existing tracks, and can be reactivated if a lost track matches a new detection. This class provides methods to manage the lifecycle of a track (activation, update, reactivation) and utility functions for bounding box format conversion.
Attributes:
| Name | Type | Description |
| -------------- | ------------- | --------------------------------------------------------------------------------------------- |
| `track_id` | `int` | Unique ID for this track. |
| `is_activated` | `bool` | Whether the track has been activated (confirmed) at least once. |
| `state` | `_TrackState` | Current state of the track (New, Tracked, Lost, or Removed). |
| `start_frame` | `int` | Frame index when this track was first activated. |
| `frame_id` | `int` | Frame index of the last update for this track (last seen frame). |
| `tracklet_len` | `int` | Number of frames this track has been in the tracked state. |
| `score` | `float` | Detection confidence score for the most recent observation of this track. |
| `obj_idx` | `int` | Index of this object's detection in the frame's results list (used for internal bookkeeping). |
### Attributes
#### ndarray
`tlbr: np.ndarray`
`property`
Returns the track's bounding box in corner format (x\_min, y\_min, x\_max, y\_max).
Returns:
| Type | Description |
| --------- | -------------------------------------------------------------------- |
| `ndarray` | np.ndarray: Bounding box in (x\_min, y\_min, x\_max, y\_max) format. |
#### ndarray
`tlwh: np.ndarray`
`property`
Returns the track's current bounding box in (x, y, w, h) format.
Returns:
| Type | Description |
| --------- | ------------------------------------------------------------- |
| `ndarray` | np.ndarray: Bounding box where (x, y) is the top-left corner. |
### STrack Methods
#### \_\_init\_\_(tlwh, ...)
`__init__(tlwh, score, obj_idx, id_counter)`
Constructor.
Parameters:
| Name | Type | Description | Default |
| ------------ | ------------ | --------------------------------------------------------------------------------- | ---------- |
| `tlwh` | `ndarray` | Initial bounding box in (x, y, w, h) format, where (x, y) is the top-left corner. | *required* |
| `score` | `float` | Detection confidence score for this object. | *required* |
| `obj_idx` | `int` | Index of this object's detection in the current frame's results list. | *required* |
| `id_counter` | `_IDCounter` | Shared counter used to generate globally unique track\_id values. | *required* |
#### activate(kalman\_filter, ...)
`activate(kalman_filter, frame_id)`
Activates this track with an initial detection.
Initializes the track's state using the provided Kalman filter, assigns a new track ID, and sets the track status to "Tracked".
Parameters:
| Name | Type | Description | Default |
| --------------- | --------------- | ---------------------------------------------- | ---------- |
| `kalman_filter` | `_KalmanFilter` | Kalman filter to associate with this track. | *required* |
| `frame_id` | `int` | Frame index at which the track is initialized. | *required* |
#### re\_activate(new\_track, ...)
`re_activate(new_track, frame_id, new_id=False)`
Reactivates a track that was previously lost, using a new detection.
Updates the track's state with the new detection's information and sets the state to "Tracked". If new\_id is True, a new track ID is assigned; otherwise, it retains the original ID.
Parameters:
| Name | Type | Description | Default |
| ----------- | -------- | ----------------------------------------------------------- | ---------- |
| `new_track` | `STrack` | New track (detection) to merge into this lost track. | *required* |
| `frame_id` | `int` | Current frame index at which the track is reactivated. | *required* |
| `new_id` | `bool` | Whether to assign a new ID to the track. Defaults to False. | `False` |
#### tlbr\_to\_tlwh(tlbr)
`tlbr_to_tlwh(tlbr)`
`staticmethod`
Converts bounding box from (top-left, bottom-right) to (top-left, width, height).
Parameters:
| Name | Type | Description | Default |
| ------ | --------- | ---------------------------------------- | ---------- |
| `tlbr` | `ndarray` | Bounding box in (x1, y1, x2, y2) format. | *required* |
Returns:
| Type | Description |
| --------- | ------------------------------------------------ |
| `ndarray` | np.ndarray: Bounding box in (x, y, w, h) format. |
#### tlwh\_to\_xyah(tlwh)
`tlwh_to_xyah(tlwh)`
`staticmethod`
Converts bounding box from (top-left x, y, width, height) to (center x, y, aspect ratio, height).
Parameters:
| Name | Type | Description | Default |
| ------ | --------- | ------------------------------------ | ---------- |
| `tlwh` | `ndarray` | Bounding box in (x, y, w, h) format. | *required* |
Returns:
| Type | Description |
| --------- | ----------------------------------------------------------------------- |
| `ndarray` | np.ndarray: Bounding box in (center x, y, aspect ratio, height) format. |
#### update(new\_track, ...)
`update(new_track, frame_id)`
Updates this track with a new matched detection.
Incorporates the detection's bounding box and score into this track's state, updates the Kalman filter prediction, and increments the track length. The track state is set to "Tracked".
Parameters:
| Name | Type | Description | Default |
| ----------- | -------- | ------------------------------------------------ | ---------- |
| `new_track` | `STrack` | The new detection track that matched this track. | *required* |
| `frame_id` | `int` | Current frame index for the update. | *required* |
## ObjectTracker
`ObjectTracker`
Bases: `ResultAnalyzerBase`
Analyzer that tracks objects across frames in a video stream.
This analyzer assigns persistent IDs to detected objects, allowing them to be tracked from frame to frame. It uses the BYTETrack multi-object tracking algorithm to match current detections with existing tracks and manage track life cycles (creation of new tracks, updating of existing ones, and removal of lost tracks). Optionally, tracking can be restricted to specific object classes via the *class\_list* parameter.
After each call to `analyze()`, the input result's detections are augmented with a `"track_id"` field for object identity. If a trail length is specified (non-zero *trail\_depth*), the result will also contain `trails` and `trail_classes` dictionaries: `trails` maps each track ID to a list of recent bounding box coordinates (the object's trail), and `trail_classes` maps each track ID to the object's class label. These facilitate drawing object paths and labeling them.
Functionality
* Unique ID assignment: Provides a unique ID for each object and maintains that ID across frames.
* Class filtering: Ignores detections whose class is not in the specified *class\_list*.
* Track retention buffer: Continues to track objects for *track\_buffer* frames after they disappear.
* Trajectory history: Keeps a history of each object's movement up to *trail\_depth* frames long.
* Overlay support: Can overlay track IDs and trails on frames for visualization.
Typical usage involves calling `analyze()` on each frame's detection results to update tracks, then `annotate()` to visualize or output the tracked results. For instance, in a video processing loop, use `tracker.analyze(detections)` followed by `tracker.annotate(detections, frame)` to maintain and display object tracks.
### ObjectTracker Methods
#### \_\_init\_\_(\*, ...)
`__init__(*, class_list=None, track_thresh=0.25, track_buffer=30, match_thresh=0.8, reset_at_scene_cut=False, anchor_point=AnchorPoint.BOTTOM_CENTER, trail_depth=0, show_overlay=True, annotation_color=None, show_only_track_ids=False)`
Constructor.
Parameters:
| Name | Type | Description | Default |
| --------------------- | ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------- |
| `class_list` | `List[str]` | List of object classes to track. If None, all detected classes are tracked. | `None` |
| `track_thresh` | `float` | Detection confidence threshold for initiating a new track. | `0.25` |
| `track_buffer` | `int` | Number of frames to keep a lost track before removing it. | `30` |
| `match_thresh` | `float` | Intersection-over-union (IoU) threshold for matching detections to existing tracks. | `0.8` |
| `reset_at_scene_cut` | `bool` | If True, resets all tracks when a scene cut is detected. Requires the result to have a `scene_cut` attribute (set by SceneCutDetector). Use this to avoid tracking objects across scene transitions in videos with cuts or edits. | `False` |
| `anchor_point` | `AnchorPoint` | Anchor point on the bounding box used for trail visualization. | `BOTTOM_CENTER` |
| `trail_depth` | `int` | Number of recent positions to keep for each track's trail. Set 0 to disable trail tracking. | `0` |
| `show_overlay` | `bool` | If True, annotate the image; if False, return the original image. | `True` |
| `annotation_color` | `Tuple[int, int, int]` | RGB tuple to use for annotations. If None, a contrasting color is chosen automatically. | `None` |
| `show_only_track_ids` | `bool` | If True, only track IDs are shown in the annotations. If False, trails and labels are also shown when available. | `False` |
#### analyze(result)
`analyze(result)`
Analyzes a detection result and maintains object tracks across frames.
Matches the current frame's detections to existing tracks, assigns track IDs to each detection, and updates or creates tracks as necessary. If trail\_depth was set, this method also updates each track's trail of past positions.
The input result is updated in-place. Each detection in result.results receives a "track\_id" identifying its track. If trails are enabled, result.trails and result.trail\_classes are updated to reflect the current active tracks.
Parameters:
| Name | Type | Description | Default |
| -------- | -------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | ---------- |
| `result` | [InferenceResults](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/postprocessor#degirum.postprocessor.inferenceresults) | Model inference result for the current frame, containing detected object bounding boxes and classes. | *required* |
#### annotate(result, ...)
`annotate(result, image)`
Draws tracking annotations on an image.
If trails are not being used, writes each object's track ID at its bounding box location. If trails are enabled, draws each object's trajectory and labels the end with the object's class name and track ID.
Parameters:
| Name | Type | Description | Default |
| -------- | -------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------- | ---------- |
| `result` | [InferenceResults](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/postprocessor#degirum.postprocessor.inferenceresults) | The inference result that was previously analyzed. | *required* |
| `image` | `ndarray` | The image (in BGR format) on which to draw the annotations. | *required* |
Returns:
| Type | Description |
| --------- | ------------------------------------------------------ |
| `ndarray` | np.ndarray: The image with tracking annotations drawn. |
# Object Selector
DeGirum Tools API Reference Guide. Select relevant objects while running inference.
{% hint style="info" %}
This API Reference is based on DeGirum Tools version 1.2.0.
{% endhint %}
## Object Selector Analyzer Module Overview
This module provides an analyzer (`ObjectSelector`) for selecting the top-K detections from object detection results based on various strategies and optional tracking. It enables intelligent filtering of detection results to focus on the most relevant objects.
Key Features
* **Selection Strategies**: Supports selecting by highest confidence score, largest bounding-box area, or by custom metric
* **Tracking Integration**: Uses `track_id` fields to persist selections across frames with configurable timeout
* **Top-K Selection**: Configurable number of objects to select per frame
* **Visual Overlay**: Draws bounding boxes for selected objects on images
* **Selection Persistence**: Maintains selection state across frames when tracking is enabled
* **Timeout Control**: Configurable frame count before removing lost objects from selection
Typical Usage
1. Create an `ObjectSelector` instance with desired selection parameters
2. Process each frame's detection results through the selector
3. Access selected objects from the augmented results
4. Optionally visualize selected objects using the annotate method
5. Use selected objects in downstream analyzers for focused processing
Integration Notes
* Works with any detection results containing bounding boxes and confidence scores
* Optional integration with `ObjectTracker` for persistent selection across frames
* Selected objects are marked in the result object for downstream processing
* Supports both frame-based and tracking-based selection modes
Key Classes
* `ObjectSelector`: Main analyzer class that processes detections and maintains selections
* `ObjectSelectionStrategies`: Enumeration of available selection strategies
Configuration Options
* `top_k`: Number of objects to select per frame
* `selection_strategy`: Strategy for ranking objects (by highest confidence score, by largest bounding box area, or by custom metric)
* `use_tracking`: Enable/disable tracking-based selection persistence
* `tracking_timeout`: Frames to wait before removing lost objects from selection
* `show_overlay`: Enable/disable visual annotations
* `annotation_color`: Customize overlay appearance
## Classes
## ObjectSelectionStrategies
`ObjectSelectionStrategies`
Bases: `Enum`
Enumeration of object selection strategies.
#### Members
* `CUSTOM_METRIC (int)`: Selects objects with the highest custom metric value.
* `HIGHEST_SCORE (int)`: Selects objects with the highest confidence scores.
* `LARGEST_AREA (int)`: Selects objects with the largest bounding-box area.
## ObjectSelector
`ObjectSelector`
Bases: `ResultAnalyzerBase`
Selects the top-K detected objects per frame based on a specified strategy.
This analyzer examines the detection results for each frame and retains only the top-K detections according to the chosen `ObjectSelectionStrategies` (e.g., highest confidence score or largest bounding-box area).
When tracking is enabled, it uses object `track_id` information to continue selecting the same objects across successive frames, removing an object from the selection if it has not appeared for a certain number of frames (the tracking timeout).
### ObjectSelector Methods
#### \_\_init\_\_(\*, ...)
`__init__(*, top_k=1, metric_threshold=0.0, selection_strategy=ObjectSelectionStrategies.HIGHEST_SCORE, custom_metric=None, use_tracking=True, tracking_timeout=30, show_overlay=True, annotation_color=None)`
Constructor.
Parameters:
| Name | Type | Description | Default |
| -------------------- | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------- |
| `top_k` | `int` | Number of objects with highest metric value to select. Default 1. When 0, metric\_threshold is used instead. | `1` |
| `metric_threshold` | `float` | Metric value threshold: if top\_k is zero, objects with metric value higher than this threshold are selected. Default 0. | `0.0` |
| `selection_strategy` | `ObjectSelectionStrategies` | Strategy for ranking objects. Default ObjectSelectionStrategies.HIGHEST\_SCORE. | `HIGHEST_SCORE` |
| `custom_metric` | `callable` | Custom metric function to use for ranking. The function should take a detection dictionary and inference result and return a numeric score. | `None` |
| `use_tracking` | `bool` | Whether to enable tracking-based selection. If True, only objects with a `track_id` field are selected (requires an ObjectTracker to precede this analyzer in the pipeline). Default True. | `True` |
| `tracking_timeout` | `int` | Number of frames to wait before removing an object from selection if it is not detected. Default 30. | `30` |
| `show_overlay` | `bool` | Whether to draw bounding boxes around selected objects on the output image. If False, the image is passed through unchanged. Default True. | `True` |
| `annotation_color` | `tuple` | RGB color for annotation boxes. Default None (uses the complement of the result overlay color). | `None` |
Raises:
| Type | Description |
| ------------ | ------------------------------------------------- |
| `ValueError` | If an unsupported selection strategy is provided. |
#### analyze(result)
`analyze(result)`
Select the top-K objects based on the configured strategy, updating the result.
Uses tracking IDs to update selected objects when tracking is enabled. All other objects not selected are removed from results.
Parameters:
| Name | Type | Description | Default |
| -------- | -------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------- | ---------- |
| `result` | [InferenceResults](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/postprocessor#degirum.postprocessor.inferenceresults) | Model result with detection information. | *required* |
Returns:
| Name | Type | Description |
| ------ | ------ | ---------------------------------------------------- |
| `None` | `None` | Modifies `result` in place; does not return a value. |
#### annotate(result, ...)
`annotate(result, image)`
Draw bounding boxes for the selected objects on the image.
Parameters:
| Name | Type | Description | Default |
| -------- | -------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | ---------- |
| `result` | [InferenceResults](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/postprocessor#degirum.postprocessor.inferenceresults) | The result containing selected objects. | *required* |
| `image` | `ndarray` | Image to annotate, shape (H, W, 3) in RGB format. | *required* |
Returns:
| Type | Description |
| --------- | ----------------------------------------------------------- |
| `ndarray` | np.ndarray: Annotated image, shape (H, W, 3) in RGB format. |
# Line Counter
DeGirum Tools API Reference Guide. Count objects as they cross virtual lines.
{% hint style="info" %}
This API Reference is based on DeGirum Tools version 1.2.0.
{% endhint %}
## Line Count Analyzer Module Overview
This module provides an analyzer (`LineCounter`) for detecting and counting objects as they cross user-defined lines within video frames. It enables precise tracking of object movements across virtual boundaries for applications like traffic monitoring and crowd management.
Key Features
* **Flexible Line Definitions**: Support for multiple lines defined by endpoints
* **Directional Counting**: Track crossings in absolute (up/down/left/right) or relative directions
* **Object Trail Tracking**: Use tracked object paths for accurate crossing detection
* **Per-Class Counting**: Maintain separate counts for different object classes
* **Visual Overlay**: Display crossing lines and count statistics on frames
* **Interactive Editing**: Optional OpenCV mouse callback for line adjustment
* **First Crossing Mode**: Option to count each object only once per line
* **Trail Analysis**: Support for analyzing entire object trails or just latest segments
Typical Usage
1. Define lines to monitor within video frames
2. Create a LineCounter instance with desired settings
3. Process inference results through the analyzer chain
4. Access crossing counts from result.line\_counts
5. Optionally visualize lines and counts using annotate method
Integration Notes
* Requires ObjectTracker analyzer upstream for trail data
* Works with any detection results containing bounding boxes
* Supports standard DeGirum PySDK result formats
* Handles partial/missing detections gracefully
Key Classes
* `LineCounter`: Main analyzer class for counting line crossings
* `SingleLineCounts`: Tracks directional counts for absolute frame directions
* `LineCounts`: Extends SingleLineCounts with per-class counting
* `SingleVectorCounts`: Tracks directional counts relative to line orientation
* `VectorCounts`: Extends SingleVectorCounts with per-class counting
Configuration Options
* `lines`: List of line coordinates (x1, y1, x2, y2) to monitor
* `anchor_point`: Bounding box point used for crossing detection
* `whole_trail`: Use entire trail or just latest segment
* `count_first_crossing`: Count each object once per line
* `absolute_directions`: Use absolute or relative directions
* `per_class_display`: Enable per-class counting
* `show_overlay`: Enable visual annotations
* `annotation_color`: Customize overlay appearance
* `window_name`: Enable interactive line adjustment
## Classes
## SingleLineCounts
`SingleLineCounts`
Holds counts of line crossings in four directions.
This class records the number of objects that crossed a line in each cardinal direction relative to the frame: leftward, rightward, upward, and downward. It is typically used within a `LineCounter` result to represent the counts for one monitored line when counting with absolute directions.
Attributes:
| Name | Type | Description |
| -------- | ----- | ------------------------------------------------------------------------------- |
| `left` | `int` | Number of objects crossing the line moving leftward (e.g., from right to left). |
| `right` | `int` | Number of objects crossing the line moving rightward (left to right). |
| `top` | `int` | Number of objects crossing the line moving upward (from bottom toward top). |
| `bottom` | `int` | Number of objects crossing the line moving downward (from top toward bottom). |
## LineCounts
`LineCounts`
Bases: `SingleLineCounts`
Extends SingleLineCounts to include per-class crossing counts.
This class tracks line crossing counts with a breakdown by object class. In addition to the total counts for all objects (inherited attributes left, right, top, bottom), it maintains a dictionary of counts for each object class label. It is typically used by `LineCounter` when `per_class_display=True` to provide class-specific crossing statistics for each line.
Attributes:
| Name | Type | Description |
| ----------- | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| `left` | `int` | Total number of objects crossing leftward (all classes combined). |
| `right` | `int` | Total number of objects crossing rightward (all classes combined). |
| `top` | `int` | Total number of objects crossing upward (all classes combined). |
| `bottom` | `int` | Total number of objects crossing downward (all classes combined). |
| `for_class` | `Dict[str, SingleLineCounts]` | Mapping from class label to a `SingleLineCounts` object for that class. Each entry holds the counts of crossings for that specific object class. |
## SingleVectorCounts
`SingleVectorCounts`
Holds counts of line crossings relative to a line's orientation.
This class is used for counting crossing events in the two opposite directions defined by a line (as opposed to absolute frame directions). It measures how many objects crossed from one side of the line to the other. Specifically, `right` represents crossings from the left side to the right side of the line (following the line's direction vector), and `left` represents crossings from the right side to the left side of the line. This is used by `LineCounter` when `absolute_directions=False` (relative direction mode).
Attributes:
| Name | Type | Description |
| ------- | ----- | ---------------------------------------------------------------------- |
| `right` | `int` | Count of objects crossing from the line's left side to its right side. |
| `left` | `int` | Count of objects crossing from the line's right side to its left side. |
## VectorCounts
`VectorCounts`
Bases: `SingleVectorCounts`
Extends SingleVectorCounts to include per-class crossing counts.
This class maintains overall crossing counts for a line (relative to its orientation) and also tracks counts per object class. It inherits the total `left` and `right` counts (for all objects) from `SingleVectorCounts`, and adds a dictionary of per-class counts. It is used by `LineCounter` when `per_class_display=True` and `absolute_directions=False`.
Attributes:
| Name | Type | Description |
| ----------- | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `left` | `int` | Total number of objects crossing from the right side to the left side of the line (all classes). |
| `right` | `int` | Total number of objects crossing from the left side to the right side of the line (all classes). |
| `for_class` | `Dict[str, SingleVectorCounts]` | Mapping from class label to a `SingleVectorCounts` object for that class. Each entry contains the left/right counts for objects of that specific class. |
## LineCounter
`LineCounter`
Bases: `ResultAnalyzerBase`
Counts objects crossing specified lines in a video stream.
This analyzer processes tracked object trajectories to detect and tally crossing events for each predefined line. It monitors a list of user-defined lines and increments the appropriate count whenever an object's trail crosses a line, determining the direction of each crossing event.
Key features
* Supports absolute (frame-axis) mode counting in four directions, or relative (line-oriented) mode counting in two directions.
* Options to use an object's entire trail versus just the latest segment for crossing detection (`whole_trail`), and to count only the first crossing per object (`count_first_crossing`).
* Can accumulate counts over multiple frames or reset counts each frame (`accumulate` flag).
* Maintains counts for each line (as `LineCounts` in absolute mode or `VectorCounts` in relative mode) and can breakdown counts by object class (if `per_class_display=True`).
* Provides an `annotate(image, result)` method to overlay the lines and current counts on video frames.
* Supports interactive line adjustment via an OpenCV window (see the `window_attach()` method).
After calling `analyze(result)` on a detection/tracking result, the `result` object is augmented with a new attribute `line_counts`. This attribute is a list of count objects (one per line) representing the crossing totals. Each element is either a `LineCounts` (for absolute directions) or `VectorCounts` (for relative directions) instance. If `per_class_display` is enabled, each count object also contains a `for_class` dictionary for per-class counts. Additionally, each detection entry in `result.results` receives a boolean list `cross_line` indicating which lines that object's trail has crossed (True/False for each monitored line).
Note
This analyzer requires that object trajectories (trails) are available in the `result` (e.g., provided by an `ObjectTracker`), since counting is based on each object's movement across frames.
### LineCounter Methods
#### \_\_init\_\_(lines, ...)
`__init__(lines, anchor_point=AnchorPoint.BOTTOM_CENTER, *, whole_trail=True, count_first_crossing=True, absolute_directions=False, accumulate=True, per_class_display=False, show_overlay=True, annotation_color=None, annotation_line_width=None, window_name=None, return_direction_in_results=False)`
Initialize a LineCounter with specified lines and counting options.
Creates a new line counter instance that will track object crossings over the specified lines. The counter can operate in either absolute (frame-axis) or relative (line-oriented) counting modes, and supports various options for trail analysis and visualization.
Parameters:
| Name | Type | Description | Default |
| ----------------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------- | --------------- |
| `lines` | `List[tuple]` | List of line coordinates, each as (x1, y1, x2, y2). | *required* |
| `anchor_point` | `AnchorPoint` | Anchor point on bbox for trails. Default is BOTTOM\_CENTER. | `BOTTOM_CENTER` |
| `whole_trail` | `bool` | Use entire trail or last segment only for intersection. Default True. | `True` |
| `count_first_crossing` | `bool` | Count only first crossing per trail if True. Default True. | `True` |
| `absolute_directions` | `bool` | Directions relative to image axes if True. Default False. | `False` |
| `accumulate` | `bool` | Accumulate counts over frames if True. Default True. | `True` |
| `per_class_display` | `bool` | Display counts per object class if True. Default False. | `False` |
| `show_overlay` | `bool` | Draw annotations if True. Default True. | `True` |
| `annotation_color` | `tuple` | RGB color for annotations. Default is complement of overlay color. | `None` |
| `annotation_line_width` | `int` | Thickness of annotation lines. | `None` |
| `window_name` | `str` | OpenCV window name to attach for interactive adjustment. | `None` |
| `return_direction_in_results` | `bool` | If True, add a `cross_line_direction` list to each detection, describing the direction of crossing for each line. Default False | `False` |
#### analyze(result)
`analyze(result)`
Analyzes object trails for line crossings and updates crossing counts.
Checks every tracked trail in `result.trails` for intersections with all lines, computes crossing direction, updates counts, and adds `line_counts` attribute to the result.
Adds a `cross_line` boolean list to each detected object's dictionary indicating which lines they crossed on this frame.
If `self._return_direction_in_results` is True, also adds a `cross_line_direction` list, aligned with `cross_line`:
{% code overflow="wrap" %}
```yaml
- In absolute mode (absolute_directions=True):
cross_line_direction[i] is a dict:
{
"horizontal": "left" or "right",
"vertical": "top" or "bottom",
}
- In relative mode (absolute_directions=False):
cross_line_direction[i] is "left" or "right"
```
{% endcode %}
#### annotate(result, ...)
`annotate(result, image)`
Draws the defined lines and crossing counts on the image.
Parameters:
| Name | Type | Description | Default |
| -------- | -------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | ---------- |
| `result` | [InferenceResults](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/postprocessor#degirum.postprocessor.inferenceresults) | Model result that contains updated `line_counts`. | *required* |
| `image` | `ndarray` | BGR image to annotate. | *required* |
Returns:
| Name | Type | Description |
| ------- | --------- | ---------------- |
| `image` | `ndarray` | Annotated image. |
#### reset
`reset()`
Reset line crossing counts and crossing history. Clears all previously counted trails and counters.
#### window\_attach(win\_name)
`window_attach(win_name)`
Attaches OpenCV window for interactive line adjustment.
Installs mouse callbacks enabling line dragging.
Parameters:
| Name | Type | Description | Default |
| ---------- | ----- | -------------------------- | ---------- |
| `win_name` | `str` | Name of the OpenCV window. | *required* |
# Event Detector
DeGirum Tools API Reference Guide. Convert analyzer outputs into high-level events.
{% hint style="info" %}
This API Reference is based on DeGirum Tools version 1.2.0.
{% endhint %}
## Event Detector Analyzer Module Overview
This module provides an analyzer (`EventDetector`) for converting analyzer outputs into high-level, human-readable events. It enables detection of complex temporal patterns and conditions based on metrics from other analyzers like zone counters and line counters.
Key Features
* **Metric-Based Events**: Convert analyzer metrics into meaningful events
* **Temporal Patterns**: Detect conditions that must hold for specific durations
* **Complex Conditions**: Combine multiple metrics with logical operators
* **Data-Driven**: Configure events using YAML or dictionary definitions
* **Ring Buffer**: Internal state management for temporal conditions
* **Integration Support**: Works with any analyzer that produces metrics
* **Schema Validation**: Ensures event definitions match required format
Typical Usage
1. Configure auxiliary analyzers (e.g., ZoneCounter, LineCounter) for required metrics
2. Create an EventDetector instance with event definitions
3. Attach it to a model or compound model
4. Access detected events via result.events\_detected
5. Use EventNotifier for event-based notifications
Integration Notes
* Requires metrics from other analyzers to be present in results
* Event definitions must match the event\_definition\_schema
* Supports both YAML and dictionary-based configuration
* Events are stored in result.events\_detected for downstream use
Key Classes
* `EventDetector`: Main analyzer class for detecting events
* `EventDefinitionSchema`: Schema for validating event definitions
Configuration Options
* `event_definitions`: YAML file or dictionary containing event definitions
* `metrics`: Dictionary mapping metric names to their sources
* `temporal_window`: Default duration for evaluating conditions
* `quantifier`: Default proportion/duration for event triggers
* `comparator`: Default operator for comparing metrics to thresholds
## Functions
#### ZoneCount(result, ...)
`ZoneCount(result, params)`
Computes the number of detected objects inside a specified zone or zones.
Requires that `result.zone_counts` is present (produced by a `ZoneCounter` analyzer). This function can filter the count by object class and aggregate counts across multiple zones if needed.
Parameters:
| Name | Type | Description | Default |
| -------- | -------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
| `result` | [InferenceResults](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/postprocessor#degirum.postprocessor.inferenceresults) | Inference results object containing zone count data. | *required* |
| `params` | `dict` | Additional parameters to filter/aggregate the count. classes (List\[str]): Class labels to include. If None, all classes are counted. index (int or str): Zone index (int) or zone name (str) to count. If None, all zones are included. aggregation (str): Aggregation function to apply across zones. One of 'sum', 'max', 'min', 'mean', 'std'. Defaults to 'sum'. | *required* |
Returns:
| Name | Type | Description |
| ------- | -------------- | -------------------------------------------------------- |
| `count` | `int \| float` | Total count of matching objects in the selected zone(s). |
Raises:
| Type | Description |
| ---------------- | --------------------------------------------------------------------- |
| `AttributeError` | If `result.zone_counts` is missing (no ZoneCounter applied upstream). |
| `ValueError` | If a specified zone index/name is out of range or not found. |
#### LineCount(result, ...)
`LineCount(result, params)`
Computes the number of object crossings on a specified line (or across all lines).
Relies on a `result.line_counts` attribute (produced by a `LineCounter` analyzer). This function can filter count by object class and crossing direction for fine-grained event definitions.
Parameters:
| Name | Type | Description | Default |
| -------- | -------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
| `result` | [InferenceResults](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/postprocessor#degirum.postprocessor.inferenceresults) | Inference results object containing line crossing counts. | *required* |
| `params` | `dict` | Filter parameters from the event's "with" clause. index (int): Line index to count. If None, all lines are considered. classes (List\[str]): Class labels to include. If None, all detected classes are counted. directions (List\[str]): Directions of crossing to include. One of 'left', 'right', 'top', 'bottom'. If None, all directions are counted. | *required* |
Returns:
| Name | Type | Description |
| ------- | ----- | ---------------------------------------------------------------- |
| `count` | `int` | Number of line-crossing events that match the specified filters. |
Raises:
| Type | Description |
| ---------------- | --------------------------------------------------------------------- |
| `AttributeError` | If `result.line_counts` is missing (no LineCounter applied upstream). |
| `ValueError` | If a specified line index is out of range for the available lines. |
#### ObjectCount(result, ...)
`ObjectCount(result, params)`
Counts the detected objects in the result, with optional class and score filtering.
This metric does not require any auxiliary analyzer; it simply tallies detections, optionally constrained by object class and minimum confidence score.
Parameters:
| Name | Type | Description | Default |
| -------- | -------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- |
| `result` | [InferenceResults](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/postprocessor#degirum.postprocessor.inferenceresults) | The inference results object containing detections. | *required* |
| `params` | `dict` | Filter parameters. classes (List\[str]): Class labels to include. If None, all detected classes are counted. min\_score (float): Minimum confidence score required for counting. If None, no score threshold is applied. | *required* |
Returns:
| Name | Type | Description |
| ------- | ----- | ---------------------------------------------------------------------------- |
| `count` | `int` | Number of detected objects that meet the specified class and score criteria. |
## Classes
## EventDetector
`EventDetector`
Bases: `ResultAnalyzerBase`
Analyzes inference results over time to detect high-level events based on metric conditions.
This analyzer monitors a chosen metric (e.g., ZoneCount, LineCount, or ObjectCount) over a sliding time window and triggers an event when a specified condition is satisfied. The condition consists of a comparison of the metric value against a threshold, and a temporal requirement that the condition holds for a certain duration or proportion of the window. When the condition is met, the event name is added to the `events_detected` set in the result.
For example, you can detect an event "PersonInZone" when the count of persons in a region (`ZoneCount`) remains above 0 for N seconds, or a "VehicleCountExceeded" event when a line-crossing count exceeds a threshold within a frame. Multiple `EventDetector` instances can be attached to the same model to detect different events in parallel.
Attributes:
| Name | Type | Description |
| --------------------- | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| `key_events_detected` | `str` | Name of the result attribute that stores detected event names (defaults to "events\_detected"). |
| `comparators` | `Dict[str, Callable[[float, float], bool]]` | Mapping of comparator keywords (e.g., "is greater than") to the corresponding comparison functions used internally. |
Note
Ensure that required analyzers (such as `ZoneCounter` or `LineCounter`) are attached before this detector so that the necessary metric data is present in `result`. The `EventDetector` maintains all state internally (using a ring buffer for timing) and is therefore stateless to callers. It is safe to use in a single-threaded inference pipeline (thread-safe per inference thread).
### EventDetector Methods
#### \_\_init\_\_(event\_description, ...)
`__init__(event_description, *, custom_metric=None, show_overlay=True, annotation_color=None, annotation_font_scale=None, annotation_pos=AnchorPoint.BOTTOM_LEFT)`
Initializes an EventDetector with a given event description and overlay settings.
The `event_description` defines the event's trigger name, metric, comparison, and timing requirements. It can be provided as a YAML string or an equivalent dictionary and must conform to the expected schema (`event_definition_schema`).
The description includes these key components:
* Trigger: Name of the event to detect
* when: Metric to evaluate (one of "ZoneCount", "LineCount", or "ObjectCount")
* Comparator: A comparison operator (e.g., "is greater than") with a threshold value
* during: Duration of the sliding window as `[value, unit]` (unit can be "frames" or "seconds")
* for at least / for at most (optional): Required portion of the window that the condition must hold true to trigger the event
**Event Definition Schema (YAML)**:
{% code overflow="wrap" %}
```python
type: object
additionalProperties: false
properties:
Trigger:
type: string
description: The name of event to raise
when:
type: string
enum: [ZoneCount, LineCount, ObjectCount, CustomMetric]
description: The name of the metric to evaluate
with:
type: object
additionalProperties: false
properties:
classes:
type: array
items:
type: string
description: The class labels to count; if not specified, all classes are counted
index:
type: integer
description: The location number (zone or line index) to count; if not specified, all locations are counted
directions:
type: array
items:
type: string
enum: [left, right, top, bottom]
description: The line intersection directions to count; if not specified, all directions are counted
min score:
type: number
description: The minimum score of the object to count
minimum: 0
maximum: 1
aggregation:
type: string
enum: [sum, max, min, mean, std]
is equal to:
type: number
description: The value to compare against
is not equal to:
type: number
description: The value to compare against
is greater than:
type: number
description: The value to compare against
is greater than or equal to:
type: number
description: The value to compare against
is less than:
type: number
description: The value to compare against
is less than or equal to:
type: number
description: The value to compare against
during:
type: array
prefixItems:
- type: number
- enum: [seconds, frames, second, frame]
items: false
description: Duration to evaluate the metric
for at least:
type: array
prefixItems:
- type: number
- enum: [percent, frames, frame]
items: false
description: Minimum duration the metric must hold true to trigger the event
for at most:
type: array
prefixItems:
- type: number
- enum: [percent, frames, frame]
items: false
description: Maximum duration the metric can hold true for the event to trigger
required: [Trigger, when, during]
oneOf:
- required: [is equal to]
type: object
- required: [is not equal to]
type: object
- required: [is greater than]
type: object
- required: [is greater than or equal to]
type: object
- required: [is less than]
type: object
- required: [is less than or equal to]
type: object
```
{% endcode %}
Parameters:
| Name | Type | Description | Default |
| ----------------------- | --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- |
| `event_description` | `Union[str, dict]` | YAML string or dictionary defining the event conditions (must match the schema above). | *required* |
| `custom_metric` | `Callable` | Custom metric function. Must be provided, if `when` key in `event_description` is set to "CustomMetric". The function accepts inference result and parameters dict of "with" clause and returns a numeric value. | `None` |
| `show_overlay` | `bool` | Whether to draw a label on the frame when the event fires. Defaults to True. | `True` |
| `annotation_color` | `tuple` | RGB color for the label background. If None, a contrasting color is auto-chosen. Defaults to None. | `None` |
| `annotation_font_scale` | `float` | Font scale for the overlay text. If None, uses a default scale. Defaults to None. | `None` |
| `annotation_pos` | `Union[AnchorPoint, tuple]` | Position for the overlay label (an `AnchorPoint` or (x,y) coordinate). Defaults to `AnchorPoint.BOTTOM_LEFT`. | `BOTTOM_LEFT` |
Raises:
| Type | Description |
| ----------------- | -------------------------------------------------------------------------- |
| `ValidationError` | If `event_description` does not conform to the required schema for events. |
| `ValueError` | If no comparison operator is specified in the event description. |
#### analyze(result)
`analyze(result)`
Evaluates the configured event condition on an inference result and updates the result if the event is detected.
The method computes the metric value, compares it to the threshold, maintains a history of condition states, and triggers the event when the condition holds for the required duration.
Parameters:
| Name | Type | Description | Default |
| -------- | -------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | ---------- |
| `result` | [InferenceResults](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/postprocessor#degirum.postprocessor.inferenceresults) | The inference result to analyze (must contain necessary metrics from prior analyzers). | *required* |
Returns:
| Type | Description |
| ------ | ---------------------------------------------------------------------------------------- |
| `None` | This method modifies the `result` in-place by adding to its `events_detected` attribute. |
Raises:
| Type | Description |
| ---------------- | ----------------------------------------------------------------------------------------------------------- |
| `AttributeError` | If the required metric data is missing in `result` (e.g., using ZoneCount without attaching `ZoneCounter`). |
#### annotate(result, ...)
`annotate(result, image)`
Draws the event label onto the image frame if the event is active for this result.
The overlay text is rendered only when all of the following conditions are true
* `self._show_overlay` is True for this EventDetector.
* The event's name is present in `result.events_detected` for the current frame.
If these conditions are met, the event name is drawn on the image at the configured position. The label's background color defaults to the complement of the model's overlay color (if no `annotation_color` was specified), and the text color is automatically chosen for optimal contrast.
Parameters:
| Name | Type | Description | Default |
| -------- | -------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | ---------- |
| `result` | [InferenceResults](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/postprocessor#degirum.postprocessor.inferenceresults) | The result object from the model (after analysis), which may contain the event in its `events_detected` set. | *required* |
| `image` | `ndarray` | The BGR image frame to annotate. | *required* |
Returns:
| Type | Description |
| --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ndarray` | np.ndarray: The same image frame with the event label overlay (if the event was detected). If no overlay was added, the image is returned unmodified. |
# Notifier
DeGirum Tools API Reference Guide. Trigger notifications when events occur.
{% hint style="info" %}
This API Reference is based on DeGirum Tools version 1.2.0.
{% endhint %}
## Notification Analyzer Module Overview
This module provides tools for generating and delivering notifications based on AI inference events. It implements the `EventNotifier` analyzer for triggering notifications and optional clip saving on events.
Key Features
* Event-Based Triggers: Generates notifications when user-defined event conditions are met
* Message Formatting: Supports `${expression}` fields for dynamic notification content
* Holdoff Control: Configurable time/frame windows to suppress repeat notifications
* Video Clip Saving: Optional video clip saving with local or cloud storage
* Visual Overlay: Annotates active notification status on images
* File Management: Handles temporary file cleanup and storage integration
Typical Usage
1. Configure notification service. For external services, use Apprise URL or config file. For console output, use "json://console" as notification\_config
2. Define event conditions and create `EventNotifier` instances
3. Process inference results through the notifier chain
4. Notifications are sent when conditions are met
Integration Notes
* Requires `EventDetector` analyzer in the chain to provide event detection
* Optional dependencies (e.g., apprise) must be installed for external notification services
* Storage configuration required for clip saving (supports both local and cloud storage)
* Supports both frame-based and time-based notification holdoff periods
Key Classes
* `EventNotifier`: Analyzer for triggering notifications based on event conditions
Configuration Options
* `notification_config`: Apprise URL or config file for notification service, or "json://console" for stdout output
* `notification_title`: Default title for notifications
* `holdoff_frames`: Number of frames to wait between notifications
* `holdoff_seconds`: Time in seconds to wait between notifications
* `clip_save`: Enable/disable video clip saving
* `storage_config`: Storage configuration for clip saving (supports local and cloud storage)
* `show_overlay`: Enable/disable visual annotations
Message Formatting
* Use `${expression}` fields in the `message` parameter to include dynamic content (e.g., `${time}` for the time the notification was sent)
* Use markdown formatting for rich text notifications (e.g. `**bold**`, `*italic*`, `[link](url)`)
* Supported variables to use in `${expression}` fields include:
* `${result}`: The inference result
* `${time}`: The time the notification was sent
* `${url}`: The URL of the uploaded file
* `${filename}`: The name of the uploaded file
Example
For local storage configuration:
{% code overflow="wrap" %}
```
clip_storage_config = ObjectStorageConfig(
endpoint=".", # path to local folder
access_key="", # not needed for local storage
secret_key="", # not needed for local storage
bucket="my_bucket_dir", # subdirectory name for local storage
)
```
{% endcode %}
## Classes
## EventNotifier
`EventNotifier`
Bases: `ResultAnalyzerBase`
Analyzer for event-based notifications.
Works in conjunction with an `EventDetector` analyzer by examining the `events_detected` set in the inference results. Generates notifications when user-defined event conditions are met.
Features
* Message Formatting: supports `${expression}` fields for dynamic notification content.
* Holdoff to suppress repeat notifications within a specified time/frame window
* Optional video clip saving upon notification trigger with local or cloud storage
* Records triggered notifications in the result object's `notifications` dictionary
* Overlay annotation of active notification status on images
### EventNotifier Methods
#### \_\_init\_\_(name, ...)
`__init__(name, condition, *, message='', notify_while_true=False, holdoff=0, notification_config=None, notification_tags=None, show_overlay=True, annotation_color=None, annotation_font_scale=None, annotation_pos=AnchorPoint.BOTTOM_LEFT, annotation_cool_down=3.0, clip_save=False, clip_sub_dir='', clip_duration=0, clip_pre_trigger_delay=0, clip_embed_ai_annotations=True, clip_target_fps=30.0, storage_config=None, notification_timeout_s=None)`
Constructor.
Parameters:
| Name | Type | Description | Default |
| --------------------------- | --------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- |
| `name` | `str` | Name of the notification. | *required* |
| `condition` | `str` | Python expression defining the condition to trigger the notification (references event names from `EventDetector`). | *required* |
| `message` | `str` | Notification message format string. If empty, uses "Notification triggered: {name}". Default is "". | `''` |
| `notify_while_true` | `bool` | Whether to notify continuously while the condition is true. Default is False: notify only when condition changes from False to True. | `False` |
| `holdoff` | `int \| float \| Tuple[float, str]` | Holdoff duration to suppress repeated notifications. If int, interpreted as frames; if float, as seconds; if tuple (value, "seconds"/"frames"), uses the specified unit. Default is 0 (no holdoff). | `0` |
| `notification_config` | `str` | Notification service config file path, Apprise URL, or "json://console" for stdout output. If None, notifications are not sent to any external service. | `None` |
| `notification_tags` | `str` | Tags to attach to notifications for filtering. Multiple tags can be separated by commas (for logical AND) or spaces (for logical OR). | `None` |
| `show_overlay` | `bool` | Whether to overlay notification text on images. Default is True. | `True` |
| `annotation_color` | `tuple` | RGB color for the annotation text background. If None, uses a complementary color to the result overlay. | `None` |
| `annotation_font_scale` | `float` | Font scale for the annotation text. If None, uses the default model font scale. | `None` |
| `annotation_pos` | `AnchorPoint \| Tuple[int, int] \| List[int]` | Position to place annotation text (either an AnchorPoint or an (x,y) coordinate). Default is AnchorPoint.BOTTOM\_LEFT. | `BOTTOM_LEFT` |
| `annotation_cool_down` | `float` | Time in seconds to display the notification text on the image. Default is 3.0. | `3.0` |
| `clip_save` | `bool` | If True, save a video clip when the notification triggers. Default is False. | `False` |
| `clip_sub_dir` | `str` | Subdirectory name in the storage bucket for saved clips. Default is "" (no subdirectory). | `''` |
| `clip_duration` | `int` | Length of the saved video clip in frames. Default is 0 (uses available frames around event). | `0` |
| `clip_pre_trigger_delay` | `int` | Number of frames to include before the trigger event in the saved clip. Default is 0. | `0` |
| `clip_embed_ai_annotations` | `bool` | If True, embed AI annotations in the saved clip. Default is True. | `True` |
| `clip_target_fps` | `float` | Frame rate (FPS) for the saved video clip. Default is 30.0. | `30.0` |
| `storage_config` | `ObjectStorageConfig` | Storage configuration for clip saving. For local storage, use endpoint="./" and local directory as bucket. For cloud storage, use S3-compatible endpoint and credentials. If None, clips are only saved locally. | `None` |
| `notification_timeout_s` | `float` | Maximum time in seconds to wait for a notification job to complete before marking it as timed out. If None, uses the default timeout. | `None` |
Raises: ValueError: If holdoff unit is not "seconds" or "frames". ImportError: If required optional packages are not installed.
Message Formatting
* Use `${expression}` fields in the `message` parameter to include dynamic content (e.g., `${time}` for the time the notification was sent)
* Use markdown formatting for rich text notifications (e.g. `**bold**`, `*italic*`, `[link](url)`)
* Supported variables to use in `${expression}` fields include:
* `${result}`: The inference result
* `${time}`: The time the notification was sent
* `${url}`: The URL of the uploaded file
* `${filename}`: The name of the uploaded file
* `re` and `json` modules are also available for advanced formatting
#### analyze(result)
`analyze(result)`
Evaluate the notification condition on the given inference result.
If the condition is satisfied (and not within a holdoff period), generates a notification message and stores it in `result.notifications`. Optionally saves a video clip when a notification is triggered, and schedules that clip for upload if storage is configured. This method modifies the input result object in-place.
Parameters:
| Name | Type | Description | Default |
| -------- | -------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- | ---------- |
| `result` | [InferenceResults](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/postprocessor#degirum.postprocessor.inferenceresults) | The inference result to analyze, which should include events detected by an `EventDetector`. | *required* |
Returns:
| Type | Description |
| ------ | ------------------------------------------------------ |
| `None` | This method modifies the input result object in-place. |
Raises:
| Type | Description |
| ---------------- | ----------------------------------------------------------------------------- |
| `AttributeError` | If the result does not contain events\_detected (EventDetector not in chain). |
#### annotate(result, ...)
`annotate(result, image)`
Draws the active notification message on the image.
Only draws the message if the notification is currently active and within its cool-down period.
Parameters:
| Name | Type | Description | Default |
| -------- | -------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------- | ---------- |
| `result` | [InferenceResults](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/postprocessor#degirum.postprocessor.inferenceresults) | The inference result object (may contain a notification entry). | *required* |
| `image` | `ndarray` | The image frame (BGR format) to annotate. | *required* |
Returns:
| Type | Description |
| --------- | -------------------------------- |
| `ndarray` | np.ndarray: The annotated image. |
#### finalize
`finalize()`
Finalize and clean up resources.
Waits for all background clip-saving threads to finish, stops the notification server, and removes the temporary clip directory if it was used.
# Clip Saver
DeGirum Tools API Reference Guide. Record video clips around trigger events.
{% hint style="info" %}
This API Reference is based on DeGirum Tools version 1.2.0.
{% endhint %}
## Clip Saving Analyzer Module Overview
This module provides an analyzer (`ClipSavingAnalyzer`) for recording video snippets triggered by events or notifications. It captures frames before and after trigger events, saving them as video clips with optional AI annotations and metadata.
Key Features
* **Pre/Post Buffering**: Configurable frame count before and after trigger events
* **Optional Overlays**: Embed AI bounding boxes and labels in the saved clips
* **Side-car JSON**: Save raw inference results alongside video files
* **Thread-Safe**: Each clip is written by its own worker thread
* **Frame Rate Control**: Configurable target FPS for saved clips
* **Event Integration**: Works with EventDetector and EventNotifier triggers
* **Storage Support**: Optional integration with object storage for clip uploads
Typical Usage
1. Create a `ClipSavingAnalyzer` instance with desired buffer and output settings
2. Process inference results through the analyzer chain
3. When triggers occur, clips are automatically saved with pre/post frames
4. Access saved clips and their associated metadata files
5. Optionally upload clips to object storage for remote access
Integration Notes
* Works with any analyzer that adds trigger names to results
* Requires video frames to be available in the result object
* Supports both local file storage and object storage uploads
* Thread-safe for concurrent clip saving operations
Key Classes
* `ClipSavingAnalyzer`: Main analyzer class for saving video clips
* `ClipSaver`: Internal class handling clip writing and buffering
Configuration Options
* `clip_duration`: Number of frames to save after trigger
* `clip_prefix`: Base path for saved clip files
* `pre_trigger_delay`: Frames to include before trigger
* `embed_ai_annotations`: Enable/disable AI overlays in clips
* `save_ai_result_json`: Enable/disable metadata saving
* `target_fps`: Frame rate for saved video clips
## Classes
## ClipSavingAnalyzer
`ClipSavingAnalyzer`
Bases: `ResultAnalyzerBase`
Result-analyzer that records short video clips whenever one of the configured trigger names appears in an [InferenceResults](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/postprocessor#degirum.postprocessor.inferenceresults). It delegates internally to [`ClipSaver`](/degirum-tools/support/video_support), which maintains a circular buffer, so every clip contains both pre-trigger and post-trigger context.
### ClipSavingAnalyzer Methods
#### \_\_init\_\_(clip\_duration, ...)
`__init__(clip_duration, triggers, file_prefix, *, pre_trigger_delay=0, embed_ai_annotations=True, save_ai_result_json=True, target_fps=30.0)`
Constructor.
Parameters:
| Name | Type | Description | Default |
| ---------------------- | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
| `clip_duration` | `int` | Total length of the output clip in frames (pre-buffer + post-buffer). | *required* |
| `triggers` | `Set[str]` | Names that fire the recorder when found in either [`EventDetector`](/degirum-tools/analyzers/event_detector#key_events_detected) or [`EventNotifier`](/degirum-tools/analyzers/notifier#key_notifications). | *required* |
| `file_prefix` | `str` | Path and filename prefix for generated files (frame number & extension are appended automatically). | *required* |
| `pre_trigger_delay` | `int` | Frames to include before the trigger. Defaults to 0. | `0` |
| `embed_ai_annotations` | `bool` | If True, use `InferenceResults.image_overlay` so bounding boxes/labels are burned into the clip. Defaults to True. | `True` |
| `save_ai_result_json` | `bool` | If True, dump a JSON file with raw inference results alongside the video. Defaults to True. | `True` |
| `target_fps` | `float` | Frame rate of the output file. Defaults to 30.0. | `30.0` |
#### analyze(result)
`analyze(result)`
Inspect a single [InferenceResults](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/postprocessor#degirum.postprocessor.inferenceresults) and forward it to internal [`ClipSaver`](/degirum-tools/support/video_support) if any trigger names are matched.
This method is called automatically for each frame when attached via [`attach_analyzers`](/degirum-tools/inference_support).
Parameters:
| Name | Type | Description | Default |
| -------- | -------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------ | ---------- |
| `result` | [InferenceResults](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/postprocessor#degirum.postprocessor.inferenceresults) | Current model output to scan for events/notifications. | *required* |
#### join\_all\_saver\_threads
`join_all_saver_threads()`
Block until all background clip-writer threads finish.
Returns:
| Name | Type | Description |
| ----- | ----- | ----------------------------------- |
| `int` | `int` | Number of threads that were joined. |
# Scene Cut Detector
DeGirum Tools API Reference Guide. Detect scene cuts in video for use with ObjectTracker and other analyzers.
{% hint style="info" %}
This API Reference is based on DeGirum Tools version 1.2.0.
{% endhint %}
## Scene Cut Detector Analyzer Module Overview
This module provides an analyzer (`SceneCutDetector`) for detecting scene cuts in video streams by comparing frame-to-frame differences using an adaptive thresholding approach. The detector is based on the PySceneDetect adaptive algorithm, which calculates differences in HSV color space and uses a rolling average to adapt to local changes.
Key Features
* **Adaptive Thresholding**: Uses rolling average of previous frames to adapt to gradual changes
* **HSV Color Space**: Analyzes differences in hue, saturation, and luminance channels
* **Fast Processing**: Supports luma-only mode and automatic frame resizing for performance
* **Configurable Parameters**: Adjustable sensitivity, minimum scene length, and window size
* **Real-time Detection**: Causal approach using only past frames for zero latency
* **Scene Cut Flag**: Adds `scene_cut` boolean attribute to inference results
Typical Usage
1. Create a `SceneCutDetector` instance with desired parameters
2. Attach it to a model or inference pipeline
3. Process video frames through the analyzer
4. Check `result.scene_cut` flag to detect scene transitions
5. Use scene cut information for downstream processing or triggering actions
Put it before ObjectTracker in the analyzer pipeline to ensure cuts are detected before tracking is applied, allowing you to reset object tracker on scene changes.
Integration Notes
* Works with any inference results that contain image data
* Can be combined with other analyzers in a pipeline
* Useful for video segmentation, activity detection, and content analysis
* Maintains internal state to track frame history
Configuration Options
* `adaptive_threshold`: Sensitivity ratio for detecting cuts (higher = less sensitive)
* `min_scene_len`: Minimum frames between detected cuts to avoid false positives
* `window_width`: Number of previous frames for rolling average calculation
* `min_content_val`: Minimum absolute change threshold for scene cuts
* `luma_only`: Use only brightness changes for faster processing
Key Classes
* `SceneCutDetector`: Main analyzer class for scene cut detection
## Classes
## SceneCutDetector
`SceneCutDetector`
Bases: `ResultAnalyzerBase`
Analyzer for detecting scene cuts using adaptive thresholding.
This analyzer examines consecutive frames and detects scene cuts when the frame-to-frame difference significantly exceeds the rolling average of recent frames. It uses HSV color space analysis to detect content changes while adapting to gradual variations like camera motion or lighting changes.
The detector adds a `scene_cut` boolean attribute to each inference result indicating whether a scene cut was detected at that frame.
Attributes:
| Name | Type | Description |
| -------------------- | ------- | ----------------------------------------------------------------------- |
| `adaptive_threshold` | `float` | Ratio threshold for scene cut detection. |
| `min_scene_len` | `int` | Minimum frames between consecutive scene cuts. |
| `window_width` | `int` | Number of previous frames used for rolling average. |
| `min_content_val` | `float` | Minimum absolute content change threshold. |
| `luma_only` | `bool` | Whether to use only luminance channel for comparison. |
| `resize_limit` | `int` | Maximum image dimension to apply frame resizing to improve performance. |
### SceneCutDetector Methods
#### \_\_init\_\_(\*, ...)
`__init__(*, adaptive_threshold=3.0, min_scene_len=15, window_width=4, min_content_val=15.0, luma_only=False, resize_limit=240)`
Initialize the scene cut detector.
Parameters:
| Name | Type | Description | Default |
| -------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- |
| `adaptive_threshold` | `float` | Ratio that the frame score must exceed relative to the average of surrounding frames to trigger a cut. Default 3.0 means the frame must have 3x more change than its neighbors. | `3.0` |
| `min_scene_len` | `int` | Minimum number of frames between detected cuts. Default 15 frames. | `15` |
| `window_width` | `int` | Number of previous frames to use for computing the rolling average. Must be at least 1. Default 4. | `4` |
| `min_content_val` | `float` | Minimum absolute change threshold. Even if the adaptive ratio is exceeded, the absolute change must be at least this value. Default 15.0. | `15.0` |
| `luma_only` | `bool` | If True, only considers changes in luminance (brightness), ignoring color information for faster processing. Default False. | `False` |
| `resize_limit` | `int` | Maximum image dimension to apply frame resizing to improve performance. Frames larger than 1.5x this limit will be resized. Default 240. | `240` |
#### analyze(result)
`analyze(result)`
Analyze a frame and detect scene cuts using adaptive thresholding.
This method processes the image from the inference result, calculates the frame-to-frame content difference, and sets `result.scene_cut` to True if a scene cut is detected, False otherwise.
Uses a causal approach: compares the current frame score against the average of previous frame scores, enabling real-time detection with no latency.
Parameters:
| Name | Type | Description | Default |
| -------- | -------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------- | ---------- |
| `result` | [InferenceResults](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/postprocessor#degirum.postprocessor.inferenceresults) | The inference result containing the image to analyze. | *required* |
Returns:
| Name | Type | Description |
| ------ | ------ | --------------------------------------------------------------- |
| `None` | `None` | Modifies `result` in place by adding the `scene_cut` attribute. |
# Support Modules
DeGirum Tools API Reference Guide. Overview of Support modules included in DeGirum Tools.
{% hint style="info" %}
This API Reference is based on DeGirum Tools version 0.24.1.
{% endhint %}
## Support modules overview
DeGirum Tools provides a suite of support modules designed to streamline common tasks in AI application development. These utilities complement PySDK and DeGirum Tools by covering media handling, math helpers, evaluation, UI elements, and storage helpers.
## Audio support
The [Audio Support module](/degirum-tools/support/audio_support) offers utilities for managing and processing audio streams.
* **Key Features**:
* Open and manage audio streams from microphones and WAV files.
* Generate audio frames with configurable buffer sizes, including support for overlapping buffers.
* Handles both blocking and non-blocking stream operations.
* **Typical Usage**: Integrating microphone input or file-based audio into AI pipelines, real-time audio processing.
## Evaluation support
The [Model Evaluation Support module](/degirum-tools/support/eval_support) provides base classes and tools for assessing the performance of AI models.
* **Key Features**:
* Abstract base class (`ModelEvaluatorBase`) for creating custom model evaluators.
* Support for evaluator configuration via YAML files.
* Tools for comparing model outputs with ground truth data and reporting results.
* **Typical Usage**: Implementing custom evaluation pipelines for various models and datasets.
## Math support
The [Math Support module](/degirum-tools/support/math_support) delivers mathematical utilities tailored for computer vision and signal processing tasks.
* **Key Features**:
* Bounding box operations: area calculation, Intersection over Union (IoU), coordinate conversions.
* Non-Maximum Suppression (NMS) with multiple selection policies.
* Image tiling utilities for fixed size or aspect ratio.
* A lightweight FIR low-pass filter for signal smoothing.
* **Typical Usage**: Post-processing object detection results, preparing image data for models, smoothing time-series data.
## Object storage support
The [Object Storage Support module](/degirum-tools/support/object_storage_support) offers helper utilities for interacting with MinIO object storage.
* **Key Classes**:
* `ObjectStorageConfig`: Manages configuration parameters for object storage connections.
* `ObjectStorage`: A wrapper for common bucket operations like upload, download, and delete.
* **Typical Usage**: Managing datasets, saving inference results, or handling other file-based assets in cloud or local storage.
## UI support
The [UI Support module](/degirum-tools/support/ui_support) provides a versatile set of tools for user interface operations.
* **Key Features**:
* Displaying images and videos (`Display` class).
* Rendering text on images with customizable options (`put_text()`).
* Tracking progress with visual bars (`Progress` class).
* Measuring and displaying Frames Per Second (FPS) (`FPSMeter` class).
* **Typical Usage**: Visualizing model outputs, creating interactive demos, monitoring application performance.
## Video support
The [Video Support module](/degirum-tools/support/video_support) offers comprehensive capabilities for using video streams with PySDK and DeGirum Tools.
* **Key Features**:
* Capture from local cameras, IP cameras, and video files.
* Save video streams with configurable quality, format, and frame rates (`VideoWriter`).
* Extract frames from video files into JPEG sequences (`video2jpegs()`).
* Manage and save event-triggered video clips with pre/post event buffering (`ClipSaver`).
* **Typical Usage**: Building video processing pipelines, security and surveillance applications, dataset creation from videos.
# Audio Support
DeGirum Tools API Reference Guide. Open microphone or file-based audio streams.
{% hint style="info" %}
This API Reference is based on DeGirum Tools version 1.2.0.
{% endhint %}
## Audio Support Module Overview
This module provides utilities for opening microphone or file-based audio streams and for generating audio frames. The module provides context managers and generators that simplify audio capture and processing workflows.
Key Features
* **Audio Stream Management**: Open and manage audio streams from microphones and files
* **Buffer Generation**: Generate audio frames with configurable buffer sizes
* **Overlapping Buffers**: Support for overlapping audio buffers
* **Format Support**: Handle WAV files and microphone input
* **Stream Control**: Non-blocking and blocking stream operations
* **Error Handling**: Robust error handling for stream operations
Typical Usage
1. Call `open_audio_stream()` to create an audio stream
2. Iterate over `audio_source()` or `audio_overlapped_source()` to process frames
3. Use non-blocking mode for real-time processing
4. Handle stream errors and cleanup
Integration Notes
* Works with PyAudio for microphone input
* Supports WAV file format
* Handles both local and remote audio sources
* Provides consistent interface across platforms
Key Functions
* `open_audio_stream()`: Context manager for microphone or WAV files
* `audio_source()`: Generator yielding audio buffers
* `audio_overlapped_source()`: Generator yielding overlapping buffers
Configuration Options
* Sampling rate
* Buffer size
* Source selection
* Blocking mode
## Functions
#### open\_audio\_stream(sampling\_rate\_hz, ...)
`open_audio_stream(sampling_rate_hz, buffer_size, audio_source=None)`
Open an audio stream.
This context manager opens a microphone or WAV file as an audio stream and automatically closes it when the context exits.
Parameters:
| Name | Type | Description | Default |
| ------------------ | ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | ---------- |
| `sampling_rate_hz` | `int` | Desired sample rate in hertz. | *required* |
| `buffer_size` | `int` | Buffer size in frames. | *required* |
| `audio_source` | `Union[int, str, None]` | Source identifier. Use an integer for a microphone index, a string for a WAV file path or URL, or `None` to use the default source. | `None` |
Yields:
| Type | Description |
| ----- | ------------------------------------------------------------- |
| `Any` | Stream-like object with get() method returning audio buffers. |
Raises:
| Type | Description |
| ----------- | ----------------------------------------------------------------------- |
| `Exception` | If the audio stream cannot be opened or the WAV file format is invalid. |
#### audio\_source(stream, ...)
`audio_source(stream, check_abort, non_blocking=False)`
Yield audio frames from a stream.
Parameters:
| Name | Type | Description | Default |
| -------------- | -------------------- | ------------------------------------------------------------------------------------------------ | ---------- |
| `stream` | `Any` | Audio stream object returned by `open_audio_stream()`. | *required* |
| `check_abort` | `Callable[[], bool]` | Callback that returns `True` to stop iteration. | *required* |
| `non_blocking` | `bool` | If `True` and no frame is available, `None` is yielded instead of blocking. Defaults to `False`. | `False` |
Yields:
| Type | Description |
| ------------------- | -------------------------------------------------------------------------------------- |
| `Optional[ndarray]` | Waveform of int16 samples or None when no data is available and non\_blocking is True. |
#### audio\_overlapped\_source(stream, ...)
`audio_overlapped_source(stream, check_abort, non_blocking=False)`
Generate audio frames with 50% overlap.
The function reads blocks from `stream` and yields frames that overlap by half of their length. Overlapping frames produce smoother results for audio analysis.
Parameters:
| Name | Type | Description | Default |
| -------------- | -------------------- | ------------------------------------------------------------------------------------------------ | ---------- |
| `stream` | `Any` | Audio stream object returned by `open_audio_stream()`. | *required* |
| `check_abort` | `Callable[[], bool]` | Callback that returns `True` to stop iteration. | *required* |
| `non_blocking` | `bool` | If `True` and no frame is available, `None` is yielded instead of blocking. Defaults to `False`. | `False` |
Yields:
| Type | Description |
| ------------------- | -------------------------------------------------------------------------------------------------------- |
| `Optional[ndarray]` | Waveform of int16 samples with 50% overlap, or None when no data is available and non\_blocking is True. |
# Model Evaluation Support
DeGirum Tools API Reference Guide. Framework and base class for model accuracy evaluation.
{% hint style="info" %}
This API Reference is based on DeGirum Tools version 1.2.0.
{% endhint %}
## Model Evaluation Support Module Overview
This module provides base classes and utilities for model evaluation, including performance metrics calculation, ground truth comparison, and evaluation result reporting. It supports various evaluation scenarios and metrics.
Key Features
* **Base Evaluator Class**: Abstract base class for model evaluators
* **YAML Configuration**: Support for evaluator configuration via YAML files
* **Flexible Evaluation**: Support for different evaluation metrics and scenarios
* **Result Reporting**: Standardized evaluation result reporting
* **Ground Truth Integration**: Support for comparing model outputs with ground truth
Typical Usage
1. Create a custom evaluator by subclassing ModelEvaluatorBase
2. Configure the evaluator using YAML or constructor parameters
3. Run evaluation on test datasets
4. Analyze and report evaluation results
Integration Notes
* Works with DeGirum PySDK models
* Supports standard evaluation metrics
* Handles various input formats
* Provides extensible evaluation framework
Key Classes
* `ModelEvaluatorBase`: Base class for model evaluators
Configuration Options
* Model parameters
* Evaluation metrics
* Dataset paths
* Ground truth format
## Classes
## ModelEvaluatorBase
`ModelEvaluatorBase`
Bases: `ABC`
Base class for model evaluators.
This abstract class initializes a model object, loads configuration parameters and defines the interface for performing evaluation.
Parameters:
| Name | Type | Description | Default |
| ---------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
| `model` | `Model` | Model instance to evaluate. | *required* |
| `**kwargs` | `Any` | Arbitrary model or evaluator parameters. Keys matching the model attributes are applied directly to the model. Remaining keys are assigned to the evaluator instance if such attributes exist. | `{}` |
Attributes:
| Name | Type | Description |
| ------- | ------- | -------------------------- |
| `model` | `Model` | The model being evaluated. |
### ModelEvaluatorBase Methods
#### \_\_init\_\_(model, ...)
`__init__(model, **kwargs)`
Initialize the evaluator.
Parameters:
| Name | Type | Description | Default |
| ---------- | ------- | ----------------------------------------------------------------------------------------------------------------------- | ---------- |
| `model` | `Model` | PySDK model object. | *required* |
| `**kwargs` | `Any` | Arbitrary model or evaluator parameters. Keys must either match model attributes or attributes of `ModelEvaluatorBase`. | `{}` |
#### evaluate(image\_folder\_path, ...)
`evaluate(image_folder_path, ground_truth_annotations_path, max_images=0)`
`abstractmethod`
Evaluate the model on a dataset.
Parameters:
| Name | Type | Description | Default |
| ------------------------------- | ----- | -------------------------------------------------------------------------- | ---------- |
| `image_folder_path` | `str` | Directory containing evaluation images. | *required* |
| `ground_truth_annotations_path` | `str` | Path to the ground truth JSON file in COCO format. | *required* |
| `max_images` | `int` | Maximum number of images to process. `0` uses all images. Defaults to `0`. | `0` |
Returns:
| Type | Description |
| ------ | ------------------------------------------- |
| `list` | Evaluation statistics (algorithm specific). |
#### init\_from\_yaml(model, ...)
`init_from_yaml(model, config_yaml)`
`classmethod`
Construct an evaluator from a YAML file.
Parameters:
| Name | Type | Description | Default |
| ------------- | ------------------------ | ---------------------------------------------------------------- | ---------- |
| `model` | `Model` | PySDK model object. | *required* |
| `config_yaml` | `Union[str, TextIOBase]` | Path or open stream with evaluator configuration in YAML format. | *required* |
Returns:
| Name | Type | Description |
| ------ | ------ | ------------------------------ |
| `Self` | `Self` | Instantiated evaluator object. |
# Math Support
DeGirum Tools API Reference Guide. Geometry, NMS, tiling and FIR filter helper functions.
{% hint style="info" %}
This API Reference is based on DeGirum Tools version 1.2.0.
{% endhint %}
## Math Support Module Overview
This module provides mathematical utilities for geometric operations and signal processing. It includes functions for bounding box manipulation, non-maximum suppression, image tiling, and a lightweight FIR low-pass filter implementation.
Key Features
* **Bounding Box Operations**: Area calculation, IoU computation, coordinate conversions
* **Non-maximum Suppression**: Multiple selection policies for detection filtering
* **Edge Box Fusion**: Handling overlapping detections with configurable thresholds
* **Image Tiling**: Utilities for fixed size or aspect ratio tiling
* **FIR Filter**: Lightweight low-pass filter implementation for signal smoothing
Typical Usage
1. Import required functions from the module
2. Use bounding box operations for detection processing
3. Apply NMS or box fusion for post-processing detections
4. Generate image tiles for processing large images
5. Use FIRFilterLP for smoothing numeric sequences
Integration Notes
* Built on NumPy for efficient array operations
* Compatible with various bounding box formats (xyxy, xywh)
* Provides consistent results across platforms
* Includes comprehensive error handling
Key Functions
* `area()`: Calculate bounding box areas
* `intersection()`: Compute intersection area of bounding boxes
* `nms()`: Apply non-maximum suppression
* `edge_box_fusion()`: Fuse overlapping edge detections
* `generate_tiles_fixed_size()`: Generate overlapping image tiles
Configuration Options
* NMS selection policies
* Box fusion thresholds
* Tile overlap parameters
* Filter coefficients
## Functions
#### area(box)
`area(box)`
Compute bounding box areas.
Parameters:
| Name | Type | Description | Default |
| ----- | --------- | ---------------------------------------------------------------------------- | ---------- |
| `box` | `ndarray` | Single box `(x1, y1, x2, y2)` or an array of such boxes with shape `(N, 4)`. | *required* |
Returns:
| Type | Description |
| --------- | ----------------------- |
| `ndarray` | Area of each input box. |
#### intersection(boxA, ...)
`intersection(boxA, boxB)`
Compute intersection area of bounding boxes.
Parameters:
| Name | Type | Description | Default |
| ------ | --------- | --------------------------------------- | ---------- |
| `boxA` | `ndarray` | First set of boxes `(x1, y1, x2, y2)`. | *required* |
| `boxB` | `ndarray` | Second set of boxes `(x1, y1, x2, y2)`. | *required* |
Returns:
| Type | Description |
| ----------------------- | ----------------------------------------- |
| `Union[float, ndarray]` | Intersection area for each pair of boxes. |
#### compute\_kmeans(embeddings, ...)
`compute_kmeans(embeddings, k=0)`
Compute k-means clustering on the given embeddings. Args: embeddings (List\[np.ndarray]): List of embedding vectors. k (int): Number of clusters. If 0, it will be deduced based on the number of embeddings. Returns: Tuple\[List\[np.ndarray], List\[int]]: Tuple containing:
* List of embeddings closest to cluster centroids
* List of their indices in the input embeddings list
#### xyxy2xywh(x)
`xyxy2xywh(x)`
Convert `(x1, y1, x2, y2)` boxes to `(x, y, w, h)` format.
#### tlbr2allcorners(x)
`tlbr2allcorners(x)`
Convert `(x1, y1, x2, y2)` boxes to all four corner points.
#### xywh2xyxy(x)
`xywh2xyxy(x)`
Convert `(x, y, w, h)` boxes to `(x1, y1, x2, y2)` format.
#### box\_iou\_batch(boxes\_true, ...)
`box_iou_batch(boxes_true, boxes_detection)`
Compute pairwise IoU between two sets of boxes.
Parameters:
| Name | Type | Description | Default |
| ----------------- | --------- | ---------------------------- | ---------- |
| `boxes_true` | `ndarray` | Ground-truth boxes `(N, 4)`. | *required* |
| `boxes_detection` | `ndarray` | Detection boxes `(M, 4)`. | *required* |
Returns:
| Type | Description |
| --------- | ----------------------------- |
| `ndarray` | IoU matrix of shape `(N, M)`. |
#### nms(detections, ...)
`nms(detections, iou_threshold=0.3, use_iou=True, box_select=NmsBoxSelectionPolicy.MOST_PROBABLE, max_wh=10000, class_agnostic=False)`
Apply non-maximum suppression to detection results.
Parameters:
| Name | Type | Description | Default |
| ---------------- | -------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | --------------- |
| `detections` | [InferenceResults](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/postprocessor#degirum.postprocessor.inferenceresults) | Iterable of detection dictionaries containing `bbox`, `score` and `class` fields. | *required* |
| `iou_threshold` | `float` | IoU/IoS threshold. Defaults to `0.3`. | `0.3` |
| `use_iou` | `bool` | If `True` use IoU, otherwise IoS. | `True` |
| `box_select` | `NmsBoxSelectionPolicy` | Box selection strategy. | `MOST_PROBABLE` |
| `max_wh` | `int` | Maximum image dimension for class separation. | `10000` |
| `class_agnostic` | `bool` | If `True` perform class-agnostic NMS. | `False` |
Returns:
| Type | Description |
| ------ | ------------------------------------------------------------ |
| `None` | `detections` is modified in place with the filtered results. |
#### edge\_box\_fusion(detections, ...)
`edge_box_fusion(detections, iou_threshold=0.55, skip_threshold=0.0, destructive=True)`
Perform box fusion on a set of edge detections. Edge detections are detections within a certain threshold of an edge.
Parameters:
| Name | Type | Description | Default |
| ---------------- | --------- | -------------------------------------------------------------------------- | ---------- |
| `detections` | `results` | A list of dictionaries that contain detection results as defined in PySDK. | *required* |
| `iou_threshold` | `float` | 1D-IoU threshold used for selecting boxes for fusion. | `0.55` |
| `skip_threshold` | `float` | Score threshold for which boxes to fuse. | `0.0` |
| `destructive` | `bool` | Keep skipped boxes (underneath the `score_threshold`) in the results. | `True` |
Returns:
| Type | Description |
| --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `results` | A list of dictionaries that contain detection results as defined in PySDK. Boxes that are not fused are kept if destructive is False, otherwise they are discarded. |
#### get\_anchor\_coordinates(xyxy, ...)
`get_anchor_coordinates(xyxy, anchor)`
Get coordinates of an anchor point inside bounding boxes.
Parameters:
| Name | Type | Description | Default |
| -------- | ------------- | ------------------------ | ---------- |
| `xyxy` | `ndarray` | Array of boxes `(N, 4)`. | *required* |
| `anchor` | `AnchorPoint` | Desired anchor location. | *required* |
Returns:
| Type | Description |
| --------- | ----------------------------------------- |
| `ndarray` | Array `(N, 2)` with `[x, y]` coordinates. |
Raises:
| Type | Description |
| ------------ | --------------------------- |
| `ValueError` | If `anchor` is unsupported. |
#### get\_image\_anchor\_point(w, ...)
`get_image_anchor_point(w, h, anchor)`
Return coordinates of an anchor point inside an image.
#### intersect(a, ...)
`intersect(a, b, c, d)`
Return `True` if two line segments intersect.
#### generate\_tiles\_fixed\_size(tile\_size, ...)
`generate_tiles_fixed_size(tile_size, image_size, min_overlap_percent)`
Generate overlapping tiles with fixed size.
Parameters:
| Name | Type | Description | Default |
| --------------------- | -------------------------- | --------------------------- | ---------- |
| `tile_size` | `Union[ndarray, Sequence]` | Tile size `(w, h)`. | *required* |
| `image_size` | `Union[ndarray, Sequence]` | Image size `(w, h)`. | *required* |
| `min_overlap_percent` | `Union[ndarray, Sequence]` | Minimum overlap in percent. | *required* |
Returns:
| Type | Description |
| --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ndarray` | Array of shape `(N, M, 4)` containing tile coordinates, where N is the number of rows in the tile grid, M is the number of columns in the tile grid, and 4 represents the coordinates `(x1, y1, x2, y2)` for each tile. |
#### generate\_tiles\_fixed\_ratio(tile\_aspect\_ratio, ...)
`generate_tiles_fixed_ratio(tile_aspect_ratio, grid_size, image_size, min_overlap_percent)`
Generate overlapping tiles with fixed aspect ratio.
Parameters:
| Name | Type | Description | Default |
| --------------------- | --------------------------------- | ------------------------ | ---------- |
| `tile_aspect_ratio` | `Union[float, ndarray, Sequence]` | Desired aspect ratio. | *required* |
| `grid_size` | `Union[ndarray, Sequence]` | Grid size `(x, y)`. | *required* |
| `image_size` | `Union[ndarray, Sequence]` | Image size `(w, h)`. | *required* |
| `min_overlap_percent` | `Union[ndarray, Sequence, float]` | Minimum overlap percent. | *required* |
Returns:
| Type | Description |
| --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ndarray` | Array of shape `(N, M, 4)` containing tile coordinates, where N is the number of rows in the tile grid, M is the number of columns in the tile grid, and 4 represents the coordinates `(x1, y1, x2, y2)` for each tile. |
## Classes
## NmsBoxSelectionPolicy
`NmsBoxSelectionPolicy`
Bases: `Enum`
Bounding box selection policy for non-maximum suppression.
This enum defines different strategies for selecting which bounding box to keep when multiple overlapping boxes are detected. Each policy has different use cases and trade-offs.
Attributes:
| Name | Type | Description |
| --------------- | ----- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| `MOST_PROBABLE` | `int` | Traditional NMS approach that keeps the box with highest confidence score. Best for high-confidence detections where false positives are costly. |
| `LARGEST_AREA` | `int` | Keeps the box with the largest area. Useful when larger detections are more likely to be correct or when object size is important. |
| `AVERAGE` | `int` | Averages the coordinates of all overlapping boxes. Good for reducing jitter in tracking applications. |
| `MERGE` | `int` | Merges all overlapping boxes into a single box. Useful when multiple detections of the same object are expected. |
## AnchorPoint
`AnchorPoint`
Bases: `Enum`
Position of a point of interest within the bounding box.
## FIRFilterLP
`FIRFilterLP`
Low-pass Finite Impulse Response (FIR) filter implementation.
This class implements a low-pass FIR filter for smoothing signals. It uses convolution with designed filter coefficients (FIR kernel) with configurable cutoff frequency and number of taps.
Attributes:
| Name | Type | Description |
| ------------------- | ------- | -------------------------------------------- |
| `normalized_cutoff` | `float` | Normalized cutoff frequency (0 to 1). |
| `taps_cnt` | `int` | Number of filter taps (order of the filter). |
| `dimension` | `int` | Number of dimensions in the input signal. |
### FIRFilterLP Methods
#### \_\_call\_\_(sample)
`__call__(sample)`
Update the filter with a new sample and return the filtered value.
This is a convenience method that calls update().
Parameters:
| Name | Type | Description | Default |
| -------- | ----------------------- | ---------------------------------------------------------------------------------------- | ---------- |
| `sample` | `Union[float, ndarray]` | New input sample. Can be a scalar or an array of length equal to the filter's dimension. | *required* |
Returns:
| Type | Description |
| --------- | -------------------------------------------------------------- |
| `ndarray` | Filtered output value with the same shape as the input sample. |
#### \_\_init\_\_(normalized\_cutoff, ...)
`__init__(normalized_cutoff, taps_cnt, dimension=1)`
Initialize the FIR filter with specified parameters.
Parameters:
| Name | Type | Description | Default |
| ------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
| `normalized_cutoff` | `float` | Normalized cutoff frequency between 0 and 1. A value of 1 corresponds to the Nyquist frequency. | *required* |
| `taps_cnt` | `int` | Number of filter taps (order of the filter). Higher values provide better frequency response but increase computational cost and delay. | *required* |
| `dimension` | `int` | Number of dimensions in the input signal. Defaults to 1 for scalar signals. | `1` |
Raises:
| Type | Description |
| ------------ | --------------------------------------------- |
| `ValueError` | If normalized\_cutoff is not between 0 and 1. |
| `ValueError` | If taps\_cnt is less than 1. |
| `ValueError` | If dimension is less than 1. |
#### get
`get()`
Get the current filtered value without updating the filter.
Returns:
| Type | Description |
| ---- | ---------------------------------------------------------- |
| | Current filtered value based on the samples in the buffer. |
#### update(sample)
`update(sample)`
Update the filter with a new sample and return the filtered value.
This method adds a new sample to the filter's buffer and computes the filtered output by convolving the input samples with the FIR filter coefficients.
Parameters:
| Name | Type | Description | Default |
| -------- | ----------------------- | ---------------------------------------------------------------------------------------- | ---------- |
| `sample` | `Union[float, ndarray]` | New input sample. Can be a scalar or an array of length equal to the filter's dimension. | *required* |
Returns:
| Type | Description |
| --------- | -------------------------------------------------------------- |
| `ndarray` | Filtered output value with the same shape as the input sample. |
Raises:
| Type | Description |
| ------------ | ----------------------------------------------------------------- |
| `ValueError` | If the input sample's shape doesn't match the filter's dimension. |
# Object Storage Support
DeGirum Tools API Reference Guide. MinIO/local object-storage wrappers for file and bucket ops.
{% hint style="info" %}
This API Reference is based on DeGirum Tools version 1.2.0.
{% endhint %}
## Object Storage Support Module Overview
Helper utilities for interacting with cloud object storage services such as MinIO. The module also provides a lightweight local file-system backend for testing without a remote service.
Key Classes
* `ObjectStorageConfig`: Configuration parameters for object storage.
* `ObjectStorage`: Convenience wrapper around common bucket operations.
Typical Usage
1. Create an `ObjectStorageConfig` with connection parameters.
2. Instantiate `ObjectStorage` using the configuration.
3. Upload, download, or delete files using the instance methods.
## Classes
## ObjectStorageConfig
`ObjectStorageConfig`
`dataclass`
Configuration for object storage connections.
Attributes:
| Name | Type | Description |
| ------------------ | ----- | ------------------------------------------ |
| `endpoint` | `str` | Object storage endpoint URL or local path. |
| `access_key` | `str` | Access key for the storage account. |
| `secret_key` | `str` | Secret key for the storage account. |
| `bucket` | `str` | Bucket name or local directory name. |
| `url_expiration_s` | `int` | Expiration time for presigned URLs. |
### ObjectStorageConfig Methods
#### construct\_direct\_url(object\_name)
`construct_direct_url(object_name)`
Construct a direct URL to an object.
Parameters:
| Name | Type | Description | Default |
| ------------- | ----- | ------------------------------------- | ---------- |
| `object_name` | `str` | Name of the object inside the bucket. | *required* |
## ObjectStorage
`ObjectStorage`
Convenience wrapper around common object storage operations.
This helper abstracts interaction with either a real MinIO server or a local directory acting as an object store. It exposes simple methods for bucket management and file uploads/downloads.
### ObjectStorage Methods
#### \_\_init\_\_(config)
`__init__(config)`
Initialize the storage helper.
Depending on `config.endpoint` this helper connects either to a real MinIO server or to a local directory used as a mock object store.
Parameters:
| Name | Type | Description | Default |
| -------- | --------------------- | ---------------------- | ---------- |
| `config` | `ObjectStorageConfig` | Storage configuration. | *required* |
#### check\_bucket\_exits(retries=1)
`check_bucket_exits(retries=1)`
Check whether the configured bucket exists.
Parameters:
| Name | Type | Description | Default |
| --------- | ----- | ------------------------- | ------- |
| `retries` | `int` | Number of retry attempts. | `1` |
Returns:
| Name | Type | Description |
| ------ | ------ | ---------------------------- |
| `bool` | `bool` | `True` if the bucket exists. |
#### check\_file\_exists\_in\_object\_storage(object\_name)
`check_file_exists_in_object_storage(object_name)`
Check whether a file exists in the configured bucket.
Parameters:
| Name | Type | Description | Default |
| ------------- | ----- | ------------------------------------- | ---------- |
| `object_name` | `str` | Name of the object within the bucket. | *required* |
#### delete\_bucket
`delete_bucket()`
Delete the bucket and all of its objects.
Raises:
| Type | Description |
| -------------- | -------------------------------- |
| `RuntimeError` | If the bucket cannot be removed. |
#### delete\_bucket\_contents
`delete_bucket_contents()`
Remove all objects from the bucket.
#### delete\_file\_from\_object\_storage(object\_name)
`delete_file_from_object_storage(object_name)`
Delete a file from the configured bucket.
Parameters:
| Name | Type | Description | Default |
| ------------- | ----- | ------------------------------------- | ---------- |
| `object_name` | `str` | Name of the object within the bucket. | *required* |
Raises:
| Type | Description |
| -------------- | ---------------------- |
| `RuntimeError` | If the deletion fails. |
#### download\_file\_from\_object\_storage(object\_name, ...)
`download_file_from_object_storage(object_name, file_path)`
Download a file from the configured bucket.
Parameters:
| Name | Type | Description | Default |
| ------------- | ----- | ---------------------------------------- | ---------- |
| `object_name` | `str` | Name of the object within the bucket. | *required* |
| `file_path` | `str` | Local path where the file will be saved. | *required* |
Raises:
| Type | Description |
| -------------- | ---------------------- |
| `RuntimeError` | If the download fails. |
#### ensure\_bucket\_exists
`ensure_bucket_exists()`
Create the bucket if it does not exist.
Raises:
| Type | Description |
| -------------- | ------------------------- |
| `RuntimeError` | If bucket creation fails. |
#### generate\_presigned\_url(object\_name)
`generate_presigned_url(object_name)`
Return a presigned download URL for an object.
Parameters:
| Name | Type | Description | Default |
| ------------- | ----- | ------------------------------------- | ---------- |
| `object_name` | `str` | Name of the object within the bucket. | *required* |
Returns:
| Type | Description |
| ----- | -------------------------------------- |
| `str` | Temporary download URL for the object. |
#### list\_bucket\_contents
`list_bucket_contents()`
List objects in the bucket.
Returns:
| Type | Description |
| ------------------ | -------------------------------------------------------------- |
| `Iterable or None` | Iterator over objects, or `None` if the bucket does not exist. |
Raises:
| Type | Description |
| -------------- | ----------------- |
| `RuntimeError` | If listing fails. |
#### upload\_file\_to\_object\_storage(file\_path, ...)
`upload_file_to_object_storage(file_path, object_name)`
Upload a file to the configured bucket.
Parameters:
| Name | Type | Description | Default |
| ------------- | ----- | ------------------------------------- | ---------- |
| `file_path` | `str` | Path of the local file to upload. | *required* |
| `object_name` | `str` | Name of the object within the bucket. | *required* |
Raises:
| Type | Description |
| -------------- | -------------------- |
| `RuntimeError` | If the upload fails. |
# UI Support
DeGirum Tools API Reference Guide. Lightweight display, FPS meter, timer and image-stack tools.
{% hint style="info" %}
This API Reference is based on DeGirum Tools version 1.2.0.
{% endhint %}
## UI Support Module Overview
This module provides a comprehensive set of utilities for user interface operations, including image display, text rendering, progress tracking, and performance monitoring. It supports both traditional GUI environments and Jupyter notebooks.
Key Features
* **Image Display**: Show images in GUI windows or Jupyter notebooks
* **Text Rendering**: Draw text with customizable fonts, colors, and positions
* **Progress Tracking**: Display progress bars with speed and percentage
* **Performance Monitoring**: Measure and display FPS
* **Environment Detection**: Auto-detect and adapt to different display environments
* **Color Utilities**: Convert between color spaces and compute complementary colors
Typical Usage
1. Use `Display` class for showing images in any environment
2. Draw text on images with `put_text()`
3. Track progress with `Progress` class
4. Monitor performance with `FPSMeter`
5. Stack images with `stack_images()`
Integration Notes
* Works in both GUI and Jupyter notebook environments
* Automatically detects and adapts to the display environment
* Supports both OpenCV and PIL image formats
* Handles video files in Jupyter notebooks
* Provides consistent interface across different platforms
Key Classes
* `Display`: Main class for showing images and videos
* `Progress`: Progress bar with speed and percentage display
* `FPSMeter`: Frames per second measurement
* `Timer`: Simple timing utility
* `stdoutRedirector`: Context manager for redirecting stdout
Configuration Options
* Font settings (face, scale, thickness)
* Color schemes (RGB/BGR)
* Progress bar appearance
* Display window properties
## Functions
#### deduce\_text\_color(bg\_color)
`deduce_text_color(bg_color)`
Return a readable text color.
Chooses black or white based on the luminance of `bg_color` so that text remains legible.
Parameters:
| Name | Type | Description | Default |
| ---------- | ------- | ----------------------------------------- | ---------- |
| `bg_color` | `tuple` | Background color as an `(R, G, B)` tuple. | *required* |
Returns:
| Type | Description |
| ---------------------- | ------------------------------------------ |
| `Tuple[int, int, int]` | `(R, G, B)` value for black or white text. |
#### color\_complement(color)
`color_complement(color)`
Return the complement of an RGB color.
Parameters:
| Name | Type | Description | Default |
| ------- | --------------- | ------------------------------- | ---------- |
| `color` | `tuple \| list` | Color specified as `(R, G, B)`. | *required* |
Returns:
| Type | Description |
| ---------------------- | ------------------------------------------ |
| `Tuple[int, int, int]` | Complementary color in `(R, G, B)` format. |
#### rgb\_to\_bgr(color)
`rgb_to_bgr(color)`
Convert an RGB color tuple to BGR.
Parameters:
| Name | Type | Description | Default |
| ------- | ------- | ---------------------------- | ---------- |
| `color` | `tuple` | Color in `(R, G, B)` format. | *required* |
Returns:
| Type | Description |
| ---------------------- | ------------------------------------------------ |
| `Tuple[int, int, int]` | Color in `(B, R, G)` order for OpenCV functions. |
#### ipython\_display(obj, ...)
`ipython_display(obj, clear=False, display_id=None)`
Display an object in IPython notebooks.
Parameters:
| Name | Type | Description | Default |
| ------------ | --------------- | ---------------------------------------------------------------------------------------------------------------------- | ---------- |
| `obj` | `Any` | Object to display. Supported types are `PIL.Image`, `numpy.ndarray` images, or a string path/URL to an image or video. | *required* |
| `clear` | `bool` | Whether to clear the previous output. Defaults to `False`. | `False` |
| `display_id` | `Optional[str]` | Custom display ID to update an existing output. | `None` |
Raises:
| Type | Description |
| ----------- | ---------------------------------- |
| `Exception` | If the object type is unsupported. |
#### put\_text(image, ...)
`put_text(image, label, corner_xy, *, corner_position=CornerPosition.TOP_LEFT, font_color, bg_color=None, font_face=cv2.FONT_HERSHEY_PLAIN, font_scale=1, font_thickness=1, line_spacing=1)`
Draw text on an image with customizable appearance and positioning.
This function draws text on an OpenCV image with support for multi-line text, background colors, and automatic positioning. The text can be placed relative to any corner of the image, and will automatically adjust to stay within image boundaries.
Parameters:
| Name | Type | Description | Default |
| ----------------- | ----------------- | ---------------------------------------------------------------------------------- | -------------------- |
| `image` | `ndarray` | Input image in OpenCV format (BGR). | *required* |
| `label` | `str` | Text to draw. Can contain newlines for multi-line text. | *required* |
| `corner_xy` | `tuple` | Base coordinates (x, y) for text placement. | *required* |
| `corner_position` | `CornerPosition` | Position of text relative to corner\_xy. Defaults to TOP\_LEFT. | `TOP_LEFT` |
| `font_color` | `tuple` | Text color in RGB format. | *required* |
| `bg_color` | `Optional[tuple]` | Background color in RGB format. If None, no background is drawn. Defaults to None. | `None` |
| `font_face` | `int` | OpenCV font face. Defaults to FONT\_HERSHEY\_PLAIN. | `FONT_HERSHEY_PLAIN` |
| `font_scale` | `float` | Font size multiplier. Defaults to 1. | `1` |
| `font_thickness` | `int` | Font thickness in pixels. Defaults to 1. | `1` |
| `line_spacing` | `float` | Multiplier for line spacing. Defaults to 1. | `1` |
Returns:
| Type | Description |
| --------- | ---------------------------- |
| `ndarray` | Image with text drawn on it. |
#### stack\_images(image1, ...)
`stack_images(image1, image2, dimension='horizontal', downscale=None, labels=None, font_color=(255, 255, 255))`
Stack two images either horizontally or vertically.
Parameters:
| Name | Type | Description | Default |
| ------------ | ------------------ | ----------------------------------------------------------- | ----------------- |
| `image1` | `ndarray \| Image` | First image. | *required* |
| `image2` | `ndarray \| Image` | Second image. | *required* |
| `dimension` | `str` | `"horizontal"` or `"vertical"`. Defaults to `"horizontal"`. | `'horizontal'` |
| `downscale` | `Optional[float]` | Scaling factor for both images if less than `1.0`. | `None` |
| `labels` | `Optional[list]` | Optional text labels for `image1` and `image2`. | `None` |
| `font_color` | `tuple` | RGB color for labels. Defaults to white. | `(255, 255, 255)` |
Returns:
| Type | Description |
| ----------------------- | ------------------------------------------------- |
| `Union[ndarray, Image]` | Combined image with optional resizing and labels. |
## Classes
## CornerPosition
`CornerPosition`
Bases: `Enum`
Enumeration of possible corner positions for text placement.
This enum defines the possible positions where text can be placed relative to a reference point in an image. The AUTO option will automatically choose the best corner based on the reference point's position.
Attributes:
| Name | Type | Description |
| -------------- | ----- | ---------------------------------------------- |
| `AUTO` | `int` | Automatically choose the best corner position. |
| `TOP_LEFT` | `int` | Place text at the top-left corner. |
| `TOP_RIGHT` | `int` | Place text at the top-right corner. |
| `BOTTOM_LEFT` | `int` | Place text at the bottom-left corner. |
| `BOTTOM_RIGHT` | `int` | Place text at the bottom-right corner. |
## FPSMeter
`FPSMeter`
Frame rate measurement utility.
This class provides functionality to measure and track frames per second (FPS) over a configurable window of time. It's useful for monitoring performance in video processing and real-time applications.
Attributes:
| Name | Type | Description |
| --------- | ----- | --------------------------------------------- |
| `avg_len` | `int` | Number of samples to use for FPS calculation. |
### FPSMeter Methods
#### \_\_init\_\_(avg\_len=100)
`__init__(avg_len=100)`
Constructor.
Parameters:
| Name | Type | Description | Default |
| --------- | ----- | -------------------------------------------------------------- | ------- |
| `avg_len` | `int` | Number of samples to use for FPS calculation. Defaults to 100. | `100` |
#### fps
`fps()`
Return current average FPS.
#### record
`record()`
Record timestamp and update average duration.
Returns current average FPS.
#### reset
`reset()`
Reset accumulators.
## Display
`Display`
Display manager for showing images and videos in various environments.
This class provides a unified interface for displaying images and videos in both GUI windows and Jupyter notebooks. It automatically detects the display environment and adapts its behavior accordingly.
Attributes:
| Name | Type | Description |
| ------------- | --------------- | ------------------------------------------------ |
| `window_name` | `str` | Name of the display window in GUI mode. |
| `show_fps` | `bool` | Whether to show FPS counter on displayed images. |
| `width` | `Optional[int]` | Target width for displayed images. |
| `height` | `Optional[int]` | Target height for displayed images. |
### Attributes
#### window\_name: str
`window_name: str`
`property`
Get the window name.
Returns:
| Type | Description |
| ----- | --------------------------- |
| `str` | Name of the display window. |
### Display Methods
#### \_\_init\_\_(capt='', ...)
`__init__(capt='', show_fps=True, w=None, h=None)`
Constructor.
Parameters:
| Name | Type | Description | Default |
| ---------- | --------------- | ---------------------------------------------------------------------- | ----------- |
| `capt` | `str` | Window title. Defaults to "". | `''` |
| `show_fps` | `bool` | Whether to show FPS counter. Defaults to True. | `True` |
| `w` | `Optional[int]` | Initial window width in pixels; None for autoscale. Defaults to None. | `None` |
| `h` | `Optional[int]` | Initial window height in pixels; None for autoscale. Defaults to None. | `None` |
Raises:
| Type | Description |
| ----------- | ------------------------- |
| `Exception` | If window title is empty. |
#### show(img, ...)
`show(img, waitkey_delay=1)`
Show image or model result.
Parameters:
| Name | Type | Description | Default |
| --------------- | ----- | ----------------------------------------------------------------------------------------------------- | ---------- |
| `img` | `Any` | Image to display. Can be a numpy array with valid OpenCV image, PIL image, or model result object. | *required* |
| `waitkey_delay` | `int` | Delay in ms for waitKey() call. Use 0 to show still images, use 1 for streaming video. Defaults to 1. | `1` |
#### show\_image(img)
`show_image(img)`
Show still image or model result.
Parameters:
| Name | Type | Description | Default |
| ----- | ----- | -------------------------------------------------------------------------------------------------- | ---------- |
| `img` | `Any` | Image to display. Can be a numpy array with valid OpenCV image, PIL image, or model result object. | *required* |
## Timer
`Timer`
Simple timer class.
### Timer Methods
#### \_\_call\_\_
`__call__()`
Get elapsed time since timer creation.
Returns:
| Type | Description |
| ------- | -------------------------------------------------- |
| `float` | Time elapsed in seconds since object construction. |
#### \_\_init\_\_
`__init__()`
Constructor. Records start time.
## Progress
`Progress`
Progress bar with speed and percentage display.
This class provides a progress bar that shows completion percentage, speed, and optional messages. It works in both GUI and Jupyter notebook environments.
Attributes:
| Name | Type | Description |
| ------------- | --------------- | ---------------------------------------------------- |
| `last_step` | `Optional[int]` | Total number of steps (None for indeterminate). |
| `start_step` | `int` | Starting step number. |
| `bar_len` | `int` | Length of the progress bar in characters. |
| `speed_units` | `str` | Units to display for speed (e.g., "FPS", "items/s"). |
### Attributes
#### step\_range: Optional\[tuple]
`step_range: Optional[tuple]`
`property`
Get start-end step range (if defined).
### Progress Methods
#### \_\_init\_\_(last\_step=None, ...)
`__init__(last_step=None, *, start_step=0, bar_len=15, speed_units='FPS')`
Constructor.
Parameters:
| Name | Type | Description | Default |
| ------------- | --------------- | ----------------------------------------------------------------------- | ------- |
| `last_step` | `Optional[int]` | Total number of steps (None for indeterminate). Defaults to None. | `None` |
| `start_step` | `int` | Starting step number. Defaults to 0. | `0` |
| `bar_len` | `int` | Progress bar length in symbols. Defaults to 15. | `15` |
| `speed_units` | `str` | Units to display for speed (e.g., "FPS", "items/s"). Defaults to "FPS". | `'FPS'` |
#### reset
`reset()`
Reset the progress bar to its initial state.
This method resets the current step, message, and timing information.
#### step(steps=1, ...)
`step(steps=1, *, message=None)`
Update progress by given number of steps.
Parameters:
| Name | Type | Description | Default |
| --------- | --------------- | ---------------------------- | ------- |
| `steps` | `int` | Number of steps to advance. | `1` |
| `message` | `Optional[str]` | Optional message to display. | `None` |
## stdoutRedirector
`stdoutRedirector`
Redirect stdout to another stream.
### stdoutRedirector Methods
#### \_\_init\_\_(stream=None)
`__init__(stream=None)`
Constructor.
Parameters:
| Name | Type | Description | Default |
| -------- | --------------- | -------------------------------------------------------------------------------- | ------- |
| `stream` | `Optional[str]` | Output stream to redirect to; None to redirect to null device. Defaults to None. | `None` |
# Video Support
DeGirum Tools API Reference Guide. Read, stream, display and save video or RTSP sources.
{% hint style="info" %}
This API Reference is based on DeGirum Tools version 1.2.0.
{% endhint %}
## Video Support Module Overview
This module provides comprehensive video stream handling capabilities, including capturing from various sources, saving to files, and managing video clips. It supports local cameras, IP cameras, video files, and YouTube videos. Key Features:
* **Multi-Source Support**: Capture from local cameras, IP cameras, video files, and YouTube
* **Video Writing**: Save video streams with configurable quality and format
* **Frame Extraction**: Convert video files to JPEG sequences
* **Clip Management**: Save video clips triggered by events with pre/post buffers
* **FPS Control**: Frame rate management for both capture and writing
* **Stream Properties**: Query video stream dimensions and frame rate Typical Usage:
1. Open video streams with `open_video_stream()`
2. Process frames using `video_source()` generator
3. Save videos with `VideoWriter` or `open_video_writer()`
4. Extract frames using `video2jpegs()`
5. Save event-triggered clips with `ClipSaver` Integration Notes:
* Works with OpenCV's VideoCapture and VideoWriter
* Supports YouTube videos through pafy
* Handles both real-time and file-based video sources
* Provides context managers for safe resource handling
* Thread-safe for concurrent video operations Key Classes:
* `VideoWriter`: Main class for saving video streams
* `ClipSaver`: Manages saving video clips with pre/post buffers Configuration Options:
* Video quality and format settings
* Frame rate control
* Clip duration and buffer size
* Output file naming and paths
## Functions
#### create\_video\_stream(video\_source=None, ...)
`create_video_stream(video_source=None, *, max_yt_quality=0, use_gstreamer=False)`
Create a video stream from various sources.
This function creates and returns video stream object working from different sources, including local cameras, IP cameras, video files, and YouTube videos.
Parameters:
| Name | Type | Description | Default |
| ---------------- | ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------- |
| `video_source` | `Union[int, str, Path, None, VideoCapture, VideoCaptureGst]` | Video source specification: - int: 0-based index for local cameras - str: IP camera URL (rtsp\://user:password\@hostname) - str: Local video file path - str: URL to mp4 video file - str: YouTube video URL - None: Use environment variable or default camera - cv2.VideoCapture or VideoCaptureGst: Pass through existing capture | `None` |
| `max_yt_quality` | `int` | Maximum video quality for YouTube videos in pixels (height). If 0, use best quality. Defaults to 0. | `0` |
| `use_gstreamer` | `bool` | If True, use GStreamer backend for video files. Only applies to .mp4 files. Defaults to False. | `False` |
Returns:
| Type | Description |
| -------------------------------------- | ---------------------------------------------------------- |
| `Union[VideoCapture, VideoCaptureGst]` | cv2.VideoCapture or VideoCaptureGst: Video capture object. |
Raises:
| Type | Description |
| ----------- | ------------------------------------- |
| `Exception` | If the video stream cannot be opened. |
#### detect\_rtsp\_cameras(subnet\_cidr, ...)
`detect_rtsp_cameras(subnet_cidr, *, timeout_s=0.5, port=554, max_workers=16)`
Scan given subnet for RTSP cameras by probing given port with OPTIONS request. Args: subnet\_cidr (str): Subnet in CIDR notation (e.g., '192.168.0.0/24'). timeout\_s (float): Timeout for each connection attempt in seconds. port (int): Port to probe for RTSP cameras (default is 554). max\_workers (int): Maximum number of concurrent threads for scanning (default is 16). Returns: dict: Dictionary with IP addresses as keys and properties as values. Properties include 'require\_auth' indicating if authentication is required.
#### open\_video\_stream(video\_source=None, ...)
`open_video_stream(video_source=None, *, max_yt_quality=0, use_gstreamer=False)`
Open a video stream from various sources.
This function provides a context manager for opening video streams from different sources. The stream is automatically closed when the context is exited. Internally it calls `create_video_stream` to create the stream.
Parameters:
| Name | Type | Description | Default |
| ---------------- | ------------------------------------------------------------ | ------------------------------------------------------ | ------- |
| `video_source` | `Union[int, str, Path, None, VideoCapture, VideoCaptureGst]` | Video source specification (see create\_video\_stream) | `None` |
| `max_yt_quality` | `int` | Maximum video quality for YouTube videos | `0` |
| `use_gstreamer` | `bool` | If True, use GStreamer backend for video files | `False` |
Yields:
| Type | Description |
| -------------------------------------- | ---------------------------------------------------------- |
| `Union[VideoCapture, VideoCaptureGst]` | cv2.VideoCapture or VideoCaptureGst: Video capture object. |
Raises:
| Type | Description |
| ----------- | ------------------------------------- |
| `Exception` | If the video stream cannot be opened. |
#### get\_video\_stream\_properties(video\_source)
`get_video_stream_properties(video_source)`
Return the dimensions and frame rate of a video source.
Parameters:
| Name | Type | Description | Default |
| -------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | ---------- |
| `video_source` | `Union[int, str, Path, None, VideoCapture, VideoCaptureGst]` | Video source identifier or an already opened capture object. | *required* |
Returns:
| Type | Description |
| ------- | ------------------------------------------------- |
| `tuple` | (width, height, fps) describing the video stream. |
#### video\_source(stream, ...)
`video_source(stream, fps=None, include_metadata=False)`
Yield frames from a video stream.
Parameters:
| Name | Type | Description | Default |
| ------------------ | -------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
| `stream` | `Union[VideoCapture, VideoCaptureGst]` | Open video stream (cv2.VideoCapture or VideoCaptureGst). | *required* |
| `fps` | `Optional[float]` | Target frame rate cap. | `None` |
| `include_metadata` | `bool` | If True, yields (frame, metadata) tuples where metadata contains timestamp, frame\_id, fps, frame dimensions. If False, yields only frames. Defaults to False. | `False` |
Yields:
| Type | Description |
| -------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Union[ndarray, Tuple[ndarray, dict]]` | If include\_metadata is False: Frames from the stream (np.ndarray). |
| `Union[ndarray, Tuple[ndarray, dict]]` | If include\_metadata is True: Tuples of (frame, metadata) where metadata is a dict containing 'timestamp', 'frame\_id', 'fps', 'frame\_width', 'frame\_height'. |
#### create\_video\_writer(fname, ...)
`create_video_writer(fname, w=0, h=0, fps=30.0)`
#### open\_video\_writer(fname, ...)
`open_video_writer(fname, w=0, h=0, fps=30.0)`
#### video2jpegs(video\_file, ...)
`video2jpegs(video_file, jpeg_path, *, jpeg_prefix='frame_', preprocessor=None)`
Convert a video file into a sequence of JPEG images.
Parameters:
| Name | Type | Description | Default |
| -------------- | ------------------------------ | ------------------------------------------------------------- | ---------- |
| `video_file` | `str` | Path to the input video file. | *required* |
| `jpeg_path` | `str` | Directory where JPEG files will be stored. | *required* |
| `jpeg_prefix` | `str` | Prefix for generated image filenames. Defaults to `"frame_"`. | `'frame_'` |
| `preprocessor` | `Callable[[ndarray], ndarray]` | Optional function applied to each frame before saving. | `None` |
Returns:
| Name | Type | Description |
| ----- | ----- | ---------------------------------------- |
| `int` | `int` | Number of frames written to `jpeg_path`. |
## Classes
## VideoWriter
`VideoWriter`
Video stream writer with configurable quality and format.
## ClipSaver
`ClipSaver`
Video clip saver with pre/post trigger buffering.
This class provides functionality to save video clips triggered by events, with configurable pre-trigger and post-trigger buffers. It maintains a circular buffer of frames and saves clips when triggers occur.
This class is primarily used by two other components in DeGirum Tools.
1. ClipSavingAnalyzer wraps ClipSaver and triggers clips from event names found in EventNotifier or EventDetector results.
2. EventNotifier can instantiate and use ClipSaver to record clips when a notification fires, optionally uploading those clips through NotificationServer.
Attributes:
| Name | Type | Description |
| ---------------------- | ------- | ------------------------------------------- |
| `clip_duration` | `int` | Total length of output clips in frames. |
| `file_prefix` | `str` | Base path for saved clip files. |
| `pre_trigger_delay` | `int` | Frames to include before trigger. |
| `embed_ai_annotations` | `bool` | Whether to include AI annotations in clips. |
| `save_ai_result_json` | `bool` | Whether to save AI results as JSON. |
| `target_fps` | `float` | Frame rate for saved clips. |
### ClipSaver Methods
#### \_\_init\_\_(clip\_duration, ...)
`__init__(clip_duration, file_prefix, *, pre_trigger_delay=0, embed_ai_annotations=True, save_ai_result_json=True, target_fps=30.0)`
Initialize the clip saver.
Parameters:
| Name | Type | Description | Default |
| ---------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
| `clip_duration` | `int` | Total length of output clips in frames (pre-buffer + post-buffer). | *required* |
| `file_prefix` | `str` | Base path for saved clip files. Frame number and extension are appended automatically. | *required* |
| `pre_trigger_delay` | `int` | Frames to include before trigger. Defaults to 0. | `0` |
| `embed_ai_annotations` | `bool` | If True, use [InferenceResults](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/postprocessor#degirum.postprocessor.inferenceresults).image\_overlay to include bounding boxes/labels in the clip. Defaults to True. | `True` |
| `save_ai_result_json` | `bool` | If True, save a JSON file with raw inference results alongside the video. Defaults to True. | `True` |
| `target_fps` | `float` | Frame rate for saved clips. Defaults to 30.0. | `30.0` |
Raises:
| Type | Description |
| ------------ | ------------------------------------------------------------- |
| `ValueError` | If clip\_duration is not positive. |
| `ValueError` | If pre\_trigger\_delay is negative or exceeds clip\_duration. |
#### forward(result, ...)
`forward(result, triggers=[])`
Process a frame and save clips if triggers occur.
This method adds the current frame to the buffer and saves clips if any triggers are present. The saved clips include pre-trigger frames from the buffer.
Parameters:
| Name | Type | Description | Default |
| ---------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
| `result` | `Any` | [InferenceResults](https://docs.degirum.com/pysdk/user-guide-pysdk/api-ref/postprocessor#degirum.postprocessor.inferenceresults) object containing the current frame and detection results. | *required* |
| `triggers` | `List[str]` | List of trigger names that occurred in this frame. Defaults to \[]. | `[]` |
Returns:
| Type | Description |
| ------------------------ | -------------------------------------------------------------- |
| `Tuple[List[str], bool]` | List of saved clip filenames and whether any clips were saved. |
Raises:
| Type | Description |
| ----------- | ----------------------------- |
| `Exception` | If the frame cannot be saved. |
#### join\_all\_saver\_threads
`join_all_saver_threads()`
Wait for all clip saving threads to complete.
This method blocks until all background clip saving threads have finished. It's useful to call this before exiting to ensure all clips are properly saved.
Returns:
| Type | Description |
| ----- | ----------------------------------- |
| `int` | Number of threads that were joined. |
## MediaServer
`MediaServer`
Manages the MediaMTX media server as a subprocess.
Starts MediaMTX using a provided config file path. If no config path is given, it runs from the MediaMTX binary's directory.
MediaMTX binary must be installed and available in the system path. Refer to for installation instructions.
### MediaServer Methods
#### \_\_del\_\_
`__del__()`
Destructor to ensure the media server is stopped.
#### \_\_enter\_\_
`__enter__()`
Enables use with context manager.
#### \_\_exit\_\_(exc\_type, ...)
`__exit__(exc_type, exc_val, exc_tb)`
Stops server when context exits.
#### \_\_init\_\_(\*, ...)
`__init__(*, config_path=None, verbose=False)`
Initializes and starts the server.
Parameters:
| Name | Type | Description | Default |
| ------------- | --------------- | ------------------------------------------------------------------------------------------------------------ | ------- |
| `config_path` | `Optional[str]` | Path to an existing MediaMTX YAML config file. If not provided, runs with config file from binary directory. | `None` |
| `verbose` | `bool` | If True, shows media server output in the console. | `False` |
#### stop
`stop()`
Stops the media server process.
## VideoStreamer
`VideoStreamer`
Streams video frames to an RTMP or RTSP server using FFmpeg. This class uses FFmpeg to stream video frames to an RTSP server. FFmpeg must be installed and available in the system path.
### VideoStreamer Methods
#### \_\_del\_\_
`__del__()`
Destructor to ensure the streamer is stopped.
#### \_\_enter\_\_
`__enter__()`
Enables use with context manager.
#### \_\_exit\_\_(exc\_type, ...)
`__exit__(exc_type, exc_value, traceback)`
Stops streamer when context exits.
#### \_\_init\_\_(stream\_url, ...)
`__init__(stream_url, width, height, *, fps=30.0, pix_fmt='bgr24', gop_size=10, verbose=False)`
Initializes the video streamer.
Parameters:
| Name | Type | Description | Default |
| ------------ | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- |
| `stream_url` | `str` | RTMP/RTSP URL to stream to (e.g., 'rtsp\://user:password\@hostname:port/stream'). Typically you use `MediaServer` class to start media server and then use its RTMP/RTSP URL like `rtsp://localhost:8554/mystream` | *required* |
| `width` | `int` | Width of the video frames in pixels. | *required* |
| `height` | `int` | Height of the video frames in pixels. | *required* |
| `fps` | `float` | Frames per second for the stream. Defaults to 30. | `30.0` |
| `pix_fmt` | `str` | Pixel format for the input frames. Defaults to 'bgr24'. Can be 'rgb24'. | `'bgr24'` |
| `gop_size` | `int` | GOP size for the video stream. Defaults to 50. | `10` |
| `verbose` | `bool` | If True, shows FFmpeg output in the console. Defaults to False. | `False` |
#### stop
`stop()`
Stops the streamer process.
#### write(img)
`write(img)`
Writes a frame to the RTSP stream. Args: img (ImageType): Frame to write. Can be:
* OpenCV image (np.ndarray)
* PIL Image
{% code overflow="wrap" %}
```
Pixel format must match the one specified in the constructor (default is 'bgr24').
```
{% endcode %}
# Environment Variables
A reference for constants and environment variables used by DeGirum Tools.
{% hint style="info" %}
This reference is based on DeGirum Tools version 0.24.1.
{% endhint %}
Environment variables supply defaults to scripts and applications built with DeGirum Tools. Storing environment variables in an `.env` or `env.ini` file automatically supplies default values.
### AI Hub, AI Server, and Model Zoos
| Variable | Purpose | Example values |
| ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ |
| `AISERVER_HOSTNAME_OR_IP` | Address of an AI Server on your LAN for local inference. Used when `@local` or AI Server backends are selected. | `localhost`, `192.168.0.101` |
| `CLOUD_ZOO_URL` | Path to a model zoo on AI Hub or your own storage. Defaults to `degirum/public` if unset. | `degirum/degirum`, `degirum/public` |
| `DEGIRUM_CLOUD_TOKEN` | API token for accessing AI Hub model zoos. Required when connecting with the `@cloud` backend. If not set, PySDK uses the token installed with the [CLI token command](https://docs.degirum.com/pysdk/user-guide-pysdk/command-line-interface#manage-ai-hub-tokens) (install subcommand). Set this variable to override the stored token. | Access token generated on the AI Hub. |
| `MODEL_ZOO_URL` | Path or URL to a model zoo. Overrides the default zoo location if set. | `/path/to/local/zoo`, |
### Input Devices
| Variable | Purpose | Example values |
| ----------- | ---------------------------------------------------- | ------------------------- |
| `AUDIO_ID` | Microphone index or WAV file path. | `0`, `input.wav` |
| `CAMERA_ID` | Camera index or video path to capture video streams. | `0`, `rtsp://host/stream` |
### S3 Object Storage
| Variable | Purpose | Example values |
| --------------- | ----------------------------------------------------------- | ------------------ |
| `S3_ACCESS_KEY` | Access key for MinIO or other S3 compatible object storage. | `minio-access-key` |
| `S3_SECRET_KEY` | Secret key for object storage services. | `minio-secret-key` |
# Remote Assets
Remote media assets for examples and tutorials. Thin catalog that exposes image/video samples from DeGirum's PySDK Examples as simple attributes and lists.
{% hint style="info" %}
This API Reference is based on DeGirum Tools version 0.24.1.
{% endhint %}
## Overview
`degirum_tools.remote_assets` is a tiny convenience module that discovers common image files (JPG, JPEG, PNG, BMP, GIF) and MP4/AVI/MOV videos published in the [PySDK Examples](https://github.com/DeGirum/PySDKExamples) repository and exposes them in two ways:
* As dynamic attributes on the module: `remote_assets.` returns a `RemoteMedia` object (a `str` subclass with helpers) that behaves like a URL string.
* As enumerations: `list_images()` and `list_videos()` return dictionaries mapping attribute names to `RemoteMedia` objects; use their keys to list available images/videos.
Each `RemoteMedia` instance is a URL-like string; `remote_assets` does not download or cache assets by itself. Any caching is handled by your application, HTTP stack, or PySDK internals.
## When to Use
* Quickstarts, demos, tests, and tutorials that need a stable sample image or video.
* Prototyping code where you want to avoid bundling media assets in your repo.
## Basic Usage
List available image and video asset names.
{% code overflow="wrap" %}
```python
from degirum_tools import remote_assets
print("Images (names):")
print(sorted(list(remote_assets.list_images().keys())))
print("Videos (names):")
print(sorted(list(remote_assets.list_videos().keys())))
```
{% endcode %}
Example output:
{% code overflow="wrap" %}
```
Images (names):
['bikes', 'car', 'cat', 'fire_place', 'license_plate', 'living_room', 'mask1', 'parking_lot', 'three_persons', 'two_cats']
Videos (names):
['cars_lp', 'example_video', 'faces_and_gender', 'hand_palm', 'parking', 'person_face_hand', 'person_pose', 'store', 'store_short', 'traffic', 'traffic2', 'traffic_hd', 'walking_people', 'walking_people2', 'walking_person']
```
{% endcode %}
Fetch URLs for commonly used assets (a cat image and a walking‑people video).
{% code overflow="wrap" %}
```python
from degirum_tools import remote_assets
# Access via attributes (preferred for readability)
cat_url = remote_assets.cat
walking_people_url = remote_assets.walking_people
print("cat:", cat_url)
print("walking_people:", walking_people_url)
```
{% endcode %}
Example output:
{% code overflow="wrap" %}
```
cat: https://raw.githubusercontent.com/DeGirum/PySDKExamples/main/images/Cat.jpg
walking_people: https://raw.githubusercontent.com/DeGirum/PySDKExamples/main/images/WalkingPeople.mp4
```
{% endcode %}
You can pass these URLs directly to PySDK models.
## Functions
`remote_assets.list_images() -> Dict[str, RemoteMedia]`
* Returns a mapping of attribute names (filename stems) to `RemoteMedia` objects for available JPG images.
`remote_assets.list_videos() -> Dict[str, RemoteMedia]`
* Returns a mapping of attribute names (filename stems) to `RemoteMedia` objects for available MP4 videos.
## Module Attributes
`remote_assets. -> RemoteMedia`
* Dynamic attribute corresponding to an asset filename without extension.
* Returns a `RemoteMedia` (subclass of `str`) pointing to a stable HTTPS URL for the asset in PySDK Examples.
## RemoteMedia
`RemoteMedia` is a lightweight `str` subclass that adds simple media-type helpers while remaining usable anywhere a URL string is expected.
Attributes:
* `kind`: media kind as a string, one of `"image"`, `"video"`, or `"other"`.
* `is_image`: `True` if the asset is an image.
* `is_video`: `True` if the asset is a video.
Example:
{% code overflow="wrap" %}
```python
from degirum_tools import remote_assets
cat = remote_assets.cat # RemoteMedia (string URL with helpers)
print(cat) # prints the URL
print(cat.kind) # "image"
print(cat.is_image) # True
print(cat.is_video) # False
vid = remote_assets.walking_people
print(vid.kind) # "video"
```
{% endcode %}
Example output:
{% code overflow="wrap" %}
```
https://raw.githubusercontent.com/DeGirum/PySDKExamples/main/images/Cat.jpg
image
True
False
video
```
{% endcode %}
# Release Notes
This page features release notes for releases of degirum-tools. You may download degirum-tools versions listed here from PyPI.org.
### Version 1.4.0 (5/15/2026)
**New Features and Modifications**
1. New `gst` subpackage providing a complete GStreamer-based video pipeline toolkit:
* `setup_gst_environment(*plugin_dirs)` — initializes GStreamer and optionally registers custom plugin directories. Must be called before any gi import.
* `GstPipelineHandler` — manages the full lifecycle of a GStreamer pipeline, with optional named appsink queues for Python frame consumption and throughput probing.
* `GstElementBase` — mixin base class for implementing custom GStreamer Python elements with minimal boilerplate. Pad templates, worker threads, and state management are handled automatically.
* `GstAiElement` — ready-to-use GStreamer filter element that accepts BGR/RGB video frames, runs AI inference via a DeGirum model, and pushes annotated frames and/or serialized JSON inference results downstream. Supports an optional secondary full-resolution input pad (sink\_full) for high-quality overlay rendering at a different resolution than the model input.
* `map_gst_buffer(buf_or_sample, readonly)` — context manager for safe `Gst.Buffer` memory mapping.
* `build_gst_pipeline(source)` — builds a gst-launch-style pipeline string for camera devices (including Windows via mfvideosrc/ksvideosrc), RTSP streams, and video files.
* `VideoCaptureGst` — `cv2.VideoCapture`-compatible class backed by a GStreamer pipeline.
2. `ObjectTracker` analyzer now extends object trails with predicted positions during tracking timeouts, using the tracker's internal motion estimate for lost tracks. This preserves trail continuity when objects temporarily leave the frame.
3. `VideoSourceGizmo` now accepts a pre-created `VideoCaptureProtocol` object (e.g., `cv2.VideoCapture` or `VideoCaptureGst`) as the video\_source argument. Additional keyword arguments `(**kwargs)` are forwarded to the `cv2.VideoCapture` constructor when opening from a path or device index.
4. `VideoStreamerGizmo` and `VideoStreamer` gain a `vcodec` parameter to configure the video codec (defaults to `libx264`).
5. `Stream.close()` gains a `force=True` option that atomically clears the queue and inserts the poison pill without blocking, preventing deadlocks when a producer has filled the queue.
**Bug Fixes**
1. `VideoStreamerGizmo` frame scheduling: duplicate frames were injected using an incorrect rate estimate that could cause a spin-loop. The logic is now based on an absolute deadline (next\_frame\_due\_s) and stops duplicating when the encoder write time exceeds the target frame interval.
2. `VideoStreamer`: `pix_fmt` was always passed to FFmpeg as bgr24 regardless of the configured pixel format. It now correctly uses the `pix_fmt` constructor argument.
3. `video_source()`: `CAP_PROP_FRAME_COUNT` can return `None` for certain capture backends (e.g., `VideoCaptureGst`), causing a `TypeError`. The comparison now handles `None` correctly.
4. `MediaServer` subprocess is now started with `start_new_session=True`, preventing signal propagation from the parent process to the media server process.
***
### Version 1.3.1 (4/21/2026)
**Bug Fixes**
`VideoStreamer` class which uses ffmpeg for video streaming, now uses TCP protocol for RTSP streaming. Previous versions used UDP protocol, which had lower latency, but may result in corrupted video streams on certain systems.
***
### Version 1.3.0 (4/3/2026)
**New Features and Modifications**
1. Performance of `ObjectTracker` analyzer is greatly improved.
2. `frame_size` parameter is added to `ZoneCounter` analyzer constructor. It specifies the frame size (width, height) when inference results do not have `.image` attribute.
3. Added new `ipc` subpackage providing transparent inter-process communication: use `ipc.spawn(MyClass, ...)` to run any class instance in an isolated child process with all public methods automatically available as ZeroMQ RPC endpoints, with MsgPack serialization and InOut/Out wrappers for mutable argument writeback.
***
### Version 1.2.2 (3/17/2026)
**Bug Fixes**
1. EventDetector analyzer raises exception when `result` argument of `analyze` method has `events_detected` attribute of `dict` type.
2. EventDetector analyzer incorrectly generates events when the event history is not fully accummulated yet.
***
### Version 1.2.1 (3/13/2026)
**New Features and Modifications**
1. `ResultAnalyzerBase` base class `analyze` and `annotate` methods are modified to ignore `None` results. This allows using analyzers with models in `non_blocking_batch_predict` mode.
**Bug Fixes**
1. `NoneType object has no attribute encoding` exception is raised during `import degirum_tools` if the process has no stdout attached.
***
### Version 1.2.0 (2/23/2026)
**New Features and Modifications**
1. `SceneCutDetector` analyzer is implemented.
`SceneCutDetector` analyzer detects scene cuts in video streams by comparing frame-to-frame differences using an adaptive thresholding approach.
Key Features:
* **Adaptive Thresholding**: Uses rolling average of previous frames to adapt to gradual changes
* **HSV Color Space**: Analyzes differences in hue, saturation, and luminance channels
* **Configurable Parameters**: Adjustable sensitivity, minimum scene length, and window size
* **Real-time Detection**: Causal approach using only past frames for zero latency
* **Scene Cut Flag**: Adds `scene_cut` boolean attribute to inference results
Typical Usage:
1. Create a `SceneCutDetector` instance with desired parameters
2. Attach it to a model or inference pipeline
3. Process video frames through the analyzer
4. Check `result.scene_cut` flag to detect scene transitions
5. Use scene cut information for downstream processing or triggering actions
Put it before `ObjectTracker` in the analyzer pipeline to ensure cuts are detected before tracking is applied, allowing you to reset object tracker on scene changes.
Integration Notes:
* Works with any inference results that contain image data
* Can be combined with other analyzers in a pipeline
* Useful for video segmentation, activity detection, and content analysis
* Maintains internal state to track frame history
Configuration Options:
* `adaptive_threshold`: Sensitivity ratio for detecting cuts (higher = less sensitive)
* `min_scene_len`: Minimum frames between detected cuts to avoid false positives
* `window_width`: Number of previous frames for rolling average calculation
* `min_content_val`: Minimum absolute change threshold for scene cuts
* `luma_only`: Use only brightness changes for faster processing
2. `reset_at_scene_cut` parameter is added to the `ObjectTracker` analyzer. When `True`, all tracks are cleared when a scene cut is detected. Requires the result to have a `scene_cut` attribute (set by `SceneCutDetector`, see above). Use this feature to avoid tracking objects across scene transitions in videos with cuts or edits.
***
### Version 1.1.0 (2/11/2026)
**New Features and Modifications**
1. `IteratorSourceGizmo`: `fps` constructor parameter is added. This is optional parameter with default value 0.0. It specifies the FPS value to be included in the metadata.
**Bug Fixes**
1. `VideoStreamerGizmo`: check for zero FPS is added to prevent division by zero in case of zero-FPS sources.
***
### Version 1.0.0 (1/27/2026)
First official release of degirum-tools.
# Overview
Overview of the DeGirumJS SDK and its key capabilities.
## What is DeGirumJS?
> **DeGirumJS** is a powerful JavaScript library designed for building and integrating AI applications directly in your browser. This SDK allows you to perform AI inference using models hosted on a local DeGirum AI Server or in the DeGirum Cloud, directly from your web browser.
***
## Quickstart
Use DeGirumJS with a single line of code. No installation is required: include the following script tag in your HTML page:
{% code overflow="wrap" %}
```html
```
{% endcode %}
## Key Features
* **Pure Vanilla JavaScript**: Written entirely in JavaScript with no external dependencies.
* **Simple Integration**: Seamlessly connect to local AI servers or the cloud for inference.
* **Real-Time Inference**: Perform AI tasks using local models or by using DeGirum's vast cloud library of models in real-time.
## Ready to dive in?
Start by exploring the [**Getting Started Guide**](/degirumjs/get-started) or browse the [**API Reference**](https://docs.degirum.com/degirumjs/0.1.5/api/index.html) for more detailed usage and options.
Or, browse the advanced topics:
* [Model Parameters](/degirumjs/guides/model-parameters)
* [Connection Modes](/degirumjs/guides/connection-modes)
* [Real-Time Batch Inference](/degirumjs/guides/batch-inference)
* [Performance & Timing Statistics](/degirumjs/guides/timing)
* [Customizing Pre-processing and Visual Overlays](/degirumjs/guides/pre-post-processing)
* [Working with Input and Output Data](/degirumjs/guides/input-output-data)
* [Device Management for Inference](/degirumjs/guides/device-management)
* [Result Object Structure + Examples](/degirumjs/guides/result-object-structure)
* [WebCodecs Example](/degirumjs/guides/web-codecs-example)
* [Release Notes](/degirumjs/all-release-notes)
Start building your browser-based AI projects with DeGirumJS.
# Getting Started
Step-by-step guide for running your first inference with DeGirumJS.
Welcome to DeGirumJS, a JavaScript AI inference SDK. This guide helps you integrate AI capabilities into your web application.
## Introduction
DeGirumJS allows you to connect to AI Server or Cloud Zoo instances, load AI models, and perform inference on various data types. This guide provides a step-by-step tutorial on how to get started.
## Core Concepts
There are 3 main objects that you will work with in DeGirumJS:
* **dg\_sdk**: The main entry point to the library.
* **zoo**: Your connection to a model repository (either local or cloud). You use this to find and load models.
* **model**: The loaded model instance that you use to run predictions.
## Setup
### Import the SDK
To start using the SDK, include the following script tag in your HTML file:
{% code overflow="wrap" %}
```html
```
{% endcode %}
## A 5-Step Guide to Your First Prediction
DeGirumJS allows you to load models from an AI server or Cloud Zoo and perform inference on the AI Server hardware or in the cloud.
{% hint style="info" %}
For local or LAN inference, run the AI Server with HTTP enabled:
{% code overflow="wrap" %}
```bash
degirum server --protocol both
```
{% endcode %}
[Click here for AI server documentation.](https://docs.degirum.com/pysdk/user-guide-pysdk/setting-up-an-ai-server)
{% endhint %}
For running cloud inference or to be able to load a model from the cloud, you need to specify your cloud token.
[Where do I get my cloud token?](https://docs.degirum.com/ai-hub/workspaces/workspace-tokens)
{% stepper %}
{% step %}
**Connect to an Inference Provider**
**Connect to an AI Server**
Instantiate the `dg_sdk` class and connect to the AI server using the `connect` method. Provide the server's IP address and port.
{% code overflow="wrap" %}
```javascript
let dg = new dg_sdk();
const AISERVER_IP = 'localhost:8779';
let zoo = await dg.connect(AISERVER_IP);
```
{% endcode %}
Have an AIServer running but want to use cloud models? For running AI Server inference on *cloud models*, include the URL of the cloud zoo and your token:
{% code overflow="wrap" %}
```javascript
let dg = new dg_sdk();
const AISERVER_IP = 'localhost:8779';
const ZOO_URL = 'https://cs.degirum.com/degirum/public';
const secretToken = prompt('Enter secret token:');
let zoo = await dg.connect(AISERVER_IP, ZOO_URL, secretToken);
```
{% endcode %}
**Connect to the Cloud**
For running Cloud inference, specify 'cloud' as the first argument, and include the URL of the cloud zoo and your token:
{% code overflow="wrap" %}
```javascript
let dg = new dg_sdk();
const ZOO_URL = 'https://cs.degirum.com/degirum/public';
const secretToken = prompt('Enter secret token:');
let zoo = await dg.connect('cloud', ZOO_URL, secretToken);
```
{% endcode %}
{% endstep %}
{% step %}
**Load a Model**
Now, you can load a model using the zoo class instance's `loadModel` method:
{% code overflow="wrap" %}
```javascript
const MODEL_NAME = 'yolo_v5s_coco--512x512_quant_n2x_cpu_1';
const modelOptions = {
overlayShowProbabilities: true
// Any other custom options for your Model (see Model Options documentation)
};
let model = await zoo.loadModel(MODEL_NAME, modelOptions);
```
{% endcode %}
You can use `zoo.listModels()` as a way to discover models available for inference on the selected inference provider.
{% endstep %}
{% step %}
**Run Inference**
Use the `predict` method to perform inference on an input image. The input for `predict` is flexible and supports a variety of types, including Blob, File, base64 string, HTMLImageElement, HTMLVideoElement, HTMLCanvasElement, ArrayBuffer, TypedArray, ImageBitmap, URL to an image (full list of supported input types can be found in the Working with Input and Output Data documentation).
{% code overflow="wrap" %}
```javascript
const image = ''; // Some input image
const result = await model.predict(image);
console.log('Result:', result);
```
{% endcode %}
{% endstep %}
{% step %}
**Understand the Output**
The result object contains the results from the model and the original imageFrame. For more details, see the [Result Object Structure](/degirumjs/guides/result-object-structure) documentation.
{% endstep %}
{% step %}
**Visualize the Results**
You can display prediction results to a `HTMLCanvasElement` or `OffscreenCanvas`:
{% code overflow="wrap" %}
```javascript
// Assuming your Canvas Element has the id 'outputCanvas'
let canvas = document.getElementById('outputCanvas');
model.displayResultToCanvas(result, canvas);
```
{% endcode %}
This will draw the inference results onto the canvas.
{% endstep %}
{% endstepper %}
## Putting It All Together: A Complete Example
To get started with a simple example page, we need the following HTML elements on the page:
* The script tag to import DeGirumJS
* A canvas element to display inference results
* An input element to browse and upload images.
Here is a HTML page that will perform inference on uploaded images and display the results:
{% code overflow="wrap" %}
```html
```
{% endcode %}
## Cleaning Up
To clean up a model instance to release resources when the model is no longer needed, use the `cleanup` method:
{% code overflow="wrap" %}
```javascript
await model.cleanup();
```
{% endcode %}
This will stop all running inferences and clean up resources used by the model instance.
## Where to Go Next
Continue exploring the following topics to learn more about DeGirumJS.
* [Model Parameters](/degirumjs/guides/model-parameters)
* [Connection Modes](/degirumjs/guides/connection-modes)
* [Real-Time Batch Inference](/degirumjs/guides/batch-inference)
* [Performance & Timing Statistics](/degirumjs/guides/timing)
* [Customizing Pre-processing and Visual Overlays](/degirumjs/guides/pre-post-processing)
* [Working with Input and Output Data](/degirumjs/guides/input-output-data)
* [Device Management for Inference](/degirumjs/guides/device-management)
* [Result Object Structure + Examples](/degirumjs/guides/result-object-structure)
* [WebCodecs Example](/degirumjs/guides/web-codecs-example)
* [Release Notes](/degirumjs/all-release-notes)
## API Reference
For detailed information on the SDK's classes, methods, and properties, refer to the [API Reference](https://assets.degirum.com/degirumjs/0.1.5/api/index.html).
# Guides
Core usage guides for connection modes, model parameters, and data handling in DeGirumJS.
# Architecture and Connection Modes
DeGirumJS offers flexible connection modes to cater to various AI inference needs, whether you're running models locally on an AI Server, entirely in the cloud, or a hybrid approach.
## Connection Modes
DeGirumJS offers three easy ways to connect for AI inference:
### Local Server
Inference runs on your own AI Server (e.g., on your machine or LAN). No internet needed, but you do need your own model zoo ready.
{% code overflow="wrap" %}
```js
const dg = new dg_sdk();
const zoo = await dg.connect('localhost:8779'); // IP/Port of an AI Server on your network
const models = await zoo.listModels();
console.log('Local models:', Object.keys(models));
```
{% endcode %}
### Hybrid (Local + Cloud Models)
Inference still runs on your local AI Server, but models are loaded from the DeGirum Cloud Zoo.
{% code overflow="wrap" %}
```js
const dg = new dg_sdk();
const zoo = await dg.connect(
'localhost:8779', // IP/Port of an AI Server on your network
'https://cs.degirum.com/degirum/public', // or another Cloud Zoo URL from AI Hub
'YOUR_TOKEN' // your AI Hub access token
);
const models = await zoo.listModels();
console.log('Cloud models via local server:', Object.keys(models));
```
{% endcode %}
### Cloud Only
Everything runs in the DeGirum Cloud - ideal for scalable, managed inference.
{% code overflow="wrap" %}
```js
const dg = new dg_sdk();
const zoo = await dg.connect(
'cloud',
'https://cs.degirum.com/degirum/public', // or another Cloud Zoo URL from AI Hub
'YOUR_TOKEN' // your AI Hub access token
);
const models = await zoo.listModels();
console.log('Cloud models:', Object.keys(models));
```
{% endcode %}
***
For more information on setting up your AI Server, refer to the [AI Server documentation](https://docs.degirum.com/pysdk/user-guide-pysdk/setting-up-an-ai-server)
# Batch Processing and Callbacks
This guide covers model.predict\_batch(), asynchronous callbacks, and how to manage the inference queue.
## Batch Processing with `predict_batch()`
The `model.predict_batch()` method is an [async generator](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/AsyncGenerator) that processes a sequence of images. This is ideal for scenarios like processing frames from a video or handling a large dataset of images efficiently.
You can provide data to `predict_batch` in two main ways:
* **Async Iterable**: Any object that implements the async iteration protocol, such as an array of image sources or a custom generator function.
* **`ReadableStream`**: A standard web API for handling streams of data, perfect for sources like the WebCodecs API.
The method processes images from the source, sends them for inference, and `yield`s the results as they become available. You consume these results using a `for await...of` loop.
### Example 1: Camera Inference Using an Async Generator Function
Here's how you can define a simple async generator to feed webcam frames to `predict_batch`.
{% code overflow="wrap" %}
```javascript
// Create video element and give it access to the webcam
const video = document.createElement('video');
video.autoplay = true;
video.style.display = 'none';
document.body.appendChild(video);
const stream = await navigator.mediaDevices.getUserMedia({ video: true });
video.srcObject = stream;
// Wait for video to be ready
await new Promise(resolve => video.onloadedmetadata = resolve);
// Frame generator yielding camera frames + frameId
async function* frameGenerator() {
let frameId = 0;
while (true) {
if (!video.videoWidth || !video.videoHeight) continue;
const bitmap = await createImageBitmap(video);
yield [bitmap, `frame_${frameId++}`];
}
}
// Run inference on the webcam frames
for await (const result of model.predict_batch(frameGenerator())) {
model.displayResultToCanvas(result, 'outputCanvas');
}
```
{% endcode %}
### Example 2: Using an Array as an Async Iterable
Here's how you can process a predefined list of image URLs. We create a simple async generator that yields the image URL and a unique frame identifier.
{% code overflow="wrap" %}
```javascript
// A simple async generator to feed predict_batch
async function* createImageSource(imageUrls) {
let frameId = 0;
for (const url of imageUrls) {
// Yield a tuple: [imageData, frameInfo]
yield [url, `frame_${frameId++}`];
}
}
// Array of image URLs to process
const urls = ['/path/to/image1.jpg', '/path/to/image2.jpg', '/path/to/image3.jpg'];
const dataSource = createImageSource(urls);
// Use for await...of to process the results
for await (const result of model.predict_batch(dataSource)) {
console.log(`Received result for: ${result.result[1]}`); // e.g., "Received result for: frame_0"
model.displayResultToCanvas(result, 'outputCanvas');
// Pause for a moment to see the result
await new Promise(resolve => setTimeout(resolve, 500));
}
console.log('Batch processing complete.');
```
{% endcode %}
### Example 3: Using a `ReadableStream`
`predict_batch` can directly consume a `ReadableStream`. This is powerful for streaming video frames, for example from a file or a live camera feed using the WebCodecs API.
{% code overflow="wrap" %}
```javascript
// Conceptual example: assuming you have a ReadableStream of VideoFrame objects ready
// let videoFrameStream = getStreamFromWebCodecs(); // some ReadableStream
// model.predict_batch can directly consume the stream
// No custom generator is needed.
for await (const result of model.predict_batch(videoFrameStream)) {
console.log('Processed a frame from the stream:', result);
model.displayResultToCanvas(result, 'outputCanvas');
}
```
{% endcode %}
Please view the [WebCodecs Example](/degirumjs/guides/web-codecs-example) for a complete demonstration of using `WebCodecs` and `ReadableStream` in DeGirumJS.
## Asynchronous Flow with Callbacks
Instead of using a `for await...of` loop to pull results, you can adopt an event-driven approach by providing a `callback` function when you load the model. When a callback is provided, `predict_batch` will *not* yield results. Instead, your callback function will be invoked automatically for each result as it arrives from the server.
This decouples the sending of frames from the receiving of results, which is ideal for real-time applications where you don't want your main loop to be blocked waiting for inference to complete.
**When to use which pattern:**
* **`for await...of` (Default):** Best for situations where you want to handle results sequentially in a straightforward, linear manner.
* **`callback` (Event-Driven):** Alternative for continuous, real-time streams. It can prevent back-pressure on your UI thread and allows your application to remain responsive.
### Example: Using a Callback
{% code overflow="wrap" %}
```javascript
// 1. Define your callback function
function handleInferenceResult(result, frameInfo) {
console.log(`Callback received result for frame: ${frameInfo}`);
// The 'result' object here is the same as the one from predict()
model.displayResultToCanvas(result, 'outputCanvas');
}
// 2. Load the model with the callback option
let model = await zoo.loadModel(
'yolo_v5s_coco--512x512_quant_n2x_cpu_1',
{ callback: handleInferenceResult }
);
// 3. Start the batch prediction. Note that we don't 'await' results here.
// The loop will run to completion, queuing up all frames.
// Results will be handled by the callback function asynchronously.
const urls = ['/path/to/image1.jpg', '/path/to/image2.jpg', '/path/to/image3.jpg'];
const dataSource = createImageSource(urls);
await model.predict_batch(dataSource); // The 'await' here just waits for all frames to be sent
console.log('All frames sent to the server. Results will arrive in the callback.');
```
{% endcode %}
## Controlling Back-pressure with `max_q_len`
When you send frames for inference, they are placed in a queue. `max_q_len` (maximum queue length) is an option you can set during model loading that defines the maximum number of frames that can be "in flight" at once.
* **`max_q_len` (default: 10 for AI Server, 80 for Cloud Server):** The size of the internal queues (`infoQ` and `resultQ`) that buffer frames and their results.
This parameter is crucial for managing system resources and preventing your application from sending data faster than the inference server can handle it. If the queue is full, your `predict()` or `predict_batch()` call will pause (asynchronously) until a space becomes available. This is a form of **back-pressure** that keeps the pipeline stable.
{% code overflow="wrap" %}
```javascript
// Load a model with a custom queue length of 5
let model = await zoo.loadModel(
'your_model_name',
{ max_q_len: 5 } // Allow up to 5 frames to be in-flight
);
```
{% endcode %}
A smaller `max_q_len` can reduce memory usage but may lower throughput if the network or server has high latency. A larger value can improve throughput by ensuring the server is never idle, but it will consume more memory and increase end-to-end latency for any single frame.
# Device Management for Inference
Configure and switch between device types when running inference with DeGirumJS.
AI Models can be run on various hardware configurations, and the DeGirumJS provides a flexible way to manage device types chosen for inference. This is particularly useful when you want to switch between different hardware accelerators or runtimes without having to significantly change your code.
Both `AIServerModel` and `CloudServerModel` classes offer flexible ways to manage device types, allowing you to configure and switch between devices dynamically.
## Supported Device Types
Each model has a set of `SupportedDeviceTypes`, which indicates the runtime/device combinations that are compatible for inference. The format for device types is `"RUNTIME/DEVICE"`, where:
* **RUNTIME** refers to the AI engine or runtime used for inference (e.g., `TENSORRT`, `OPENVINO`).
* **DEVICE** refers to the hardware type (e.g., `CPU`, `GPU`, `NPU`).
## AIServerModel / CloudServerModel Device Management
In the `AIServerModel` and `CloudServerModel` classes, device management is integrated into both the initialization and runtime phases of the model lifecycle. Below are key scenarios and examples:
### Default Device Type Selection
When you load a model without specifying a device type, the default device type **specified in the model parameters** is selected.
{% code overflow="wrap" %}
```javascript
let model = await zoo.loadModel('your_model_name');
console.log(model.deviceType); // Outputs: "DefaultRuntime/DefaultAgent"
```
{% endcode %}
### Switching Device Types After Initialization
You can change the device type even after the model has been initialized. The model will validate the requested device type against the system’s supported device types.
{% code overflow="wrap" %}
```javascript
model.deviceType = 'RUNTIME2/CPU';
console.log(model.deviceType); // Outputs: "RUNTIME2/CPU"
```
{% endcode %}
If the requested device type is not valid, an error will be thrown.
### Specifying a Device Type During Initialization
You can specify a device type when loading the model. The model will start with the specified device type if it’s available.
{% code overflow="wrap" %}
```javascript
let model = await zoo.loadModel('your_model_name', { deviceType: 'RUNTIME2/CPU' });
console.log(model.deviceType); // Outputs: "RUNTIME2/CPU"
```
{% endcode %}
### Handling Multiple Device Types
The SDK allows you to provide a list of device types. The **first available** option in the list will be selected.
{% code overflow="wrap" %}
```javascript
model.deviceType = ['RUNTIME3/CPU', 'RUNTIME1/CPU'];
console.log(model.deviceType); // Outputs: "RUNTIME3/CPU" if available, otherwise "RUNTIME1/CPU"
```
{% endcode %}
### Fallback and Error Handling
If none of the specified device types are supported, the model will throw an error, ensuring that only valid configurations are used.
{% code overflow="wrap" %}
```javascript
try {
model.deviceType = ['INVALID/DEVICE', 'ANOTHER_INVALID/DEVICE'];
} catch (e) {
console.error('Error: Invalid device type selection');
}
```
{% endcode %}
### Supported Device Types
You can check the supported device types for a model using the `supportedDeviceTypes` property.
{% code overflow="wrap" %}
```javascript
console.log(model.supportedDeviceTypes); // Outputs: ["RUNTIME1/CPU", "RUNTIME2/CPU"]
```
{% endcode %}
### System Supported Device Types
You can check the system’s list of supported devices for inference using the `getSupportedDevices()` method of the `dg_sdk` class.
{% code overflow="wrap" %}
```javascript
let dg = new dg_sdk();
let aiserverDevices = dg.getSupportedDevices('targetAIServerIp');
console.log(aiserverDevices); // Outputs: ["RUNTIME1/CPU", "RUNTIME2/CPU", "RUNTIME3/CPU"]
let cloudDevices = dg.getSupportedDevices('cloud');
console.log(cloudDevices); // Outputs: ["RUNTIME1/CPU", "RUNTIME2/CPU", "RUNTIME3/CPU"]
```
{% endcode %}
Device management in both `AIServerModel` and `CloudServerModel` is designed to be flexible, allowing you to fine-tune the inference environment. You can easily switch between device types, handle fallbacks, and ensure that your models are always running on supported configurations.
# Model Parameters
Overview of model parameters available when loading or configuring models.
When loading a model with **`zoo.loadModel()`** you can supply an **`options`** object that lets you control various functionalities of the model. You can also set these parameters on a model instance after it has been loaded.
## Model Parameters
The following parameters can be configured to control the behavior of the model, including preprocessing, post-processing, and connection settings. Parameters are categorized by their effect:
* **JSON-only parameters**: These are passed directly to the server as model configuration and do not directly alter the JavaScript SDK's model class behavior.
* **Model-only parameters**: These exclusively affect the model class's behavior within the JavaScript SDK and are not sent to the server.
* **Hybrid parameters**: These both influence the model class's functionality in JavaScript and are also set in the model parameter JSON sent to the server.
| Parameter Name | Description | Type | Valid Values / Constraints | Category |
| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
| `deviceType` | Specifies the target device type for inference. This can be a single device or an array of preferred devices. The SDK will attempt to use the first supported device in the list. | `string` or `Array` | Examples: `'RUNTIME/DEVICE'` (e.g., `'OPENVINO/CPU'`) or `['RUNTIME1/DEVICE1', 'RUNTIME2/DEVICE2']`. Must be a device type supported by the model and available on the system. | Hybrid |
| `labelWhitelist` | An array of strings. If set, only detection labels present in this list will be displayed in the overlay. | `Array` | An array of strings, e.g., `['person', 'car']`. | Model-only |
| `labelBlacklist` | An array of strings. If set, detection labels present in this list will be excluded from the overlay. | `Array` | An array of strings, e.g., `['background', 'noise']`. | Model-only |
| `overlayColor` | Sets the color(s) used for drawing bounding boxes, labels, and segmentation masks. Can be a single RGB triplet `[R, G, B]` or an array of such triplets `[[R1, G1, B1], [R2, G2, B2]]` for cycling through colors. Defaults to auto-generated colors. | `Array` or `Array>` | Numbers between 0 and 255 for RGB components. Example: `[255, 0, 0]` for red, or `[[255, 0, 0], [0, 255, 0]]`. | Model-only |
| `overlayLineWidth` | Sets the line width (in pixels) for drawing bounding boxes and connections in pose detection. | `number` | A positive number, e.g., `3`. | Model-only |
| `overlayShowLabels` | A boolean value. If `true`, labels (e.g., class names) will be displayed in the overlay. | `boolean` | `true` or `false`. | Model-only |
| `overlayShowProbabilities` | A boolean value. If `true`, probabilities or confidence scores will be displayed alongside labels in the overlay. | `boolean` | `true` or `false`. | Model-only |
| `overlayAlpha` | Sets the transparency percentage of the overlay elements (bounding boxes, masks, etc.). | `number` | A number between 0 (fully transparent) and 1 (fully opaque). | Model-only |
| `overlayFontScale` | Sets the scaling factor for the font size of text displayed in the overlay. | `number` | A positive number, e.g., `1.5` for 150% size. | Model-only |
| `inputLetterboxFillColor` | Sets the RGB fill color used for letterboxing when the `inputPadMethod` is set to `'letterbox'`. | `Array` | An array of 3 numbers between 0 and 255, e.g., `[0, 0, 0]` for black. | Model-only |
| `inputPadMethod` | Specifies the method used to resize and pad the input image to match the model's expected input dimensions. | `string` | One of: `'stretch'`, `'letterbox'`, `'crop-first'`, or `'crop-last'`. | Model-only |
| `saveModelImage` | A boolean value. If `true`, the preprocessed image (as a Blob) that was sent to the model will be included in the result object. | `boolean` | `true` or `false`. | Model-only |
| `inputCropPercentage` | For `'crop-first'` and `'crop-last'` padding methods, this specifies the percentage of the image to crop. | `number` | A number between 0 and 1. | Model-only |
| `autoScaleDrawing` | A boolean value. If `true`, overlay elements like font size and line width will automatically scale based on the display canvas size to maintain visual consistency. | `boolean` | `true` or `false`. | Model-only |
| `targetDisplayWidth` | Reference width (in pixels) used for `autoScaleDrawing`. This helps determine the scaling factor for overlay elements. | `number` | A number representing the target width, defaults to `1920`. | Model-only |
| `targetDisplayHeight` | Reference height (in pixels) used for `autoScaleDrawing`. This helps determine the scaling factor for overlay elements. | `number` | A number representing the target height, defaults to `1080`. | Model-only |
| `cloudToken` | Sets the authentication token required for connecting to the DeGirum cloud inference service. | `string` | A valid authentication token string. | JSON-only |
| `cloudURL` | Sets the base URL for the DeGirum cloud server. | `string` | A valid URL string, e.g., `'https://cloud.degirum.com'`. | JSON-only |
| `outputConfidenceThreshold` | Sets the minimum confidence score (probability) for a detection or classification to be included in the model's output. Results below this threshold are filtered out. | `number` | A number between 0 and 1. | JSON-only |
| `outputMaxDetections` | Sets the maximum total number of detections that the model should return for a single inference. | `number` | An integer. | JSON-only |
| `outputMaxDetectionsPerClass` | Sets the maximum number of detections allowed per individual class. | `number` | An integer. | JSON-only |
| `outputMaxClassesPerDetection` | Sets the maximum number of classes to report for each detected object. Useful for models that can classify a single detection into multiple categories. | `number` | An integer. | JSON-only |
| `outputNmsThreshold` | Sets the Non-Maximum Suppression (NMS) Intersection Over Union (IOU) threshold. This parameter is used to filter overlapping bounding boxes, keeping only the most confident ones. | `number` | A number between 0 and 1. | JSON-only |
| `outputPostprocessType` | Specifies the type of post-processing to apply to the model's raw output. This dictates how the model's predictions are interpreted and formatted. | `string` | One of: `'None'`, `'Base'`, `'Classification'`, `'MultiLabelClassification'`, `'Detection'`, `'DetectionYolo'`, `'DetectionYoloV8'`, `'DetectionYoloV10'`, `'DetectionYoloPlates'`, `'DetectionYoloV8Plates'`, `'FaceDetection'`, `'PoseDetection'`, `'PoseDetectionYoloV8'`, `'HandDetection'`, `'Segmentation'`, `'SegmentationYoloV8`, `Dequantization`, `Null`. | JSON-only |
| `outputTopK` | For classification models, this parameter specifies the number of top predictions (classes with the highest confidence scores) to return. | `number` | An integer. | JSON-only |
| `outputUseRegularNms` | A boolean value. If `true`, the model will use regular Non-Maximum Suppression (NMS); otherwise, it may use a faster, approximate NMS algorithm. | `boolean` | `true` or `false`. | JSON-only |
| `measureTime` | A boolean value. If `true`, enables detailed performance timing statistics for various stages of the inference process (client-side preprocessing, server-side inference, client-side post-processing). These statistics can be retrieved using `getTimeStats()` and `printLatencyInfo()`. | `boolean` | `true` or `false`. | Hybrid |
| `eagerBatchSize` | Sets the batch size for eager execution, influencing how many frames are processed together in a single batch. | `number` | A positive integer. | JSON-only |
| `outputPoseThreshold` | Sets the confidence threshold for individual keypoints in pose detection results. Keypoints with scores below this threshold may be filtered out. | `number` | A number between 0 and 1. | JSON-only |
| `inputShape` | Defines the expected input shape(s) for the model. This can be an array of arrays, where each inner array represents the dimensions (N, H, W, C) for a specific input. | `Array>` | An array of integer arrays, e.g., `[[1, 224, 224, 3]]` for a single input of batch size 1, height 224, width 224, and 3 channels. | Hybrid |
# Performance and Timing Statistics
Interpret performance and latency metrics collected during inference.
Once you enable `measureTime` on a Model instance, every `predict` and `predict_batch` result contains timing statistics. These are accumulated by the Model instance inside a `timeStats` object.
## Available methods
1. `getTimeStats()`: Use this method to return a formatted string of all the statistics collected so far.
2. `resetTimeStats()`: Use this method to delete all your old statistics and create a fresh `timeStats` object to collect more statistics with.
3. To access the `timeStats` object directly, you can use `modelName.timeStats.stats["statName"]`, where the `statName` is one of the operations tracked.
4. `printLatencyInfo()` logs a brief, human-readable summary of average timings into the console.
### Example usage
{% code overflow="wrap" %}
```javascript
let model = await zoo.loadModel('your_model_name', { measureTime: true });
let result = await model.predict(image);
console.log(model.getTimeStats()); // Pretty print time stats
// Access client-side and server-side timing stats
let preprocessDuration = model.timeStats.stats["ImagePreprocessDuration_ms"]; // Get image preprocess duration (min, avg, max, count)
let preprocessMin = model.timeStats.stats["ImagePreprocessDuration_ms"].min; // Get min image preprocess duration
let inferenceDuration = model.timeStats.stats["CoreInferenceDuration_ms"]; // Get core inference duration (min, avg, max, count)
let inferenceMax = model.timeStats.stats["CoreInferenceDuration_ms"].max; // Get max core inference duration
let frameTotalDuration = model.timeStats.stats["FrameTotalDuration_ms"]; // Get total time taken for the entire frame processing
let deviceTemp = model.timeStats.stats["DeviceTemperature_C"]; // Get device temperature if available
model.resetTimeStats(); // Reset time stats
```
{% endcode %}
## Client-Side Timings
These metrics are measured within the JavaScript SDK running in the user's browser.
| Key | Description |
| ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `FrameTotalDuration_ms` | (End-to-End) The total wall-clock time from the moment `predict` or `predict_batch` is called until the final processed result is ready for the user. This is the most comprehensive client-side metric. |
| `MutexWait_ms` | The time spent waiting to acquire a lock before starting to process a new frame. Only relevant for synchronous `predict()` calls. This value will be high if you are calling `predict()` faster than the model can process frames, indicating contention. |
| `InputFrameConvert_ms` | The time taken to validate and convert the user's input (e.g., a URL, base64 string, or `HTMLImageElement`) into a standardized format ready for preprocessing. (before preprocessing) |
| `ImagePreprocessDuration_ms` | The time spent on client-side image manipulation. This primarily consists of resizing the image to the model's required input dimensions and applying padding/cropping methods. |
| `EncodeEmit_ms` |
The time taken to encode the image data and send it over the network. - For AIServerModel, this is just socket.send(blob). - For CloudServerModel, this involves encoding the data with msgpack and then socket.emit().
|
| `ResultProcessing_ms` | The time spent processing a result after it has been received from the server. This includes matching it with the original frame info, applying label filters, and pushing it into the result queue. |
| `ResultQueueWaitingTime_ms` | The time a processed result sits in the output queue (`resultQ`) before being returned to the user's code. This measures back-pressure if the user code is consuming results slower than the model produces them. |
| `SocketConnectWait_ms` | A one-time (or per-reconnect) cost of establishing the network connection to the server. This will not appear for every frame. |
## Server-Side Timings
These metrics are measured on the AI Server or Cloud Server and are included in the result payload sent back to the client. The SDK simply extracts and records them.
| Key | Description |
| ----------------------------- | -------------------------------------------------------------------------------------------------------------- |
| `PythonPreprocessDuration_ms` | Duration of client-side pre-processing step including data loading time and data conversion time |
| `CorePreprocessDuration_ms` | Duration of server-side pre-processing step |
| `CoreInferenceDuration_ms` | Duration of server-side AI inference step |
| `CorePostprocessDuration_ms` | Duration of server-side post-processing step |
| `CoreInputFrameSize_bytes` | The size of received input frame |
| `DeviceInferenceDuration_ms` | (DeGirum ORCA models only) Duration of AI inference computations on AI accelerator IC excluding data transfers |
| `DeviceTemperature_C` | (DeGirum ORCA models only) Internal temperature of AI accelerator IC in C |
| `DeviceFrequency_MHz` | (DeGirum ORCA models only) Working frequency of AI accelerator IC in MHz |
# Preprocessing and Visual Overlays
Customize preprocessing and drawing parameters for DeGirumJS models.
DeGirumJS provides a rich set of user-facing properties that allow you to precisely control how input data is handled before inference (pre-processing) and how inference results are visually presented (overlays). This flexibility enables you to tailor the SDK's behavior to your specific application needs and aesthetic preferences.
The following examples show the parameters set as options when you load a model using `zoo.loadModel()`. You can always set the parameters by simply invoking the corresponding setter methods on a model instance after it has been loaded.
## Input Handling (Pre-processing)
The SDK automatically resizes and prepares your input images to match the dimensions required by the AI model. You can customize this process using the following parameters:
### inputPadMethod
This parameter determines how the input image is scaled and positioned within the model's input frame. It determines how your input image is sent to the model.
#### 'letterbox' (Default)
The image is resized to fit within the model's input dimensions while preserving its original aspect ratio. Any empty space (padding) around the image is filled with the color specified by `inputLetterboxFillColor`. This method prevents distortion and is generally recommended for most vision models.
{% code overflow="wrap" %}
```javascript
let model = await zoo.loadModel('your_model', { inputPadMethod: 'letterbox' });
```
{% endcode %}
#### 'stretch'
The image is stretched or shrunk to exactly match the model's input dimensions, regardless of its original aspect ratio. This can lead to image distortion but ensures the entire image fills the input frame.
{% code overflow="wrap" %}
```javascript
let model = await zoo.loadModel('your_model', { inputPadMethod: 'stretch' });
```
{% endcode %}
#### 'crop-first'
The image is first cropped to match the aspect ratio of the model's input, and then resized. The `inputCropPercentage` determines how much of the original image is retained.
{% code overflow="wrap" %}
```javascript
let model = await zoo.loadModel('your_model', { inputPadMethod: 'crop-first', inputCropPercentage: 0.9 });
```
{% endcode %}
#### 'crop-last'
The image is resized first, and then cropped to fit the model's input dimensions.
{% code overflow="wrap" %}
```javascript
let model = await zoo.loadModel('your_model', { inputPadMethod: 'crop-last', inputCropPercentage: 0.9 });
```
{% endcode %}
### inputLetterboxFillColor
When `inputPadMethod` is `'letterbox'`, this parameter sets the RGB color of the padded areas.
**Type**: `Array` (e.g., `[R, G, B]`, where each component is 0-255)
**Default**: `[0, 0, 0]` (black)
{% code overflow="wrap" %}
```javascript
let model = await zoo.loadModel('your_model', {
inputPadMethod: 'letterbox',
inputLetterboxFillColor: [255, 0, 0] // Red letterbox
});
```
{% endcode %}
### inputCropPercentage
This parameter is used in conjunction with `'crop-first'` and `'crop-last'` `inputPadMethod` values. It specifies the percentage of the image (after initial scaling for `'crop-last'`) that should be retained after cropping.
**Type**: `number` (between 0 and 1)
**Default**: `1.0`
{% code overflow="wrap" %}
```javascript
let model = await zoo.loadModel('your_model', {
inputPadMethod: 'crop-first',
inputCropPercentage: 0.8 // Retain 80% of the cropped image
});
```
{% endcode %}
## Overlay Customization
The `model.displayResultToCanvas()` method draws visual overlays (like bounding boxes, labels, and keypoints) on a canvas. You can customize the appearance of these overlays using the following parameters:
### overlayAlpha
Controls the transparency of the drawn overlays. A value of `1.0` means fully opaque, while `0.0` means fully transparent.
**Type**: `number` (between 0 and 1)
**Default**: `0.75`
{% code overflow="wrap" %}
```javascript
let model = await zoo.loadModel('your_model', { overlayAlpha: 0.5 }); // 50% transparent overlays
```
{% endcode %}
### overlayColor
Sets the color(s) for drawing overlays. You can provide a single RGB triplet for a uniform color or an array of RGB triplets to cycle through different colors for different detected objects/classes.
**Type**: `Array` (single `[R, G, B]`) or `Array>` (multiple `[[R, G, B], ...]`)
**Default**: `[-1, -1, -1]` (triggers automatic generation of distinct, bright colors)
{% code overflow="wrap" %}
```javascript
let model = await zoo.loadModel('your_model', { overlayColor: [255, 0, 0] }); // All overlays will be red
```
{% endcode %}
{% code overflow="wrap" %}
```javascript
let model = await zoo.loadModel('your_model', {
overlayColor: [[255, 0, 0], [0, 255, 0], [0, 0, 255]] // Cycles red, green, blue
});
```
{% endcode %}
### overlayFontScale
Adjusts the size of text labels (e.g., class names, probabilities) drawn on the overlay. A value of `1.0` is the default size.
**Type**: `number` (positive number)
**Default**: `1.0`
{% code overflow="wrap" %}
```javascript
let model = await zoo.loadModel('your_model', { overlayFontScale: 1.5 }); // 50% larger text
```
{% endcode %}
### overlayLineWidth
Sets the width of lines used in overlays, such as bounding box borders and connections in pose detection.
**Type**: `number` (positive number)
**Default**: `2`
{% code overflow="wrap" %}
```javascript
let model = await zoo.loadModel('your_model', { overlayLineWidth: 4 }); // Thicker lines
```
{% endcode %}
### overlayShowLabels
A boolean flag to control the visibility of text labels (e.g., "person", "car") on the overlay.
**Type**: `boolean`
**Default**: `true`
{% code overflow="wrap" %}
```javascript
let model = await zoo.loadModel('your_model', { overlayShowLabels: false }); // Hide labels
```
{% endcode %}
### overlayShowProbabilities
A boolean flag to control the visibility of confidence scores (probabilities) alongside labels on the overlay.
**Type**: `boolean`
**Default**: `false`
{% code overflow="wrap" %}
```javascript
let model = await zoo.loadModel('your_model', { overlayShowProbabilities: true }); // Show probabilities
```
{% endcode %}
### autoScaleDrawing
When set to `true`, the SDK automatically scales the drawn overlays (bounding boxes, labels, keypoints) to appear consistent regardless of the input image's original dimensions or the canvas size. It uses `targetDisplayWidth` and `targetDisplayHeight` as reference.
**Type**: `boolean`
**Default**: `false`
{% code overflow="wrap" %}
```javascript
let model = await zoo.loadModel('your_model', { autoScaleDrawing: true });
```
{% endcode %}
### targetDisplayWidth / targetDisplayHeight
These optional parameters are used in conjunction with `autoScaleDrawing`. They define a reference canvas size (e.g., `1920x1080`) against which the overlay elements are scaled. If your target canvas size is different from the default, you can adjust these values to ensure optimal visual presentation.
**Type**: `number`
**Defaults**: `1920` for width, `1080` for height
{% code overflow="wrap" %}
```javascript
let model = await zoo.loadModel('your_model', {
autoScaleDrawing: true,
targetDisplayWidth: 1280,
targetDisplayHeight: 720
});
```
{% endcode %}
## Label Filtering
You can control which detected objects or classification results are included in the final output by filtering them based on their labels. This is particularly useful when you are only interested in a subset of the classes a model can detect.
### labelBlacklist
An array of strings. Any result whose `label` matches an entry in this list will be *excluded* from the final output.
**Type**: `Array`
**Default**: `null` (no labels are blacklisted by default)
{% code overflow="wrap" %}
```javascript
let model = await zoo.loadModel('your_model', { labelBlacklist: ['cat', 'dog'] }); // Exclude cats and dogs
```
{% endcode %}
### labelWhitelist
An array of strings. If this list is provided, *only* results whose `label` matches an entry in this list will be *included* in the final output. All other labels will be filtered out.
**Type**: `Array`
**Default**: `null` (no labels are whitelisted by default, all are included unless blacklisted)
{% code overflow="wrap" %}
```javascript
let model = await zoo.loadModel('your_model', { labelWhitelist: ['person', 'car'] }); // Only include persons and cars
```
{% endcode %}
{% hint style="info" %}
If both `labelWhitelist` and `labelBlacklist` are provided, the whitelist takes precedence. Only items in the whitelist are considered, and any of those also present in the blacklist are removed. Use one list at a time for clarity.
{% endhint %}
# Result Object Structure
Understand the structure of prediction results returned by DeGirumJS.
The AIServerModel and CloudServerModel classes return a result object that contains the inference results from the `predict` and `predict_batch` functions.
Example:
{% code overflow="wrap" %}
```javascript
let someResult = await someModel.predict(image);
console.log(someResult);
```
{% endcode %}
For example, the result can be structured like this:
{% code overflow="wrap" %}
```json
{
"result": [
[
{ "category_id": 1, "label": "foo", "score": 0.2 },
{ "category_id": 0, "label": "bar", "score": 0.1 }
],
"frame123"
],
"imageFrame": imageBitmap
}
```
{% endcode %}
**Accessing the Result Data**
* **Inference Results**: Access the main results using `someResult.result[0]`.
* **Frame Info / Number**: Get the frame information or frame number using `someResult.result[1]`.
* **Original Input Image**: Access the original input image with `someResult.imageFrame`.
***
## Inference Result Types
The inference results can be one of the following types:
* **Detection**: Contains `bbox`, `category_id`, `label`, `score`, optional `landmarks` and `mask` for instance segmentation and pose estimation.
* **Classification**: Contains `label` and `score` for single-label classification.
* **Multi-Label Classification**: Contains `classifier` name and a `results` array with multiple `{ label, score }` entries.
* **Segmentation Mask**: Contains `shape` and `data`, where `data` is a flat array (or `Uint8Array`) representing class IDs per pixel.
* **Pose Detection**: Contains `landmarks` array with each landmark having `category_id`, `landmark` coordinates, optional `score`, `label`, and connectivity information.
***
## Example Results
### Detection Result
Detection results include bounding boxes (`bbox`) along with category IDs, labels, and confidence scores. Optionally, masks and landmarks may be included for segmentation and pose.
{% code overflow="wrap" %}
```json
[
{
"category_id": 2,
"label": "person",
"score": 0.98,
"bbox": [0.1, 0.2, 0.4, 0.8]
},
{
"category_id": 3,
"label": "dog",
"score": 0.87,
"bbox": [0.5, 0.3, 0.9, 0.7],
"mask": {
"width": 128,
"height": 128,
"data": ""
}
}
]
```
{% endcode %}
### Classification Result
Single-label classification results contain a `label`, `score`, and `category_id`.
{% code overflow="wrap" %}
```json
[
{
"category_id": 0,
"label": "dog",
"score": 0.76
}
]
```
{% endcode %}
### Multi-Label Classification Result
Multi-label classification results include a classifier name and an array of label-score pairs.
{% code overflow="wrap" %}
```json
[
{
"classifier": "scene_tags",
"results": [
{ "label": "beach", "score": 0.85 },
{ "label": "sunset", "score": 0.65 }
]
}
]
```
{% endcode %}
### Segmentation Mask Result
Semantic segmentation results provide a `shape` and a `data` array, which is typically returned as a `Uint8Array`.
{% code overflow="wrap" %}
```json
[
{
"shape": [1, 256, 256],
"data": [0, 1, 1, 0, /* ... repeated for 256*256 entries ... */]
}
]
```
{% endcode %}
### Pose Detection Result
Pose detection results include `landmarks` for each detected person, where each landmark has coordinates, optional score and label, and connectivity information.
{% code overflow="wrap" %}
```json
[
{
"category_id": 0,
"label": "person",
"score": 0.98,
"bbox": [0.1, 0.2, 0.4, 0.8],
"landmarks": [
{
"category_id": 0,
"landmark": [0.15, 0.25],
"connect": [1],
"label": "nose",
"score": 0.99
},
{
"category_id": 1,
"landmark": [0.15, 0.35],
"connect": [0],
"label": "left_eye",
"score": 0.97
}
]
}
]
```
{% endcode %}
***
# WebCodecs Example
Examples for using predict\_batch with the WebCodecs API.
If your browser [supports the WebCodecs API](https://caniuse.com/webcodecs), you can create efficient video processing pipelines with DeGirumJS.
The WebCodecs API provides low-level access to the individual frames of a video stream. This allows for highly efficient and flexible video processing pipelines directly in the browser. When combined with DeGirumJS's `predict_batch()` method, you can perform real-time AI inference on a live webcam stream with minimal latency.
The core components of this pipeline are:
1. **`MediaStreamTrackProcessor`**: Takes a `MediaStreamTrack` (like from a webcam) and exposes its frames as a `ReadableStream` of `VideoFrame` objects.
2. **`predict_batch()`**: The DeGirumJS method that can directly consume a `ReadableStream` of `VideoFrame` objects and efficiently process them for inference.
3. **`MediaStreamTrackGenerator`**: Takes a stream of processed `VideoFrame` objects and exposes them as a new `MediaStreamTrack`, which can be displayed in a `