LogoLogo
AI HubCommunityWebsite
  • Start Here
  • AI Hub
    • Overview
    • Quickstart
    • Teams
    • Device Farm
    • Browser Inference
    • Model Zoo
      • Hailo
      • Intel
      • MemryX
      • BrainChip
      • Google
      • DeGirum
      • Rockchip
    • View and Create Model Zoos
    • Compiler
    • PySDK Integration
  • PySDK
    • Overview
    • Quickstart
    • Installation
    • Runtimes and Drivers
      • Hailo
      • OpenVINO
      • MemryX
      • BrainChip
      • Rockchip
      • ONNX
    • PySDK User Guide
      • Core Concepts
      • Organizing Models
      • Setting Up an AI Server
      • Loading an AI Model
      • Running AI Model Inference
      • Model JSON Structure
      • Command Line Interface
      • API Reference Guide
        • PySDK Package
        • Model Module
        • Zoo Manager Module
        • Postprocessor Module
        • AI Server Module
        • Miscellaneous Modules
      • Older PySDK User Guides
        • PySDK 0.16.1
        • PySDK 0.16.0
        • PySDK 0.15.2
        • PySDK 0.15.1
        • PySDK 0.15.0
        • PySDK 0.14.3
        • PySDK 0.14.2
        • PySDK 0.14.1
        • PySDK 0.14.0
        • PySDK 0.13.4
        • PySDK 0.13.3
        • PySDK 0.13.2
        • PySDK 0.13.1
        • PySDK 0.13.0
    • Release Notes
      • Retired Versions
    • EULA
  • DeGirum Tools
    • Overview
      • Streams
        • Streams Base
        • Streams Gizmos
      • Compound Models
      • Analyzers
        • Clip Saver
        • Event Detector
        • Line Count
        • Notifier
        • Object Selector
        • Object Tracker
        • Zone Count
      • Inference Support
      • Support Modules
        • Audio Support
        • Model Evaluation Support
        • Math Support
        • Object Storage Support
        • UI Support
        • Video Support
  • DeGirumJS
    • Overview
    • Get Started
    • Understanding Results
    • Release Notes
    • API Reference Guides
      • DeGirumJS 0.1.3
      • DeGirumJS 0.1.2
      • DeGirumJS 0.1.1
      • DeGirumJS 0.1.0
      • DeGirumJS 0.0.9
      • DeGirumJS 0.0.8
      • DeGirumJS 0.0.7
      • DeGirumJS 0.0.6
      • DeGirumJS 0.0.5
      • DeGirumJS 0.0.4
      • DeGirumJS 0.0.3
      • DeGirumJS 0.0.2
      • DeGirumJS 0.0.1
  • Orca
    • Overview
    • Benchmarks
    • Unboxing and Installation
    • M.2 Setup
    • USB Setup
    • Thermal Management
    • Tools
  • Resources
    • External Links
Powered by GitBook

Get Started

  • AI Hub Quickstart
  • PySDK Quickstart
  • PySDK in Colab

Resources

  • AI Hub
  • Community
  • DeGirum Website

Social

  • LinkedIn
  • YouTube

Legal

  • PySDK EULA
  • Terms of Service
  • Privacy Policy

© 2025 DeGirum Corp.

On this page
  • Model Evaluation Support Module Overview
  • Classes
  • ModelEvaluatorBase
  • Functions
  • __init__(model, ...)
  • evaluate(image_folder_path, ...)
  • init_from_yaml(model, ...)

Was this helpful?

  1. DeGirum Tools
  2. Overview
  3. Support Modules

Model Evaluation Support

This API Reference is based on DeGirum Tools version 0.16.7.

Model Evaluation Support Module Overview

This module provides base classes and utilities for model evaluation, including performance metrics calculation, ground truth comparison, and evaluation result reporting. It supports various evaluation scenarios and metrics.

Key Features

  • Base Evaluator Class: Abstract base class for model evaluators

  • YAML Configuration: Support for evaluator configuration via YAML files

  • Flexible Evaluation: Support for different evaluation metrics and scenarios

  • Result Reporting: Standardized evaluation result reporting

  • Ground Truth Integration: Support for comparing model outputs with ground truth

Typical Usage

  1. Create a custom evaluator by subclassing ModelEvaluatorBase

  2. Configure the evaluator using YAML or constructor parameters

  3. Run evaluation on test datasets

  4. Analyze and report evaluation results

Integration Notes

  • Works with DeGirum PySDK models

  • Supports standard evaluation metrics

  • Handles various input formats

  • Provides extensible evaluation framework

Key Classes

  • ModelEvaluatorBase: Base class for model evaluators

Configuration Options

  • Model parameters

  • Evaluation metrics

  • Dataset paths

  • Ground truth format

Classes

ModelEvaluatorBase

ModelEvaluatorBase

Bases: ABC

Base class for model evaluators.

This abstract class initializes a model object, loads configuration parameters and defines the interface for performing evaluation.

Parameters:

Name
Type
Description
Default

model

Model

Model instance to evaluate.

required

**kwargs

Any

Arbitrary model or evaluator parameters. Keys matching the model attributes are applied directly to the model. Remaining keys are assigned to the evaluator instance if such attributes exist.

{}

Attributes:

Name
Type
Description

model

Model

The model being evaluated.

Functions

__init__(model, ...)

__init__(model, **kwargs)

Initialize the evaluator.

Parameters:

Name
Type
Description
Default

model

Model

PySDK model object.

required

**kwargs

Any

Arbitrary model or evaluator parameters. Keys must either match model attributes or attributes of ModelEvaluatorBase.

{}

evaluate(image_folder_path, ...)

evaluate(image_folder_path, ground_truth_annotations_path, max_images=0)

abstractmethod

Evaluate the model on a dataset.

Parameters:

Name
Type
Description
Default

image_folder_path

str

Directory containing evaluation images.

required

ground_truth_annotations_path

str

Path to the ground truth JSON file in COCO format.

required

max_images

int

Maximum number of images to process. 0 uses all images. Defaults to 0.

0

Returns:

Type
Description

list

Evaluation statistics (algorithm specific).

init_from_yaml(model, ...)

init_from_yaml(model, config_yaml)

classmethod

Construct an evaluator from a YAML file.

Parameters:

Name
Type
Description
Default

model

Model

PySDK model object.

required

config_yaml

Union[str, TextIOBase]

Path or open stream with evaluator configuration in YAML format.

required

Returns:

Name
Type
Description

Self

Self

Instantiated evaluator object.

PreviousAudio SupportNextMath Support

Last updated 2 days ago

Was this helpful?