> For the complete documentation index, see [llms.txt](https://docs.degirum.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.degirum.com/degirum-tools/support/eval_support.md).

# Model Evaluation Support

DeGirum Tools API Reference Guide. Framework and base class for model accuracy evaluation.

{% hint style="info" %}
This API Reference is based on DeGirum Tools version 1.2.0.
{% endhint %}

## Model Evaluation Support Module Overview <a href="#model-evaluation-support-module-overview" id="model-evaluation-support-module-overview"></a>

This module provides base classes and utilities for model evaluation, including performance metrics calculation, ground truth comparison, and evaluation result reporting. It supports various evaluation scenarios and metrics.

Key Features

* **Base Evaluator Class**: Abstract base class for model evaluators
* **YAML Configuration**: Support for evaluator configuration via YAML files
* **Flexible Evaluation**: Support for different evaluation metrics and scenarios
* **Result Reporting**: Standardized evaluation result reporting
* **Ground Truth Integration**: Support for comparing model outputs with ground truth

Typical Usage

1. Create a custom evaluator by subclassing ModelEvaluatorBase
2. Configure the evaluator using YAML or constructor parameters
3. Run evaluation on test datasets
4. Analyze and report evaluation results

Integration Notes

* Works with DeGirum PySDK models
* Supports standard evaluation metrics
* Handles various input formats
* Provides extensible evaluation framework

Key Classes

* `ModelEvaluatorBase`: Base class for model evaluators

Configuration Options

* Model parameters
* Evaluation metrics
* Dataset paths
* Ground truth format

## Classes <a href="#classes" id="classes"></a>

## ModelEvaluatorBase <a href="#modelevaluatorbase" id="modelevaluatorbase"></a>

`ModelEvaluatorBase`

Bases: `ABC`

Base class for model evaluators.

This abstract class initializes a model object, loads configuration parameters and defines the interface for performing evaluation.

Parameters:

| Name       | Type    | Description                                                                                                                                                                                    | Default    |
| ---------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
| `model`    | `Model` | Model instance to evaluate.                                                                                                                                                                    | *required* |
| `**kwargs` | `Any`   | Arbitrary model or evaluator parameters. Keys matching the model attributes are applied directly to the model. Remaining keys are assigned to the evaluator instance if such attributes exist. | `{}`       |

Attributes:

| Name    | Type    | Description                |
| ------- | ------- | -------------------------- |
| `model` | `Model` | The model being evaluated. |

### ModelEvaluatorBase Methods <a href="#modelevaluatorbase-methods" id="modelevaluatorbase-methods"></a>

#### \_\_init\_\_(model, ...) <a href="#init" id="init"></a>

`__init__(model, **kwargs)`

Initialize the evaluator.

Parameters:

| Name       | Type    | Description                                                                                                             | Default    |
| ---------- | ------- | ----------------------------------------------------------------------------------------------------------------------- | ---------- |
| `model`    | `Model` | PySDK model object.                                                                                                     | *required* |
| `**kwargs` | `Any`   | Arbitrary model or evaluator parameters. Keys must either match model attributes or attributes of `ModelEvaluatorBase`. | `{}`       |

#### evaluate(image\_folder\_path, ...) <a href="#evaluate" id="evaluate"></a>

`evaluate(image_folder_path, ground_truth_annotations_path, max_images=0)`

`abstractmethod`

Evaluate the model on a dataset.

Parameters:

| Name                            | Type  | Description                                                                | Default    |
| ------------------------------- | ----- | -------------------------------------------------------------------------- | ---------- |
| `image_folder_path`             | `str` | Directory containing evaluation images.                                    | *required* |
| `ground_truth_annotations_path` | `str` | Path to the ground truth JSON file in COCO format.                         | *required* |
| `max_images`                    | `int` | Maximum number of images to process. `0` uses all images. Defaults to `0`. | `0`        |

Returns:

| Type   | Description                                 |
| ------ | ------------------------------------------- |
| `list` | Evaluation statistics (algorithm specific). |

#### init\_from\_yaml(model, ...) <a href="#init_from_yaml" id="init_from_yaml"></a>

`init_from_yaml(model, config_yaml)`

`classmethod`

Construct an evaluator from a YAML file.

Parameters:

| Name          | Type                     | Description                                                      | Default    |
| ------------- | ------------------------ | ---------------------------------------------------------------- | ---------- |
| `model`       | `Model`                  | PySDK model object.                                              | *required* |
| `config_yaml` | `Union[str, TextIOBase]` | Path or open stream with evaluator configuration in YAML format. | *required* |

Returns:

| Name   | Type   | Description                    |
| ------ | ------ | ------------------------------ |
| `Self` | `Self` | Instantiated evaluator object. |