Get Started

This page provides a step-by-step guide for setting up and using the DeGirumJS JavaScript AI Inference SDK.

Welcome to the DeGirumJS: a JavaScript AI Inference SDK! This guide will help you get started with integrating AI inference capabilities into your web application. Follow the steps below to set up your environment, connect to an AI server or the cloud, and run inference.

Table of Contents

• Getting Started with DeGirumJS

  • Table of Contents

  • Introduction

  • Setup

    • Import the SDK

  • Basic Usage

    • Connect to an AI Server

    • Connect to the Cloud

    • Load a Model

    • Perform Inference

    • Displaying Results

    • Simple Example HTML Page

  • Model Options

    • Device Management for Inference

      • Supported Device Types

      • AIServerModel / CloudServerModel Device Management

    • Measure Time Usage

      • Operations Tracked

      • Available methods

      • Example usage

  • API Reference

Introduction

The JavaScript AI Inference SDK allows you to connect to AI Server or Cloud Zoo instances, load AI models, and perform inference on various data types. This guide provides a step-by-step tutorial on how to use the SDK effectively.

Setup

Import the SDK

To start using the SDK, include the following script tag in your HTML file:

<script src="https://assets.degirum.com/degirumjs/0.1.3/degirum-js.min.obf.js"></script>

Basic Usage

DeGirumJS allows you to load models from an AI server or Cloud Zoo and perform inference on the AI Server hardware or in the cloud. For AI server inference (local or LAN server), you need to have an AI Server running with http protocol enabled :

degirum server --protocol both

See AI Server documentation

For running cloud inference or to be able to load a model from the cloud, you need to specify your cloud token.

Connect to an AI Server

Instantiate the dg_sdk class and connect to the AI server using the connect method:

let dg = new dg_sdk();
const AISERVER_IP = 'ws://localhost:8779';

let inference_manager = await dg.connect(AISERVER_IP);

For running AI Server inference on cloud models, include the URL of the cloud zoo and your token:

let dg = new dg_sdk();
const AISERVER_IP = 'ws://localhost:8779';
const ZOO_URL = 'https://cs.degirum.com/degirum/public';
const secretToken = prompt('Enter secret token:');

let inference_manager = await dg.connect(AISERVER_IP, ZOO_URL, secretToken);

Connect to the Cloud

For running Cloud inference, specify 'cloud' as the first argument, and include the URL of the cloud zoo and your token:

let dg = new dg_sdk();
const ZOO_URL = 'https://cs.degirum.com/degirum/public';
const secretToken = prompt('Enter secret token:');

let inference_manager = await dg.connect('cloud', ZOO_URL, secretToken);

Load a Model

Now, you can load a model using the zoo class instance's loadModel method:

const MODEL_NAME = 'yolo_v5s_face_det--512x512_quant_n2x_cpu_1';
const modelOptions = {
    overlayShowProbabilities: true
    // Any other custom options for your Model (see Model Options section below)
};

let model = await inference_manager.loadModel(MODEL_NAME, modelOptions);

Perform Inference

Use the predict method to perform inference on an input image:

const image = ''; // Some image. Can be Blob, File, base64string, HTMLImageElement, HTMLVideoElement, HTMLCanvasElement, ArrayBuffer, TypedArray, ImageBitmap, URL to an image
const result = await model.predict(image);
console.log('Result:', result);

Displaying Results

You can display prediction results to a HTMLCanvasElement:

// Assuming your Canvas Element has the id 'outputCanvas'
let canvas = document.getElementById('outputCanvas');
model.displayResultToCanvas(result, canvas);

This will draw the inference results onto the canvas.

Understanding the Result Object Structure The result object contains the predictions made by the model, such as detected objects, classes, probabilities, bounding boxes, and more.

Simple Example HTML Page

To get started with a simple example page, we need the following HTML elements on the page:

The script tag to import DeGirumJS
A canvas element to display inference results.
An input element to browse and upload images.

Here is a HTML page that will perform inference on uploaded images and display the results:

<script src="https://assets.degirum.com/degirumjs/0.1.3/degirum-js.min.obf.js"></script>
<canvas id="outputCanvas" width="400" height="400"></canvas>
<input type="file" id="imageInput" accept="image/*">
<script type="module">
    // Grab the outputCanvas and imageInput elements by ID:
    const canvas = document.getElementById('outputCanvas');
    const input = document.getElementById('imageInput');
    
    // Initialize the SDK
    let dg = new dg_sdk();
    // Query the user for the cloud token:
    const secretToken = prompt('Enter secret token:');
    // Inference settings
    const MODEL_NAME = 'yolo_v5s_face_det--512x512_quant_n2x_cpu_1';
    const ZOO_URL = 'https://cs.degirum.com/degirum/public';
    const AISERVER_IP = 'ws://localhost:8779';
    
    // Connect to the cloud zoo
    let inference_manager = await dg.connect(AISERVER_IP, ZOO_URL, secretToken);
    
    // Model options
    const modelOptions = {
        overlayShowProbabilities: true
    };
    // Load the model with the options
    let model = await inference_manager.loadModel(MODEL_NAME, modelOptions);
    
    // Function to run inference on uploaded files
    input.onchange = async function () {
        let file = input.files[0];
        // Predict
        let result = await model.predict(file);
        console.log('Result from file:', result);
        // Display result to canvas
        model.displayResultToCanvas(result, canvas);
    }
</script>

Model Options

When loading a model, you can specify various options to customize its behavior:

• inputCropPercentage: Set the percentage to crop the input image for crop-first and crop-last padding methods.

• inputLetterboxFillColor: Set the color for letterbox background.

• inputPadMethod: Set the padding method for input. Options: letterbox(default), stretch, crop-first, crop-last.

• labelBlacklist: Specify labels to exclude.

• labelWhitelist: Specify labels to include.

• outputConfidenceThreshold: Set the confidence threshold for outputs.

• outputMaxClassesPerDetection: Set the maximum number of classes per detection.

• outputMaxDetections: Set the maximum number of detections.

• outputMaxDetectionsPerClass: Set the maximum number of detections per class.

• outputNmsThreshold: Set the non-maximum suppression threshold.

• outputPoseThreshold: Set the pose threshold.

• outputPostprocessType: Set the post-process type.

• outputTopK: Set the top K results to output.

• outputUseRegularNms: Use regular non-maximum suppression.

• overlayAlpha: Set the transparency of the overlay.

• overlayColor: Set the color for overlay. Can be one RGB color or an array of RGB colors.

• overlayFontScale: Set the font scale for overlay text.

• overlayLineWidth: Set the width of the overlay lines.

• overlayShowLabels: Show or hide labels in the overlay.

• overlayShowProbabilities: Show or hide probabilities in the overlay.

• saveModelImage: Flag to enable attaching the model input image to the results.

• autoScaleDrawing: Flag to enable auto-scaling of the inference results drawn by displayResultToCanvas() based on a target display resolution. By default, it uses a reference size of 1920x1080 for optimal viewing. If targetDisplayWidth or targetDisplayHeight are specified, they will override the defaults and be used as the reference resolution for scaling. This ensures that visual elements like labels, bounding boxes, and keypoints appear consistent across various input image sizes and display resolutions.

• targetDisplayWidth: (Optional) Set the target display width for auto-scaling the results overlay (if autoScaleDrawing is enabled). Defaults to 1920 if not specified. This allows you to customize the reference width for scaling drawn elements.

• targetDisplayHeight: (Optional) Set the target display height for auto-scaling the results overlay (if autoScaleDrawing is enabled). Defaults to 1080 if not specified. This allows you to customize the reference height for scaling drawn elements.

• deviceType: Specify the runtime and hardware for inference. See Device Management for Inference.
• measureTime: Flag to enable collection of max, min, count, and average of time taken for inference duration and preprocessing duration in milliseconds. Off by default.

To destroy / clean up a model instance, use the cleanup method:

await model.cleanup();

This will stop all running inferences and clean up resources used by the model instance.

Device Management for Inference

Both AIServerModel and CloudServerModel classes offer flexible ways to manage device types, allowing you to configure and switch between devices dynamically.

Supported Device Types

Each model has a set of SupportedDeviceTypes, which indicates the runtime/device combinations that are compatible for inference. The format for device types is "RUNTIME/DEVICE", where:

• RUNTIME refers to the AI engine or runtime used for inference (e.g., TENSORRT, OPENVINO).

• DEVICE refers to the hardware type (e.g., CPU, GPU, NPU).

AIServerModel / CloudServerModel Device Management

In the AIServerModel and CloudServerModel classes, device management is integrated into both the initialization and runtime phases of the model lifecycle. Below are key scenarios and examples:

1. Default Device Type Selection: When you load a model without specifying a device type, the default device type specified in the model parameters is selected.

   let model = await inference_manager.loadModel('your_model_name');
   console.log(model.deviceType); // Outputs: "DefaultRuntime/DefaultAgent"

2. Switching Device Types After Initialization: You can change the device type even after the model has been initialized. The model will validate the requested device type against the system’s supported device types.

   model.deviceType = 'RUNTIME2/CPU';
   console.log(model.deviceType); // Outputs: "RUNTIME2/CPU"

   If the requested device type is not valid, an error will be thrown.

3. Specifying a Device Type During Initialization: You can specify a device type when loading the model. The model will start with the specified device type if it’s available.

   let model = await inference_manager.loadModel('your_model_name', { deviceType: 'RUNTIME2/CPU' });
   console.log(model.deviceType); // Outputs: "RUNTIME2/CPU"

4. Handling Multiple Device Types: The SDK allows you to provide a list of device types. The first available option in the list will be selected.

   model.deviceType = ['RUNTIME3/CPU', 'RUNTIME1/CPU'];
   console.log(model.deviceType); // Outputs: "RUNTIME3/CPU" if available, otherwise "RUNTIME1/CPU"

5. Fallback and Error Handling: If none of the specified device types are supported, the model will throw an error, ensuring that only valid configurations are used.

   try {
       model.deviceType = ['INVALID/DEVICE', 'ANOTHER_INVALID/DEVICE'];
   } catch (e) {
       console.error('Error: Invalid device type selection');
   }

6. Supported Device Types: You can check the supported device types for a model using the supportedDeviceTypes property.

   console.log(model.supportedDeviceTypes); // Outputs: ["RUNTIME1/CPU", "RUNTIME2/CPU"]

7. System Supported Device Types You can check the system’s list of supported devices for inference using the getSupportedDevices() method of the dg_sdk class.

   let dg = new dg_sdk();
   let aiserverDevices = dg.getSupportedDevices('targetAIServerIp');
   console.log(supportedDevices); // Outputs: ["RUNTIME1/CPU", "RUNTIME2/CPU", "RUNTIME3/CPU"]
   let cloudDevices = dg.getSupportedDevices('cloud');
   console.log(supportedDevices); // Outputs: ["RUNTIME1/CPU", "RUNTIME2/CPU", "RUNTIME3/CPU"]

Device management in both AIServerModel and CloudServerModel is designed to be flexible, allowing you to fine-tune the inference environment. You can easily switch between device types, handle fallbacks, and ensure that your models are always running on supported configurations.

Measure Time Usage

Enabling the measureTime flag will create a timeStats object within the model which holds various statistics (max, min, count, average) to track how long certain operations took.

Operations Tracked

1. ImagePreprocessDuration_ms: Time taken for preprocessing the input image.

2. CorePreprocessDuration_ms: Duration of server-side pre-processing step.

3. CoreInferenceDuration_ms: Time taken for the actual inference operation on the server (between sending frame and receiving results).

4. CoreLoadResultDuration_ms: Duration of server-side data movement step.

5. CorePostprocessDuration_ms: Duration of server-side post-processing step.

6. PythonPreprocessDuration_ms: Duration of client-side pre-processing step including data loading and data conversion time.

7. FrameTotalDuration_ms: Total duration from calling the predict() or predict_batch() method to receiving the results.

8. DeviceInferenceDuration_ms: (Orca models only) Duration of AI inference computations on AI accelerator hardware (DeGirum Orca).

9. DeviceTemperature_C: (Orca models only) Internal temperature of AI accelerator hardware in Celsius (DeGirum Orca).

10. DeviceFrequency_MHz: (Orca models only) Working frequency of AI accelerator hardware in MHz (DeGirum Orca).

Available methods

1. getTimeStats(): Use this method to return a formatted string of all the statistics collected so far.

2. resetTimeStats(): Use this method to delete all your old statistics and create a fresh timeStats object to collect more statistics with.

3. To access the timeStats object directly, you can use modelName.timeStats.stats["statName"], where the statName is one of the operations tracked.

Example usage

let model = await inference_manager.loadModel('your_model_name', { measureTime: true });
let result = await model.predict(image);
console.log(model.getTimeStats()); // Pretty print time stats

// Access client-side and server-side timing stats
let preprocessDuration = model.timeStats.stats["ImagePreprocessDuration_ms"]; // Get image preprocess duration (min, avg, max, count)
let preprocessMin = model.timeStats.stats["ImagePreprocessDuration_ms"].min; // Get min image preprocess duration

let inferenceDuration = model.timeStats.stats["CoreInferenceDuration_ms"]; // Get core inference duration (min, avg, max, count)
let inferenceMax = model.timeStats.stats["CoreInferenceDuration_ms"].max; // Get max core inference duration

let frameTotalDuration = model.timeStats.stats["FrameTotalDuration_ms"]; // Get total time taken for the entire frame processing

let deviceTemp = model.timeStats.stats["DeviceTemperature_C"]; // Get device temperature if available

model.resetTimeStats(); // Reset time stats

API Reference

For detailed information on the SDK's classes, methods, and properties, refer to the API Reference.