Inference with cloud models

Run inference on a local AI server while fetching models from DeGirum’s public cloud zoo—ideal for hybrid setups where compute is local, but model access is remote.

Estimated read time: 2 minutes

This setup runs inference on local hardware (e.g., inference_host_address="localhost" or inference_host_address="<host_ip>:<port>") while fetching models from the cloud zoo (zoo_url="degirum/hailo"). It's useful when compute is local, but you want to pull models directly from DeGirum's public AI Hub.

Client and AI Server on the same host

In this case, the AI Server and client application run on the same machine.

To start the server, run:

degirum server

You should see output similar to:

DeGirum asio server is started at TCP port 8778
Local model zoo is served from '.' directory.  
Press Enter to stop the server

The server runs until you press ENTER in the terminal. By default, it listens on TCP port 8778. To specify a different port, use the --port argument.

degirum server --port <your_port>

Example ModelSpec

This ModelSpec runs inference on a local AI Server using a model fetched from DeGirum's public Hailo model zoo:

# Example ModelSpec
model_spec = ModelSpec(
    model_name="yolov8n_coco--640x640_quant_hailort_multidevice_1",
    zoo_url="degirum/hailo",
    inference_host_address="localhost:8778",
    model_properties={"device_type": ["HAILORT/HAILO8L", "HAILORT/HAILO8"]}
)

• zoo_url="degirum/hailo": loads the model from DeGirum's public Hailo model zoo.

• inference_host_address="localhost:8778": runs inference using Hailo devices managed by a local AI server.

Client and AI Server on different hosts

You can also run the AI server on a remote host. On that remote machine, launch the server:

degirum server

Expected output:

DeGirum asio server is started at TCP port 8778
Local model zoo is served from '.' directory.  
Press Enter to stop the server

The server runs until you press ENTER in the terminal. By default, it listens on TCP port 8778. To specify a different port, use the --port argument:

degirum server --port <your_port>

Example ModelSpec

This ModelSpec runs inference on a remote AI Server using a model fetched from DeGirum's public Hailo model zoo.

# Example ModelSpec
model_spec = ModelSpec(
    model_name="yolov8n_coco--640x640_quant_hailort_multidevice_1",
    zoo_url="degirum/hailo",
    inference_host_address="<host_ip>:<port>",
    model_properties={"device_type": ["HAILORT/HAILO8L", "HAILORT/HAILO8"]}
)

• zoo_url="degirum/hailo": fetches the model from DeGirum's public Hailo model zoo.

• inference_host_address="<host_ip>:<port>": runs inference using Hailo devices managed by the remote AI server.