Skip to main content

How to Deploy Self-Hosted Agents

In this guide, we will explore how to deploy agents on your own infrastructure using the Agents SDK.

Prerequisites​

Before you begin, make sure you have completed the Getting Started with the Agents SDK tutorial and you are familiar with the Agent Configuration Reference.

Serving the REST API​

The Agents SDK serves a REST API that allows you to interact with the agents. The API contains streaming and non-streaming endpoints for sending a conversation and receiving a response from the agent. To serve the REST API, you need to run the following command:

za agents serve "<agents project directory>"

This will start the API server on 127.0.0.1 at port 8000 by default. To override the default host, use the --host option. You can access the API documentation at http://<host>:8000/docs.

Deploying the Server​

To deploy the server on your own infrastructure, you can use any cloud provider or hosting service that supports running your own applications.

Stateful Conversations and Crash Recovery​

za agents serve runs a stateless server: every request must carry the full conversation, and a stream can only be reattached on the process that produced it, while it is still running. That is fine for a single instance behind sticky sessions.

To get stateful sessions, resume-after-crash, and stream reattachment across replicas, serve the app through your own entrypoint and wire the persistence stores into setup_app. The SDK ships file-backed stores that work for any number of processes sharing one directory; for a multi-replica deployment you implement the same interfaces on your own infrastructure. Persisting Conversations and State has the wiring example, the interface contracts, and the adapter rules.

Two operational rules follow:

  • Every replica must point at the same store backend; cross-replica resume and reattachment read the rows another replica wrote.
  • Validate the wiring with za agents perf: kill a server mid-turn and resume on the other (CLI reference).

Using your Deployed Agents in the Zeta Alpha Platform​

Once you have deployed your agents, you can use them in the Zeta Alpha Platform by updating your tenant configuration. For example you can add the following to your chat_bot_setups configuration, one entry for each agent:

{
  "bot_identifier": "<agent identifier>",
  "agent_name": "<agent name>",
  "llm_configuration_name": "",
  "serving_url": "https://<host>:<port>"
}

Replace <agent identifier>, <agent name>, <host>, and <port> with the appropriate values for your deployed agents. Note that for self-hosted agents, the llm_client_configuration will be taken from the agent_setups.json file, therefore we leave llm_configuration_name empty.

Using your Deployed Agents outside the Zeta Alpha Platform​

The REST API provided by the Agents SDK allows you to interact with the agents from any application or service. You can send a conversation to the agent and receive a response by making HTTP requests to the API endpoints.

warning

When running in production, it is recommended to secure the API endpoints with authentication and authorization mechanisms to protect your agents and data. You can achieve this by using API gateways, firewalls, and other security measures that your hosting service provides.

Alternatively, you can also extend the API server to add custom endpoints and security features.