Skip to main content

​
The problem

As a concept, guardrailing has a few areas that, when unoptimized, can introduce latency and be extremely resource-expensive. The main two areas are:
  • Guardrailing orchestration; and
  • ML models that validate a single guard
These are resource-heavy in slightly different ways. ML models can run with low latency on GPU-equipped machines. (Some ML models used for validation run in tens of seconds on CPUs, while they run in milliseconds on GPUs.) Meanwhile, guardrailing orchestration benefits from general memory and compute resources.

​
The Guardrails approach

The Guardrails library lets you separate the execution of orchestration from the execution of ML-based validation. Instead of running a validator’s ML model in the same process as your application, you can host that model behind an API and configure the validator to call that remote endpoint. The endpoint presents a unified interface that all validator models implement.
Remote validation inferencing is available in Guardrails versions 0.5.0 and above.
Guardrails previously offered hosted inference endpoints for some validators as a free preview. Those hosted endpoints are being shut down on August 6, 2026. Run validator models locally (the default), or host your own endpoint as described below.

​
Running a validator’s model locally (default)

By default, a validator runs its ML model in-process. Install the validator and use it directly:

​
Hosting your own endpoint

For higher throughput — or to keep heavy ML models off your application hosts — run a validator’s model behind your own inference endpoint. Any endpoint that implements the interface Guardrails validators expect will work; that interface is defined in the _inference_remote method of the validator. After implementing the interface, host it (for example, using gunicorn and Flask) and point the validator at it with the validation_endpoint constructor argument:
Hosting the model separately increases speed and throughput compared to running locally, which makes use cases such as streaming more viable in production.

​
Learn more

To learn more about hosting your own validator models, see the Host Remote Validator Models doc. To learn more about writing your own validators, see the Custom validators doc.