Handling Fix Results for Streaming in Guardrails
Overview
This document describes how Guardrails handles fix results for streaming, addressing the challenges that arise from the new streaming architecture implemented to reduce latency and cost of validation.
Fix Actions
Fix actions are a feature in Guardrails that allow you to specify an action to take when a particular validator fails. Some validators support a "FIX" on_fail action, which programmatically corrects faulty LLM output.
Examples
-
Detect PII Validator:
Input: John lives in San Francisco.
Output: <PERSON> lives in <LOCATION> -
Lowercase Validator:
Input: JOHN lives IN san FRANCISCO.
Output: john lives in san francisco.
In non-streaming scenarios, fix results from one validator can be piped into subsequent validators.
Challenges with Streaming
The new streaming architecture allows validators to specify how much context to accumulate before running validation. However, this poses challenges for fix actions:
- Validators cannot run in sequence due to different chunk thresholds.
- Each validator accumulates chunks independently.
- Validators are unaware of fixes applied by other validators.
Solution: Merging Algorithm
To address these challenges, Guardrails implements the following solution:
- Wait for all validators to accumulate enough chunks to validate and output a fix value.
- Run a merging algorithm on the fixes from all validators.
Merging Process
The merging algorithm combines fix outputs from multiple validators. For example:
LLM Output: JOE is FUNNY and LIVES in NEW york
PII Fix Output: <PERSON> is FUNNY and lives in <LOCATION>
Lowercase Fix: joe is funny and lives in new york
Merged Output: <PERSON> is funny and lives in <LOCATION>
Implementation Details
- The merging algorithm is a modified version of the
three-merge
package. - It uses Google's
diff-match-patch
algorithm under the hood.
Limitations and Edge Cases
While the merging algorithm works well for most cases, there are some limitations:
- Edge cases may occur when replacement ranges overlap between multiple validators.
- In rare instances, the merging algorithm might produce unexpected results.
Reporting Issues
If you encounter any bugs related to stream fixes, please:
- File an issue on the Guardrails GitHub repository.
- Mention @nichwch in the issue description.