Check whether an LLM response contains PII (Personally Identifiable Information)

Using the PIIFilter validator

This is a simple check that looks for the presence of a few common PII patterns It is not intended to be a comprehensive check for PII and to be a quick check that can be used to filter out responses that are likely to contain PII. It uses the Microsoft Presidio library to check for PII.

# Install the necessary packages
 pip install presidio-analyzer presidio-anonymizer -q
 python -m spacy download en_core_web_lg -q

    
    [notice] A new release of pip is available: 23.0.1 -> 23.3.1
    [notice] To update, run: pip install --upgrade pip
    /home/zayd/workspace/guardrails/.venv/lib/python3.9/site-packages/torch/cuda/__init__.py:611: UserWarning: Can't initialize NVML
      warnings.warn("Can't initialize NVML")
    
    [notice] A new release of pip is available: 23.0.1 -> 23.3.1
    [notice] To update, run: pip install --upgrade pip
    ✔ Download and installation successful
    You can now load the package via spacy.load('en_core_web_lg')

# Import the guardrails package
import guardrails as gd
from guardrails.validators import PIIFilter
from rich import print

    /home/zayd/workspace/guardrails/.venv/lib/python3.9/site-packages/torch/cuda/__init__.py:611: UserWarning: Can't initialize NVML
      warnings.warn("Can't initialize NVML")

# Create Guard object with this validator
# One can specify either pre-defined set of PII or SPI (Sensitive Personal Information) entities by passing in the `pii` or `spi` argument respectively.
# It can be passed either durring intialization or later through the metadata argument in parse method.

# One can also pass in a list of entities supported by Presidio to the `pii_entities` argument.
guard = gd.Guard.from_string(
    validators=[PIIFilter(pii_entities="pii", on_fail="fix")],
    description="testmeout",
)

    nlp_engine not provided, creating default.
    configuration file /home/zayd/workspace/guardrails/.venv/lib/python3.9/site-packages/conf/default.yaml not found.  Using default config: {'nlp_engine_name': 'spacy', 'models': [{'lang_code': 'en', 'model_name': 'en_core_web_lg'}]}.
    configuration file is missing 'ner_model_configuration'. Using default
    model_to_presidio_entity_mapping is missing from configuration, using default
    low_score_entity_names is missing from configuration, using default
    labels_to_ignore is missing from configuration, using default
    Created NLP engine: spacy. Loaded models: ['en']
    registry not provided, creating default.
    Loaded recognizer: UsBankRecognizer
    Loaded recognizer: UsLicenseRecognizer
    Loaded recognizer: UsItinRecognizer
    Loaded recognizer: UsPassportRecognizer
    Loaded recognizer: UsSsnRecognizer
    Loaded recognizer: NhsRecognizer
    Loaded recognizer: SgFinRecognizer
    Loaded recognizer: AuAbnRecognizer
    Loaded recognizer: AuAcnRecognizer
    Loaded recognizer: AuTfnRecognizer
    Loaded recognizer: AuMedicareRecognizer
    Loaded recognizer: InPanRecognizer
    Loaded recognizer: CreditCardRecognizer
    Loaded recognizer: CryptoRecognizer
    Loaded recognizer: DateRecognizer
    Loaded recognizer: EmailRecognizer
    Loaded recognizer: IbanRecognizer
    Loaded recognizer: IpRecognizer
    Loaded recognizer: MedicalLicenseRecognizer
    Loaded recognizer: PhoneRecognizer
    Loaded recognizer: UrlRecognizer
    Loaded recognizer: SpacyRecognizer
    nlp_engine not provided, creating default.
    configuration file /home/zayd/workspace/guardrails/.venv/lib/python3.9/site-packages/conf/default.yaml not found.  Using default config: {'nlp_engine_name': 'spacy', 'models': [{'lang_code': 'en', 'model_name': 'en_core_web_lg'}]}.
    configuration file is missing 'ner_model_configuration'. Using default
    model_to_presidio_entity_mapping is missing from configuration, using default
    low_score_entity_names is missing from configuration, using default
    labels_to_ignore is missing from configuration, using default
    Created NLP engine: spacy. Loaded models: ['en']
    registry not provided, creating default.
    Loaded recognizer: UsBankRecognizer
    Loaded recognizer: UsLicenseRecognizer
    Loaded recognizer: UsItinRecognizer
    Loaded recognizer: UsPassportRecognizer
    Loaded recognizer: UsSsnRecognizer
    Loaded recognizer: NhsRecognizer
    Loaded recognizer: SgFinRecognizer
    Loaded recognizer: AuAbnRecognizer
    Loaded recognizer: AuAcnRecognizer
    Loaded recognizer: AuTfnRecognizer
    Loaded recognizer: AuMedicareRecognizer
    Loaded recognizer: InPanRecognizer
    Loaded recognizer: CreditCardRecognizer
    Loaded recognizer: CryptoRecognizer
    Loaded recognizer: DateRecognizer
    Loaded recognizer: EmailRecognizer
    Loaded recognizer: IbanRecognizer
    Loaded recognizer: IpRecognizer
    Loaded recognizer: MedicalLicenseRecognizer
    Loaded recognizer: PhoneRecognizer
    Loaded recognizer: UrlRecognizer
    Loaded recognizer: SpacyRecognizer

# Parse the text
text = "My email address is demo@lol.com, and my phone number is 1234567890"
output = guard.parse(
    llm_output=text,
)

# Print the output
print(output)

Entity DOMAIN_NAME doesn't have the corresponding recognizer in language : en

ValidationOutcome(
    raw_llm_output='My email address is demo@lol.com, and my phone number is 1234567890',
    validated_output='My email address is <EMAIL_ADDRESS>, and my phone number is <PHONE_NUMBER>',
    reask=None,
    validation_passed=True,
    error=None
)

Here, both EMAIL_ADDRESS and PHONE_NUMBER are detected as PII.

# Let's test with passing through metadata for the same guard object
# This will take precendence over the entities passed in during initialization
output = guard.parse(
    llm_output=text,
    metadata={"pii_entities": ["EMAIL_ADDRESS"]},
)

# Print the output
print(output)

ValidationOutcome(
    raw_llm_output='My email address is demo@lol.com, and my phone number is 1234567890',
    validated_output='My email address is <EMAIL_ADDRESS>, and my phone number is 1234567890',
    reask=None,
    validation_passed=True,
    error=None
)

As you can see here, only EMAIL_ADDRESS is detected as PII, and the PHONE_NUMBER is not detected as PII.

# Let's try with SPI entities
# Create a new guard object
guard = gd.Guard.from_string(
    validators=[PIIFilter(pii_entities="spi", on_fail="fix")],
    description="testmeout",
)

    nlp_engine not provided, creating default.
    configuration file /home/zayd/workspace/guardrails/.venv/lib/python3.9/site-packages/conf/default.yaml not found.  Using default config: {'nlp_engine_name': 'spacy', 'models': [{'lang_code': 'en', 'model_name': 'en_core_web_lg'}]}.
    configuration file is missing 'ner_model_configuration'. Using default
    model_to_presidio_entity_mapping is missing from configuration, using default
    low_score_entity_names is missing from configuration, using default
    labels_to_ignore is missing from configuration, using default
    Created NLP engine: spacy. Loaded models: ['en']
    registry not provided, creating default.
    Loaded recognizer: UsBankRecognizer
    Loaded recognizer: UsLicenseRecognizer
    Loaded recognizer: UsItinRecognizer
    Loaded recognizer: UsPassportRecognizer
    Loaded recognizer: UsSsnRecognizer
    Loaded recognizer: NhsRecognizer
    Loaded recognizer: SgFinRecognizer
    Loaded recognizer: AuAbnRecognizer
    Loaded recognizer: AuAcnRecognizer
    Loaded recognizer: AuTfnRecognizer
    Loaded recognizer: AuMedicareRecognizer
    Loaded recognizer: InPanRecognizer
    Loaded recognizer: CreditCardRecognizer
    Loaded recognizer: CryptoRecognizer
    Loaded recognizer: DateRecognizer
    Loaded recognizer: EmailRecognizer
    Loaded recognizer: IbanRecognizer
    Loaded recognizer: IpRecognizer
    Loaded recognizer: MedicalLicenseRecognizer
    Loaded recognizer: PhoneRecognizer
    Loaded recognizer: UrlRecognizer
    Loaded recognizer: SpacyRecognizer
    nlp_engine not provided, creating default.
    configuration file /home/zayd/workspace/guardrails/.venv/lib/python3.9/site-packages/conf/default.yaml not found.  Using default config: {'nlp_engine_name': 'spacy', 'models': [{'lang_code': 'en', 'model_name': 'en_core_web_lg'}]}.
    configuration file is missing 'ner_model_configuration'. Using default
    model_to_presidio_entity_mapping is missing from configuration, using default
    low_score_entity_names is missing from configuration, using default
    labels_to_ignore is missing from configuration, using default
    Created NLP engine: spacy. Loaded models: ['en']
    registry not provided, creating default.
    Loaded recognizer: UsBankRecognizer
    Loaded recognizer: UsLicenseRecognizer
    Loaded recognizer: UsItinRecognizer
    Loaded recognizer: UsPassportRecognizer
    Loaded recognizer: UsSsnRecognizer
    Loaded recognizer: NhsRecognizer
    Loaded recognizer: SgFinRecognizer
    Loaded recognizer: AuAbnRecognizer
    Loaded recognizer: AuAcnRecognizer
    Loaded recognizer: AuTfnRecognizer
    Loaded recognizer: AuMedicareRecognizer
    Loaded recognizer: InPanRecognizer
    Loaded recognizer: CreditCardRecognizer
    Loaded recognizer: CryptoRecognizer
    Loaded recognizer: DateRecognizer
    Loaded recognizer: EmailRecognizer
    Loaded recognizer: IbanRecognizer
    Loaded recognizer: IpRecognizer
    Loaded recognizer: MedicalLicenseRecognizer
    Loaded recognizer: PhoneRecognizer
    Loaded recognizer: UrlRecognizer
    Loaded recognizer: SpacyRecognizer

# Parse text
text = "My email address is demo@xyz.com, and my account number is 1234789012367654."

output = guard.parse(
    llm_output=text,
)

# Print the output
print(output)

ValidationOutcome(
    raw_llm_output='My email address is demo@xyz.com, and my account number is 1234789012367654.',
    validated_output='My email address is demo@xyz.com, and my account number is <US_BANK_NUMBER>.',
    reask=None,
    validation_passed=True,
    error=None
)

Here, only the US_BANK_NUMBER is detected as PII, as specified in the "spi" entities. Refer to the documentation for more information on the "pii" and "spi" entities. Obviosuly, you can pass in any Presidio-supported entities through the metadata.

# Another example
text = "My ITIN is 923756789 and my driver's license number is 87651239"

output = guard.parse(
    llm_output=text,
    metadata={"pii_entities": ["US_ITIN", "US_DRIVER_LICENSE"]},
)

# Print the output
print(output)

ValidationOutcome(
    raw_llm_output="My ITIN is 923756789 and my driver's license number is 87651239",
    validated_output="My ITIN is <US_ITIN> and my driver's license number is <US_DRIVER_LICENSE>",
    reask=None,
    validation_passed=True,
    error=None
)

In this way, any PII entity that you want to check for can be passed in through the metadata and masked by Guardrails for your LLM outputs. Of-course, like all other examples, you can integrate this into your own code and workflows through the complete Guard execution.

Check whether an LLM response contains PII (Personally Identifiable Information)

In this way, any PII entity that you want to check for can be passed in through the metadata and masked by Guardrails for your LLM outputs. Of-course, like all other examples, you can integrate this into your own code and workflows through the complete Guard execution.​

In this way, any PII entity that you want to check for can be passed in through the metadata and masked by Guardrails for your LLM outputs. Of-course, like all other examples, you can integrate this into your own code and workflows through the complete Guard execution.