Introduction

In today's business landscape, data privacy and security are top-of-mind concerns for enterprises. This concern is heightened by the expanding use of Large Language Models (LLMs) in customer interactions, making the need for robust strategies to manage access to Personally Identifiable Information (PII) more relevant than ever. As organizations increasingly depend on LLM services for a range of applications, the imperative to safeguard sensitive information becomes critical. In the context of Data Privacy LLM, utilizing an LLM Gateway has become an important tool for businesses to manage Personally Identifiable Information (PII) effectively. Companies are challenged to find a balance between ensuring traceability, being able to run LLM Analytics on natural language traffic, and controlling PII data. This blog post investigates various strategies to redact or anonymize PII in LLM traffic, utilizing an LLM Gateway to provide privacy and security.

Why Do We Need to Remove PII?

The primary reason for removing PII from LLM gateway logs is to protect individual privacy and comply with data protection regulations such as GDPR, CCPA, and others. PII includes any data that can be used to identify an individual, such as names, addresses, phone numbers, and social security numbers. If this information is exposed, it can lead to privacy breaches, identity theft, and other forms of cybercrime. Additionally, businesses handling PII are obligated to ensure its confidentiality, integrity, and availability, failing which can result in hefty fines and loss of reputation. For further insights, consider the example of Large Language Models And EU Data Protection: Mapping (Some) Of The Problems.

LLM Services' Approach to Privacy and Security

Providers such as Azure, which offer the OpenAI Azure service, provide strong privacy and security measures. These providers approach to Data Privacy LLM is, however, not always specifically tailored to equip users with tools for the removal or redaction of PII data (Example: https://learn.microsoft.com/en-us/legal/cognitive-services/openai/data-privacy). The offerings from the LLM API providers typically include :

• Data Privacy and Security. This aspect guarantees that customer data, including prompts (inputs) and completions (outputs), remains inaccessible to other customers, OpenAI, or for use in enhancing OpenAI models. It's also not utilized for automatically improving Azure OpenAI models unless explicitly fine-tuned with your training data. The Azure OpenAI models are stateless, ensuring that fine-tuned models are exclusively available for the customer's use.

• Abuse Monitoring and Content Filtering: Azure OpenAI Service includes features for content filtering and abuse monitoring to prevent harmful content generation. These features operate synchronously as the service processes prompts to generate content. However, for customers who require processing of sensitive, highly confidential, or legally-regulated input data, there is an option to apply for an exemption from abuse monitoring and human review.

Data used for LLM Analytics

Most companies beginning to explore generative AI and integrating it into their customer services will soon realize the importance of employing LLM DevOps methods. These methods are essential for facilitating rapid iterations and improvements. Generating valuable data for LLM Analytics is a crucial component of LLM DevOps. In this context, a data-generating LLM Gateway such as Gecholog.ai can be exceptionally effective. However, certain challenges arise, for example:

1. How can we remove PII content before the data is sent to the LLM API?

2. How to acquire comprehensive metrics while limiting the presence of PII in the LLM Analytics database?

Enhancing Data Privacy with LLM Gateway Techniques

Use Processors!

A cutting-edge data-generating LLM Gateway like Gecholog.ai offers several features for managing traffic and logs. A key feature is the "processor" concept, which facilitates the execution of custom microservices to augment or modify the payload. These processors can operate before the LLM API on the request data, after the LLM API response is returned to the requester, or as post-processing microservices to redact, mask, modify or change the final stored logs.

Image: Flow Chart showing the Execution Order of Custom Processors in the LLM Gateway Gecholog.ai

Use PII Identification Services!

By leveraging a custom processor, it's possible to integrate any external API for identifying PII data in your payload, headers, or any other request data. For instance, Azure offers a suite of tools for Personal Identifiable Information detection. Alternatively, open-source tools like spaCy or various models from Hugging Face can be employed for entity recognition. (See our docs site a simple spaCy Python entity tagger.) The custom processor will use the external PII Identification Service on the payload, redact and put it back to the flow.

Choose what Data to Store!

Another capability in Gecholog.ai is the option to filter, i.e., select which fields in the log to store (see logger section of the Gecholog.ai configuration). This functionality allows for the user to configure storage of selected metadata only, such as timestamps, tags, token consumption, system messages, etc., which are necessary for tracking performance but do not contain any PII.

Summarization of PII Redaction Techniques Using LLM Gateway

Real-time Techniques – Control the Payloads

• Blocking: Implementing a mechanism to check requests in real-time and deny any that contain PII. This proactive approach prevents sensitive data from ever reaching the LLM API.

• Redact Ingress: Running pre-processing scripts to redact, replace, mask, or anonymize PII in the payload before it is sent to the LLM API. This step ensures that the LLM service processes only data that has been sanitized.

Post-Process Techniques – Control the Logs

• Redact Post: Proceeding with the request as usual, but ensuring that the logs are redacted of any PII. This technique involves scrubbing the logs post-call to remove sensitive information.

• Store Only Metrics: An alternative approach is to remove payload data from the logs while retaining metrics and other non-PII data. This strategy aids in preserving valuable analytical data without risking individual privacy.

Conclusion

In conclusion, removing PII from LLM logs can be essential for preserving user privacy and adhering to data protection regulations. Although LLM services offer various features to assist in this process, organizations should also consider additional techniques, like those involving an LLM Gateway as described in this article, effective in both real-time and post-processing scenarios, to ensure comprehensive protection. By integrating a mix of these strategies, businesses can harness the capabilities of LLM Analytics while maintaining the utmost standards of data privacy and security.