Privacy Screen

• 1: Server Setup
• 2: Connecting to the Server
• 3: Text Redaction
• 4: Concurrency
• 5: Privacy Screen Client
• 6: Redaction Categories
• 7: Redaction Languages
• 8: Prerequisites and System Requirements
• 9: Proto API Reference

1 - Server Setup

Installing Cobalt Privacy Screen

Cobalt distributes a docker-compose file that orchestrates three docker images, one for each of the following services:

• Privacy Screen Server (frontend for accepting text / audio streams)
• Transcribe Server (for recognizing text in audio files)
• Redaction Backend Engine (for redacting text data)

Having these components as separate images facilitates large deployments where each image can be auto-scaled independently based on request traffic.

Installing Server

1. Contact Cobalt to get a link to the image files in AWS S3 and the docker-compose configuration file. This link will expire in two weeks, so be sure to download the file to your own server.

2. Download with the AWS CLI if you have it, or with curl:

   URL="the url sent by Cobalt"
   FILE_NAME="name you want to give the file (should end with the same extension as the url, usually tar.bz2)"
   curl $URL -L -o $FILE_NAME
   

3. Untar the file, and load the docker images. The tar file will also contain the docker-compose.yaml file.

   tar -xvjf $FILE_NAME -C ./
   docker load < *.bz2
   

4. Copy the cobalt license file into the server folder

5. Copy the deid license file into the server folder

6. Start the services using docker-compose:

   docker-compose up --build
   

The server will be running in the container and listening on port 2728 for gRPC requests from clients.

2 - Connecting to the Server

Once you have the Cobalt Privacy Screen server up and running, you are ready to create a client connection.

First, you need to know the address (host:port) where the server is running. This document will assume the values 127.0.0.1:9002, but these can be replaced with your server address in actual code.

Default Connection

The following code snippet connects to the server and queries its version. It uses our recommended default setup, expecting the server to be listening on a TLS encrypted connection.

• Go
• Python

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/cobaltspeech/sdk-trifid/grpc/go-trifid"
)

const serverAddr = "127.0.0.1:9002"

func main() {
	client, err := trifid.NewClient(serverAddr)
	if err != nil {
		log.Fatal(err)
	}

	// Be sure to close the client when we are done with it.
	defer client.Close()
}

import trifid

client = trifid.Client(server_address="localhost:9002")

Insecure Connection

It is sometimes required to connect to Privacy Screen server without TLS enabled (during debugging, for example). Note that if the server has TLS enabled, attempting to connect with an insecure client will fail.

To create an insecure connection, do the following when creating the client:

• Go
• Python

client, err := trifid.NewClient(serverAddr, trifid.WithInsecure())

client = trifid.Client(server_address="localhost:9002", insecure=True)

Client Authentication

In our recommended default setup, TLS is enabled in the gRPC setup, and when connecting to the server, clients validate the server’s SSL certificate to make sure they are talking to the right party. This is similar to how “https” connections work in web browsers.

In some setups, it may be desired that the server should also validate clients connecting to it and only respond to the ones it can verify. If your Privacy Screen server is configured to do client authentication, you will need to present the appropriate certificate and key when connecting to it.

Please note that in the client-authentication mode, the client will still also verify the server’s certificate, and therefore this setup uses mutually authenticated TLS. This can be done with:

• Go
• Python

// certPem and keyPem are the bytes of the client certificate and key
// provided to you.
client, err := trifid.NewClient(serverAddr, trifid.WithClientCert(certPem, keyPem))

# cert_pem and key_pem are the contents of the client certificate and key
# provided to you.
client = trifid.Client(server_address="localhost:9002", client_certificate=cert_pem client_key=key_pem)

Server Information

The client provides two methods to get information about the server - Version and ListModels.

Version

The Version method provides information about the version of the Privacy Screen server the client is connected to, as well as information about other relevant services and packages the server uses.

• Go
• Python

// Request the server version info
ver, err := client.Version(context.Background())
fmt.Printf("Server Version: %v\n", ver)

# Request the server version info
ver = client.version()
print(f"Server Version: {ver}")

List Models

The ListModels method fetches a list of models available on the Privacy Screen server. On the server side, the models are specified as part of the server’s config file.

• Go
• Python

// Request the list of models
modelList, err := client.ListModels(context.Background())
fmt.Printf("Available Models:\n")
for _, mdl := range modelList.Models {
	fmt.Printf("  ID: %v\n", mdl.Id)
	fmt.Printf("    Name: %v\n", mdl.Name)
	fmt.Printf("    Redaction Classes: %v\n", mdl.RedactionClasses)
}

# Request the list of models
model_list = client.list_models()
print("Available Models:")
for mdl in model_list:
    print(f"  ID: {mdl.id}")
    print(f"    Name: {mdl.name}")
    print(f"    Redaction Classes: {mdl.redaction_classes}")

3 - Text Redaction

TODO

4 - Concurrency

The recommended level of concurrency, i.e. the optimal number of simultaneous requests to make to the container is covered below for the CPU and GPU containers. The recommended concurrency level is driven primarily by the compute requirement of the Neural Network models, such as for PII detection.

CPU

For Neural Network inference workloads, CPUs don’t require inputs to be batched together to achieve good hardware utilization. In practice, due to network overhead and pre/post-processing code it is best to use a low level of concurrency such as 2 per container instance. If latency isn’t a concern, a value of 32 is recommended.

GPU

Unlike CPUs, GPUs require inputs to be batched together and processed as a single large input to achieve optimal hardware utilization. This means that there is a tradeoff between latency and throughput. A concurrency level of 32 per container instance is a good tradeoff between latency and throughput, however concurrency levels as low as 8 do not significantly impact throughput. If latency isn’t a concern, a value of 128 will ensure maximum hardware utilization.

5 - Privacy Screen Client

This release includes a PrivacyScreen client that can be used to quickly send audio/transcripts to the server. This reads audio files in WAV (PCM16SLE) format and transcripts in the JSON format, you can see examples for txt and json files:

• txt
• json

Jack and Jill went up to 224 North Hill drive to fetch a pail of water. Jack fell down broke his crown and Jill called 4125555555.

{
    "utterances": [
        {
            "start_time_ms": 30,
            "duration_ms": 4230,
            "audio_channel": 0,
            "words": [
                {
                    "start_time_ms": 30,
                    "duration_ms": 390,
                    "text": "Jack"
                },
                {
                    "start_time_ms": 420,
                    "duration_ms": 120,
                    "text": "and"
                },
                {
                    "start_time_ms": 540,
                    "duration_ms": 240,
                    "text": "Jill"
                },
                {
                    "start_time_ms": 780,
                    "duration_ms": 240,
                    "text": "went"
                },
                {
                    "start_time_ms": 1020,
                    "duration_ms": 150,
                    "text": "up"
                },
                {
                    "start_time_ms": 1170,
                    "duration_ms": 60,
                    "text": "to"
                },
                {
                    "start_time_ms": 1230,
                    "duration_ms": 1080,
                    "text": "224"
                },
                {
                    "start_time_ms": 2310,
                    "duration_ms": 300,
                    "text": "North"
                },
                {
                    "start_time_ms": 2610,
                    "duration_ms": 150,
                    "text": "Hill"
                },
                {
                    "start_time_ms": 2760,
                    "duration_ms": 300,
                    "text": "drive"
                },
                {
                    "start_time_ms": 3060,
                    "duration_ms": 90,
                    "text": "to"
                },
                {
                    "start_time_ms": 3150,
                    "duration_ms": 270,
                    "text": "fetch"
                },
                {
                    "start_time_ms": 3420,
                    "duration_ms": 60,
                    "text": "a"
                },
                {
                    "start_time_ms": 3480,
                    "duration_ms": 270,
                    "text": "pail"
                },
                {
                    "start_time_ms": 3750,
                    "duration_ms": 120,
                    "text": "of"
                },
                {
                    "start_time_ms": 3870,
                    "duration_ms": 390,
                    "text": "water."
                }
            ]
        },
        {
            "start_time_ms": 9300,
            "duration_ms": 5324,
            "audio_channel": 1,
            "words": [
                {
                    "start_time_ms": 9300,
                    "duration_ms": 420,
                    "text": "Jack"
                },
                {
                    "start_time_ms": 9720,
                    "duration_ms": 210,
                    "text": "fell"
                },
                {
                    "start_time_ms": 9930,
                    "duration_ms": 420,
                    "text": "down"
                },
                {
                    "start_time_ms": 10410,
                    "duration_ms": 270,
                    "text": "broke"
                },
                {
                    "start_time_ms": 10680,
                    "duration_ms": 150,
                    "text": "his"
                },
                {
                    "start_time_ms": 10830,
                    "duration_ms": 450,
                    "text": "crown"
                },
                {
                    "start_time_ms": 11310,
                    "duration_ms": 180,
                    "text": "and"
                },
                {
                    "start_time_ms": 11490,
                    "duration_ms": 210,
                    "text": "Jill"
                },
                {
                    "start_time_ms": 11700,
                    "duration_ms": 330,
                    "text": "called"
                },
                {
                    "start_time_ms": 12030,
                    "duration_ms": 2594,
                    "text": "4125555555."
                }
            ]
        }
    ]
}

Examples of client calls

There are several ways the client interacts with the server. These examples are always run from the same path as the client binary is. When in doubt, run ./privacy-screen-gprc-client -h to get more information the parameters needed to run the client.

Redact Text

./privacy-screen-grpc-client redact-text \
        --insecure \
        --model-id general \
        --input-text input.txt \
        --output-result redacted_token.json

Redact Transcript

./privacy-screen-grpc-client redact-transcript \
        --insecure \
        --model-id general \
        --input-transcript testdata/input.json \
        --output-transcript redacted_output.json

Redact Transcribed Audio

./privacy-screen-grpc-client redact-transcribed-audio \
        --insecure \
        --model-id general \
        --input-audio testdata/input.wav \
        --input-transcript testdata/input.json \
        --output-audio redacted_output.wav \
        --output-transcript redacted_output.json \
        --timeout 5m

Transcribe and Redact

./privacy-screen-grpc-client transcribe-and-redact \
        --insecure \
        --model-id en_US \
        --input-audio testdata/input.wav \
        --output-audio redacted_output.wav \
        --output-transcript redacted_output.json \
        --output-unredacted-transcript unredacted_output.json \
        --timeout 5m

6 - Redaction Categories

Personally Identifiable Information (PII)

Protected Health Information (PHI)

Payment Credit Industry Information (PCI)

Beta Entity Types

Note that Beta support for the following entity types is currently only available with our English models.

International Entity Mapping

In the tables below, you can find localized variants of our entity types. For each entity type, there is a description, an example, and the label under which the entity falls. This section does not include entity types that may vary regionally, but still directly correspond to one of the entities listed above (e.g., PHONE_NUMBER, PASSPORT_NUMBER, DRIVER_LICENSE, LOCATION_ADDRESS, CREDIT_CARD_EXPIRATION). The following numbers are commonly used across many countries, hence are not included in each country’s table: GST (Goods and Services Tax), HST (Harmonized Sales Tax). These numbers are redacted as NUMERICAL_PII.

Asia Pacific

Australia

China

India

Japan

Korea

New Zealand

Philippines

Europe

Belgium

Germany

France

Italy

Netherlands

Portugal

Russia

Spain

Ukraine

United Kingdom

North and South America

Brazil

Canada

Mexico

United States

7 - Redaction Languages

Cobalt features core support for 14 languages and extended support for 39 additional languages, with core languages featuring the highest level of performance. The complete list of supported languages below details which languages have core support, which have extended or beta support, and which are upcoming additions. New languages are continually being added, please contact us if you require a language not in the list below.

In addition to supporting 50+ languages, Cobalt offers support for regional language varieties in recognition of the large differences in vocabulary and grammar that can exist in the same language when spoken in different regions. So far, this includes support for varieties of English (US, UK, Canada and Australia), Spanish (Spain and Mexico), French (France and Canada), and Portuguese (Portugal and Brazil). Cobalt also supports code-switching, or mixing of different languages. This means that, in a phrase such as J’ai payé 76,88RM por ein Haarschnitt da 范玉菲 habang ko ay nasa Україна, multilingual PII is accurately de-identified. The selection of supported regional language varieties is continually being expanded, please let us know if there is a specific request.

Cobalt’s supported entity types function across each supported language, with multilingual equivalents of different PII (Personally Identifiable Information) entities, PHI (Protected Health Information) entities, and PCI (Payment Card Industry) entities being detected in each language. Our Supported Entity Types page provides a more detailed look at our coverage of language and region-specific entity equivalents. The solution is also sensitive to cross-linguistic differences in how names are structured, how place names are referred to, and how monetary units are described in different languages, among other differences.

Core Support

Extended Support

8 - Prerequisites and System Requirements

Prerequisites

The following prerequisites are required to run the container:

• Container engine, such as Docker (can be installed using the official instructions )
• (GPU only) Nvidia Container Toolkit with Nvidia driver version 515 or higher (can be installed using the following installation guide )

All other dependencies, such as CUDA are included with the container and don’t need to be installed separately.

System Requirements

• Docker and docker-compose installed.

The image comes in two different build flavours:

• A compact, CPU-only container that runs on any Intel or AMD CPU and a container with GPU acceleration. The CPU container is highly optimised for the majority of use cases, as the container uses hand-coded AMX/AVX2/AVX512/AVX512 VNNI instructions in conjunction with Neural Network compression techniques to deliver a ~25X speedup over a reference transformer-based system.
• A GPU container is designed for large-scale deployments making billions of API calls or processing terabytes of data per month.

Minumum Requirements

The minimum system requirements for the container image are as follows:

Recommended Requirements

While CPU-based container will run on any x86-compatible instance, the below cloud instance types give optimal throughput and latency per dollar:

• In the event when a lower latency is required, the instance type should be scaled; e.g. using an M7i.xlarge in place of a M7i.large. While the Cobalt docker solution can make use of all available CPU cores, it delivers best throughput per dollar using a single CPU core machine. Scaling CPU cores does not result in a linear increase in performance.

Similarly for the GPU-based image, it is recommended the following Nvidia T4 GPU-equipped instance types:

9 - Proto API Reference

The API is defined as a protobuf spec, so native bindings can be generated in any language with gRPC support. We recommend using buf to generate the bindings.

This section of the documentation is auto-generated from the protobuf spec. The service contains the methods that can be called, and the “messages” are the data structures (objects, classes or structs in the generated code, depending on the language) passed to and from the methods.

TrifidService

Service that implements the Cobalt Trifid Redaction Engine API.

Version

Version(VersionRequest) VersionResponse

Returns version information from the server.

ListModels

ListModels(ListModelsRequest) ListModelsResponse

ListModels returns information about the models the server can access.

RedactText

RedactText(RedactTextRequest) RedactTextResponse Redact text using a redaction engine that is configured with the provided redaction configuration.

RedactTranscript

RedactTranscript(RedactTranscriptRequest) RedactTranscriptResponse

redacts transcript using a redaction engine that is configured with the provided redaction configuration.

StreamingRedactTranscribedAudio

StreamingRedactTranscribedAudio(StreamingRedactTranscribedAudioRequest) StreamingRedactTranscribedAudioResponse

Performs bidirectional streaming redaction on transcribed audio. Receive redacted audio while sending audio. The transcription of audio data must be ready before sending the audio.

StreamingTranscribeAndRedact

StreamingTranscribeAndRedact(StreamingTranscribeAndRedactRequest) StreamingTranscribeAndRedactResponse

Performs bidirectional streaming speech recognition and redaction. Receive redacted audio and transcriptions while sending audio.

Messages

• If two or more fields in a message are labeled oneof, then each method call using that message must have exactly one of the fields populated
• If a field is labeled repeated, then the generated code will accept an array (or struct, or list depending on the language).

ListModelsRequest

The top-level message sent by the client for the ListModels method.

ListModelsResponse

The message returned to the client by the ListModels method.

Fields

• models (ModelInfo repeated) List of models available for use on Trifid server.

ModelInfo

Description of a Trifid Model

Fields

• id (string ) Unique identifier of the model. This identifier is used to choose the model that should be used for recognition, and is specified in the RedactionConfig message.

• name (string ) Model name. This is a concise name describing the model, and may be presented to the end-user, for example, to help choose which model to use for their recognition task.

• redaction_classes (string repeated) List of supported redaction classes.

RedactTextRequest

The top-level message sent by the client for the ListModels method.

Fields

• redaction_config (RedactionConfig)
• text (string ) Unique identifier of the model. This identifier is used to choose the model that should be used for recognition, and is specified in the RedactionConfig message.

RedactionConfig

Configuration for setting up a redaction engine.

Fields

• model_id (string) Unique identifier of the model to use, as obtained from a ModelInfo message.
• redaction_classes (string repeated) List of whitelisted redaction classes. If the list is empty, server default redaction class list will be considered.
• disable_streaming (bool ) This is an optional field. If this is set to true, Cobalt Privacy Screen will redact entire transcript at once, by doing so, redaction accuracy will increase at the cost of higher latency. If set to false, Cobalt Privacy Screen will redact one utterance at a time and return the result as soon as possible. The default is false.
• custom_classes (CustomClasses repeated) This is an optional field. If set, then provided list will be used to extend the list of redaction classes.

CustomClasses

CustomClass allows the client to define a new redaction class. Patterns defined here will override default redaction class for matching tokens.

Fields

• redaction_class string This is name of the new redaction class. For example, this could be “COMPANY_NAME”.
• pattern string Pattern defines a Python regex expression that would be used to identify tokens in text that get redacted to this new redaction class. For example, “COBALT|GOOGLE|MICROSOFT”, or more complex patterns such as “^COMPANY-[\d]{4}$”.

ListModelsResponse

The message returned to the client by the ListModels method.

Fields

• models (ModelInfo repeated) List of models available for use on Trifid server.

ModelInfo

Description of a Trifid Model

Fields

• id (string ) Unique identifier of the model. This identifier is used to choose the model that should be used for recognition, and is specified in the RedactionConfig message.

• name (string ) Model name. This is a concise name describing the model, and may be presented to the end-user, for example, to help choose which model to use for their recognition task.

• redaction_classes (string repeated) List of supported redaction classes.

RedactTranscriptRequest

The top-level messages sent by the client for the RedactTranscript method. Contains redaction config and a transcription to redact.

Fields

• config (RedactionConfig ) redaction config

• transcript (Transcript ) transcription to redact

RedactTranscriptResponse

The top-level message sent by the server for the RedactTranscript method. Contains redacted transcript.

Fields

• transcript (Transcript )

RedactionConfig

Configuration for setting up a redaction engine.

Fields

• model_id (string ) Unique identifier of the model to use, as obtained from a ModelInfo message.

• redaction_classes (string repeated) List of whitelisted redaction classes. If the list is empty, server default redaction class list will be considered.

• disable_streaming (bool ) This is an optional field. If this is set to true, Trifid will redact entire transcript at once, by doing so, redaction accuracy will increase at the cost of higher latency. If set to false, Trifid will redact one utterance at a time and return the result as soon as possible. The default is false.

Transcript

Transcript contains multiple utterance of the audio.

Fields

• utterances (Utterance repeated)

Utterance

Utterance of the audio

Fields

• text (string ) Text representing the utterance of the audio.

• audio_channel (uint32 ) Channel of the audio file associate with this utterance. Channels are 0-indexed, so the for mono audio data, this value will always be 0.

• start_time_ms (uint64 ) Time offset in milliseconds relative to the beginning of audio corresponding to the start of this utterance.

• duration_ms (uint64 ) Duration in milliseconds of the current utterance in the audio.

• asr_confidence (double ) ASR confidence estimate between 0 and 1. A higher number represents a higher likelihood of the output being correct.

• words_info (WordInfo repeated) Word-level information corresponding to the utterance. This field contains word-level timestamps, which are essential as input for audio redaction. This field is only available in an output utterance if enable_word_info was set to true in the RedactionConfig.

StreamingRedactTranscribedAudioRequest

The top-level messages sent by the client for the StreamingRedactTranscribedAudio method. In this streaming call, multiple StreamingRedactTranscribedAudioRequest messages should be sent. The first message must contain a RedactTranscribedAudioConfig message only and all subsequent messages must contain audio data only.

Fields

• oneof request.config (RedactTranscribedAudioConfig)
• oneof request.audio (bytes)

RedactTranscribedAudioConfig

Configuration for setting up a StreamingRedactTranscribedAudio method.

Fields

• redaction_config(RedactionConfig) Text redaction config.
• transcript(Trancript) Transcription of the entire audio. This must be ready before sending the audio.

StreamingRedactTranscribedAudioResponse

The top-level message sent by the server for the StreamingRedactTranscribedAudio method. In this streaming call, multiple StreamingRedactTranscribedAudioResponse messages contain either Utterance or redacted audio data will be returned.

Fields

• oneof result.utterance (Utterance)
• oneof result.audio (bytes)

StreamingTranscribeAndRedactRequest

The top-level messages sent by the client for the StreamingTranscribeAndRedact method. In this streaming call, multiple StreamingTranscribeAndRedactRequest messages should be sent. The first message must contain a TranscribeAndRedactConfig message only and all subsequent messages must contain audio data only.

Fields

• oneof request.config (TranscribeAndRedactConfig)
• oneof request.audio (bytes)

TranscribeAndRedactConfig

Configuration for setting up a StreamingTranscribeAndRedact method.

Fields

• redaction_config(RedactionConfig) Text redaction config.
• enable_unredacted_transcript(bool) This is an optional field. If this is set to true, each utterance result will include unredacted utterance. If set to false, no unredacted utterance will be returned. The default is false.

StreamingTranscribeAndRedactResponse

The top-level message sent by the server for the StreamingTranscribeAndRedact method. In this streaming call, multiple StreamingTranscribeAndRedactResponse messages contain either TranscribeAndRedactUtterance or redacted audio data will be returned.

Fields

• oneof result.utterance (Utterance)
• oneof result.audio (bytes)

VersionRequest

The top-level message sent by the client for the Version method.

VersionResponse

The top-level message sent by the server for the Version method.

Fields

• version (string ) Version of the server handling these requests.

WordInfo

Word level details for words in a utterance.

Fields

• text (string ) The actual word corresponding to the utterance.

• asr_confidence (double ) ASR confidence estimate between 0 and 1. A higher number represents a higher likelihood that the word was correctly recognized.

• start_time_ms (uint64 ) Time offset in milliseconds relative to the beginning of audio received by the recognizer and corresponding to the start of this spoken word.

• duration_ms (uint64 ) Duration in milliseconds of the current word in the spoken audio.

• is_redacted (bool ) If this is set to true, it denotes that the curent word is redacted word or an original word of a redacted word.

• redaction_class (string ) Recognized redaction class. This is available only if the current word is a redacted word.

• redaction_confidence (double ) Redactio confidence estimate between 0 and 1. A higher number represents a higher likelihood that the word was correctly recognized. This is available only if the current word is a redacted word.

Enums

Scalar Value Types