Voice Intelligence

• 1: Privacy Screen
• 1.1: Server Setup
• 1.2: Connecting to the Server
• 1.3: Text Redaction
• 1.4: Concurrency
• 1.5: Privacy Screen Client
• 1.6: Redaction Categories
• 1.7: Redaction Languages
• 1.8: Prerequisites and System Requirements
• 1.9: Proto API Reference
• 2: VoiceBio
• 2.1: Getting Started
• 2.2: Generating SDKs
• 2.3: Connecting to the Server
• 2.4: Streaming Enrollment
• 2.5: Streaming Verification
• 2.6: Streaming Identification
• 2.7: Comparing Voiceprints
• 2.8: Vectorizing Voiceprints
• 2.9: API Reference
• 2.10:

1 - Privacy Screen

Demo

Cobalt’s Privacy Screen engine can redact various categories of sensitive information automatically from text and audio. Every business that collects or deals with personal data should redact sensitive information in order to protect customer privacy, comply with laws and regulations, and discover new business opportunities.

Privacy Screen makes audio and text redaction possible in real-time with the advantage of our low latency and accurate speech recognition engine, Transcribe, combined with a robust redaction backend engine that identifies several types of sensitive or confidential information. There are several categories:

• Personally Identifying Information (PII) such as names, addresses, phone numbers etc.
• Protected Health Information (PHI) such as medical conditions, injuries, names of medication etc.
• Payment Card Industry (PCI) such as credit card and bank details.

A detailed list of all the categories that are identified by Privacy Screen can be found here.

How does redaction work

Sensitive information redaction typically works as a two step process. First, a machine learning model detects and classifies the desired entities in the text. Then, this classification is used to determine if the entity needs to be redacted, and if it does, the entity is replaced with an entity label in the redacted transcript. Currently, Cobalt uses state-of-the-art deep neural network (DNN) model for PII, PHI and PCI redaction.

There are three different options of using Cobalt’s redaction solution:

• Redact PII from a text transcript
• Redact PII from an audio file
• Redact PII from an audio file with a text transcript

Each of these services can be used in two operating modes:

• Streaming mode: Redaction will run utterance by utterance, and output will be streamed out as soon as the result is ready.
• Batch mode: All input audio/transcript will be processed, redacted in one batch and the output will be available at the end of the process.

Redact PII from a text transcript

In this use case, you can identify and redact sensitive PII from an input text transcript. Detected PII entities are replaced with an appropriate PII token in the redacted text transcript. Both the input and redacted transcripts are specified as JSON with a list of utterances. Each utterance has a list of words that has:

• Text
• Redaction class
• Redaction confidence score

You can specify the desired redaction classes applicable for your use case in the config file.

Redact PII from an audio file

In this case, the input audio file is first transcribed using Cobalt’s transcribe API and then text redaction is applied on the ASR generated transcript. Detected PII entities are replaced with an appropriate PII token in the redacted text transcript. In the output, you can get:

• Redacted text transcript
• Unredacted text transcript
• Redacted audio file where the PII has been masked with a beep sound The redacted text transcript contains a redaction confidence score, ASR confidence score, and associated starting and ending timestamps for each utterance and/or word.

Redact PII from both an audio file with a text transcript

In this use case, an audio file and associated transcript is given as input in order to get the redacted transcript and redacted audio file as output. The input transcript should be specified as JSON with a list of utterances:

• Each utterance has:
  • Audio Channel in the audio file. Indexed from 0
  • A list of words. Each word has:
    • Text
    • Timestamp in the audio file where this word starts (in milliseconds)
    • Duration of this word in the audio file (in milliseconds)

Output transcript has the same format as the input, except each word has extra fields such as “redaction_class”, “redaction_confidence”, and “is_redacted”.

Text Redaction

Here is an example of text redaction:

System requirements

Minimum requirements

Recommended requirements for CPU container

Recommended requirements for GPU container

1.1 - Server Setup

Installing Cobalt Privacy Screen

Cobalt distributes a docker-compose file that orchestrates three docker images, one for each of the following services:

• Privacy Screen Server (frontend for accepting text / audio streams)
• Transcribe Server (for recognizing text in audio files)
• Redaction Backend Engine (for redacting text data)

Having these components as separate images facilitates large deployments where each image can be auto-scaled independently based on request traffic.

Installing Server

1. Contact Cobalt to get a link to the image files in AWS S3 and the docker-compose configuration file. This link will expire in two weeks, so be sure to download the file to your own server.

2. Download with the AWS CLI if you have it, or with curl:

   URL="the url sent by Cobalt"
   FILE_NAME="name you want to give the file (should end with the same extension as the url, usually tar.bz2)"
   curl $URL -L -o $FILE_NAME
   

3. Untar the file, and load the docker images. The tar file will also contain the docker-compose.yaml file.

   tar -xvjf $FILE_NAME -C ./
   docker load < *.bz2
   

4. Copy the cobalt license file into the server folder

5. Copy the deid license file into the server folder

6. Start the services using docker-compose:

   docker-compose up --build
   

The server will be running in the container and listening on port 2728 for gRPC requests from clients.

1.2 - Connecting to the Server

Once you have the Cobalt Privacy Screen server up and running, you are ready to create a client connection.

First, you need to know the address (host:port) where the server is running. This document will assume the values 127.0.0.1:9002, but these can be replaced with your server address in actual code.

Default Connection

The following code snippet connects to the server and queries its version. It uses our recommended default setup, expecting the server to be listening on a TLS encrypted connection.

• Go
• Python

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/cobaltspeech/sdk-trifid/grpc/go-trifid"
)

const serverAddr = "127.0.0.1:9002"

func main() {
	client, err := trifid.NewClient(serverAddr)
	if err != nil {
		log.Fatal(err)
	}

	// Be sure to close the client when we are done with it.
	defer client.Close()
}

import trifid

client = trifid.Client(server_address="localhost:9002")

Insecure Connection

It is sometimes required to connect to Privacy Screen server without TLS enabled (during debugging, for example). Note that if the server has TLS enabled, attempting to connect with an insecure client will fail.

To create an insecure connection, do the following when creating the client:

• Go
• Python

client, err := trifid.NewClient(serverAddr, trifid.WithInsecure())

client = trifid.Client(server_address="localhost:9002", insecure=True)

Client Authentication

In our recommended default setup, TLS is enabled in the gRPC setup, and when connecting to the server, clients validate the server’s SSL certificate to make sure they are talking to the right party. This is similar to how “https” connections work in web browsers.

In some setups, it may be desired that the server should also validate clients connecting to it and only respond to the ones it can verify. If your Privacy Screen server is configured to do client authentication, you will need to present the appropriate certificate and key when connecting to it.

Please note that in the client-authentication mode, the client will still also verify the server’s certificate, and therefore this setup uses mutually authenticated TLS. This can be done with:

• Go
• Python

// certPem and keyPem are the bytes of the client certificate and key
// provided to you.
client, err := trifid.NewClient(serverAddr, trifid.WithClientCert(certPem, keyPem))

# cert_pem and key_pem are the contents of the client certificate and key
# provided to you.
client = trifid.Client(server_address="localhost:9002", client_certificate=cert_pem client_key=key_pem)

Server Information

The client provides two methods to get information about the server - Version and ListModels.

Version

The Version method provides information about the version of the Privacy Screen server the client is connected to, as well as information about other relevant services and packages the server uses.

• Go
• Python

// Request the server version info
ver, err := client.Version(context.Background())
fmt.Printf("Server Version: %v\n", ver)

# Request the server version info
ver = client.version()
print(f"Server Version: {ver}")

List Models

The ListModels method fetches a list of models available on the Privacy Screen server. On the server side, the models are specified as part of the server’s config file.

• Go
• Python

// Request the list of models
modelList, err := client.ListModels(context.Background())
fmt.Printf("Available Models:\n")
for _, mdl := range modelList.Models {
	fmt.Printf("  ID: %v\n", mdl.Id)
	fmt.Printf("    Name: %v\n", mdl.Name)
	fmt.Printf("    Redaction Classes: %v\n", mdl.RedactionClasses)
}

# Request the list of models
model_list = client.list_models()
print("Available Models:")
for mdl in model_list:
    print(f"  ID: {mdl.id}")
    print(f"    Name: {mdl.name}")
    print(f"    Redaction Classes: {mdl.redaction_classes}")

1.3 - Text Redaction

TODO

1.4 - Concurrency

The recommended level of concurrency, i.e. the optimal number of simultaneous requests to make to the container is covered below for the CPU and GPU containers. The recommended concurrency level is driven primarily by the compute requirement of the Neural Network models, such as for PII detection.

CPU

For Neural Network inference workloads, CPUs don’t require inputs to be batched together to achieve good hardware utilization. In practice, due to network overhead and pre/post-processing code it is best to use a low level of concurrency such as 2 per container instance. If latency isn’t a concern, a value of 32 is recommended.

GPU

Unlike CPUs, GPUs require inputs to be batched together and processed as a single large input to achieve optimal hardware utilization. This means that there is a tradeoff between latency and throughput. A concurrency level of 32 per container instance is a good tradeoff between latency and throughput, however concurrency levels as low as 8 do not significantly impact throughput. If latency isn’t a concern, a value of 128 will ensure maximum hardware utilization.

1.5 - Privacy Screen Client

This release includes a PrivacyScreen client that can be used to quickly send audio/transcripts to the server. This reads audio files in WAV (PCM16SLE) format and transcripts in the JSON format, you can see examples for txt and json files:

• txt
• json

Jack and Jill went up to 224 North Hill drive to fetch a pail of water. Jack fell down broke his crown and Jill called 4125555555.

{
    "utterances": [
        {
            "start_time_ms": 30,
            "duration_ms": 4230,
            "audio_channel": 0,
            "words": [
                {
                    "start_time_ms": 30,
                    "duration_ms": 390,
                    "text": "Jack"
                },
                {
                    "start_time_ms": 420,
                    "duration_ms": 120,
                    "text": "and"
                },
                {
                    "start_time_ms": 540,
                    "duration_ms": 240,
                    "text": "Jill"
                },
                {
                    "start_time_ms": 780,
                    "duration_ms": 240,
                    "text": "went"
                },
                {
                    "start_time_ms": 1020,
                    "duration_ms": 150,
                    "text": "up"
                },
                {
                    "start_time_ms": 1170,
                    "duration_ms": 60,
                    "text": "to"
                },
                {
                    "start_time_ms": 1230,
                    "duration_ms": 1080,
                    "text": "224"
                },
                {
                    "start_time_ms": 2310,
                    "duration_ms": 300,
                    "text": "North"
                },
                {
                    "start_time_ms": 2610,
                    "duration_ms": 150,
                    "text": "Hill"
                },
                {
                    "start_time_ms": 2760,
                    "duration_ms": 300,
                    "text": "drive"
                },
                {
                    "start_time_ms": 3060,
                    "duration_ms": 90,
                    "text": "to"
                },
                {
                    "start_time_ms": 3150,
                    "duration_ms": 270,
                    "text": "fetch"
                },
                {
                    "start_time_ms": 3420,
                    "duration_ms": 60,
                    "text": "a"
                },
                {
                    "start_time_ms": 3480,
                    "duration_ms": 270,
                    "text": "pail"
                },
                {
                    "start_time_ms": 3750,
                    "duration_ms": 120,
                    "text": "of"
                },
                {
                    "start_time_ms": 3870,
                    "duration_ms": 390,
                    "text": "water."
                }
            ]
        },
        {
            "start_time_ms": 9300,
            "duration_ms": 5324,
            "audio_channel": 1,
            "words": [
                {
                    "start_time_ms": 9300,
                    "duration_ms": 420,
                    "text": "Jack"
                },
                {
                    "start_time_ms": 9720,
                    "duration_ms": 210,
                    "text": "fell"
                },
                {
                    "start_time_ms": 9930,
                    "duration_ms": 420,
                    "text": "down"
                },
                {
                    "start_time_ms": 10410,
                    "duration_ms": 270,
                    "text": "broke"
                },
                {
                    "start_time_ms": 10680,
                    "duration_ms": 150,
                    "text": "his"
                },
                {
                    "start_time_ms": 10830,
                    "duration_ms": 450,
                    "text": "crown"
                },
                {
                    "start_time_ms": 11310,
                    "duration_ms": 180,
                    "text": "and"
                },
                {
                    "start_time_ms": 11490,
                    "duration_ms": 210,
                    "text": "Jill"
                },
                {
                    "start_time_ms": 11700,
                    "duration_ms": 330,
                    "text": "called"
                },
                {
                    "start_time_ms": 12030,
                    "duration_ms": 2594,
                    "text": "4125555555."
                }
            ]
        }
    ]
}

Examples of client calls

There are several ways the client interacts with the server. These examples are always run from the same path as the client binary is. When in doubt, run ./privacy-screen-gprc-client -h to get more information the parameters needed to run the client.

Redact Text

./privacy-screen-grpc-client redact-text \
        --insecure \
        --model-id general \
        --input-text input.txt \
        --output-result redacted_token.json

Redact Transcript

./privacy-screen-grpc-client redact-transcript \
        --insecure \
        --model-id general \
        --input-transcript testdata/input.json \
        --output-transcript redacted_output.json

Redact Transcribed Audio

./privacy-screen-grpc-client redact-transcribed-audio \
        --insecure \
        --model-id general \
        --input-audio testdata/input.wav \
        --input-transcript testdata/input.json \
        --output-audio redacted_output.wav \
        --output-transcript redacted_output.json \
        --timeout 5m

Transcribe and Redact

./privacy-screen-grpc-client transcribe-and-redact \
        --insecure \
        --model-id en_US \
        --input-audio testdata/input.wav \
        --output-audio redacted_output.wav \
        --output-transcript redacted_output.json \
        --output-unredacted-transcript unredacted_output.json \
        --timeout 5m

1.6 - Redaction Categories

Personally Identifiable Information (PII)

Protected Health Information (PHI)

Payment Credit Industry Information (PCI)

Beta Entity Types

Note that Beta support for the following entity types is currently only available with our English models.

International Entity Mapping

In the tables below, you can find localized variants of our entity types. For each entity type, there is a description, an example, and the label under which the entity falls. This section does not include entity types that may vary regionally, but still directly correspond to one of the entities listed above (e.g., PHONE_NUMBER, PASSPORT_NUMBER, DRIVER_LICENSE, LOCATION_ADDRESS, CREDIT_CARD_EXPIRATION). The following numbers are commonly used across many countries, hence are not included in each country’s table: GST (Goods and Services Tax), HST (Harmonized Sales Tax). These numbers are redacted as NUMERICAL_PII.

Asia Pacific

Australia

China

India

Japan

Korea

New Zealand

Philippines

Europe

Belgium

Germany

France

Italy

Netherlands

Portugal

Russia

Spain

Ukraine

United Kingdom

North and South America

Brazil

Canada

Mexico

United States

1.7 - Redaction Languages

Cobalt features core support for 14 languages and extended support for 39 additional languages, with core languages featuring the highest level of performance. The complete list of supported languages below details which languages have core support, which have extended or beta support, and which are upcoming additions. New languages are continually being added, please contact us if you require a language not in the list below.

In addition to supporting 50+ languages, Cobalt offers support for regional language varieties in recognition of the large differences in vocabulary and grammar that can exist in the same language when spoken in different regions. So far, this includes support for varieties of English (US, UK, Canada and Australia), Spanish (Spain and Mexico), French (France and Canada), and Portuguese (Portugal and Brazil). Cobalt also supports code-switching, or mixing of different languages. This means that, in a phrase such as J’ai payé 76,88RM por ein Haarschnitt da 范玉菲 habang ko ay nasa Україна, multilingual PII is accurately de-identified. The selection of supported regional language varieties is continually being expanded, please let us know if there is a specific request.

Cobalt’s supported entity types function across each supported language, with multilingual equivalents of different PII (Personally Identifiable Information) entities, PHI (Protected Health Information) entities, and PCI (Payment Card Industry) entities being detected in each language. Our Supported Entity Types page provides a more detailed look at our coverage of language and region-specific entity equivalents. The solution is also sensitive to cross-linguistic differences in how names are structured, how place names are referred to, and how monetary units are described in different languages, among other differences.

Core Support

Extended Support

1.8 - Prerequisites and System Requirements

Prerequisites

The following prerequisites are required to run the container:

• Container engine, such as Docker (can be installed using the official instructions )
• (GPU only) Nvidia Container Toolkit with Nvidia driver version 515 or higher (can be installed using the following installation guide )

All other dependencies, such as CUDA are included with the container and don’t need to be installed separately.

System Requirements

• Docker and docker-compose installed.

The image comes in two different build flavours:

• A compact, CPU-only container that runs on any Intel or AMD CPU and a container with GPU acceleration. The CPU container is highly optimised for the majority of use cases, as the container uses hand-coded AMX/AVX2/AVX512/AVX512 VNNI instructions in conjunction with Neural Network compression techniques to deliver a ~25X speedup over a reference transformer-based system.
• A GPU container is designed for large-scale deployments making billions of API calls or processing terabytes of data per month.

Minumum Requirements

The minimum system requirements for the container image are as follows:

Recommended Requirements

While CPU-based container will run on any x86-compatible instance, the below cloud instance types give optimal throughput and latency per dollar:

• In the event when a lower latency is required, the instance type should be scaled; e.g. using an M7i.xlarge in place of a M7i.large. While the Cobalt docker solution can make use of all available CPU cores, it delivers best throughput per dollar using a single CPU core machine. Scaling CPU cores does not result in a linear increase in performance.

Similarly for the GPU-based image, it is recommended the following Nvidia T4 GPU-equipped instance types:

1.9 - Proto API Reference

The API is defined as a protobuf spec, so native bindings can be generated in any language with gRPC support. We recommend using buf to generate the bindings.

This section of the documentation is auto-generated from the protobuf spec. The service contains the methods that can be called, and the “messages” are the data structures (objects, classes or structs in the generated code, depending on the language) passed to and from the methods.

TrifidService

Service that implements the Cobalt Trifid Redaction Engine API.

Version

Version(VersionRequest) VersionResponse

Returns version information from the server.

ListModels

ListModels(ListModelsRequest) ListModelsResponse

ListModels returns information about the models the server can access.

RedactText

RedactText(RedactTextRequest) RedactTextResponse Redact text using a redaction engine that is configured with the provided redaction configuration.

RedactTranscript

RedactTranscript(RedactTranscriptRequest) RedactTranscriptResponse

redacts transcript using a redaction engine that is configured with the provided redaction configuration.

StreamingRedactTranscribedAudio

StreamingRedactTranscribedAudio(StreamingRedactTranscribedAudioRequest) StreamingRedactTranscribedAudioResponse

Performs bidirectional streaming redaction on transcribed audio. Receive redacted audio while sending audio. The transcription of audio data must be ready before sending the audio.

StreamingTranscribeAndRedact

StreamingTranscribeAndRedact(StreamingTranscribeAndRedactRequest) StreamingTranscribeAndRedactResponse

Performs bidirectional streaming speech recognition and redaction. Receive redacted audio and transcriptions while sending audio.

Messages

• If two or more fields in a message are labeled oneof, then each method call using that message must have exactly one of the fields populated
• If a field is labeled repeated, then the generated code will accept an array (or struct, or list depending on the language).

ListModelsRequest

The top-level message sent by the client for the ListModels method.

ListModelsResponse

The message returned to the client by the ListModels method.

Fields

• models (ModelInfo repeated) List of models available for use on Trifid server.

ModelInfo

Description of a Trifid Model

Fields

• id (string ) Unique identifier of the model. This identifier is used to choose the model that should be used for recognition, and is specified in the RedactionConfig message.

• name (string ) Model name. This is a concise name describing the model, and may be presented to the end-user, for example, to help choose which model to use for their recognition task.

• redaction_classes (string repeated) List of supported redaction classes.

RedactTextRequest

The top-level message sent by the client for the ListModels method.

Fields

• redaction_config (RedactionConfig)
• text (string ) Unique identifier of the model. This identifier is used to choose the model that should be used for recognition, and is specified in the RedactionConfig message.

RedactionConfig

Configuration for setting up a redaction engine.

Fields

• model_id (string) Unique identifier of the model to use, as obtained from a ModelInfo message.
• redaction_classes (string repeated) List of whitelisted redaction classes. If the list is empty, server default redaction class list will be considered.
• disable_streaming (bool ) This is an optional field. If this is set to true, Cobalt Privacy Screen will redact entire transcript at once, by doing so, redaction accuracy will increase at the cost of higher latency. If set to false, Cobalt Privacy Screen will redact one utterance at a time and return the result as soon as possible. The default is false.
• custom_classes (CustomClasses repeated) This is an optional field. If set, then provided list will be used to extend the list of redaction classes.

CustomClasses

CustomClass allows the client to define a new redaction class. Patterns defined here will override default redaction class for matching tokens.

Fields

• redaction_class string This is name of the new redaction class. For example, this could be “COMPANY_NAME”.
• pattern string Pattern defines a Python regex expression that would be used to identify tokens in text that get redacted to this new redaction class. For example, “COBALT|GOOGLE|MICROSOFT”, or more complex patterns such as “^COMPANY-[\d]{4}$”.

ListModelsResponse

The message returned to the client by the ListModels method.

Fields

• models (ModelInfo repeated) List of models available for use on Trifid server.

ModelInfo

Description of a Trifid Model

Fields

• id (string ) Unique identifier of the model. This identifier is used to choose the model that should be used for recognition, and is specified in the RedactionConfig message.

• name (string ) Model name. This is a concise name describing the model, and may be presented to the end-user, for example, to help choose which model to use for their recognition task.

• redaction_classes (string repeated) List of supported redaction classes.

RedactTranscriptRequest

The top-level messages sent by the client for the RedactTranscript method. Contains redaction config and a transcription to redact.

Fields

• config (RedactionConfig ) redaction config

• transcript (Transcript ) transcription to redact

RedactTranscriptResponse

The top-level message sent by the server for the RedactTranscript method. Contains redacted transcript.

Fields

• transcript (Transcript )

RedactionConfig

Configuration for setting up a redaction engine.

Fields

• model_id (string ) Unique identifier of the model to use, as obtained from a ModelInfo message.

• redaction_classes (string repeated) List of whitelisted redaction classes. If the list is empty, server default redaction class list will be considered.

• disable_streaming (bool ) This is an optional field. If this is set to true, Trifid will redact entire transcript at once, by doing so, redaction accuracy will increase at the cost of higher latency. If set to false, Trifid will redact one utterance at a time and return the result as soon as possible. The default is false.

Transcript

Transcript contains multiple utterance of the audio.

Fields

• utterances (Utterance repeated)

Utterance

Utterance of the audio

Fields

• text (string ) Text representing the utterance of the audio.

• audio_channel (uint32 ) Channel of the audio file associate with this utterance. Channels are 0-indexed, so the for mono audio data, this value will always be 0.

• start_time_ms (uint64 ) Time offset in milliseconds relative to the beginning of audio corresponding to the start of this utterance.

• duration_ms (uint64 ) Duration in milliseconds of the current utterance in the audio.

• asr_confidence (double ) ASR confidence estimate between 0 and 1. A higher number represents a higher likelihood of the output being correct.

• words_info (WordInfo repeated) Word-level information corresponding to the utterance. This field contains word-level timestamps, which are essential as input for audio redaction. This field is only available in an output utterance if enable_word_info was set to true in the RedactionConfig.

StreamingRedactTranscribedAudioRequest

The top-level messages sent by the client for the StreamingRedactTranscribedAudio method. In this streaming call, multiple StreamingRedactTranscribedAudioRequest messages should be sent. The first message must contain a RedactTranscribedAudioConfig message only and all subsequent messages must contain audio data only.

Fields

• oneof request.config (RedactTranscribedAudioConfig)
• oneof request.audio (bytes)

RedactTranscribedAudioConfig

Configuration for setting up a StreamingRedactTranscribedAudio method.

Fields

• redaction_config(RedactionConfig) Text redaction config.
• transcript(Trancript) Transcription of the entire audio. This must be ready before sending the audio.

StreamingRedactTranscribedAudioResponse

The top-level message sent by the server for the StreamingRedactTranscribedAudio method. In this streaming call, multiple StreamingRedactTranscribedAudioResponse messages contain either Utterance or redacted audio data will be returned.

Fields

• oneof result.utterance (Utterance)
• oneof result.audio (bytes)

StreamingTranscribeAndRedactRequest

The top-level messages sent by the client for the StreamingTranscribeAndRedact method. In this streaming call, multiple StreamingTranscribeAndRedactRequest messages should be sent. The first message must contain a TranscribeAndRedactConfig message only and all subsequent messages must contain audio data only.

Fields

• oneof request.config (TranscribeAndRedactConfig)
• oneof request.audio (bytes)

TranscribeAndRedactConfig

Configuration for setting up a StreamingTranscribeAndRedact method.

Fields

• redaction_config(RedactionConfig) Text redaction config.
• enable_unredacted_transcript(bool) This is an optional field. If this is set to true, each utterance result will include unredacted utterance. If set to false, no unredacted utterance will be returned. The default is false.

StreamingTranscribeAndRedactResponse

The top-level message sent by the server for the StreamingTranscribeAndRedact method. In this streaming call, multiple StreamingTranscribeAndRedactResponse messages contain either TranscribeAndRedactUtterance or redacted audio data will be returned.

Fields

• oneof result.utterance (Utterance)
• oneof result.audio (bytes)

VersionRequest

The top-level message sent by the client for the Version method.

VersionResponse

The top-level message sent by the server for the Version method.

Fields

• version (string ) Version of the server handling these requests.

WordInfo

Word level details for words in a utterance.

Fields

• text (string ) The actual word corresponding to the utterance.

• asr_confidence (double ) ASR confidence estimate between 0 and 1. A higher number represents a higher likelihood that the word was correctly recognized.

• start_time_ms (uint64 ) Time offset in milliseconds relative to the beginning of audio received by the recognizer and corresponding to the start of this spoken word.

• duration_ms (uint64 ) Duration in milliseconds of the current word in the spoken audio.

• is_redacted (bool ) If this is set to true, it denotes that the curent word is redacted word or an original word of a redacted word.

• redaction_class (string ) Recognized redaction class. This is available only if the current word is a redacted word.

• redaction_confidence (double ) Redactio confidence estimate between 0 and 1. A higher number represents a higher likelihood that the word was correctly recognized. This is available only if the current word is a redacted word.

Enums

Scalar Value Types

2 - VoiceBio

2.1 - Getting Started

Using Cobalt VoiceBio

• A typical VoiceBio release, provided as a compressed archive, will contain a linux binary (voicebio-server) for the required native CPU architecture, appropriate Dockerfile and models.

• Cobalt VoiceBio runs either locally on linux or using Docker.

• Cobalt VoiceBio will serve the VoiceBio GRPC API on port 2727.

• To quickly try out VoiceBio, first start the server as shown below and use the SDK in your preferred language to use VoiceBio from the command line or within your application.

Running VoiceBio Server Locally on Linux

./voicebio-server

• By default, the binary assumes the presence of a configuration file, located in the same directory, named: voicebio-server.cfg.toml. A different config file may be specified using the --config argument.

Running VoiceBio Server as a Docker Container

To build and run the Docker image for VoiceBio, run:

docker build -t cobalt-voicebio .
docker run -p 2727:2727 -p 8080:8080 cobalt-voicebio

How to Get a Copy of the VoiceBio Server and Models

Contact us for getting a release best suited to your requirements.

The release you will receive is a compressed archive (tar.bz2) and is generally structured accordingly:

release.tar.bz2
├── COPYING
├── README.md
├── voicebio-server
├── voicebio-server.cfg.toml
├── Dockerfile
├── models
│   └── en_US-16khz
│
└── cobalt.license.key [ provided separately, needs to be copied over ]

• The README.md file contains information about this release and instructions for how to start the server on your system.

• The voicebio-server is the server program which is configured using the voicebio-server.cfg.toml file.

• The Dockerfile can be used to create a container that will let you run VoiceBio server on non-linux systems such as MacOS and Windows.

• The models directory contains the speaker ID models. The content of these directory will depend on the models you are provided.

System Requirements

Cobalt VoiceBio runs on Linux. You can run it directly as a linux application.

You can evaluate the product on Windows or Linux using Docker Desktop but we would not recommend this setup for use in a production environment.

A Cobalt VoiceBio release typically includes a single VoiceBio model together with binaries and config files. The general purpose VoiceBio models take up to 100MB of disk space, and need a minimum of 2GB RAM when evaluating locally. For production workloads, we recommend configuring containerized applications with each instance allocated with 4 CPUs and 4GB RAM.

Cobalt VoiceBio runs on x86_64 CPUs. We also support Arm64 CPUs, including processors such as the Graviton (AWS c7g EC2 instances). VoiceBio is significantly more cost effective to run on C7g instances compared to similarly sized Intel or AMD processors, and we can provide you an Arm64 release on request.

To integrate Cobalt VoiceBio into your application, please follow the next steps to install or generate the SDK in a language of your choice.

2.2 - Generating SDKs

• APIs for all Cobalt’s services are defined as a protocol buffer specification or simply a proto file and be found in the cobaltspeech/proto github repository.

• The proto file allows a developer to auto-generate client SDKs for a number of different programming languages. Step by step instructions for generating your own SDK can be found below.

• We provide pre-generated SDKs for a couple of languages. You can choose to use these instead of generating your own. These are listed here along with instructions on how to install / import them into your projects.

Pre-generated SDKs

Golang

• Pre-generated SDK files for Golang can be found in the cobaltspeech/go-genproto repo

• To use it in your Go project, simply import it:

import voicebiopb "github.com/cobaltspeech/go-genproto/cobaltspeech/voicebio/v1"

• An example client using the above repo can be found here.

Python

• Pre-generated SDK files for Python can be found in the cobaltspeech/py-genproto repo

• The Python SDK depends on Python >= 3.5. You may use pip to perform a system-wide install, or use virtualenv for a local install. To use it in your Python project, install it:

pip install --upgrade pip
pip install "git+https://github.com/cobaltspeech/py-genproto"

Generating SDKs

Step 1. Installing buf

• To work with proto files, we recommend using buf, a user-friendly command line tool that can be configured generate documentation, schemas and SDK code for different languages.

• Linux (from Github)
• MacOS
• Reference

# Latest version as of March 14th, 2023.

COBALT="${HOME}/cobalt"
  mkdir -p "${COBALT}/bin"

VERSION="1.15.1"
URL="https://github.com/bufbuild/buf/releases/download/v${VERSION}/buf-$(uname -s)-$(uname -m)"
  curl -L ${URL} -o "${COBALT}/bin/buf"

# Give executable permissions and adding to $PATH.

chmod +x "${COBALT}/bin/buf"
  export PATH="${PATH}:${COBALT}/bin"

brew install bufbuild/buf/buf

Step 2. Getting proto files

• Clone the cobaltspeech/proto repository:

COBALT="${HOME}/cobalt"
mkdir -p "${COBALT}/git"

# Change this to where you want to clone the repo to.
PROTO_REPO="${COBALT}/git/proto"

git clone https://github.com/cobaltspeech/proto "${PROTO_REPO}"

Step 3. Generating code

• The cobaltspeech/proto repo provides a buf.gen.yaml config file to get you started with a couple of languages.

• Other plugins can be added to the buf.gen.yaml file to generate SDK code for more languages.

• To generate the SDKs, simply run the following (assuming the buf binary is in your $PATH)

cd "${PROTO_REPO}"

# Removing any previously generated files.
rm -rf ./gen

# Generating code for all proto files inside the `proto` directory.
buf generate proto

• You should now have a folder called gen inside ${PROTO_REPO} that contains the generated code. The latest version of the VoiceBio API is v1. You can import / include / copy the generated files into your projects as per the conventions of different languages.

• Python
• Golang

gen
├── ... other languages ...
└── py
  └── cobaltspeech
    ├── ... other services ...
    └── voicebio
      └── v1
        ├── voicebio_pb2_grpc.py
        ├── voicebio_pb2.py
        └── voicebio_pb2.pyi

gen
├── ... other languages ...
└── go
   ├── cobaltspeech
   │ ├── ...
   │   └── voicebio
   │      └── v1
   │        ├── voicebio_grpc.pb.go
   │        └── voicebio.pb.go
   └── gw
     └── cobaltspeech
       ├── ...
       └── voicebio
         └── v1
            └── voicebio.pb.gw.go

Step 4. Installing gPRC and protobuf

• A couple of gRPC and protobuf dependencies are required along with the code generated above. The method of installing them depends on the programming language being used.
• These dependencies and the most common way of installing/ / including them are listed below for some chosen languages.

• Python
• Golang
• C++

# It is encouraged to this inside a python virtual environment

# to avoid creating version conflicts for other scripts that may

# be using these libraries.

pip install --upgrade protobuf
pip install --upgrade grpcio
pip install --upgrade google-api-python-client

go get google.golang.org/protobuf
go get google.golang.org/grpc
go get google.golang.org/genproto

# More details on grpc installation can be found at:

# https://grpc.io/docs/languages/cpp/quickstart/

COBALT="${HOME}/cobalt"
mkdir -p "${COBALT}/git"

# Latest version as of 14th March, 2023.

VERSION="v1.52.0"
GRPC_REPO="${COBALT}/git/grpc-${VERSION}"

git clone \
 --recurse-submodules --depth 1 --shallow-submodules \
 -b "${VERSION}" \
 https://github.com/grpc/grpc ${GRPC_REPO}

cd "${GRPC_REPO}"
mkdir -p cmake/build

# Change this to where you want to install libprotobuf and libgrpc.

# It is encouraged to install gRPC locally as there is no easy way to

# uninstall gRPC after you’ve installed it globally.

INSTALL_DIR="${COBALT}"

cd cmake/build
cmake \
 -DgRPC_INSTALL=ON \
 -DgRPC_BUILD_TESTS=OFF \
 -DCMAKE_INSTALL_PREFIX=${INSTALL_DIR} \
 ../..

make -j
make install

2.3 - Connecting to the Server

• Once you have your VoiceBio server up and running, and have installed or generated the SDK for your project, you can connect to a running instance of VoiceBio server, by “dialing” a gRPC connection.

• First, you need to know the address where the server is running: e.g. host:grpc_port. By default, this is localhost:2727 and should be logged to the terminal when you first start VoiceBio server as grpcAddr:

2023/08/14 10:49:38 info  {"license":"Copyright © 2023--present. Cobalt Speech and Language, Inc.  For additional details, including information about open source components used in this software, please see the COPYING file bundled with this program."}
2023/08/14 10:49:38 info  {"msg":"reading config file","path":"configs/voicebio-server.config.toml"}
2023/08/14 10:49:38 info  {"msg":"server initializing"}
2023/08/14 10:49:38 info  {"msg":"license verified"}
2023/08/14 10:49:41 info  {"msg":"runtime initialized","model_count":"2","init_time_taken":"2.512935646s"}
2023/08/14 10:49:41 info  {"msg":"server started","grpcAddr":"[::]:2727","httpApiAddr":"[::]:8080","httpOpsAddr":"[::]:8081"}

Default Connection

The following code snippet connects to the server and queries its version. It connects to the server using an “insecure” gRPC channel. This would be the case if you have just started up a local instance of VoiceBio server without TLS enabled.

• Python
• Go

import grpc
import cobaltspeech.voicebio.v1.voicebio_pb2_grpc as stub
import cobaltspeech.voicebio.v1.voicebio_pb2 as voicebio

serverAddress = "localhost:2727"

# Using a channel without TLS enabled.
channel = grpc.insecure_channel(serverAddress)
client = stub.VoiceBioServiceStub(channel)

# Get server version.
versionResp = client.Version(voicebio.VersionRequest())
print(versionResp)

package main

import (
	"context"
	"fmt"
	"os"

	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials/insecure"

	voicebiopb "github.com/cobaltspeech/go-genproto/cobaltspeech/voicebio/v1"
)

func main() {
	const (
		serverAddress  = "localhost:2727"
	)

	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	opts := []grpc.DialOption{
		grpc.WithTransportCredentials(insecure.NewCredentials()), // Using a channel without TLS enabled.
		grpc.WithBlock(),
		grpc.WithReturnConnectionError(),
		grpc.FailOnNonTempDialError(true),
	}

	conn, err := grpc.DialContext(ctx, serverAddress, opts...)
	if err != nil {
		fmt.Printf("failed to dial gRPC connection: %v\n", err)
		os.Exit(1)
	}

	client := voicebiopb.NewVoiceBioServiceClient(conn)

	// Get server version.
	versionResp, err := client.Version(ctx, &voicebiopb.VersionRequest{})
	if err != nil {
		fmt.Printf("failed to get server version: %v\n", err)
		os.Exit(1)
	}

	fmt.Printf("%v\n", versionResp)
}

Connect with TLS

• In our recommended setup for deployment, TLS is enabled in the gRPC connection, and when connecting to the server, clients validate the server’s SSL certificate to make sure they are talking to the right party. This is similar to how “https” connections work in web browsers.

• The following snippets show how to connect to a VoiceBio Server that has TLS enabled. They use the cobalt’s self-hosted demo server at demo.cobaltspeech.com:2727, but you obviously use your own server instance.

• Python
• Go

import grpc
import cobaltspeech.voicebio.v1.voicebio_pb2_grpc as stub
import cobaltspeech.voicebio.v1.voicebio_pb2 as voicebio

serverAddress = "demo.cobaltspeech.com:2727"

# Setup a gRPC connection with TLS. You can optionally provide your own
# root certificates and private key to grpc.ssl_channel_credentials()
# for mutually authenticated TLS.
creds = grpc.ssl_channel_credentials()
channel = grpc.secure_channel(serverAddress, creds)
client = stub.VoiceBioServiceStub(channel)

# Get server version.
versionResp = client.Version(voicebio.VersionRequest())
print(versionResp)

package main

import (
	"context"
	"crypto/tls"
	"fmt"
	"os"
	"time"

	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials"

	voicebiopb "github.com/cobaltspeech/go-genproto/cobaltspeech/voicebio/v1"
)

func main() {
	const (
		serverAddress  = "demo.cobaltspeech.com:2727"
		connectTimeout = 10 * time.Second
	)

	// Setup a gRPC connection with TLS. You can optionally provide your own
	// root certificates and private key through tls.Config for mutually
	// authenticated TLS.
	tlsCfg := tls.Config{}
	creds := credentials.NewTLS(&tlsCfg)

	ctx, cancel := context.WithTimeout(context.Background(), connectTimeout)
	defer cancel()

	opts := []grpc.DialOption{
		grpc.WithTransportCredentials(creds),
		grpc.WithBlock(),
		grpc.WithReturnConnectionError(),
		grpc.FailOnNonTempDialError(true),
	}

	conn, err := grpc.DialContext(ctx, serverAddress, opts...)
	if err != nil {
		fmt.Printf("failed to dial gRPC connection: %v\n", err)
		os.Exit(1)
	}

	client := voicebiopb.NewVoiceBioServiceClient(conn)

	// Get server version.
	versionResp, err := client.Version(ctx, &voicebiopb.VersionRequest{})
	if err != nil {
		fmt.Printf("failed to get server version: %v\n", err)
		os.Exit(1)
	}

	fmt.Printf("%v\n", versionResp)
}

Client Authentication

• In some setups, it may be desired that the server should also validate clients connecting to it and only respond to the ones it can verify. If your VoiceBio server is configured to do client authentication, you will need to present the appropriate certificate and key when connecting to it.

• Please note that in the client-authentication mode, the client will still also verify the server’s certificate, and therefore this setup uses mutually authenticated TLS.

• The following snippets show how to present client certificates when setting up the credentials. These could then be used in the same way as the examples above to connect to a TLS enabled server.

• Python
• Go

creds = grpc.ssl_channel_credentials(
  root_certificates=root_certificates,  # PEM certificate as byte string
  private_key=private_key,              # PEM client key as byte string 
  certificate_chain=certificate_chain,  # PEM client certificate as byte string
)

package main

import (
	// ...

	"crypto/tls"
	"crypto/x509"
	"fmt"
	"os"

	// ..
)

func main() {
	// ...

	// Root PEM certificate for validating self-signed server certificate
	var rootCert []byte

	// Client PEM certificate and private key.
	var certPem, keyPem []byte

	caCertPool := x509.NewCertPool()
	if ok := caCertPool.AppendCertsFromPEM(rootCert); !ok {
		fmt.Printf("unable to use given caCert\n")
		os.Exit(1)
	}

	clientCert, err := tls.X509KeyPair(certPem, keyPem)
	if err != nil {
		fmt.Printf("unable to use given client certificate and key: %v\n", err)
		os.Exit(1)
	}

	tlsCfg := tls.Config{
		RootCAs:      caCertPool,
		Certificates: []tls.Certificate{clientCert},
	}

	creds := credentials.NewTLS(&tlsCfg)

	// ...
}

2.4 - Streaming Enrollment

• The following example shows how to stream audio using VoiceBio’s StreamingEnroll request and generate a voiceprint. The stream can come from a file on disk or be directly from a microphone in real time.

Streaming from an audio file

• We support several headered file formats including WAV, MP3, FLAC etc. For more details, please see the protocol buffer specification here. For best accuracy, it is recommended to use an uncompressed / loss-less compression audio format like WAV or FLAC.

• The examples below use a WAV file as input. We will query the server for available models and use the first model to generate the voiceprint.

• Generated Voiceprints can be updated and made more robust by re-enrolling them with additional audio. Please see the re-enrollment section.

• Python
• Go

import grpc
import cobaltspeech.voicebio.v1.voicebio_pb2_grpc as stub
import cobaltspeech.voicebio.v1.voicebio_pb2 as voicebio

serverAddress = "localhost:2727"

# Using a channel without TLS enabled.
channel = grpc.insecure_channel(serverAddress)
client = stub.VoiceBioServiceStub(channel)

# Get server version.
versionResp = client.Version(voicebio.VersionRequest())
print(versionResp)

# Get list of models on the server.
modelResp = client.ListModels(voicebio.ListModelsRequest())

print("Models:")
for model in modelResp.models:
    print(model)

# Select a model ID from the list above. Going with the first model
# in this example.
modelID = modelResp.models[0].id

# Set the enrollment config. We don't set the audio format and let the
# server auto-detect the format from the file header.
cfg = voicebio.EnrollmentConfig(
    model_id=modelID,
    previous_voiceprint=None,
)

# The first request to the server should only contain the
# configuration. Subsequent requests should contain audio
# bytes. We can write a simple generator to do this.
def stream(cfg, audio, bufferSize=1024):
    yield voicebio.StreamingEnrollRequest(config=cfg)
    
    data = audio.read(bufferSize)
    while len(data) > 0:
        yield voicebio.StreamingEnrollRequest(audio=voicebio.Audio(data=data))
        data = audio.read(bufferSize)

# Streaming audio to the server.
with open("test.wav", "rb") as audio:
  result = client.StreamingEnroll(stream(cfg, audio))

# A certain minimum duration of speech is required for completing enrollment.
# The enrollment status contains information on Whether that has been met or
# whether additional audio is required.  
print(f"enrollment Status:\n{result.enrollment_status}\n")

# Saving the voiceprint data to a file. This can be provided again
# in another StreamingEnroll request (for continuing enrollment) or
# submitted for verification / identification requests.
with open("voiceprint.bin", 'w') as f:
  f.write(result.voiceprint.data)

package main

import (
	"context"
	"errors"
	"fmt"
	"io"
	"os"

	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials/insecure"

	voicebio "github.com/cobaltspeech/go-genproto/cobaltspeech/voicebio/v1"
)

func main() {
	const (
		serverAddress = "localhost:2727"
	)

	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	opts := []grpc.DialOption{
		grpc.WithTransportCredentials(insecure.NewCredentials()), // Using a channel without TLS enabled.
		grpc.WithBlock(),
		grpc.WithReturnConnectionError(),
		grpc.FailOnNonTempDialError(true),
	}

	conn, err := grpc.DialContext(ctx, serverAddress, opts...)
	if err != nil {
		fmt.Printf("failed to dial gRPC connection: %v\n", err)
		os.Exit(1)
	}

	client := voicebio.NewVoiceBioServiceClient(conn)

	// Get server version.
	versionResp, err := client.Version(ctx, &voicebio.VersionRequest{})
	if err != nil {
		fmt.Printf("failed to get server version: %v\n", err)
		os.Exit(1)
	}

	fmt.Printf("%v\n", versionResp)

	// Get list model of models on the server.
	modelResp, err := client.ListModels(ctx, &voicebio.ListModelsRequest{})
	if err != nil {
		fmt.Printf("failed to get model list: %v\n", err)
		os.Exit(1)
	}

	fmt.Println("Models:")
	for _, m := range modelResp.Models {
		fmt.Println(m)
	}
	fmt.Println()

	// Selecting the first model.
	cfg := &voicebio.EnrollmentConfig{
		ModelId:            modelResp.Models[0].Id,
		PreviousVoiceprint: nil,
	}

	// Opening audio file.
	audio, err := os.Open("test.wav")
	if err != nil {
		fmt.Printf("failed to open audio file: %v\n", err)
		os.Exit(1)
	}

	defer audio.Close()

	// Starting enrollment.
	result, err := StreamingEnroll(ctx, client, cfg, audio)
	if err != nil {
		fmt.Printf("failed to run streaming enrollment: %v\n", err)
		os.Exit(1)
	}

	// A certain minimum duration of speech is required for completing enrollment.
	// The enrollment status contains information on Whether that has been met or
	// whether additional audio is required.
	fmt.Printf("Enrollment Status: %v\n", result.EnrollmentStatus)

	// Saving the voiceprint data to a file. This can be provided again
	// in another StreamingEnroll request (for continuing enrollment) or
	// submitted for verification / identification requests.
	if err := os.WriteFile("voiceprint.bin", []byte(result.Voiceprint.Data), os.ModePerm); err != nil {
		fmt.Printf("failed to write voiceprint data: %v\n", err)
		os.Exit(1)
	}
}

// StreamingEnroll wraps the streaming API for performing speaker enrollment
// (i.e. voiceprint generation) using  the given cfg.
//
// Data is read from the given audio reader into a buffer and streamed to VoiceBio
// server. The default buffer size may be overridden using Options when creating
// the Client.
//
// If any error occurs while reading the audio or sending it to the server, this
// method will immediately exit, returning that error.
func StreamingEnroll(
	ctx context.Context,
	client voicebio.VoiceBioServiceClient,
	cfg *voicebio.EnrollmentConfig,
	audio io.Reader,
) (*voicebio.StreamingEnrollResponse, error) {
	const (
		streamingBufSize = 1024
	)

	// Creating stream.
	stream, err := client.StreamingEnroll(ctx)
	if err != nil {
		return nil, err
	}

	// Sending audio.
	if err := sendAudio(stream, cfg, audio, streamingBufSize); err != nil && !errors.Is(err, io.EOF) {
		// if sendAudio encountered io.EOF, it's only a
		// notification that the stream has closed.  The actual
		// status will be obtained in the CloseAndRecv call. We
		// therefore return on non-EOF errors here.
		return nil, err
	}

	// Returning result.
	return stream.CloseAndRecv()
}

// sendAudio sends the config and audio to a stream.
func sendAudio(
	stream voicebio.VoiceBioService_StreamingEnrollClient,
	cfg *voicebio.EnrollmentConfig,
	audio io.Reader,
	bufSize uint32,
) error {
	// The first message needs to be a config message, and all subsequent
	// messages must be audio messages.

	// Send the config.
	if err := stream.Send(&voicebio.StreamingEnrollRequest{
		Request: &voicebio.StreamingEnrollRequest_Config{Config: cfg},
	}); err != nil {
		// if this failed, we don't need to CloseSend
		return err
	}

	// Stream the audio.
	buf := make([]byte, bufSize)
	for {
		n, err := audio.Read(buf)
		if n > 0 {
			if err2 := stream.Send(&voicebio.StreamingEnrollRequest{
				Request: &voicebio.StreamingEnrollRequest_Audio{
					Audio: &voicebio.Audio{Data: buf[:n]},
				},
			}); err2 != nil {
				// if we couldn't Send, the stream has
				// encountered an error and we don't need to
				// CloseSend.
				return err2
			}
		}

		if err != nil {
			// err could be io.EOF, or some other error reading from
			// audio.  In any case, we need to CloseSend, send the
			// appropriate error to errCh and return from the function
			if err2 := stream.CloseSend(); err2 != nil {
				return err2
			}

			if err != io.EOF {
				return err
			}

			return nil
		}
	}
}

Streaming from microphone

• Streaming audio from microphone input basically requires a reader interface that can provided audio samples recorded from a microphone; typically this requires interaction with system libraries. Another option is to use an external command line tool like sox to record and pipe audio into the client.

• The examples below use the latter approach by using the rec command provided with sox to record and stream the audio.

• Python
• Go

#!/usr/bin/env python3

# This example assumes sox is installed on the system and is available
# in the system's PATH variable. Instead of opening a regular file from
# disk, we open a subprocess that executes sox's rec command to record
# audio from the system's default microphone.

import subprocess
import grpc
import cobaltspeech.voicebio.v1.voicebio_pb2_grpc as stub
import cobaltspeech.voicebio.v1.voicebio_pb2 as voicebio

serverAddress = "localhost:2727"

# Using a channel without TLS enabled.
channel = grpc.insecure_channel(serverAddress)
client = stub.VoiceBioServiceStub(channel)

# Get server version.
versionResp = client.Version(voicebio.VersionRequest())
print(versionResp)

# Get list of models on the server.
modelResp = client.ListModels(voicebio.ListModelsRequest())

print("Models:")
for model in modelResp.models:
    print(model)

# Select a model ID from the list above. Going with the first model
# in this example.
m = modelResp.models[0]
modelID = m.id

# Setting audio format to be raw 16-bit signed little endian audio samples
# recorded at the sample rate expected by the model.
cfg = voicebio.EnrollmentConfig(
    model_id=modelID,
    previous_voiceprint=None,
    audio_format=voicebio.AudioFormat(
      audio_format_raw=voicebio.AudioFormatRAW(
        encoding="AUDIO_ENCODING_SIGNED",
        bit_depth=16,
        byte_order="BYTE_ORDER_LITTLE_ENDIAN",
        sample_rate=m.attributes.sample_rate,
        channels=1,
      )
    ),
)

# Open microphone stream using sox's rec command and record
# audio using the config specified above for *10 seconds*.
maxDuration = 10
cmd = f"rec -t raw -r {m.attributes.sample_rate} -e signed -b 16 -L -c 1 - trim 0 {maxDuration}"
mic = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE)
audio = mic.stdout

try:
  _ = audio.read(1024) # Trying to read some bytes as sanity check.
except Exception as err:
    print(f"[ERROR] failed to read audio from mic stream: {err}")

print(f"\n[INFO] recording {maxDuration} seconds of audio microphone ... \n")

# The first request to the server should only contain the
# recognition configuration. Subsequent requests should contain
# audio bytes. We can write a simple generator to do this.
def stream(cfg, audio, bufferSize=1024):
    yield voicebio.StreamingEnrollRequest(config=cfg)

    data = audio.read(bufferSize)
    while len(data) > 0:
        yield voicebio.StreamingEnrollRequest(audio=voicebio.Audio(data=data))
        data = audio.read(bufferSize)

# Streaming audio to the server.
result = client.StreamingEnroll(stream(cfg, audio))

# A certain minimum duration of speech is required for completing enrollment.
# The enrollment status contains information on Whether that has been met or
# whether additional audio is required.  
print(f"enrollment Status:\n{result.enrollment_status}\n")

# Saving the voiceprint data to a file. This can be provided again
# in another StreamingEnroll request (for continuing enrollment) or
# submitted for verification / identification requests.
with open("voiceprint.bin", 'w') as f:
  f.write(result.voiceprint.data)

audio.close()
mic.kill()

package main

import (
	"context"
	"errors"
	"fmt"
	"io"
	"os"
	"os/exec"
	"strings"

	"golang.org/x/sync/errgroup"
	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials/insecure"

	voicebio "github.com/cobaltspeech/go-genproto/cobaltspeech/voicebio/v1"
)

func main() {
	const (
		serverAddress = "localhost:2727"
	)

	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	opts := []grpc.DialOption{
		grpc.WithTransportCredentials(insecure.NewCredentials()), // Using a channel without TLS enabled.
		grpc.WithBlock(),
		grpc.WithReturnConnectionError(),
		grpc.FailOnNonTempDialError(true),
	}

	conn, err := grpc.DialContext(ctx, serverAddress, opts...)
	if err != nil {
		fmt.Printf("failed to dial gRPC connection: %v\n", err)
		os.Exit(1)
	}

	client := voicebio.NewVoiceBioServiceClient(conn)

	// Get server version.
	versionResp, err := client.Version(ctx, &voicebio.VersionRequest{})
	if err != nil {
		fmt.Printf("failed to get server version: %v\n", err)
		os.Exit(1)
	}

	fmt.Printf("%v\n", versionResp)

	// Get list model of models on the server.
	modelResp, err := client.ListModels(ctx, &voicebio.ListModelsRequest{})
	if err != nil {
		fmt.Printf("failed to get model list: %v\n", err)
		os.Exit(1)
	}

	fmt.Println("Models:")
	for _, m := range modelResp.Models {
		fmt.Println(m)
	}
	fmt.Println()

	// Selecting first model.
	m := modelResp.Models[0]

	// Setting audio format to be raw 16-bit signed little endian audio samples
	// recorded at the sample rate expected by the model.
	cfg := &voicebio.EnrollmentConfig{
		ModelId:            m.Id,
		PreviousVoiceprint: nil,
		AudioFormat: &voicebio.AudioFormat{AudioFormat: &voicebio.AudioFormat_AudioFormatRaw{
			AudioFormatRaw: &voicebio.AudioFormatRAW{
				Encoding:   voicebio.AudioEncoding_AUDIO_ENCODING_SIGNED,
				SampleRate: m.Attributes.SampleRate,
				BitDepth:   16,
				ByteOrder:  voicebio.ByteOrder_BYTE_ORDER_LITTLE_ENDIAN,
				Channels:   1,
			},
		},
		},
	}

	// Open microphone stream using sox's rec command and record
	// audio using the config specified above for *10 seconds*.
	maxDuration := 10
	args := fmt.Sprintf("-t raw -r %d -e signed -b 16 -L -c 1 - trim 0 %d", m.Attributes.SampleRate, maxDuration)
	cmd := exec.CommandContext(ctx, "rec", strings.Fields(args)...)
	cmd.Stderr = os.Stderr

	audio, err := cmd.StdoutPipe()
	if err != nil {
		fmt.Printf("failed to open microphone stream: %v\n", err)
		os.Exit(1)
	}

	// Starting routines to record from microphone and stream to server
	// using an errgroup.Group that returns if either one encounters an error.
	eg, ctx := errgroup.WithContext(ctx)

	eg.Go(func() error {
		fmt.Printf("\n[INFO] recording %d seconds from microphone \n", maxDuration)

		if err := cmd.Run(); err != nil {
			return fmt.Errorf("record from microphone: %w", err)
		}

		return nil
	})

	// Starting enrollment.
	result, err := StreamingEnroll(ctx, client, cfg, audio)
	if err != nil {
		fmt.Printf("failed to run streaming enrollment: %v\n", err)
		os.Exit(1)
	}

	if err := eg.Wait(); err != nil {
		fmt.Printf("%v\n", err)
		os.Exit(1)
	}

	// A certain minimum duration of speech is required for completing enrollment.
	// The enrollment status contains information on Whether that has been met or
	// whether additional audio is required.
	fmt.Printf("Enrollment Status: %v\n", result.EnrollmentStatus)

	// Saving the voiceprint data to a file. This can be provided again
	// in another StreamingEnroll request (for continuing enrollment) or
	// submitted for verification / identification requests.
	if err := os.WriteFile("voiceprint.bin", []byte(result.Voiceprint.Data), os.ModePerm); err != nil {
		fmt.Printf("failed to wriet voiceprint data: %v\n", err)
		os.Exit(1)
	}
}

// StreamingEnroll wraps the streaming API for performing speaker enrollment
// (i.e. voiceprint generation) using  the given cfg.
//
// Data is read from the given audio reader into a buffer and streamed to VoiceBio
// server. The default buffer size may be overridden using Options when creating
// the Client.
//
// If any error occurs while reading the audio or sending it to the server, this
// method will immediately exit, returning that error.
func StreamingEnroll(
	ctx context.Context,
	client voicebio.VoiceBioServiceClient,
	cfg *voicebio.EnrollmentConfig,
	audio io.Reader,
) (*voicebio.StreamingEnrollResponse, error) {
	const (
		streamingBufSize = 1024
	)

	// Creating stream.
	stream, err := client.StreamingEnroll(ctx)
	if err != nil {
		return nil, err
	}

	// Sending audio.
	if err := sendAudio(stream, cfg, audio, streamingBufSize); err != nil && !errors.Is(err, io.EOF) {
		// if sendAudio encountered io.EOF, it's only a
		// notification that the stream has closed.  The actual
		// status will be obtained in the CloseAndRecv call. We
		// therefore return on non-EOF errors here.
		return nil, err
	}

	// Returning result.
	return stream.CloseAndRecv()
}

// sendAudio sends audio to a stream.
func sendAudio(
	stream voicebio.VoiceBioService_StreamingEnrollClient,
	cfg *voicebio.EnrollmentConfig,
	audio io.Reader,
	bufSize uint32,
) error {
	// The first message needs to be a config message, and all subsequent
	// messages must be audio messages.

	// Send the config.
	if err := stream.Send(&voicebio.StreamingEnrollRequest{
		Request: &voicebio.StreamingEnrollRequest_Config{Config: cfg},
	}); err != nil {
		// if this failed, we don't need to CloseSend
		return err
	}

	// Stream the audio.
	buf := make([]byte, bufSize)
	for {
		n, err := audio.Read(buf)
		if n > 0 {
			if err2 := stream.Send(&voicebio.StreamingEnrollRequest{
				Request: &voicebio.StreamingEnrollRequest_Audio{
					Audio: &voicebio.Audio{Data: buf[:n]},
				},
			}); err2 != nil {
				// if we couldn't Send, the stream has
				// encountered an error and we don't need to
				// CloseSend.
				return err2
			}
		}

		if err != nil {
			// err could be io.EOF, or some other error reading from
			// audio.  In any case, we need to CloseSend, send the
			// appropriate error to errCh and return from the function
			if err2 := stream.CloseSend(); err2 != nil {
				return err2
			}

			if err != io.EOF {
				return err
			}

			return nil
		}
	}
}