• 1: Speech Recognition & Transcription
• 1.1: Transcribe
• 1.1.1: Getting Started
• 1.1.2: Running Quick Tests
• 1.1.3: Generating SDKs
• 1.1.4: Connecting to the Server
• 1.1.5: Streaming Recognition
• 1.1.6: Recognition Configurations
• 1.1.7: Recognition Context
• 1.1.8: Hybrid vs End-to-End Models
• 1.1.9: API Reference
• 1.1.10: FAQ
• 1.1.11:
• 2: Voice Intelligence
• 2.1: Privacy Screen
• 2.1.1: Server Setup
• 2.1.2: Connecting to the Server
• 2.1.3: Text Redaction
• 2.1.4: Concurrency
• 2.1.5: Privacy Screen Client
• 2.1.6: Redaction Categories
• 2.1.7: Redaction Languages
• 2.1.8: Prerequisites and System Requirements
• 2.1.9: Proto API Reference
• 2.2: VoiceBio
• 2.2.1: Getting Started
• 2.2.2: Generating SDKs
• 2.2.3: Connecting to the Server
• 2.2.4: Streaming Enrollment
• 2.2.5: Streaming Verification
• 2.2.6: Streaming Identification
• 2.2.7: Comparing Voiceprints
• 2.2.8: Vectorizing Voiceprints
• 2.2.9: API Reference
• 2.2.10:
• 3: Voice User Interfaces
• 3.1: VoiceGen
• 3.1.1: Getting Started
• 3.1.2: Generating SDKs
• 3.1.3: Connecting to the Server
• 3.1.4: Streaming Synthesis
• 3.1.5: API Reference
• 3.1.6:

1 - Speech Recognition & Transcription

Demos

• Transcribe Demo

1.1 - Transcribe

Web Demo

Cobalt’s Transcribe engine is a state-of-the-art speech recognition system. Cobalt Transcribe supports two different DNN architectures:

• Hybrid models combine separately tunable Acoustic Models, Lexicons, and Language Models, making them highly customizable for specific use cases. Hybrid models support extremely low-latency partial results.

• End-to-end models go straight from sounds to words in the same DNN. They tend to be more accurate for general use cases, particularly for systems in which sub-second response time is not required.

Cobalt Transcribe is a highly flexible system that can run on-premise, in your private cloud, or fully embedded on your device. Your data – both the audio and the transcripts – never leave your control.

The SDK is based on a gRPC API and client code can be easily generated for different languages using the proto definition, including C++, C#, Go, Java and Python, and can add support for more languages as required.

Once running, Transcribe’s API provides a method to which you can stream audio. This audio can either be from a microphone or a file. We recommend uncompressed WAV as the encoding, but support other formats such as MP3, ulaw etc.

Transcribe’s API provides a number of options for returning the speech recognition results. The results are passed back using Google’s protobuf library, allowing them to be handled natively by your application. Transcribe can estimate its confidence in the transcription result at the word or utterance level, along with timestamps of the words. Confidence scores are in the range 0-1. Transcribe’s output options are described below.

Automatic Transcription Results

The simplest result that Transcribe returns is its best guess at the transcription of your audio. Transcribe recognizes the audio you are streaming, listens for the end of each utterance, and returns the speech recognition result.

Transcribe maintains its transcriptions in an N-best list, i.e. is the top N transcriptions from the recognizer. The best ASR result is the first entry in this list.

{
  "alternatives": [
    {
      "transcript": "TOMORROW IS A NEW DAY",
      "confidence": 0.514
    },
    {
      "transcript": "TOMORROW IS NEW DAY",
      "confidence": 0.201
    },
    {
      "transcript": "TOMORROW IS A <UNK> DAY",
      "confidence": 0.105
    },
    {
      "transcript": "TOMORROW IS ISN'T NEW DAY",
      "confidence": 0.093
    },
    {
      "transcript": "TOMORROW IS A YOUR DAY",
      "confidence": 0.087
    }
  ],
}

A single stream may consist of multiple utterances separated by silence. Transcribe handles each utterance separately.

For longer utterances, it is often useful to see the partial speech recognition results while the audio is being streamed. For example, this allows you to see what the ASR system is predicting in real-time while someone is speaking. Transcribe supports both partial and final ASR results.

Confusion Network

A Confusion Network is a form of speech recognition output that’s been turned into a compact graph representation of many possible transcriptions, as here:

Note that <eps> in this representation is silence.

{
  "cnet": {
    "links": [
      {
        "duration": "1.350s",
        "arcs": [
          {
            "word": "<eps>",
            "confidence": 1.0
          }
        ],
        "startTime": "0s"
      },
      {
        "duration": "0.690s",
        "arcs": [
          {
            "word": "TOMORROW",
            "confidence": 1.0
          }
        ],
        "startTime": "1.350s"
      },
      {
        "duration": "0.080s",
        "arcs": [
          {
            "word": "<eps>",
            "confidence": 1.0
          }
        ],
        "startTime": "2.040s"
      },
      {
        "duration": "0.168s",
        "arcs": [
          {
            "word": "IS",
            "confidence": 0.892
          },
          {
            "word": "<eps>",
            "confidence": 0.108
          }
        ],
        "startTime": "2.120s"
      },
      {
        "duration": "0.010s",
        "arcs": [
          {
            "word": "<eps>",
            "confidence": 1.0
          }
        ],
        "startTime": "2.288s"
      },
      {
        "duration": "0.093s",
        "arcs": [
          {
            "word": "A",
            "confidence": 0.620
          },
          {
            "word": "<eps>",
            "confidence": 0.233
          },
          {
            "word": "ISN'T",
            "confidence": 0.108
          },
          {
            "word": "THE",
            "confidence": 0.039
          }
        ],
        "startTime": "2.298s"
      },
      {
        "duration": "0.005s",
        "arcs": [
          {
            "word": "<eps>",
            "confidence": 1.0
          }
        ],
        "startTime": "2.391s"
      },
      {
        "duration": "0.273s",
        "arcs": [
          {
            "word": "NEW",
            "confidence": 0.661
          },
          {
            "word": "<UNK>",
            "confidence": 0.129
          },
          {
            "word": "YOUR",
            "confidence": 0.107
          },
          {
            "word": "YOU",
            "confidence": 0.102
          }
        ],
        "startTime": "2.396s"
      },
      {
        "duration": "0s",
        "arcs": [
          {
            "word": "<eps>",
            "confidence": 1.0
          }
        ],
        "startTime": "2.670s"
      },
      {
        "duration": "0.420s",
        "arcs": [
          {
            "word": "DAY",
            "confidence": 0.954
          },
          {
            "word": "TODAY",
            "confidence": 0.044
          },
          {
            "word": "<UNK>",
            "confidence": 0.002
          }
        ],
        "startTime": "2.670s"
      },
      {
        "duration": "0.270s",
        "arcs": [
          {
            "word": "<eps>",
            "confidence": 1.0
          }
        ],
        "startTime": "3.090s"
      }
    ]
  }
}

Formatted output

Many speech recognition systems typically output raw words exactly as spoken, without any formatting which can improve intelligibility. Cobalt Transcribe’s customizable formatting suite enables a variety of intelligent formatting options:

• Capitalizing the first letter of the utterance
• Numbers: “cobalt’s atomic number is twenty seven” -> “Cobalt’s atomic number is 27”
• TrueCasing: “the iphone was launched in two thousand and seven” -> “The iPhone was launched in 2007”
• Ordinals: “summer solstice is twenty first june” -> “Summer solstice is 21st June”

1.1.1 - Getting Started

Using Cobalt Transcribe

• A typical Transcribe release, provided as a compressed archive, will contain a linux binary (transcribe-server) for the required native CPU architecture, appropriate Dockerfile and models.

• Cobalt Transcribe runs either locally on linux or using Docker.

• Cobalt Transcribe will serve the Transcribe GRPC API on port 2727. A web demo will be enabled on port 8080.

• To quickly try out Transcribe, first start the server as shown below and open the web demo at http://localhost:8080 in your browser to send live microphone input or upload an audio file for transcription. You can also use the SDK to use Transcribe from within your application or just command line.

Running Transcribe Server Locally on Linux

./transcribe-server

• By default, the binary assumes the presence of a configuration file, located in the same directory, named: transcribe-server.cfg.toml. A different config file may be specified using the --config argument.

Running Transcribe Server as a Docker Container

To build and run the Docker image for Transcribe, run:

docker build -t cobalt-transcribe .
docker run -p 2727:2727 -p 8080:8080 cobalt-transcribe

How to Get a Copy of the Transcribe Server and Models

Please contact us for finding a product release or Transcribe model best suited to your requirements.

The demo release you will receive is a compressed archive (tar.bz2) and is structured accordingly:

release.tar.bz2
├── COPYING
├── README.md
├── transcribe-server
├── transcribe-server.cfg.toml
├── cobalt.license.key
├── Dockerfile
├── models
│   └── en_US-16khz
├── formatters
│   └── en_US-16khz
│
└── cobalt.license.key [ provided separately, needs to be copied over ]

• The README.md file contains information about this release and instructions for how to start the server on your system.

• The transcribe-server is the server program which is configured using the transcribe-server.cfg.toml file.

• The Dockerfile can be used to create a container that will let you run Transcribe server on non-linux systems such as MacOS and Windows.

• The models and formatters directories contain your speech recognition and text formatting models. The content of these directories will depend on the models you downloaded.

System Requirements

Cobalt Transcribe runs on Linux. You can run it directly as a linux application, or using Docker.

You can evaluate the product on Windows or Linux using Docker Desktop but we would not recommend this setup for use in a production environment.

A Cobalt Transcribe release typically includes a single Transcribe model together with binaries and config files. The general purpose Transcribe models take up to 4GB disk space, and need a minimum of 4GB RAM when evaluating locally. For production workloads, we recommend configuring containerized applications with each instance allocated with 4 CPUs and 8GB RAM.

Cobalt Transcribe runs on x86_64 CPUs. We also support Arm64 CPUs, including processors such as the Graviton (AWS c7g EC2 instances). Transcribe is significantly more cost effective to run on C7g instances compared to similarly sized Intel or AMD processors, and we can provide you an Arm64 release on request.

To integrate Cobalt Transcribe into your application, please follow the next steps to Generate the SDK in a language of your choice.

1.1.2 - Running Quick Tests

In the release package you received, we have bundled a prebuilt application, a command-line client, that you can use to connect to the server you now have running. You can use this client to send files to the server for saving the transcription, either as a stream of text, or as JSON with additional information.

Obtaining the Client Application

In the release package of Cobalt Transcribe, you should find a folder called transcribe-client, that contains client binaries for multiple platforms.

transcribe-client/
└── bin
    ├── darwin_amd64
    │   └── transcribe-client
    └── linux_amd64
        └── transcribe-client

This client is implemented in Go, and the code is available for reference.

If you need to build the client yourself, instead of using the pre-packaged version in the release, you can install it using:

go install github.com/cobaltspeech/examples-go/transcribe/transcribe-client@latest

Running The Client

After you follow the Getting Started instructions, you should have Cobalt Transcribe server running and listening on port 2727 on the local machine.

You can run the transcribe client in various ways:

# Transcribe a file using the default address of the Cobalt Transcribe Server (localhost:2727)
transcribe-client recognize input.wav

# Transcribe a file, pointing to some other server address
transcribe-client recognize input.wav --server host:port

# Transcribe a file using the default server address, and save output as a JSON file.
transcribe-client recognize input.wav --output-json output.json

# (Advanced use): List information about the models available on the server.
transcribe-client list

# (Advanced use): Transcribe a file, get word level timestamps and confidences.
transcribe-client recognize input.wav --output-json output.json --recognition-config '{"enable_word_details": true}'

For more details on the recognition-config struct, please see the API spec

# Getting usage information
transcribe-client --help

1.1.3 - Generating SDKs

• APIs for all Cobalt’s services are defined as a protocol buffer specification or simply a proto file and be found in the cobaltspeech/proto github repository.

• The proto file allows a developer to auto-generate client SDKs for a number of different programming languages. Step by step instructions for generating your own SDK can be found below.

• We provide pre-generated SDKs for a couple of languages. You can choose to use these instead of generating your own. These are listed here along with instructions on how to install / import them into your projects.

Pre-generated SDKs

Golang

• Pre-generated SDK files for Golang can be found in the cobaltspeech/go-genproto repo

• To use it in your Go project, simply import it:

import transcribepb "github.com/cobaltspeech/go-genproto/cobaltspeech/transcribe/v5"

• An example client using the above repo can be found here.

Python

• Pre-generated SDK files for Python can be found in the cobaltspeech/py-genproto repo

• The Python SDK depends on Python >= 3.5. You may use pip to perform a system-wide install, or use virtualenv for a local install. To use it in your Python project, install it:

pip install --upgrade pip
pip install "git+https://github.com/cobaltspeech/py-genproto"

Generating SDKs

Step 1. Installing buf

• To work with proto files, we recommend using buf, a user-friendly command line tool that can be configured generate documentation, schemas and SDK code for different languages.

• Linux (from Github)
• MacOS
• Reference

# Latest version as of March 14th, 2023.

COBALT="${HOME}/cobalt"
  mkdir -p "${COBALT}/bin"

VERSION="1.15.1"
URL="https://github.com/bufbuild/buf/releases/download/v${VERSION}/buf-$(uname -s)-$(uname -m)"
  curl -L ${URL} -o "${COBALT}/bin/buf"

# Give executable permissions and adding to $PATH.

chmod +x "${COBALT}/bin/buf"
  export PATH="${PATH}:${COBALT}/bin"

brew install bufbuild/buf/buf

Step 2. Getting proto files

• Clone the cobaltspeech/proto repository:

COBALT="${HOME}/cobalt"
mkdir -p "${COBALT}/git"

# Change this to where you want to clone the repo to.
PROTO_REPO="${COBALT}/git/proto"

git clone https://github.com/cobaltspeech/proto "${PROTO_REPO}"

Step 3. Generating code

• The cobaltspeech/proto repo provides a buf.gen.yaml config file to get you started with a couple of languages.

• Other plugins can be added to the buf.gen.yaml file to generate SDK code for more languages.

• To generate the SDKs, simply run the following (assuming the buf binary is in your $PATH)

cd "${PROTO_REPO}"

# Removing any previously generated files.
rm -rf ./gen

# Generating code for all proto files inside the `proto` directory.
buf generate proto

• You should now have a folder called gen inside ${PROTO_REPO} that contains the generated code. The latest version of the transcribe API is v5. You can import / include / copy the generated files into your projects as per the conventions of different languages.

• Python
• Golang
• C++

gen
├── ... other languages ...
└── py
└── cobaltspeech
├── ... other services ...
└── transcribe
└── v5
├── transcribe_pb2_grpc.py
├── transcribe_pb2.py
└── transcribe_pb2.pyi

gen
├── ... other languages ...
└── go
   ├── cobaltspeech
   │ ├── ...
   │   └── transcribe
   │   └── v5
   │   ├── transcribe_grpc.pb.go
   │   └── transcribe.pb.go
   └── gw
   └── cobaltspeech
   ├── ...
   └── transcribe
   └── v5
   └── transcribe.pb.gw.go

gen
├── ... other languages ...
└── cpp
   └── cobaltspeech
   ├── ...
   └── transcribe
   └── v5
   ├── transcribe.grpc.pb.cc
   ├── transcribe.grpc.pb.h
   ├── transcribe.pb.cc
   ├── transcribe.pb.h
   ├── transcribe.pb.validate.cc
   └── transcribe.pb.validate.h

Step 4. Installing gPRC and protobuf

• A couple of gRPC and protobuf dependencies are required along with the code generated above. The method of installing them depends on the programming language being used.
• These dependencies and the most common way of installing/ / including them are listed below for some chosen languages.

• Python
• Golang
• C++

# It is encouraged to this inside a python virtual environment

# to avoid creating version conflicts for other scripts that may

# be using these libraries.

pip install --upgrade protobuf
pip install --upgrade grpcio
pip install --upgrade google-api-python-client

go get google.golang.org/protobuf
go get google.golang.org/grpc
go get google.golang.org/genproto

# More details on grpc installation can be found at:

# https://grpc.io/docs/languages/cpp/quickstart/

COBALT="${HOME}/cobalt"
mkdir -p "${COBALT}/git"

# Latest version as of 14th March, 2023.

VERSION="v1.52.0"
GRPC_REPO="${COBALT}/git/grpc-${VERSION}"

git clone \
 --recurse-submodules --depth 1 --shallow-submodules \
 -b "${VERSION}" \
 https://github.com/grpc/grpc ${GRPC_REPO}

cd "${GRPC_REPO}"
mkdir -p cmake/build

# Change this to where you want to install libprotobuf and libgrpc.

# It is encouraged to install gRPC locally as there is no easy way to

# uninstall gRPC after you’ve installed it globally.

INSTALL_DIR="${COBALT}"

cd cmake/build
cmake \
 -DgRPC_INSTALL=ON \
 -DgRPC_BUILD_TESTS=OFF \
 -DCMAKE_INSTALL_PREFIX=${INSTALL_DIR} \
 ../..

make -j
make install

1.1.4 - Connecting to the Server

• Once you have your Transcribe server up and running, and have generated the SDK for your project, you can connect to a running instance of Transcribe server, by “dialing” a gRPC connection.

• First, you need to know the address where the server is running: e.g. host:grpc_port. By default, this is localhost:2727 and should be logged to the terminal when you first start Transcribe server as grpcAddr:

2023/03/15 07:54:01 info  {"license":"Copyright © 2015--present. Cobalt Speech and Language, Inc.  For additional details, including information about open source components used in this software, please see the COPYING file bundled with this program."}
2023/03/15 07:54:01 info  {"msg":"reading config file","path":"transcribe-server.cfg.toml"}
2023/03/15 07:54:01 info  {"msg":"version","server":"v5.3.5-b70948b","built":"2023-03-14"}
2023/03/15 07:54:01 info  {"msg":"server initializing"}
2023/03/15 07:54:01 info  {"msg":"server started","grpcAddr":"[::]:8027","httpApiAddr":"[::]:8030","httpOpsAddr":"[::]:8031"}

• The default binding address and port for the gRPC / http server (bundled webpage demo) can be configured in the transcribe-server config file.

Default Connection

• The following code snippet connects to the server and queries its version. It connects to the server using an “insecure” gRPC channel. This would be the case if you have just started up a local instance of Transcribe server without TLS enabled.

• Python
• Go

import grpc
import cobaltspeech.transcribe.v5.transcribe_pb2_grpc as stub
import cobaltspeech.transcribe.v5.transcribe_pb2 as transcribe

serverAddress = "localhost:2727"

# Using a channel without TLS enabled.
channel = grpc.insecure_channel(serverAddress)
client = stub.TranscribeServiceStub(channel)

# Get server version.
versionResp = client.Version(transcribe.VersionRequest())
print(versionResp)

package main

import (
	"context"
	"fmt"
	"os"
	"time"

	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials/insecure"

	transcribepb "github.com/cobaltspeech/go-genproto/cobaltspeech/transcribe/v5"
)

func main() {
	const (
		serverAddress  = "localhost:2727"
	)

	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	opts := []grpc.DialOption{
		grpc.WithTransportCredentials(insecure.NewCredentials()), // Using a channel without TLS enabled.
		grpc.WithBlock(),
		grpc.WithReturnConnectionError(),
		grpc.FailOnNonTempDialError(true),
	}

	conn, err := grpc.DialContext(ctx, serverAddress, opts...)
	if err != nil {
		fmt.Printf("failed to dial gRPC connection: %v\n", err)
		os.Exit(1)
	}

	client := transcribepb.NewTranscribeServiceClient(conn)

	// Get server version.
	versionResp, err := client.Version(ctx, &transcribepb.VersionRequest{})
	if err != nil {
		fmt.Printf("failed to get server version: %v\n", err)
		os.Exit(1)
	}

	fmt.Printf("%v\n", versionResp)
}

Connect with TLS

• In our recommended setup for deployment, TLS is enabled in the gRPC connection, and when connecting to the server, clients validate the server’s SSL certificate to make sure they are talking to the right party. This is similar to how “https” connections work in web browsers.

• The following snippets show how to connect to a Transcribe Server that has TLS enabled. They use the cobalt’s self-hosted demo server at demo.cobaltspeech.com:2727, but you obviously use your own server instance.

• Python
• Go

import grpc
import cobaltspeech.transcribe.v5.transcribe_pb2_grpc as stub
import cobaltspeech.transcribe.v5.transcribe_pb2 as transcribe

serverAddress = "demo.cobaltspeech.com:2727"

# Setup a gRPC connection with TLS. You can optionally provide your own
# root certificates and private key to grpc.ssl_channel_credentials()
# for mutually authenticated TLS.
creds = grpc.ssl_channel_credentials()
channel = grpc.secure_channel(serverAddress, creds)
client = stub.TranscribeServiceStub(channel)

# Get server version.
versionResp = client.Version(transcribe.VersionRequest())
print(versionResp)

package main

import (
	"context"
	"crypto/tls"
	"fmt"
	"os"
	"time"

	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials"

	transcribepb "github.com/cobaltspeech/go-genproto/cobaltspeech/transcribe/v5"
)

func main() {
	const (
		serverAddress  = "demo.cobaltspeech.com:2727"
		connectTimeout = 10 * time.Second
	)

	// Setup a gRPC connection with TLS. You can optionally provide your own
	// root certificates and private key through tls.Config for mutually
	// authenticated TLS.
	tlsCfg := tls.Config{}
	creds := credentials.NewTLS(&tlsCfg)

	ctx, cancel := context.WithTimeout(context.Background(), connectTimeout)
	defer cancel()

	opts := []grpc.DialOption{
		grpc.WithTransportCredentials(creds),
		grpc.WithBlock(),
		grpc.WithReturnConnectionError(),
		grpc.FailOnNonTempDialError(true),
	}

	conn, err := grpc.DialContext(ctx, serverAddress, opts...)
	if err != nil {
		fmt.Printf("failed to dial gRPC connection: %v\n", err)
		os.Exit(1)
	}

	client := transcribepb.NewTranscribeServiceClient(conn)

	// Get server version.
	versionResp, err := client.Version(ctx, &transcribepb.VersionRequest{})
	if err != nil {
		fmt.Printf("failed to get server version: %v\n", err)
		os.Exit(1)
	}

	fmt.Printf("%v\n", versionResp)
}

Client Authentication

• In some setups, it may be desired that the server should also validate clients connecting to it and only respond to the ones it can verify. If your Transcribe server is configured to do client authentication, you will need to present the appropriate certificate and key when connecting to it.

• Please note that in the client-authentication mode, the client will still also verify the server’s certificate, and therefore this setup uses mutually authenticated TLS.

• The following snippets show how to present client certificates when setting up the credentials. These could then be used in the same way as the examples above to connect to a TLS enabled server.

• Python
• Go

creds = grpc.ssl_channel_credentials(
  root_certificates=root_certificates,  # PEM certificate as byte string
  private_key=private_key,              # PEM client key as byte string 
  certificate_chain=certificate_chain,  # PEM client certificate as byte string
)

package main

import (
	// ...

	"crypto/tls"
	"crypto/x509"
	"fmt"
	"os"

	// ..
)

func main() {
	// ...

	// Root PEM certificate for validating self-signed server certificate
	var rootCert []byte

	// Client PEM certificate and private key.
	var certPem, keyPem []byte

	caCertPool := x509.NewCertPool()
	if ok := caCertPool.AppendCertsFromPEM(rootCert); !ok {
		fmt.Printf("unable to use given caCert\n")
		os.Exit(1)
	}

	clientCert, err := tls.X509KeyPair(certPem, keyPem)
	if err != nil {
		fmt.Printf("unable to use given client certificate and key: %v\n", err)
		os.Exit(1)
	}

	tlsCfg := tls.Config{
		RootCAs:      caCertPool,
		Certificates: []tls.Certificate{clientCert},
	}

	creds := credentials.NewTLS(&tlsCfg)

	// ...
}

1.1.5 - Streaming Recognition

• The following example shows how to transcribe a audio stream using Transcribe’s StreamingRecognize request. The stream can come from a file on disk or be directly from a microphone in real time.

Streaming from an audio file

• We support several headered file formats including WAV, MP3, FLAC etc. For more details, please see the protocol buffer specification here.

• The examples below use a WAV file as input to the streaming recognition. We will query the server for available models and use the first model to transcribe the speech.

• Python
• Go

import grpc
import cobaltspeech.transcribe.v5.transcribe_pb2_grpc as stub
import cobaltspeech.transcribe.v5.transcribe_pb2 as transcribe

serverAddress = "localhost:2727"

# Using a channel without TLS enabled.
channel = grpc.insecure_channel(serverAddress)
client = stub.TranscribeServiceStub(channel)

# Get server version.
versionResp = client.Version(transcribe.VersionRequest())
print(versionResp)

# Get list of models on the server.
modelResp = client.ListModels(transcribe.ListModelsRequest())
for model in modelResp.models:
    print(model)

# Select a model ID from the list above. Going with the first model
# in this example.
modelID = modelResp.models[0].id

# Set the recognition config. We don't set the audio format and let the
# server auto-detect the format from the file header.
cfg = transcribe.RecognitionConfig(
    model_id=modelID,
)

# Open audio file.
audio = open("test.wav", "rb")

# The first request to the server should only contain the
# recognition configuration. Subsequent requests should contain
# audio bytes. We can write a simple generator to do this.
def stream(cfg, audio, bufferSize=1024):
    yield transcribe.StreamingRecognizeRequest(config=cfg)
    
    data = audio.read(bufferSize)
    while len(data) > 0:
        yield transcribe.StreamingRecognizeRequest(
          audio=transcribe.RecognitionAudio(data=data),
        )
        data = audio.read(bufferSize)

# We also define a callback function to execute for each response.
# The example below just prints the formatted transcript to stdout.
def processResponse(resp):
    result = resp.result
    hyp = result.alternatives[0]                    # 1-best hypothesis.
    transcript = hyp.transcript_formatted           # Formatted transcript.
    start = hyp.start_time_ms / 1000.0              # Converting to seconds.
    end = start + hyp.duration_ms / 1000.0          # Converting to seconds.
    newLine = "\r" if result.is_partial else "\n\n" # Will not move to new line for partial results.
    print(f"[{start:0.2f}:{end:0.2f}] {transcript}", end=newLine)

# Streaming requests to the server.
for resp in client.StreamingRecognize(stream(cfg, audio)):
    processResponse(resp)

package main

import (
	"context"
	"errors"
	"fmt"
	"io"
	"os"
	"sync"

	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials/insecure"

	transcribe "github.com/cobaltspeech/go-genproto/cobaltspeech/transcribe/v5"
)

func main() {
	const (
		serverAddress = "localhost:2727"
	)

	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	opts := []grpc.DialOption{
		grpc.WithTransportCredentials(insecure.NewCredentials()), // Using a channel without TLS enabled.
		grpc.WithBlock(),
		grpc.WithReturnConnectionError(),
		grpc.FailOnNonTempDialError(true),
	}

	conn, err := grpc.DialContext(ctx, serverAddress, opts...)
	if err != nil {
		fmt.Printf("failed to dial gRPC connection: %v\n", err)
		os.Exit(1)
	}

	client := transcribe.NewTranscribeServiceClient(conn)

	// Get server version.
	versionResp, err := client.Version(ctx, &transcribe.VersionRequest{})
	if err != nil {
		fmt.Printf("failed to get server version: %v\n", err)
		os.Exit(1)
	}

	fmt.Printf("%v\n", versionResp)

	// Get list model of models on the server.
	modelResp, err := client.ListModels(ctx, &transcribe.ListModelsRequest{})
	if err != nil {
		fmt.Printf("failed to get model list: %v\n", err)
		os.Exit(1)
	}

	for _, m := range modelResp.Models {
		fmt.Println(m)
	}
	fmt.Println()

	// Selecting the first model.
	cfg := &transcribe.RecognitionConfig{
		ModelId: modelResp.Models[0].Id,
	}

	// Opening audio file.
	audio, err := os.Open("test.wav")
	if err != nil {
		fmt.Printf("failed to open audio file: %v\n", err)
		os.Exit(1)
	}

	defer audio.Close()

	// Starting recognition.
	err = StreamingRecognize(ctx, client, cfg, audio, printTranscript)
	if err != nil {
		fmt.Printf("failed to run streaming recognition: %v\n", err)
		os.Exit(1)
	}
}

// StreamingRecognize wraps the bidirectional streaming API for performing
// speech recognition. It sets up recognition using the given cfg.
//
// Data is read from the given audio reader into a buffer and streamed to cubic
// server. The default buffer size may be overridden using Options when creating
// the Client.
//
// As results are received from Transcribe server, they will be sent to the
// provided handlerFunc.
//
// If any error occurs while reading the audio or sending it to the server, this
// method will immediately exit, returning that error.
//
// This function returns only after all results have been passed to the
// resultHandler.
func StreamingRecognize(
	ctx context.Context,
	client transcribe.TranscribeServiceClient,
	cfg *transcribe.RecognitionConfig,
	audio io.Reader,
	handlerFunc func(*transcribe.StreamingRecognizeResponse),
) error {
	const (
		streamingBufSize = 1024
	)

	// Creating stream.
	stream, err := client.StreamingRecognize(ctx)
	if err != nil {
		return err
	}

	// There are two concurrent processes going on. We will create a new
	// goroutine to read audio and stream it to the server.  This goroutine
	// will receive results from the stream.  Errors could occur in both
	// go routines.  We therefore setup a channel, errCh, to hold these
	// errors. Both go routines are designed to send up to one error, and
	// return immediately. Therefore we use a buffered channel with a
	// capacity of two.
	errCh := make(chan error, 2)

	// start streaming audio in a separate goroutine
	var wg sync.WaitGroup
	wg.Add(1)
	go func() {
		if err := sendAudio(stream, cfg, audio, streamingBufSize); err != nil && !errors.Is(err, io.EOF) {
			// if sendAudio encountered io.EOF, it's only a
			// notification that the stream has closed.  The actual
			// status will be obtained in a subsequent Recv call, in
			// the other goroutine below.  We therefore only forward
			// non-EOF errors.
			errCh <- err
		}

		wg.Done()
	}()

	// Receive results from the stream.
	for {
		in, err := stream.Recv()
		if errors.Is(err, io.EOF) {
			break
		}

		if err != nil {
			errCh <- err
			break
		}

		handlerFunc(in)
	}

	wg.Wait()

	select {
	case err := <-errCh:
		// There may be more than one error in the channel, but it is
		// very likely they are related (e.g. connection reset causing
		// both the send and recv to fail) and we therefore return the
		// first error and discard the other.
		return err
	default:
		return nil
	}
}

// printTranscript is a callback function given to StreamingRecognize method to
// print results that are returned though the gRPC stream.
func printTranscript(resp *transcribe.StreamingRecognizeResponse) {
	if resp.Error != nil {
		fmt.Printf("\n[ERROR] server returned an error: %v\n", resp.Error)
		return
	}

	hyp := resp.Result.Alternatives[0]
	startTime := float32(hyp.StartTimeMs) / 1000.0
	endTime := startTime + float32(hyp.DurationMs)/1000.0

	if resp.Result.IsPartial {
		fmt.Printf("\r[%0.2f:%0.2f] %s", startTime, endTime, hyp.TranscriptFormatted)
	} else {
		fmt.Printf("[%0.2f:%0.2f] %s\n\n", startTime, endTime, hyp.TranscriptFormatted)
	}
}

// sendAudio sends audio to a stream.
func sendAudio(
	stream transcribe.TranscribeService_StreamingRecognizeClient,
	cfg *transcribe.RecognitionConfig,
	audio io.Reader,
	bufSize uint32,
) error {
	// The first message needs to be a config message, and all subsequent
	// messages must be audio messages.

	// Send the recognition config
	if err := stream.Send(&transcribe.StreamingRecognizeRequest{
		Request: &transcribe.StreamingRecognizeRequest_Config{Config: cfg},
	}); err != nil {
		// if this failed, we don't need to CloseSend
		return err
	}

	// Stream the audio.
	buf := make([]byte, bufSize)
	for {
		n, err := audio.Read(buf)
		if n > 0 {
			if err2 := stream.Send(&transcribe.StreamingRecognizeRequest{
				Request: &transcribe.StreamingRecognizeRequest_Audio{
					Audio: &transcribe.RecognitionAudio{Data: buf[:n]},
				},
			}); err2 != nil {
				// if we couldn't Send, the stream has
				// encountered an error and we don't need to
				// CloseSend.
				return err2
			}
		}

		if err != nil {
			// err could be io.EOF, or some other error reading from
			// audio.  In any case, we need to CloseSend, send the
			// appropriate error to errCh and return from the function
			if err2 := stream.CloseSend(); err2 != nil {
				return err2
			}
			if err != io.EOF {
				return err
			}
			return nil

		}
	}
}

Streaming from microphone

• Streaming audio from microphone input basically requires a reader interface that can provided audio samples recorded from a microphone; typically this requires interaction with system libraries. Another option is to use an external command line tool like sox to record and pipe audio into the client.

• The examples below use the latter approach by using the rec command provided with sox to record and stream the audio.

• Python
• Go

#!/usr/bin/env python3

# This example assumes sox is installed on the system and is available
# in the system's PATH variable. Instead of opening a regular file from
# disk, we open a subprocess that executes sox's rec command to record
# audio from the system's default microphone.

import grpc
import cobaltspeech.transcribe.v5.transcribe_pb2_grpc as stub
import cobaltspeech.transcribe.v5.transcribe_pb2 as transcribe
import subprocess

serverAddress = "localhost:2727"

# Using a channel without TLS enabled.
channel = grpc.insecure_channel(serverAddress)
client = stub.TranscribeServiceStub(channel)

# Get server version.
versionResp = client.Version(transcribe.VersionRequest())
print(versionResp)

# Get list of models on the server.
modelResp = client.ListModels(transcribe.ListModelsRequest())
for model in modelResp.models:
    print(model)

# Select a model ID from the list above. Going with the first model
# in this example.
m = modelResp.models[0]
modelID = m.id

# Setting audio format to be raw 16-bit signed little endian audio samples
# recorded at the sample rate expected by the model.
cfg = transcribe.RecognitionConfig(
    model_id=modelID,
	audio_format_raw=transcribe.AudioFormatRAW(
      encoding="AUDIO_ENCODING_SIGNED",
      bit_depth=16,
      byte_order="BYTE_ORDER_LITTLE_ENDIAN",
      sample_rate=m.attributes.sample_rate,
      channels=1,
    )
)

# Open microphone stream using sox's rec command and record
# audio using the config specified above.
cmd = f"rec --no-show-progress -t raw -r {m.attributes.sample_rate} -e signed -b 16 -L -c 1 -"
mic = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE)
audio = mic.stdout

try:
  _ = audio.read(1024) # Trying to read some bytes as sanity check.
except Exception as err:
  	print(f"[ERROR] failed to read audio from mic stream: {err}")

print("\n[INFO] recording from microphone ... Press ctrl + c to exit\n")

# The first request to the server should only contain the
# recognition configuration. Subsequent requests should contain
# audio bytes. We can write a simple generator to do this.
def stream(cfg, audio, bufferSize=1024):
    yield transcribe.StreamingRecognizeRequest(config=cfg)
    
    data = audio.read(bufferSize)
    while len(data) > 0:
        yield transcribe.StreamingRecognizeRequest(
          audio=transcribe.RecognitionAudio(data=data),
        )
        data = audio.read(bufferSize)

# We also define a callback function to execute for each response.
# The example below just prints the formatted transcript to stdout.
def processResponse(resp):
    result = resp.result
    hyp = result.alternatives[0]                    # 1-best hypothesis.
    transcript = hyp.transcript_formatted           # Formatted transcript.
    start = hyp.start_time_ms / 1000.0              # Converting to seconds.
    end = start + hyp.duration_ms / 1000.0          # Converting to seconds.
    newLine = "\r" if result.is_partial else "\n\n" # Will not move to new line for partial results.
    print(f"[{start:0.2f}:{end:0.2f}] {transcript}", end=newLine)

# Streaming requests to the server.
try:
  for resp in client.StreamingRecognize(stream(cfg, audio)):
      processResponse(resp)
except KeyboardInterrupt:
	# Stop streaming when ctrl + c pressed.
	pass
except Exception as err:
	print(f"[ERROR] failed to stream audio: {err}")

audio.close()
mic.kill()

package main

import (
	"context"
	"errors"
	"fmt"
	"io"
	"os"
	"os/exec"
	"os/signal"
	"strings"
	"sync"
	"syscall"

	"golang.org/x/sync/errgroup"

	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials/insecure"

	transcribe "github.com/cobaltspeech/go-genproto/cobaltspeech/transcribe/v5"
)

func main() {
	const (
		serverAddress = "localhost:2727"
	)

	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	opts := []grpc.DialOption{
		grpc.WithTransportCredentials(insecure.NewCredentials()), // Using a channel without TLS enabled.
		grpc.WithBlock(),
		grpc.WithReturnConnectionError(),
		grpc.FailOnNonTempDialError(true),
	}

	conn, err := grpc.DialContext(ctx, serverAddress, opts...)
	if err != nil {
		fmt.Printf("failed to dial gRPC connection: %v\n", err)
		os.Exit(1)
	}

	client := transcribe.NewTranscribeServiceClient(conn)

	// Get server version.
	versionResp, err := client.Version(ctx, &transcribe.VersionRequest{})
	if err != nil {
		fmt.Printf("failed to get server version: %v\n", err)
		os.Exit(1)
	}

	fmt.Printf("%v\n", versionResp)

	// Get list model of models on the server.
	modelResp, err := client.ListModels(ctx, &transcribe.ListModelsRequest{})
	if err != nil {
		fmt.Printf("failed to get model list: %v\n", err)
		os.Exit(1)
	}

	for _, m := range modelResp.Models {
		fmt.Println(m)
	}
	fmt.Println()

	// Selecting first model.
	m := modelResp.Models[0]

	// Setting audio format to be raw 16-bit signed little endian audio samples
	// recorded at the sample rate expected by the model.
	cfg := &transcribe.RecognitionConfig{
		ModelId: m.Id,
		AudioFormat: &transcribe.RecognitionConfig_AudioFormatRaw{
			AudioFormatRaw: &transcribe.AudioFormatRAW{
				Encoding:   transcribe.AudioEncoding_AUDIO_ENCODING_SIGNED,
				SampleRate: m.Attributes.SampleRate,
				BitDepth:   16,
				ByteOrder:  transcribe.ByteOrder_BYTE_ORDER_LITTLE_ENDIAN,
				Channels:   1,
			},
		},
	}

	// Open microphone stream using sox's rec command and record
	// audio using the config specified above.
	args := fmt.Sprintf("--no-show-progress -t raw -r %d -e signed -b 16 -L -c 1 -", m.Attributes.SampleRate)
	cmd := exec.CommandContext(ctx, "rec", strings.Fields(args)...)

	audio, err := cmd.StdoutPipe()
	if err != nil {
		fmt.Printf("failed to open microphone stream: %v\n", err)
		os.Exit(1)
	}

	// Starting routines to record from microphone and stream to server
	// using an errgroup.Group that returns if either one encounters an error.
	eg, ctx := errgroup.WithContext(ctx)

	eg.Go(func() error {
		fmt.Printf("\n[INFO] recording from microphone ... Press ctrl + c to exit\n")

		if err := cmd.Run(); err != nil {
			return fmt.Errorf("record from microphone: %w", err)
		}

		return nil
	})

	eg.Go(func() error { return StreamingRecognize(ctx, client, cfg, audio, printTranscript) })

	// Also using a routine to monitor for interrupts.
	eg.Go(func() error {
		const maxInterrupts = 10
		interrupt := make(chan os.Signal, maxInterrupts)
		signal.Notify(interrupt, os.Interrupt, syscall.SIGTERM)

		<-interrupt
		cancel()

		return ctx.Err()
	})

	if err := eg.Wait(); err != nil && !errors.Is(err, ctx.Err()) {
		fmt.Printf("failed to run streaming recognition: %v\n", err)
	}
}

// StreamingRecognize wraps the bidirectional streaming API for performing
// speech recognition. It sets up recognition using the given cfg.
//
// Data is read from the given audio reader into a buffer and streamed to cubic
// server. The default buffer size may be overridden using Options when creating
// the Client.
//
// As results are received from Transcribe server, they will be sent to the
// provided handlerFunc.
//
// If any error occurs while reading the audio or sending it to the server, this
// method will immediately exit, returning that error.
//
// This function returns only after all results have been passed to the
// resultHandler.
func StreamingRecognize(
	ctx context.Context,
	client transcribe.TranscribeServiceClient,
	cfg *transcribe.RecognitionConfig,
	audio io.Reader,
	handlerFunc func(*transcribe.StreamingRecognizeResponse),
) error {
	const (
		streamingBufSize = 1024
	)

	// Creating stream.
	stream, err := client.StreamingRecognize(ctx)
	if err != nil {
		return err
	}

	// There are two concurrent processes going on. We will create a new
	// goroutine to read audio and stream it to the server.  This goroutine
	// will receive results from the stream.  Errors could occur in both
	// go routines.  We therefore setup a channel, errCh, to hold these
	// errors. Both go routines are designed to send up to one error, and
	// return immediately. Therefore we use a buffered channel with a
	// capacity of two.
	errCh := make(chan error, 2)

	// start streaming audio in a separate goroutine
	var wg sync.WaitGroup
	wg.Add(1)
	go func() {
		if err := sendAudio(stream, cfg, audio, streamingBufSize); err != nil && !errors.Is(err, io.EOF) {
			// if sendAudio encountered io.EOF, it's only a
			// notification that the stream has closed.  The actual
			// status will be obtained in a subsequent Recv call, in
			// the other goroutine below.  We therefore only forward
			// non-EOF errors.
			errCh <- err
		}

		wg.Done()
	}()

	// Receive results from the stream.
	for {
		in, err := stream.Recv()
		if errors.Is(err, io.EOF) {
			break
		}

		if err != nil {
			errCh <- err
			break
		}

		handlerFunc(in)
	}

	wg.Wait()

	select {
	case err := <-errCh:
		// There may be more than one error in the channel, but it is
		// very likely they are related (e.g. connection reset causing
		// both the send and recv to fail) and we therefore return the
		// first error and discard the other.
		return err
	default:
		return nil
	}
}

// printTranscript is a callback function given to StreamingRecognize method to
// print results that are returned though the gRPC stream.
func printTranscript(resp *transcribe.StreamingRecognizeResponse) {
	if resp.Error != nil {
		fmt.Printf("\n[ERROR] server returned an error: %v\n", resp.Error)
		return
	}

	hyp := resp.Result.Alternatives[0]
	startTime := float32(hyp.StartTimeMs) / 1000.0
	endTime := startTime + float32(hyp.DurationMs)/1000.0

	if resp.Result.IsPartial {
		fmt.Printf("\r[%0.2f:%0.2f] %s", startTime, endTime, hyp.TranscriptFormatted)
	} else {
		fmt.Printf("[%0.2f:%0.2f] %s\n\n", startTime, endTime, hyp.TranscriptFormatted)
	}
}

// sendAudio sends audio to a stream.
func sendAudio(
	stream transcribe.TranscribeService_StreamingRecognizeClient,
	cfg *transcribe.RecognitionConfig,
	audio io.Reader,
	bufSize uint32,
) error {
	// The first message needs to be a config message, and all subsequent
	// messages must be audio messages.

	// Send the recognition config
	if err := stream.Send(&transcribe.StreamingRecognizeRequest{
		Request: &transcribe.StreamingRecognizeRequest_Config{Config: cfg},
	}); err != nil {
		// if this failed, we don't need to CloseSend
		return err
	}

	// Stream the audio.
	buf := make([]byte, bufSize)
	for {
		n, err := audio.Read(buf)
		if n > 0 {
			if err2 := stream.Send(&transcribe.StreamingRecognizeRequest{
				Request: &transcribe.StreamingRecognizeRequest_Audio{
					Audio: &transcribe.RecognitionAudio{Data: buf[:n]},
				},
			}); err2 != nil {
				// if we couldn't Send, the stream has
				// encountered an error and we don't need to
				// CloseSend.
				return err2
			}
		}

		if err != nil {
			// err could be io.EOF, or some other error reading from
			// audio.  In any case, we need to CloseSend, send the
			// appropriate error to errCh and return from the function
			if err2 := stream.CloseSend(); err2 != nil {
				return err2
			}
			if err != io.EOF {
				return err
			}
			return nil

		}
	}
}

1.1.6 - Recognition Configurations

• An in-depth explanation of the methods, data structures and types in the auto-generated SDKs can be found in the API Reference section. The sub-section on the RecognitionConfig object is particularly important here. This page discusses the common combinations of values set in RecognitionConfig sent to the server.

• First, here’s a quick overview of the fields in RecognitionConfig.

Use cases

Transcribing Headered Files

• The most basic use case is getting a formatted transcript for a headered audio file such foo.wav. This would simply need a config such as the following:

{
    "model_id": "1",
}

• Transcribe will return one or more results depending on partial result frequency, end points in speech etc. each of which would like the following:

{
  "error": null,
  "result": {
    "alternatives": [
      {
        "transcript_formatted": "Tomorrow is a new day.",
        "transcript_raw": "TOMORROW IS A NEW DAY",
        "start_time_ms": 180,
        "duration_ms": 1425,
        "confidence": 0.870,
      },
      {
        "transcript_formatted": "Tomorrow is a you day.",
        "transcript_raw": "TOMORROW IS A YOU DAY",
        "start_time_ms": 180,
        "duration_ms": 1425,
        "confidence": 0.130,
      }
      // ...
      // Other alternative hypotheses.
      // ...
    ]
  }
}

• If some sort of non-fatal error was encountered, then Transcribe will populate the error field. One such case maybe sending audio sampled at a lower sample rate than what the model is configured for (e.g. sending 8 kHz audio to a 16 kHz model):

{
  "error": {
    "message": "potential accuracy loss: input sample rate (8000) is lower than required (16000)"
  },
  "results": {
    // ...
    // Results
    // ...
  },
}

Transcribing Raw Audio Stream

• For transcribing raw audio streams, such as those coming in from a live microphone, the details of the audio samples such as their sampling rate, encoding etc. must be specified in the RecognitionConfig like so:

{
    "model_id": "1",
    "audio_format_raw": {
      encoding="SIGNED",
      bit_depth=16,
      byte_order="LITTLE_ENDIAN",
      sample_rate=16000,
      channels=1,
    }
}

• For various other encoding formats for raw samples, check AudioFormatRaw in the API specification.

Getting Word-level Details

• If you need to know the word-level details such as word timestamps, to align subtitles with a video, for example, then you can use the following config to enable those word-level timestamps.

{
    "model_id": "1",
    "enable_word_details": true
}

• Each alternative hypothesis in the returned results will have a word_details field containing details for both formatted and raw words:

{
  "error": null,
  "result": {
    "alternatives": [
      {
        "transcript_formatted": "Tomorrow is a new day.",
        "transcript_raw": "TOMORROW IS A NEW DAY",
        "start_time_ms": 180,
        "duration_ms": 1425,
        "confidence": 0.870,
        "word_details": {
          "formatted": [
            { "word": "Tomorrow", "confidence": 1.0, "start_time_ms": 180, "duration_ms": 800 },
            { "word": "is", "confidence": 1.0, "start_time_ms": 980, "duration_ms": 120 },
            { "word": "a", "confidence": 1.0, "start_time_ms": 1100, "duration_ms": 120 },
            { "word": "new", "confidence": 0.870, "start_time_ms": 1220, "duration_ms": 210 },
            { "word": "day.", "confidence": 1.0, "start_time_ms": 1450, "duration_ms": 155 },          
          ],
          "raw": [
            { "word": "TOMORROW", "confidence": 1.0, "start_time_ms": 180, "duration_ms": 800 },
            { "word": "IS", "confidence": 1.0, "start_time_ms": 980, "duration_ms": 120 },
            { "word": "A", "confidence": 1.0, "start_time_ms": 1100, "duration_ms": 120 },
            { "word": "NEW", "confidence": 0.870, "start_time_ms": 1220, "duration_ms": 210 },
            { "word": "DAY", "confidence": 1.0, "start_time_ms": 1450, "duration_ms": 155 },          
          ],
        }
      },
      // ...
      // Other alternative hypotheses.
      // ...
    ]
  }
}

Getting Confusion Networks

• For applications that need more than the one-best transcription, the most comprehensive and detailed results are found in the confusion network. Please refer to the in-depth confusion network documentation to see what is included.

• To enable the confusion network, the following config can be used:

{
    "model_id": "1",
    "enable_confusion_network": true
}

• The confusion network will be accessible at the cnet field in the results returned:

{
  "error": null,
  "result": {
    "alternatives": [
        {
          "transcript_formatted": "Tomorrow is a new day.",
          "transcript_raw": "TOMORROW IS A NEW DAY",
          "start_time_ms": 180,
          "duration_ms": 1425,
          "confidence": 0.870,
        },
        {
          "transcript_formatted": "Tomorrow is a you day.",
          "transcript_raw": "TOMORROW IS A YOU DAY",
          "start_time_ms": 180,
          "duration_ms": 1425,
          "confidence": 0.130,
        }
        // ...
        // Other alternative hypotheses.
        // ...
    ],
    "cnet": {
      "links": [
        { 
          "start_time_ms": 180,
          "duration_ms": 800,
          "arcs": [
            { "word": "TOMORROW", "confidence": 1.0 }
          ]
        },
        { 
          "start_time_ms": 980,
          "duration_ms": 120,
          "arcs": [
            { "word": "IS", "confidence": 1.0 }
          ]
        },
        { 
          "start_time_ms": 1100,
          "duration_ms": 120,
          "arcs": [
            { "word": "A", "confidence": 1.0 }
          ]
        },
        { 
          "start_time_ms": 1220,
          "duration_ms": 210,
          "arcs": [
            { "word": "NEW", "confidence": 0.870 },
            { "word": "YOU", "confidence": 0.130 }
          ]
        },
        { 
          "start_time_ms": 1450,
          "duration_ms": 155,
          "arcs": [
            { "word": "DAY", "confidence": 1.0 }
          ]
        }        
      ]
    }
  }
}

1.1.7 - Recognition Context

• Cobalt Transcribe allows users to send context information with a recognition request which may aid the speech recognition. For example, if you have a list of names that you want to make sure the Transcribe model recognizes correctly, with the correct spelling, then you may provide the list in the form of a RecognitionContext object along with the RecognitionConfig before streaming data.

• Transcribe models allow different sets of “context tokens” each of which can be paired with a list of words or phrases. For example, a Transcribe model may have a context token for airport names, and you can provide a list of airport names you want to be recognized correctly for this context token. Likewise, models may also be configured with tokens for “contact list names”, “menu items”, “medical jargon” etc.

To ensure that there is no added latency in processing the list of words or phrases during a recognition request, we have a API method called CompileContext() that allows the user to compile the list into a compact, efficient format for passing to the StreamingRecognize() method.

Compiling Recognition Context

• The following snippet shows an example of how to compile context data and then send it during a recognition request.

• Python
• Go

import grpc
import cobaltspeech.transcribe.v5.transcribe_pb2_grpc as stub
import cobaltspeech.transcribe.v5.transcribe_pb2 as transcribe

serverAddress = "localhost:2727"

# Using a channel without TLS enabled.
channel = grpc.insecure_channel(serverAddress)
client = stub.TranscribeServiceStub(channel)

# Get server version.
versionResp = client.Version(transcribe.VersionRequest())
print(versionResp)

# Get list of models on the server. 
modelResp = client.ListModels(transcribe.ListModelsRequest())
for model in modelResp.models:
    print(model)


# Select a model ID from the list above. Going with the first model
# in this example. Also printing list of allowed context tokens.
m = modelResp.models[0]
print(f"context tokens =  {m.attributes.context_info.allowed_context_tokens}")

# Let's say this model has an allowed context token called "airport_names" and
# we have a list of airport names that we want to make sure the recognizer gets
# right. We compile the list of names using the CompileContext(), save the compiled
# data and send it back with subsequent recognize requests to customize and improve
# the results.
#
# More typically, general models have a "catch-all" token called "unk:default" which
# can be used to boost the probabilities of any type word, as well as add words that
# are not in the model's vocabulary.
phrases = ["NARITA", "KUALA LUMPUR INTERNATIONAL", "ISTANBUL ATATURK", "LAGUARDIA"]
token = m.attributes.context_info.allowed_context_tokens[0]  # "unk:default" 

compileReq = transcribe.CompileContextRequest(
    model_id=m.id,
    token=token,
    phrases=[ transcribe.ContextPhrase(text=t) for t in phrases ],
)

# Sending compilation request. 
compiledResp = client.CompileContext(compileReq)

# Saving the compiled result for later use; note this compiled data is only
# compatible with the model whose ID was provided in the CompileContext call
compiledContexts = []
compiledContexts.append(compiledResp.context)

# Set the recognition config. We don't set the audio format and let the
# server auto-detect the format from the file header.
cfg = transcribe.RecognitionConfig(
    model_id=m.id,
    context=transcribe.RecognitionContext(compiled=compiledContexts),
)

# Open audio file.
audio = open("test.wav", "rb")

# The first request to the server should only contain the
# recognition configuration. Subsequent requests should contain
# audio bytes. We can write a simple generator to do this.
def stream(cfg, audio, bufferSize=1024):
    yield transcribe.StreamingRecognizeRequest(config=cfg)
    
    data = audio.read(bufferSize)
    while len(data) > 0:
        yield transcribe.StreamingRecognizeRequest(
          audio=transcribe.RecognitionAudio(data=data),
        )
        data = audio.read(bufferSize)

# We also define a callback function to execute for each response.
# The example below just prints the formatted transcript to stdout.
def processResponse(resp):
    result = resp.result
    hyp = result.alternatives[0]                    # 1-best hypothesis.
    transcript = hyp.transcript_formatted           # Formatted transcript.
    start = hyp.start_time_ms / 1000.0              # Converting to seconds.
    end = start + hyp.duration_ms / 1000.0          # Converting to seconds.
    newLine = "\r" if result.is_partial else "\n\n" # Will not move to new line for partial results.
    print(f"[{start:0.2f}:{end:0.2f}] {transcript}", end=newLine)

# Streaming requests to the server.
for resp in client.StreamingRecognize(stream(cfg, audio)):
    processResponse(resp)