Taff, the TeamTalk Audio File Fixer
Doug Lee
Last Revised May, 2020

This utility is designed to create versions of TeamTalk audio recordings that can be played in standard media players. See the Technical Details section for details on what this means and why it is necessary.

Taff creates versions of TeamTalk Opus .ogg files with the same basename but a .opus extension. The created files should play in many players and be supported by many decoders that fail to handle original TeamTalk recordings. No data is lost in the conversion process, and this document includes an algorithm for recovering the exact original recording .ogg file in case this is of interest.

Taff is a console application and can be run either with command-line options or without arguments in an interactive mode. It can process single files, multiple files, folders of files, or an entire file tree at once and can preserve or delete the original .ogg files as it goes. In the Taff author's experience, Taff file processing can be extremely fast: The autor fixed 74 files totaling 2.9 gigabytes in nine seconds. (The machine used was a Mac Mini with SSD storage.)

Taff is Copyright (c) 2019-2020 by Doug Lee and is covered by version 3 of the GNU General Public License. This program comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions. See the full GNU version 3 license for details. A copy of this license is also included as file copying.txt.

Table of Contents

System Requirements

On Windows, users can run the executable (.exe) version of this utility. Those wishing to run this utility from source on Windows must first download and install a version of Python 3.7.

On MacOS, Linux, and other Python-supporting platforms, make sure to run this utility via Python 3.7.

Installation Instructions

To install this utility:

  1. Download the Taff zip file
  2. Extract the zip's contents into a location of your choosing.
  3. Run the utility from its extracted location. On Windows, run taff.exe. (If desired, make a Windows shortcut for your Desktop to run the utility more conveniently.) On other operating systems, or on Windows with Python 3.7 installed, run taff.py via the correct Python command.

Taff Usage

This document will use the command taff to indicate launching Taff. If you are using the Windows executable, it might be necessary to type the full path to Taff, such as c:\taff\taff.exe instead. To run Taff from source, on Windows after installing Python 3.7 or on any other operating system, it will more likely be necessary to type something like python c:\taff\taff.py or python3 c:\taff\taff.py.

Taff is designed not to require being launched from within a shell window (e.g., cmd on Windows) for most common functionality. This is why Taff will often stop before closing with a message like "Press Enter to exit." This facilitates use of Taff via a Windows or similar shortcut or from Windows Explorer or similar.

If the output of Taff is redirected to a file or program (a pipe) such as more, many final prompts will not occur. Prompts that do occur will be sent to both the screen and the output destination, and the user response will also be sent to the output destination for reference. Much other output will not be duplicated to the screen, however.

If run without command-line arguments, Taff will prompt for the arguments to use. All that follows will work if included on the command line or if typed at this prompt.

Taff supports the following options and functions besides its main functionality:

-h or --help
Print help text and exit. The help text includes the Taff version number, license information, usage help, and usage examples.
-v or --version
Print the Taff version number and exit.
-i or --inspect
Inspect files and describe their content in detail. This option is meant for developers and will produce copious output. To inspect a file, list its name after the option. Wildcard patterns like *.ogg are also allowed and will produce output for each matched file.

With the above exceptions, Taff command lines consist of the following elements, in order:

Available global options:

Targets and target options that may occur in any order after global options:

Examples:

taff .
Scans files in the current folder.
taff -f -r .
Fix files in the current folder and all subfolders.
taff -d -r c:/TeamTalk/recordings
Fix files in the given folder and in all of its subfolders. Also delete each original .ogg file as its fixed .opus version is completed.

Known Issues

Issues shown in this section may be fixed in future Taff versions.

Fixing a file that is currently being written to by TeamTalk may produce unpredictable results and will certainly result in a .opus file that goes out of date with respect to its .ogg original.

Though it is possible to rebuild the original .ogg file from a .opus file that this utility created, the option to do so is currently not implemented. The above issue with a currently active .ogg file is one reason for this.

The -i or --inspect option produces output that normally ends with the confusing message, "Incomplete header (got 0 bytes out of 27)." This is normal unless the number of bytes is not 0. The inspect option is also not coded to stop for files that are not of the proper format; e.g., throw a .flac file at it, and you're likely to get reams of completely useless output.

Technical Details

This section is mostly for the incurably curious (hello, that's me!) and the highly technically inclined. A simpler summary of what appears below:

Readers of the following subsections may find it helpful to refer to "RFC 7845 - Ogg Encapsulation for the Opus Audio Codec" and/or "RFC 3533 - The Ogg Encapsulation Format Version 0" for information on the precise meanings of terms like "granule position," "pre-skip," and "end-of-stream flag."

How a Problematic TeamTalk Recording Is Identified

The following algorithm determines what fixes should be applied to a file. See the next subsection for the specific meanings of the terms "OpusTags issue" and "granule position issue."

Note that TeamTalk 5.5 recordings will need the granule position issue fixed, and TeamTalk 5.4 and older recordings will need both OpusTags and granule position issues fixed.

The Problems Being Solved

TeamTalk versions before 5.5 produced .ogg files with the following deviations from the Opus specifications:

  1. The required OpusTags, or comment, page of the file was omitted. This alone caused many decoders and players to reject TeamTalk recordings. This document will now begin referring to this issue as "the OpusTags issue."
  2. The "granule position" field for the first audio page was all zeros. Section 4.5 of "RFC 7845 - Ogg Encapsulation for the Opus Audio Codec" indicates that the granule position of the first page may not be smaller than the number of audio samples in packets that complete on that page. Since the first audio page contains a significant amount of audio data as evidenced by its size in bytes, it is reasonable to assume that 0 is too small to be considered permissible. The above-referenced specification also says that decoders must consider a stream invalid if this requirement is not met, which is likely why many decoders and players still reject TeamTalk recordings even if an OpusTags comment header page is inserted. This document will now refer to this issue as "the granual position issue."

Subsequent to much discussion of the OpusTags issue on TeamTalk forums, the TeamTalk author kindly fixed the OpusTags issue in TeamTalk version 5.5. Unfortunately, this only served to reveal the granule position issue, which was previously unnoticed. The granule position issue remained a problem in TeamTalk 5.5 recordings.

TeamTalk recordings may sometimes have one more issue, here referred to as the end-of-stream issue: The final Ogg page in a stream should indicate that it ends the stream with an "end of stream" flag. This flag is not always present in a TeamTalk recording. Decoders and players are to handle this condition gracefully, however; so this issue should not represent a problem. Taff does not address the end-of-stream issue but simply copies the original data without altering the end-of-stream flag. This Taff author considers it likely that the final Ogg page's end-of-stream flag is set or left unset depending on what caused the recording to end (e.g., the user turning off recording, the user leaving a channel, the transmitting user leaving a channel, etc.).

The Fixes In Detail

Taff is designed to fix recordings made by TeamTalk 5.4 and older, and recordings made by TeamTalk 5.5. The method used appears to make most if not all decoders and players that handle Opus files capable of handling these.

Taff writes the fixed version of an original TeamTalk .ogg file as a .opus file with the same basename. It is this author's observation that use of .opus as the file extension, rather than .ogg, can solve a few problems as well, such as those caused by Windows opening a different player for Ogg files than for Opus files even if the Ogg player is not able to handle the Opus format.

All fixes are performed by creating a new .opus file with the same basename (and location) as the original .ogg file. The new Opus file includes text that Taff can subsequently recognize. This lets Taff distinguish between Opus files generated by Taff and those generated by other programs.

For TeamTalk 5.4 and older Opus recordings with no OpusTags page (the OpusTags issue), the Taff OpusTags page serves to fill in the missing page and thus solves that problem. This does create a page with a page sequence number that duplicates that of the next page. The next page is the first audio page and is the page where the granule position issue fix is applied. That page, whether its page sequence number is duplicated or not, will be ignored by decoders as will be shortly explained.

For both TeamTalk 5.4 and older and TeamTalk 5.5 files, Taff fixes the granule position issue sufficiently to satisfy all tested players and decoders. The method used is unusual, however, and thus deserves explanation.

Setting the granule position to a value above the expected number of samples in the first page seems not to satisfy decoders. However, allowing the Cyclic Redundancy Check (CRC) of this page to be invalid causes all tested players and decoders silently to ignore the page but then accept the remainder of the stream. The fix employed, therefore, is simply to change the granule position to something not used elsewhere, but without recalculating the CRC. This means that all original Ogg data remains either unchanged or reliably retrievable in the Opus file, even if a small portion of it will not be used by decoders.

Why Does This Work?

Opus is an asymptotically convergent predictive codec, which means that the output at one moment will depend on what the decoder has seen recently along with what is being seen now. This in turn requires there to be a history that extends before the first audio actually heard. The Opus specification allows an Opus file creator to specify how many initial samples to skip when producing audio, so that this history may be created and properly used. The Opus specification refers to the number of data bytes in this skip zone as "pre-skip."

TeamTalk sets the pre-skip count to 0, whereas other encoders such as OpusEnc (an Opus reference implementation) uses a higher value, such as 312. Based on below-explained experimentation, this Taff author believes that the data in the initial audio page invalidated by this granule position issue fix trick would either have been ignored anyway or would only serve to avoid an initial sound spike in the recorded audio that was not actually present when the audio was recorded. Experiments so far have not indicated that this possible variation is at all significant and that it is limited in duration to a small number of milliseconds.

Experimental Results In Detail

This author conducted a number of experiments while writing Taff to verify the usability of resulting audio files:

Because the original Ogg data can reliably be retrieved from a Taff-generated Opus file, it is clear that any deviations in audio from what the Ogg would produce comes from decoder interpretation of data and does not represent data loss incurred by Taff.

Recreating Original Data

A precise explanation of how to retrieve the original data, for which the code is not currently written:

  1. Consider a Taff-generated input file audio.opus and a being-restored original TeamTalk recording file audio.ogg.
  2. Read the first three Ogg pages from audio.opus. These will now be called page0, page1, and page2, respectively. page0 will be the Opus identification (OpusHead) page, page1 the Opus comments (OpusTags) page, and page2 the first page of audio data. If there are not three pages in the audio.opus file, stop; this is not a Taff-generated file.
  3. If page1 does not contain the Taff vendor id, stop; this is not a Taff-generated file.
  4. Copy page0 to audio.ogg without change.
  5. If page2 has page sequence number 1, this is a file generated from a TeamTalk 5.4 or older recording. To recreate the original recording's OpusTags issue, ignore page1 completely and do not copy it to audio.ogg.
  6. If the page2 page sequence number is instead 2, this is a file generated from a TeamTalk 5.5 recording. To recreate the original file, emit into audio.opus a comment (OpusTags) page with no vendor id string or comments. Do not copy page1 from audio.opus into audio.ogg.
  7. To revert the granule position fix for all recordings, replace the granule position field (8 bytes) in page2 with zeros. There will be no need to recalculate the CRC; but as an added verification step, the actual CRC of the resulting page2 can be checked against the CRC stored in it (when calculating CRC, use zeros for the CRC field itself in the calculation; see the calc_crc method in the Taff source code). Copy the resulting page to audio.ogg instead of copying the original page2.
  8. Copy all remaining pages of audio.opus unchanged into audio.ogg. audio.ogg should now be exactly what Taff originally fixed.

Revision History

Here is the revision history of Taff, most recent entry first:

Version 1.0
First public release.