This page documents, in some considerable detail for anyone curious, the history of the TeamTalk Commander (TTCom), also sometimes called the TeamTalk Console, before its ultimate publication under the GNU Public License on September 7, 2014.
This page was last revised on September 7, 2014.
I was first motivated to learn about the TeamTalk communication protocols by a very focused attack against one TeamTalk server I frequented often. Besides writing Python code to analyze and report on over 349 million lines of server logs (all generated in about one week) and more code to adapt a firewall dynamically in response to these attacks, I decided to examine the traffic I was attempting to control. I found two types of traffic: UDP for audio/video, and TCP for client/server commands and events containing no audio or video data. The latter proved to be easy to interpret and generate, but the former appeared to require either much effort, a very expensive Software Development Kit (SDK), or both.
TTCom actually began as nothing more than a proof of concept in September, 2011: Chris Nestrud (ChrisN) ran a web page the purpose of which was to display who was on any of various TeamTalk servers at a given time. His code, however, caused Windows XP TeamTalk clients to hang for roughly two seconds every time he logged in. (This appears to have been due to an anomaly in Windows XP and not to any fault in Chris's code.) His method of collecting information to post was to login every minute or so; therefore, this problem became very widespread and frequent, although many people did not know it's cause for a long time.
Frustrated by this problem and its effect on a fair number of people, I wrote in two days what later became the foundation of TTCom as a demonstration of how to avoid the need to login every few minutes while collecting information even more frequently for his page. From the very beginning of the TTCom version control logs, the first five entries:
* The appearance of "{rel1_ChrisN}" indicates that Chris Nestrud received this TTCom revision. Braced items like this are version control tags, and I include in tag names the names of code recipients.
- 1 Doug Lee 2011-09-25
- Kick-off: Code from ChrisN that he gave to Simon, then to me on Sat Sep 24, or maybe early the next morning.
- 2 Doug Lee 2011-09-25
- Current versions of my
attrdictandmycmdclasses.- 3 Doug Lee 2011-09-25
- Substantial build/rewrite following on from Chris Nestrud's work: A full Teamtalk server connection manager class, with automatic keep-alive pinging, asynchronous incoming data handling, asynchronous connection state management, decent output for many server events, and initial (though untested) support for sending commands from code.
- 4 Doug Lee 2011-09-25
- Functioning program and default sample config file for it.
- 5 Doug Lee 2011-09-26 {rel1_ChrisN}*
- Various fixes before first release. Kick and possibly other events still don't work, all event lines are logged, and there is an effort to avoid thread exceptions/crashes.
I should note that, at this time, the logging of events was simply to give me a way to learn of new event types and formats so I could make TTCom handle them appropriately. In early November of 2011, I wrote a small utility to gather and report all seen event types and parameter keywords based on these logs.
From there, I began expanding and fixing functionality with the goal of producing a freely distributable, multi-server TeamTalk text client, so that TeamTalk users could more readily find each other when frequenting multiple servers and could sometimes do things more efficiently than permitted by the regular TeamTalk clients. Development with this intent continued for roughly another month. I also gave Simon a copy of the code on October 1, 2011.
In November of 2011, I began to address two new concerns:
First, a new TeamTalk release with different audio codecs caused a lot of confusion by allowing people to be in the same channel with different client versions but yet be unable to hear each other. I added TTCom code that announced such an occasion when a person with a client too old or too new for a specific server joined a channel, so both the joiner and the other participants would see the problem and the joiner could correct it. Clearly though, this service would not work well if run by multiple instances of TTCom on the same TeamTalk server. I thus realized that releasing the entire TTCom project for public use might cause problems.
Also at about this time,the Ctrl+Shift+U command was added to TeamTalk, making TTCom connections visible. Concerns surfaced that TeamTalk users and admins might not like permanent connections to park on TeamTalk servers with no corresponding voice connection. Perhaps because of the earlier reporting service, admins remained happy to let my connection remain; but many agreed that my release of TTCom, in any form, would be a bad idea.
Regardless of the now ambiguous aims for TTCom, development continued. As previously mentioned, I eventually wrote a small utility to catalog event types and their parameters:
- 67 Doug Lee 2011-12-30
- Little utility from Nov 5 (14:15 or so) for listing all parameter keywords used on all event lines sent by the server.
- 68 Doug Lee 2011-12-30
- Enhanced keys.py to include sorted command lists to which each keyword applies, and the output of that process on the logs accumulated so far.
In January of 2012, I finally succeeded at a second aim for Chris
Nestrud's benefit: I managed to simulate just enough of TeamTalk's UDP
protocol to avoid the Windows XP hang previously mentioned. I sent
this code to Chris for his own use in his web collector and included
it in TTCom so that, even though it rarely issued a login command,
it would not bother XP clients even then.
Though I remained ambivalent about the full public release of TTCom as already explained, I decided to share the code with Nick Giannak, admin and owner of the HKC Radio TeamTalk server, in gratitude to him for his help and server permissions that allowed me to write the code in the first place:
- 82 Doug Lee 2012-01-09 {rel10_nick}
- Manifest added, and the files in it, plus a Nick-specific ttcom.conf, delivered to Nick on his HKC server with a bit of hands-on training. Other people in channel included JDX, Monte, Serena, a probably unattentive Alyssa, possibly Rich C, Venison, and at the very start, Keao.
Development continued, including discovery (from the aforementioned log scanner for event types and parameters) of new event keywords. I also continued to allow Simon to keep a fairly updated TTCom for his own use.
Up to this point, I believe it is accurate to say that I only attached TTCom to TeamTalk servers whose owners explicitly expressed acceptance of the idea. On January 16, 2012, I made the decision to place TTCom on servers that appeared in Chris Nestrud's public web listing of server participants. This effectively made both his and my monitoring activities more obvious to TeamTalk users. Again from version control logs (I had by now begun versioning my personal ttcom.conf file along with the code itself):
- 90 Doug Lee 2012-01-16
- 11 more servers from a list by ChrisN just received.
I quickly proved, not to my surprise, that attaching TTCom to the official public TeamTalk servers produced way more noise than I liked to see in the TTCom window; so I disabled a few servers very soon after adding them. In early February of 2012, I also created my own TeamTalk server instance, partly to allow experimentation without imposing on anyone else.
I believe it was about this time that I first witnessed, via TTCom and/or its logs, the attack and takeover of a TeamTalk server by someone who had acquired an admin account and password and who then deleted all users and channels and banned a number of users, including TTCom itself. This caused me to discover and fix a perhaps amusing bug:
- 98 Doug Lee 2012-02-10
- Server bans no longer cause continuous login retries.
This event also caused me to add kick, ban, and kb (kick-ban) commands as well as yet another feature that I was afraid to distribute publicly:
- 102 Doug Lee 2012-02-12 {rel11_simon}
- Support for printing
autoLoginchange on refresh, and support ofautoLogin=2for persistent relogin after kick (untested yet).
The purpose of autoLogin=2 was to allow me to see the
activities of a server takeover even if the person doing the takeover
attempted to kick TTCom off the server. On a number of subsequent
occasions, this proved very useful in postmortem analyses of server
takeovers.
On or near February 25, 2012, another server attack occurred, this time against the server run by the Cisco Academy for the Vision Impaired (CAVI). As this attack directly and adversely impacted an operating business (the server was used for social events and occasionally classwork discussions as far as I could tell), I felt it appropriate to offer my assistance in diagnosing the server problems when a CAVI representative asked me for help. I therefore wrote a log scanner that produced appropriately redacted output and sent a CAVI representative the results. "Appropriately redacted" in this case meant the removal from all logged events any passwords and numeric channel keys. As of this writing, this is the only instance on which I passed on TTCom log material to anyone. It is also when the logs began to serve more purpose than merely a way to learn new TeamTalk TCP protocol elements.
As significant evidence suggested that the mentioned two server attacks were carried out by the same person, and as reports consistently over time indicated this behavior was not unique to that individual, I spent the next week or so crafting a means of detecting that person's appearance on servers via sounds issued from my computer. This is the origin of "triggers" in TTCom. The system went live on March 2, 2012, at which time I began informing TeamTalk server owners and admins of its existence and nature. I did, however, keep the new system out of any code copies I gave to Simon and eventually stopped sending him updates for a while.
In late February of 2012, I started including a version number with TTCom logins that ended with "C," to identify TTCom as something other than a "normal" TeamTalk client. I used this rather than a nickname because version numbers could not be changed for regular clients, thus making the "C" unique to TTCom and thus a clear indicator of my presence on a server.
Around March 26, 2012, I became aware (through Simon) of someone who had managed to find a way to join a TeamTalk channel without being seen. This included the ability to talk and receive voice communication invisibly. It also included the ability to send numerous private text messages where each message would open its own window on the recipient's computer. This presented both a privacy issue and a potential Denial of Service (DOS) attack against specific TeamTalk users. Simon and I began referring to this behavior as "cloaking" and began working out a way to detect it. We also reported it to the author of TeamTalk, who fixed the bug it exploited in future TeamTalk releases. By mid April of 2012, I enabled code that automatically kicked cloakers off of servers whose owners wanted this service.
Simon's help with this incident earned him another TTCom update. The incident also spawned a multi-person, somewhat concerted attempt to teach the original discoverer and user of this cloaking tactic a better sense of social responsibility. During the process of working out detection and reporting of cloaking behavior, the person remarked on at least one occasion (though I think without knowing the author or name of TTCom) that he was being prevented from cloaking on various servers because of the presence of a "console." (This may be the origin of that term to describe TTCom, though I am not sure of this.)
Around March 26, 2012, another TeamTalk server was shut down and effectively wiped out by an attack. This was to be the first of two such attacks on the same server in a short time, and both by the same person (possibly with help from one other party) - and according to evidence, by the same person that had carried out attacks already mentioned (not the cloaking person).
Perhaps fueled by both this attack and by the aforementioned effort to teach social responsibility, the cloaker began helping me with both ideas and small code snippets to detect and report cloaking and other errant behaviors.
On March 26, 2012, Simon and I and several others held an informal, TeamTalk-based discussion of what out of TTCom might make sense to release for public use. I took notes from this meeting:
The notes read as follows:
- 192 Doug Lee 2012-03-27
- Ideas for a stripped-down TTCom version, largely drawn from a chat last night with Tyler Spivey and Simon Jaeger among a few others.
Before publication: Logging off by default. Events maybe off by default. autoLogin 0 or 1 by default? To admins: Don't kick cons just for being present. React to behavior, not predictions. End to admins. End before publication. Nice to have for publication: Read actual client config file. Refresh automatically on change to that file. End nice to have for publication.
Concerns represented by those notes:
Development continued and included some code reorganization as part of an attempt to find a safe way to publish TTCom without publishing source, the fear inspiring this being that the source would be too easy to use maliciously against TeamTalk servers:
- 241 Doug Lee 2012-04-06
- Most main-module code split out into submodule TTComCmd in preparation for non-source distribution possibilities.
On or before April 10, 2012, Tyler Spivey and Simon discovered what we came to call "UDP masquerading," a trick by which a TeamTalk user not in a channel could replace a user in that channel and both talk and hear as if being that user. With some help, and as best I could, I added a detector of this behavior to TTCom so I could alert people if I saw it happen in future:
- 254 Doug Lee 2012-04-10
- Detector for the UDP masquerade discovered today by Tyler and Simon. Tyler helped me test this code.
As by now security issues were a frequent occurrence, and as I happened to have significant time, I put considerable effort in April, 2012, into restructuring TTCom better to handle threading and responsiveness issues. TTCom thus became my platform on which to learn more about managing threaded programs in Python. I also fixed a few problems with the Windows XP support fix and sent Chris both revised code and a full-fledged, single-server TeamTalk scanner in Python for him to use or adapt for his web site. Finally, still uncertain of the public or private future of TTCom, I continued to make changes that might eventually make publication easier. I also tried to arrange a chat with the TeamTalk author about the future of TTCom but did not succeed.
After early May of 2012, there was a relatively calm period among the TeamTalk servers to which TTCom was attached. After several months of no ill reports, in September of 2012, I removed all triggers that were aimed at reporting known troublemakers in real time. Between October and December of 2012, there were occasional but brief needs to use triggers of this sort.
This period also saw the first use of TTCom as an assistant for a specific user: On request, I wrote a trigger to automate moving an Edified Access server user into her channel of choice whenever she joined the server. (This was either before we knew this could be done via TeamTalk client configuration or before such a thing became possible there.)
Near the end of December of 2012, yet another round of problems beset a server on my list, causing one of its owners to grant me admin access and request help. Again, evidence indicated the problems to stem from the same user as for most other attacks observed. Though at one time the users of that server were in disagreement over whether I should attach with TTCom, this event appeared to cement everyone into agreement that I should - for a while.
The "while" ended on February 9, 2013, however, when I became annoyed at inconsistent treatment ranging between "please help" and kicking me off the server without warning. I removed TTCom from that server though the primary owner seemed to like it there, on the grounds that he also chose his admins and knew of their behavior for long enough to hold some responsibility for it.
It was also around this time that I became aware of what I came to call "nickname spoofing," meaning one person appearing on a server deliberately using a nickname commonly known to belong to someone else, with the apparent intention of either annoying people or attempting to acquire sensitive information normally accessible to the person being spoofed. Over time, this tactic permitted at least one person, on multiple occasions, to acquire admin passwords and to take down TeamTalk servers with that information. It could also cast blame for aberrant behavior on the wrong person.
Accordingly, I began adding code to detect such behavior in some limited cases. I also spent some time writing means of scanning logs specifically to connect the nicknames used by a single person together, as a way to help identify the real person behind such activities. A functional scanner finally resulted on February 26, 2013, though it could sometimes take considerable time (several minutes) to find the nicknames for just one person.. I went on to use this scanner in a reactive fashion to determine who had caused problems after receiving reports of problems from server owners and admins.
Despite my working repeatedly with Chris on fixing his web site scanner/builder so it would not stall Windows XP users every few minutes, the problem persisted. In March of 2013, I wrote a detector for this into TTCom as well, so I could quantize the problem and report this information to him. This piece of code stayed in action for over a year, until sometime after Windows XP was officially retired by Microsoft.
On March 7, 2013, I received from Tyler Spivey a URL pointing to a TeamTalk forum post that listed numerous TeamTalk servers of which I had not been aware. Some required server passwords, and I believe at least one required a user account; although the post included all required information to log into each server listed.
Believing, perhaps against better judgment, that a TeamTalk forum post was sufficient indication of public status despite the aforementioned password requirements, I simply connected TTCom to a number of these servers without comment, expecting that this might give me a better chance of detecting new or hitherto-unnoticed events or event parameter keywords, and possibly, new and interesting places for me to visit with my normal client. I silenced events from these servers so they would not clutter my TTCom screen, reducing the effect of my presence (on my side, anyway) to mere inclusion of these servers in the results of the TTCom shortSum command that lists, for each server with people in channels, the server name and membership per channel.
I did indeed make at least one useful discovery from this: I learned that it is possible to have a TeamTalk server that is so busy as to freeze a normal TeamTalk client for several minutes on login, while TTCom could connect and process events with no notable performance problems.
The server that caused me to learn this, however, turned out to be a server intended for classes - and not in English, either. I was soon politely requested to pull TTCom off the server, which I did. One or two other servers from this list dismissed me as well, with different levels of aplomb.
Perhaps due to the above tactical error, the number of questions about TTCom increased. I was finally motivated to try answering them once and for all with a web page, so I wrote what became TTCom.php on April 5, 2013. The page stayed unmodified from that day until May 24, 2014, at which time I expanded it to explain more TTCom functions as more had by then been added to TTCom itself.
On August 7, 2013, while I was hanging out on a server with several people, I was asked, perhaps whimsically, to implement a feature for that specific server whereby participants would be instantly moved into or out of a certain channel based on current nickname. Dubbed the "ghost" service based on the channel and nickname requirements involved, this service eventually went live on August 31, to the apparent delight and amusement of many, including of course the server owner.
Surely more amusing to programmers than to TeamTalk users, I also
encountered (in mid September of 2013) a server whose
userTimeout (required ping frequency to stay connected)
was set to 0. Normal TeamTalk clients could not stay connected to this
server easily if at all. I managed to code TTCom to hang on to such a
server successfully by using a 300 millisecond ping time:
It could be said, I suppose, that this made TTCom a ghost that would not so easily die again... but also a means by which I could verify and report my discovery of why that server would not work for most if not all would-be users.
- 449 Doug Lee 2013-09-12
usertimeout=0is now handled successfully, despite the fact that the classic client at least can't do this, by using a timeout of 300 milliseconds in this case. 500 does not work. 400 was not tried.
Development of TTCom slowed considerably, but occasional modifications still occurred:
During the week of September 1, 2014, a number of regular frequenters of one TeamTalk server (at least one being an admin) expressed dislike for TTCom being attached to that server. The dissenters curiously did not appear to include the server's owner, who had previously expressed appreciation for TTCom's presence. After a few sometimes heated exchanges with server participants, and in line with my previously mentioned decision to honor non-owner requests after a time when the owner expressed no clear preference, I removed TTCom from that server.
But the disagreement, having spread onto Twitter, became bitter and involved more than just members of that one server. There was considerable misinformation, but out of the comments I found that did not involve misinformation, I drew the following list of concerns from those who did not like TTCom appearing:
Up to this point, I had assumed the following would occur and be completely sufficient to cover any disputes:
I considered taking a vote on whether to continue running TTCom in the capacity of a multi-server security assistant, but I realized two problems:
So with only guesses but no clear prediction of consequences, I decided to release the TTCom code, first for public inspection and second for potential public use. Criteria supporting my decision were, likely in this order:
As of the publication of this code, my interest in running TTCom becomes mostly one of personal convenience: It is easiest for me to find people and activity via its use. Where TTCom is not wanted, I am likely simply to devise a way to scan for people on demand, as Chris has long done except automatically on a schedule in his case, so I can maintain this personal convenience without annoying server participants that resent permanent connections.
For both client and server efficiency though, I expect to remain connected to those servers that do not mind the connection, so that the number of login sequences is significantly reduced. Logging into a TeamTalk server causes the server to send potentially many long lines of data to the client: one for each created channel, one for each logged-in user, and one for each participant in a channel; thus, a server with 50 channels and 10 users currently in channels will send approximately 71 lines of data, the last being the login for the now-connecting user.
At this writing, I do not intend to take down the services TTCom now provides en mass. I acknowledge, however, the possibility that individual server owners will begin running their own TTCom instances to provide some of these. Such owners should notify me of such changes so that I may turn the corresponding services off in my TTCom instance for that server.
TTCom, in my opinion and I believe in that of many server owners and participants alike, has over its three-year life so far served well to help the TeamTalk user community. The recent revelations regarding data collection by the National Security Agency (NSA) may have contributed to public concerns over my data collection, though I assert that mine was much more overt in nature particularly as of my publication of the "All About the TeamTalk Console" page. Be that as it may, perhaps the time has indeed come for a change.
It is difficult for me to predict among several possible outcomes of this release:
Whatever happens, all may rest assured that my intent with respect to TeamTalk and its users in general remains, as it always has, to be of service.