Rustock Botnet Using TLS Encryption to Push Spam

PCWorld reported that the volume of spam being sent by the notorious Rustock botnet using TLS encryption has surged in recent weeks, establishing an important new trend in botnet behavior, security companies have said.

Two weeks ago, Symantec's MessageLabs division reported noticing large volumes of spam using TLS (Transport Layer Security), an encryption protocol successor to the better-known SSL (Secure Sockets Layer), and normally a way of securing the contents of an email between server and client.

At that point, the percentage of spam encrypted by Rustock using TLS was around the 35 percent mark, a figure the company says in its latest Intelligence Report this week has surged to as much as 77 percent of its activity during the month.(See "Don't be Dragooned into the Botnet Army.")

The challenge is that TLS imposes higher processing demands on mail servers compared to non-TLS traffic, estimated to be around 1 kilobyte overhead for every spam email. Given that most email is now spam, the accumulated overhead on mail servers has the potential to be high whether the messages are detected as spam or not.

Rustock itself is a top three player in the botnet stakes, responsible for various huge spam campaigns since it first appeared in 2008. It was one of the bots said to have been most affected by the closing of McColo in early 2009, a trauma that might have had some bearing on the latest self-defence evolution.

MessageLabs is not the only hosted messaging provider to notice Rustock's use of TLS. Around the same time as the company put out its first Rustock warning, a blog by Terry Zink of Microsoft's Forefront Online Security division mentioned a similar issue with Rustock and its use of the TLS protocol.

Other providers contacted by Techworld this week reported seeing more TLS-encrypted spam included CronLab of Sweden and US-UK messaging outfit, M86 Security.

"We set up a node in our Labs with TLS and confirmed that some Rustock botnets were indeed using TLS," said Phil Hay, a spam expert at M86 Security. "Our statistics show that Rustock is still the leading source of spam output and this new use of TLS highlights an escalating level of sophistication."

"In essence this means that organisations can't rely on enforcing TLS as a means for reducing spam. It does have an effect on system resources however, as all forms of encryption do," said Hay.

Why Rustock has adopted this technique is open to debate. Adding TLS to outbound spam slows the rate at which spam can be delivered, which would seem to hurt the spammer's intention to spread non-legitimate email as far and fast as possible. It is also the case that TLS-encrypted email is no longer automatically trusted by receiving servers, so it is unlikely to be a simple evasion technique.

Experts such as Zink speculate that it could be connected to the recent clampdown on several botnets by law enforcement, including the bust that led police in Spain to arrest three men accused of running the Mariposa botnet. Infiltrating Mariposa - indeed infiltrating any botnet - involves cracking its command and control layer. TLS could be a defence that makes such interception harder.

[Editor's note: While this refers to email based spam; it appears that increasingly malware is using SSL based transport, making SSL (and TLS) just as dangerous as in the clear connections. SSL proxies that check for malware will become increasingly important in fighting these threats]