CESNET Project SSERV

CESNET technical report 29/2010

Pavel Vachek

Received 3. 12. 2010

Other formats: PDF, EPUB

Abstract

The Shadowserver Foundation, an Internet security volunteer watchdog group, generates and publishes its day-to-day reports on malware, botnet activities, DDoS and electronic fraud. This technical report describes a simple system for receiving and processing the Shadowserver reports by the CESNET-CERTS SSERV system and for sending appropriate Security Incident Reports to the member and customer network administrators.

Keywords: Security, Incident, Shadowserver, CESNET, CESNET-CERTS, Linux, Perl

1  Introduction

A simple and useful SSERV system has been developed in CESNET, the National Research and Educational Network of the Czech Republic. It has been operating since November 2010, informing administrators of the CESNET2 networks about any security incidents which the Shadowserver project reported and which originated from networks they manage. As a result, these administrators are informed soon; they can check suspect machines, disinfect or disconnect them from the Internet, and thus protect their users from possible data loss, financial loss or identity theft, as well as keep up the CESNET reputation. The main goal of this Technical Report is to inform CERT/CSIRT security teams and network managers, especially those of ISPs or large University networks, about the advantages of using the Shadowserver data and to help them install a similar system to significantly improve the security of their constituency.

2  The Shadowserver Project

The Shadowserver Foundation is an Internet security volunteer watchdog group whose mission is to improve the security of the Internet, especially by monitoring the Internet traffic on honeypots and spam traps, and by malware analysis. The Foundation makes available the resulting data free of charge to Internet Service Providers and to large end users who own and manage their IP address space or preferably an Autonomous System.

The instructions on how to request a free subscription to the Shadowserver Foundation's reporting service are available from Shadowserver web.

The Shadowserver Foundation makes available reports on various nefarious network activities, e.g. Botnet Command and Control servers, infected systems, spam relays and suspect URLs found in spam. Reports describing these activities within respective ASNs or networks are sent every day or every week to those administrators who requested them if any incidents are detected.

3  The SSERV Project

The SSERV project receives and analyses e-mail messages sent by the Shadowserver project, selects the important data, adds descriptions and other useful information and finally sends them to appropriate network administrators to fix the problems. Main components of the CESNET SSERV project are these four programs written in the Perl language:

4  Shadowserver File Receiver

Ssfrx, the Shadowserver File Receiver, is a program whose standard input accepts incoming mail from the local SMTP server. Its only function is the reception of incident reports sent by the Shadowserver Foundation as follows:

The Shadowserver Foundation mail server does not use PGP signatures or X.509 certificates. As a result, mail sending and receiving is simpler and faster but the only authenticity test ssfrx can perform is checking the sender IP address and other invariant fields in the mail header.

5  Shadowserver File Processor

Ssfp, the Shadowserver File Processor, is the most important part of the SSERV project. Ssfp reads and processes the following files residing in the $SSERV directory:

Every report line contains an IP address of a suspect machine. To determine its administrator's e-mail address, ssfp reads a file generated previously by the add2cont program which is described below. This file contains a sorted list of all CESNET2 networks (decimal start and end addresses), their abuse contacts as well as NETNAMEs. As a result, assigning an appropriate abuse contact to every incident report is fast and simple.

By default, Shadowserver sends all incident reports in the CSV format. All information regarding one security incident fits on a single line which may be rather long, however. For example, one recent report regarding Command and Control servers contained a 1228-character line which, in addition to data on a server from our networks, contained also a list of other C&C servers involved in the communication as well as extensive geolocation data and autonomous systems of all C&C servers listed. This information is important for the Shadowserver project but it is superfluous for our local network administrators who need to know especially the data which directly concerns them, that is, in this case, the IP address and port number of the IRC server together with the IRC channel identifier.

A similar situation exists in the `botnet-drone' and `sinkhole-http-drone' reports containing information on infected or most-likely-infected machines. For example, the Shadowsever reports contain this information for every security incident listed in the `sinkhole-http-drone' file:

Clearly, to identify and check a suspect machine, network administrators need to know much simpler data. This is why the ssfp program converts the Shadowserver reports to a simpler form which will be shown later.

Ssfp takes care also of another potential problem. CESNET once received a 900-KByte `Spam URL Report' containing over 4000 lines reporting virtually a single security incident concerning a cracked user account. We realised that while Shadowserver compresses the data on drones very effectively, it does not seem to limit the volume of the Spam URL Reports. Therefore, ssfp limits the number of lines concerning a single suspect machine IP address from a Spam URL Report which is sent to the local network administrators.

According to this author's previous experience, administrators of certain CESNET networks prefer receiving not only a single e-mail containing incident reports on the whole network, but also many short e-mails containing incident report(s), each for a single IP address. Some administrators also wish to ignore data concerning selected IP addresses or even whole networks (the latter concerns those who subscribed to their own data feed of the Shadowserver Project). Ssfp allows all of that.

Program ssfp stores all files containing the processed data in the $RESULTS directory. All filenames are in format `YYMMDD:NNN' where `YYMMDD' gives the date of report and `NNN' describes the intended recipients. Filenames intended for network administrators who prefer receiving also the individual files for each suspected IP address are in format `YYMMDD:NNN.SSS'.

In addition, a BZIP2-compressed archive file YYMMDD.bz2 is created which contains data concerning all security incidents of the day; these are sorted by IP address and by date. As a result, all incidents may be retrieved later, e.g. for statistical purposes.

6  Shadowserver Report Sender

Ssrep – Shadowserver Report Sender is a simple program which performs the following:

Sending e-mail to a large number of recipients can bring embarrassment if incorrect data is sent or if wrong recipients are involved. To prevent this, ssrep provides two test modes of operation which help debugging the system thoroughly.

After all files are sent correctly, ssrep renames them and leaves them in the $RESULTS directory for a possible short-term administrator's reference; however, next run of ssfp will delete them.

7  IP Address to Abuse Contact Translator

Add2cont – IP Address to Abuse contact translator performs a single task. On every working day in the early morning, it requests an inetnum record from the RIPE database (RIPE DB in Figure 1) for every network within the CESNET Autonomous System whose abuse contacts are new or may have changed recently. From each such inetnum record, add2cont extracts the allocation start and end addresses, abuse contacts and NETNAMEs, and creates a sorted list of all CESNET networks in a format similar to:

3221225984:3221226111:abuse@example1.cz:EXAMPLE1-NET
3221226112:3221226239:abuse@example2.cz:EXAMPLE2-NET

Requesting the person or role records is not necessary because all inetnum records belonging to the CESNET Autonomous system have the following format:

inetnum:192.0.2.0 – 192.0.2.127
netname:EXAMPLE1-NET
descr:Customer Name
descr:Location
country:CZ
admin-c:XY123-RIPE
tech-c: AB456-RIPE
status:ASSIGNED XX
remarks:Please report network abuse -> abuse@example1.cz

The bulk transfer of many inetnum records might seem wasteful. This author thinks that currently this solution is in fact better: we need information for some 60 to 70 networks every day and this would require some 60 to 70 separate RIPE database requests, while the bulk transfer generates only three requests. Of course, this program may be remade after our networks are disinfected and hardened against new attacks and after the number of infected networks decreases significantly.

[Image]

Figure 1. CESNET SSERV Project Data Flow.

8  Installation

Installing all four programs is quite simple. All of them are installed under an ordinary unprivileged username sserv in directory /home/sserv/bin.

8.1  Ssfrx Installation

File .forward contains a single line

"|/home/sserv/bin/ssfrxlink || exit 75"

where ssfrxlink points to the latest version of ssfrx. No other steps for ssfrx installation are necessary.

8.2  Ssfp Installation

Program ssfp should run after ssfrx receives all e-mails for the day. We are not sure how many e-mails will arrive; usually, we get two or three files but this number may change any time. All e-mails arrive usually within a short time limit of some five minutes, usually between 20 p.m. and 1 a.m. GMT. Therefore, the simplest solution is to start ssfp well after that but before the morning shift starts. We add these two lines to crontab:

# Processing Shadowserver files every Mon-Fri morning:
17 4 * * 1-5 /home/sserv/bin/ssfplink

where ssfplink points to the latest version of ssfp. No other steps for ssfp installation are necessary.

8.3  Ssrep Installation

Program ssrep should start immediately after ssfp terminates but only if any result files were generated. As ssfp generates exit code = 0 if and only if any result files are generated, just modifying the two above mentioned line in crontab is enough:

# Processing Shadowserver files and mailing results every Mon-Fri morning:
17 4 * * 1-5 /home/sserv/bin/ssfplink && /home/sserv/bin/ssreplink

where ssreplink points to the latest version of ssrep.

Program ssrep requires installation of module Mail::Sender. Its current version 0.8.16 is dated 14 Jul 2008.

No other steps for ssrep installation are necessary.

8.4  Add2cont Installation

Program add2cont should run some time before ssfp starts – preferably at a time when the RIPE database is not loaded much, for example around 3 a.m. We add these two lines to crontab:

# Building a list of all our allocations every Mon-Fri morning:
2 3 * * 1-5 /home/sserv/bin/add2cont

No symbolic link is used here because this program should not change much.

9  Hardware Requirements

The SSERV system can run on any decent machine (server or even a PC) capable of running Linux and connected to a local area network. According to this author's experience, programs add2cont, ssfrx and ssfp run just several seconds per day. Program ssrep runs longer because it sleeps for several seconds after each e-mail is sent.

Storage requirements depend on how long the data should be stored and on the number of detected incident reports. In our case, compressed e-mail attachments sent by Shadowserver usually take several KB to several tens of KB; so does the archive file YYMMDD.bz2 generated by ssfp.

10  Example of a Message Sent to Local Administrators

From: CESNET SSERV
To: abuse@dom.ain.cz
Date: 29. 11. 2010 05:19
Subject: [SSERV101129:012] Suspect machine 192.0.2.73 = dom73.ain.cz, ...

Dear Administrator,

The Shadowserver project monitors malware, botnet traffic and
elektronic fraud in Internet networks. Suspect or dangerous traffic
has been detected on these machines from your network:

192.0.2.73 = dom73.ain.cz
192.0.2.99 = dom99.ain.cz

Shadowserver detects several kinds of problems. These can be
distinguished by the last character on each line, i.e. C, D, H and S.

A list of machines which joined HTTP sinkhole servers originally built
by malicious domains. Short description of record fields follows:
Src IP add.|Src TCP port → Dst IP add.|Dst. TCP port|(infection type)|time|code=H

192.0.2.73 51216 -> 192.168.123.32 80 (downadup) 2010-11-27_10:55:32 H
192.0.2.99 31933 -> 192.168.123.34 80 (downadup) 2010-11-27_21:35:35 H

One list of tools suitable for Conficker/Downadup removal can be found
here:

http://www.shadowserver.org/wiki/pmwiki.php/Stats/Conficker#toc5

Conficker Eye Chart:

http://www.confickerworkinggroup.org/infection_test/cfeyechart.html

Asterisks denote unavailable data. Time zone = GMT.
(Current time zone in the Czech Republic = CET = GMT+1.)

Would you please investigate and solve this and/or inform all
responsible parties?

Best regards, ...

Note: This message is generated automatically; sending replies is unnecessary.
More information on CESNET SSERV project at http://......
Copyright © 2010 The Shadowserver Foundation. Reproduced with permission.

11  Conclusion

Shadowserver Foundation collects worldwide information about malicious Internet traffic and makes its appropriate parts available free of charge to interested Internet service providers. Shadowserver writes that over 12 thousand reports are sent every night which means that many ISPs are not aware that this service exists. Our SSERV project attempts not only to use the Shadowserver data within CESNET but also to inform other ISPs and thus to help bridge the gap between the Foundation and potential users of its services.

The SSERV system is simple to install, runs on almost any contemporary PC or server and brings the data necessary for solving security incidents to the network administrators of our end users. Perl programs written at CESNET-CERTS are easy to comprehend and modify. Most of our network administrators are happy about this system.

This author keeps working on new features of CESNET SSERV according to our experience and our network managers' wishes.

Administrators of CERT or CSIRT teams interested in the SSERV project may send a note to this author who will be glad to help.

12  Acknowledgement

This work is supported by the research intent MSM6383917201 of the Ministry of Education, Youth and Sports of the Czech Republic.

další weby:fond rozvojemetacentrumCzechLightpřenosyvideoservereduroameduID.cz