From an operations perspective, there is very little that is less useful and more aggravating than vendor magic. What I mean by this is anything that "happens" in the background that you have no visibility into. While many organizations enjoy the simplicity provided by this, when you need to McGyver some solution to a security issue that vendors haven't addressed yet, you just might feel like simply setting fire to equipment that got in your way. Not that I'm endorsing that.
This is one of the main strengths of open source software. If you know what you're doing, you can uncover all the magic so you know exactly what you're dealing with and you can fix it up if you need to. Snort-wise, one of the things that it does in the background for you is normalize data and put it into various buffers. At one point the list of buffers was fairly small: normalized and raw. At this point you have the following buffers:
| Buffer | Internal Representation | Notes * |
| normalized | CONTENT_BUF_NORMALIZED | This is the default buffer that Snort matches against. Also contains gzip decoded data. |
| raw | CONTENT_BUF_RAW | Not used often, mainly for looking at non-normalized TELNET and FTP data. |
| http_uri | CONTENT_BUF_URI | |
| http_raw_uri | CONTENT_BUF_RAW_URI | |
| http_cookie | CONTENT_BUF_COOKIE | Config option required to activate cookie parsing. |
| http_raw_cookie | CONTENT_BUF_RAW_COOKIE | |
| http_header | CONTENT_BUF_HEADER | |
| http_raw_header | CONTENT_BUF_RAW_HEADER | |
| http_method | CONTENT_BUF_METHOD | Parsed from header, not normalized. |
| http_stat_code | CONTENT_BUF_STAT_CODE | Parsed from header, not normalized. |
| http_stat_msg | CONTENT_BUF_STAT_MSG | Parsed from header, not normalized. |
| file_data:mime | BUF_FILE_DATA_MIME | Buffer holds the mime decoded data for SMTP |
| file_data | BUF_FILE_DATA | Not actually a buffer, but a pointer into normalized buffer |
| base64_decode | BUF_BASE64_DECODE |
* see the labs_buffers.c file for additional commentary on the buffers
Buffers aren't the only place where Snort massages the data. Both fragmentation and stream reassembly occur and can impact detection. So between parsing, normalization, defragmentation and stream reassembly, the final data blob looked at by Snort can be significantly different than what you see on Wireshark. This can make rule writing and debugging difficult. To help with this I've written a set of .SO rules that print out the buffers exactly as Snort views them for each packet in a PCAP. They've been really useful, so we're releasing them on the VRT Labs site (currently tested against Snort 2.9.0.3, so don't yell at me if it doesn't work on anything before that).
Once you download them, move them to your .SO directory and modify the following line in your Makefile:
libs := icmp p2p dos exploit bad-traffic web-activex web-client web-iis netbios misc nntp smtp web-misc sql imap chat multimedia pop3to:
libs := labsThen run "make", and modify your Snort conf to include the new labs.rules file. It should be something like:
include $RULE_PATH/../so_rules/labs.rulesThe labs.rules file should look like this:
# Autogenerated skeleton rules file. Do NOT edit by hand
alert tcp any any -> any any (msg:"VRT LABS: All Ports Two-Way Packet Description"; sid:100005; gid:3; rev:1; classtype:misc-activity; metadata: engine shared, soid 3|100005;)
alert tcp any any -> any $HTTP_PORTS (msg:"VRT LABS: HTTP_PORTS Client to Server Packet Description"; sid:100000; gid:3; rev:1; classtype:misc-activity; metadata: engine shared, soid 3|100000;)
alert tcp any $HTTP_PORTS -> any any (msg:"VRT LABS: HTTP_PORTS Server to Client Packet Description"; sid:100001; gid:3; rev:1; classtype:misc-activity; metadata: engine shared, soid 3|100001;)
alert tcp any any -> any 25 (msg:"VRT LABS: SMTP Client to Server Packet Description"; sid:100111; gid:3; rev:1; classtype:misc-activity; metadata: engine shared, soid 3|100111;)To get started, I would recommend commenting out all but the first rule. This will show you all the goodies you need. When you're working specifically with http data, I'd enable one or both of the second and third rules. Finally, when looking at SMTP client-to-server traffic (where you'll see mime-decoded data), you can enable only the fourth rule. If you have all rules on, you'll get multiple decoding (probably two per packet).
Hey, listen up: This is only designed to be run on pcap files with TCP data, where you have all the time in the world to read, parse and write data. If you run this on a running sensor it will probably melt, so don't. Also, it is what it is, so don't run it on anything important.
Each packet will start with a header as follows:
****************************** NEW PACKET *****************************
Timestamp: 2009-08-27 18:08:29:16274
Src IP: 195.2.253.95:80
Dst IP: 10.11.250.196:1075
TCP Flags: ACK
The top line lets you know you have a new packet (easy to miss if you have a lot of data) and then you have a time-stamp (conveniently formatted in Wireshark format) and more IP/TCP header information. If this is a pseudo packet rebuilt by the stream5 preprocessor, you see this instead of the NEW PACKET line above:
************************ NEW REASSEMBLED PACKET ***********************Then we start pulling apart the buffers. First we check if there is data, and if there isn't any, we simply write:
[-No data in this packet-]Otherwise we write the raw buffer out and check to see if the normalized buffer is different than the raw buffer. If it isn't, you'll see the raw packet data and then:
[NORMALIZED/GZIP BUFFER DATA] (IDENTICAL TO RAW BUF)If the data isn't the same, it will print the normalized data.
After this we get into specificly parsed buffers. After the jump, we have two packets that are an example of a packet broken out. It is a client request and server response over http, so you can see how we break things out:
****************************** NEW PACKET *****************************
Timestamp: 2009-08-27 18:08:28:886026
Src IP: 10.11.250.196:1075
Dst IP: 10.2.253.95:80
TCP Flags: PSH ACK
****** BUFFER INFORMATION ******
[RAW BUFFER DATA (0xacc1fe0)]:
0x0000 47 45 54 20 2f 41 14 41 41 41 41 2f 63 6f 6e 66 GET /AAAAAA/conf
0x0010 69 67 32 2e 62 69 6e 20 48 54 54 50 2f 31 2e 31 ig2.bin HTTP/1.1
0x0020 0d 0a 41 63 63 65 70 74 3a 20 2a 2f 2a 0d 0a 55 ..Accept: */*..U
0x0030 73 65 72 2d 41 67 65 6e 74 3a 20 4d 6f 7a 69 6c ser-Agent: Mozil
0x0040 6c 61 2f 34 2e 30 20 28 63 6f 6d 70 61 74 69 62 la/4.0 (compatib
0x0050 6c 65 3b 20 4d 53 49 45 20 37 2e 30 3b 20 57 69 le; MSIE 7.0; Wi
0x0060 6e 64 6f 77 73 20 4e 54 20 35 2e 31 29 0d 0a 48 ndows NT 5.1)..H
0x0070 6f 73 74 3a 20 31 39 35 2e 32 2e 32 35 33 2e 39 ost: 195.2.253.9
0x0080 35 0d 0a 50 72 61 67 6d 61 3a 20 6e 6f 2d 63 61 5..Pragma: no-ca
0x0090 63 68 65 0d 0a 0d 0a che....
[NORMALIZED/GZIP BUFFER DATA] (IDENTICAL TO RAW BUF)
[HTTP_HEADER BUFFER DATA (0x8b1fb00)]:
0x0000 41 63 63 65 70 74 3a 20 2a 2f 2a 0d 0a 55 73 65 Accept: */*..Use
0x0010 72 2d 41 67 65 6e 74 3a 20 4d 6f 7a 69 6c 6c 61 r-Agent: Mozilla
0x0020 2f 34 2e 30 20 28 63 6f 6d 70 61 74 69 62 6c 65 /4.0 (compatible
0x0030 3b 20 4d 53 49 45 20 37 2e 30 3b 20 57 69 6e 64 ; MSIE 7.0; Wind
0x0040 6f 77 73 20 4e 54 20 35 2e 31 29 0d 0a 48 6f 73 ows NT 5.1)..Hos
0x0050 74 3a 20 31 39 35 2e 32 2e 32 35 33 2e 39 35 0d t: 195.2.253.95.
0x0060 0a 50 72 61 67 6d 61 3a 20 6e 6f 2d 63 61 63 68 .Pragma: no-cach
0x0070 65 0d 0a 0d 0a e....
[HTTP_HEADER_RAW BUFFER DATA (0xacc2002)]:
0x0000 41 63 63 65 70 74 3a 20 2a 2f 2a 0d 0a 55 73 65 Accept: */*..Use
0x0010 72 2d 41 67 65 6e 74 3a 20 4d 6f 7a 69 6c 6c 61 r-Agent: Mozilla
0x0020 2f 34 2e 30 20 28 63 6f 6d 70 61 74 69 62 6c 65 /4.0 (compatible
0x0030 3b 20 4d 53 49 45 20 37 2e 30 3b 20 57 69 6e 64 ; MSIE 7.0; Wind
0x0040 6f 77 73 20 4e 54 20 35 2e 31 29 0d 0a 48 6f 73 ows NT 5.1)..Hos
0x0050 74 3a 20 31 39 35 2e 32 2e 32 35 33 2e 39 35 0d t: 195.2.253.95.
0x0060 0a 50 72 61 67 6d 61 3a 20 6e 6f 2d 63 61 63 68 .Pragma: no-cach
0x0070 65 0d 0a 0d 0a e....
[HTTP_URI BUFFER DATA (0xacc1fe4)]:
0x0000 2f 41 41 41 41 41 41 2f 63 6f 6e 66 69 67 32 2e /AAAAAA/config2.
0x0010 62 69 6e bin
[HTTP_URI_RAW BUFFER DATA (0xacc1fe4)]:
0x0000 2f 41 41 41 41 41 41 2f 63 6f 6e 66 69 67 32 2e /AAAAAA/config2.
0x0010 62 69 6e bin[HTTP_POST BUFFER DATA (NO DATA)]
[HTTP_METHOD BUFFER DATA (0xacc1fe0)]:
0x0000 47 45 54 GET
[HTTP_COOKIE BUFFER DATA (NO DATA)]
[HTTP_COOKIE_RAW BUFFER DATA] (NO DATA)]
********** END PACKET **********
****************************** NEW PACKET *****************************
Timestamp: 2009-08-27 18:08:29:16274
Src IP: 10.2.253.95:80
Dst IP: 10.11.250.196:1075
TCP Flags: ACK
****** BUFFER INFORMATION ******
[RAW BUFFER DATA (0xacc1fe0)]:
0x0000 48 54 54 50 2f 31 2e 31 20 32 30 30 20 4f 4b 0d HTTP/1.1 200 OK.
0x0010 0a 44 61 74 65 3a 20 54 68 75 2c 20 32 37 20 41 .Date: Thu, 27 A
0x0020 75 67 20 32 30 30 39 20 30 37 3a 34 39 3a 33 30 ug 2009 07:49:30
0x0030 20 47 4d 54 0d 0a 53 65 72 76 65 72 3a 20 41 70 GMT..Server: Ap
0x0040 61 63 68 65 2f 32 2e 32 2e 31 31 20 28 46 72 65 ache/2.2.11 (Fre
0x0050 65 42 53 44 29 20 6d 6f 64 5f 73 73 6c 2f 32 2e eBSD) mod_ssl/2.
0x0060 32 2e 31 31 20 4f 70 65 6e 53 53 4c 2f 30 2e 39 2.11 OpenSSL/0.9
0x0070 2e 37 65 2d 70 31 20 44 41 56 2f 32 20 50 48 50 .7e-p1 DAV/2 PHP
0x0080 2f 35 2e 32 2e 38 20 77 69 74 68 20 53 75 68 6f /5.2.8 with Suho
0x0090 73 69 6e 2d 50 61 74 63 68 0d 0a 4c 61 73 74 2d sin-Patch..Last-
0x00a0 4d 6f 64 69 66 69 65 64 3a 20 57 65 64 2c 20 32 Modified: Wed, 2
0x00b0 36 20 41 75 67 20 32 30 30 39 20 31 38 3a 30 39 6 Aug 2009 18:09
0x00c0 3a 34 33 20 47 4d 54 0d 0a 45 54 61 67 3a 20 22 :43 GMT..ETag: "
0x00d0 61 61 30 32 62 33 2d 63 37 63 34 2d 34 37 32 30 aa02b3-c7c4-4720
0x00e0 66 35 61 66 36 32 37 63 30 22 0d 0a 41 63 63 65 f5af627c0"..Acce
0x00f0 70 74 2d 52 61 6e 67 65 73 3a 20 62 79 74 65 73 pt-Ranges: bytes
0x0100 0d 0a 43 6f 6e 74 65 6e 74 2d 4c 65 6e 67 74 68 ..Content-Length
0x0110 3a 20 35 31 31 34 30 0d 0a 43 6f 6e 74 65 6e 74 : 51140..Content
0x0120 2d 54 79 70 65 3a 20 61 70 70 6c 69 63 61 74 69 -Type: applicati
0x0130 6f 6e 2f 6f 63 74 65 74 2d 73 74 72 65 61 6d 0d on/octet-stream.
0x0140 0a 0d 0a f2 2a 25 3f 37 50 e7 02 09 f5 10 cf 63 ....*%?7P......c
0x0150 47 4e 5f 2a b3 ac 05 b6 fe 42 cd fe c0 9a ec 6f GN_*.....B.....o
0x0160 bb 3c 98 d5 75 f7 6a 61 6c 30 88 6a 5c e5 20 65 .<..u.jal0.j\. e
0x0170 75 6f 51 ba 91 63 61 52 5a c8 91 cd 79 84 7e 96 uoQ..caRZ...y.~.
0x0180 96 58 e1 3e 20 f8 04 12 82 61 59 1e b6 18 d1 9b .X.> ....aY.....
0x0190 56 3b f3 e7 5b bb 12 66 10 19 92 8e f8 e1 d0 ea V;..[..f........
0x01a0 42 77 fd 8e a7 4e 0e 1f fa 83 32 f6 df 9c 91 79 Bw...N....2....y
[NORMALIZED/GZIP BUFFER DATA] (IDENTICAL TO RAW BUF)
[HTTP_HEADER BUFFER DATA (0x8b24b00)]:
0x0000 44 61 74 65 3a 20 54 68 75 2c 20 32 37 20 41 75 Date: Thu, 27 Au
0x0010 67 20 32 30 30 39 20 30 37 3a 34 39 3a 33 30 20 g 2009 07:49:30
0x0020 47 4d 54 0d 0a 53 65 72 76 65 72 3a 20 41 70 61 GMT..Server: Apa
0x0030 63 68 65 2f 32 2e 32 2e 31 31 20 28 46 72 65 65 che/2.2.11 (Free
0x0040 42 53 44 29 20 6d 6f 64 5f 73 73 6c 2f 32 2e 32 BSD) mod_ssl/2.2
0x0050 2e 31 31 20 4f 70 65 6e 53 53 4c 2f 30 2e 39 2e .11 OpenSSL/0.9.
0x0060 37 65 2d 70 31 20 44 41 56 2f 32 20 50 48 50 2f 7e-p1 DAV/2 PHP/
0x0070 35 2e 32 2e 38 20 77 69 74 68 20 53 75 68 6f 73 5.2.8 with Suhos
0x0080 69 6e 2d 50 61 74 63 68 0d 0a 4c 61 73 74 2d 4d in-Patch..Last-M
0x0090 6f 64 69 66 69 65 64 3a 20 57 65 64 2c 20 32 36 odified: Wed, 26
0x00a0 20 41 75 67 20 32 30 30 39 20 31 38 3a 30 39 3a Aug 2009 18:09:
0x00b0 34 33 20 47 4d 54 0d 0a 45 54 61 67 3a 20 22 61 43 GMT..ETag: "a
0x00c0 61 30 32 62 33 2d 63 37 63 34 2d 34 37 32 30 66 a02b3-c7c4-4720f
0x00d0 35 61 66 36 32 37 63 30 22 0d 0a 41 63 63 65 70 5af627c0"..Accep
0x00e0 74 2d 52 61 6e 67 65 73 3a 20 62 79 74 65 73 0d t-Ranges: bytes.
0x00f0 0a 43 6f 6e 74 65 6e 74 2d 4c 65 6e 67 74 68 3a .Content-Length:
0x0100 20 35 31 31 34 30 0d 0a 43 6f 6e 74 65 6e 74 2d 51140..Content-
0x0110 54 79 70 65 3a 20 61 70 70 6c 69 63 61 74 69 6f Type: applicatio
0x0120 6e 2f 6f 63 74 65 74 2d 73 74 72 65 61 6d 0d 0a n/octet-stream..
0x0130 0d 0a
[HTTP_HEADER_RAW BUFFER DATA (0xacc1ff1)]:
0x0000 44 61 74 65 3a 20 54 68 75 2c 20 32 37 20 41 75 Date: Thu, 27 Au
0x0010 67 20 32 30 30 39 20 30 37 3a 34 39 3a 33 30 20 g 2009 07:49:30
0x0020 47 4d 54 0d 0a 53 65 72 76 65 72 3a 20 41 70 61 GMT..Server: Apa
0x0030 63 68 65 2f 32 2e 32 2e 31 31 20 28 46 72 65 65 che/2.2.11 (Free
0x0040 42 53 44 29 20 6d 6f 64 5f 73 73 6c 2f 32 2e 32 BSD) mod_ssl/2.2
0x0050 2e 31 31 20 4f 70 65 6e 53 53 4c 2f 30 2e 39 2e .11 OpenSSL/0.9.
0x0060 37 65 2d 70 31 20 44 41 56 2f 32 20 50 48 50 2f 7e-p1 DAV/2 PHP/
0x0070 35 2e 32 2e 38 20 77 69 74 68 20 53 75 68 6f 73 5.2.8 with Suhos
0x0080 69 6e 2d 50 61 74 63 68 0d 0a 4c 61 73 74 2d 4d in-Patch..Last-M
0x0090 6f 64 69 66 69 65 64 3a 20 57 65 64 2c 20 32 36 odified: Wed, 26
0x00a0 20 41 75 67 20 32 30 30 39 20 31 38 3a 30 39 3a Aug 2009 18:09:
0x00b0 34 33 20 47 4d 54 0d 0a 45 54 61 67 3a 20 22 61 43 GMT..ETag: "a
0x00c0 61 30 32 62 33 2d 63 37 63 34 2d 34 37 32 30 66 a02b3-c7c4-4720f
0x00d0 35 61 66 36 32 37 63 30 22 0d 0a 41 63 63 65 70 5af627c0"..Accep
0x00e0 74 2d 52 61 6e 67 65 73 3a 20 62 79 74 65 73 0d t-Ranges: bytes.
0x00f0 0a 43 6f 6e 74 65 6e 74 2d 4c 65 6e 67 74 68 3a .Content-Length:
0x0100 20 35 31 31 34 30 0d 0a 43 6f 6e 74 65 6e 74 2d 51140..Content-
0x0110 54 79 70 65 3a 20 61 70 70 6c 69 63 61 74 69 6f Type: applicatio
0x0120 6e 2f 6f 63 74 65 74 2d 73 74 72 65 61 6d 0d 0a n/octet-stream..
0x0130 0d 0a ..
[HTTP_STAT_CODE BUFFER DATA (0xacc1fe9)]:
0x0000 32 30 30 200
[HTTP_STAT_MSG BUFFER DATA (0xacc1fed)]:
0x0000 4f 4b 0d 0a OK..
********** END PACKET **********
I have some things left to do on this. For example, raw buffers that are the same as their normalized buffers don't need to be printed. As we update things, we'll announce on @VRT_Sourcefire and @kpyke. Let us know how you're using this or if you notice any bugs at research@sourcefire.com.