From an operations perspective, there is very little that is less useful and more aggravating than vendor magic. What I mean by this is anything that "happens" in the background that you have no visibility into. While many organizations enjoy the simplicity provided by this, when you need to McGyver some solution to a security issue that vendors haven't addressed yet, you just might feel like simply setting fire to equipment that got in your way. Not that I'm endorsing that.

This is one of the main strengths of open source software. If you know what you're doing, you can uncover all the magic so you know exactly what you're dealing with and you can fix it up if you need to. Snort-wise, one of the things that it does in the background for you is normalize data and put it into various buffers. At one point the list of buffers was fairly small: normalized and raw. At this point you have the following buffers:

Buffer Internal Representation Notes *
normalized CONTENT_BUF_NORMALIZED This is the default buffer that Snort matches against.  Also contains gzip decoded data.
raw CONTENT_BUF_RAW Not used often, mainly for looking at non-normalized TELNET and FTP data.
http_uri CONTENT_BUF_URI
http_raw_uri CONTENT_BUF_RAW_URI
http_cookie CONTENT_BUF_COOKIE Config option required to activate cookie parsing.
http_raw_cookie CONTENT_BUF_RAW_COOKIE
http_header CONTENT_BUF_HEADER
http_raw_header CONTENT_BUF_RAW_HEADER
http_method CONTENT_BUF_METHOD Parsed from header, not normalized.
http_stat_code CONTENT_BUF_STAT_CODE Parsed from header, not normalized.
http_stat_msg CONTENT_BUF_STAT_MSG Parsed from header, not normalized.
file_data:mime BUF_FILE_DATA_MIME Buffer holds the mime decoded data for SMTP
file_data BUF_FILE_DATA Not actually a buffer, but a pointer into normalized buffer
base64_decode BUF_BASE64_DECODE

* see the labs_buffers.c file for additional commentary on the buffers

Buffers aren't the only place where Snort massages the data.  Both fragmentation and stream reassembly occur and can impact detection.  So between parsing, normalization, defragmentation and stream reassembly, the final data blob looked at by Snort can be significantly different than what you see on Wireshark.  This can make rule writing and debugging difficult.  To help with this I've written a set of .SO rules that print out the buffers exactly as Snort views them for each packet in a PCAP.  They've been really useful, so we're releasing them on the VRT Labs site (currently tested against Snort 2.9.0.3, so don't yell at me if it doesn't work on anything before that).

Once you download them, move them to your .SO directory and modify the following line in your Makefile:

libs := icmp p2p dos exploit bad-traffic web-activex web-client web-iis netbios misc nntp smtp web-misc sql imap chat multimedia pop3

to:

libs := labs

Then run "make", and modify your Snort conf to include the new labs.rules file.  It should be something like:

include $RULE_PATH/../so_rules/labs.rules

The labs.rules file should look like this:

# Autogenerated skeleton rules file.  Do NOT edit by hand
alert tcp any any -> any any (msg:"VRT LABS: All Ports Two-Way Packet Description"; sid:100005; gid:3; rev:1; classtype:misc-activity; metadata: engine shared, soid 3|100005;)
alert tcp any any -> any $HTTP_PORTS (msg:"VRT LABS: HTTP_PORTS Client to Server Packet Description"; sid:100000; gid:3; rev:1; classtype:misc-activity; metadata: engine shared, soid 3|100000;)
alert tcp any $HTTP_PORTS -> any any (msg:"VRT LABS: HTTP_PORTS Server to Client Packet Description"; sid:100001; gid:3; rev:1; classtype:misc-activity; metadata: engine shared, soid 3|100001;)
alert tcp any any -> any 25 (msg:"VRT LABS: SMTP Client to Server Packet Description"; sid:100111; gid:3; rev:1; classtype:misc-activity; metadata: engine shared, soid 3|100111;)

To get started, I would recommend commenting out all but the first rule.  This will show you all the goodies you need.  When you're working specifically with http data, I'd enable one or both of the second and third rules.  Finally, when looking at SMTP client-to-server traffic (where you'll see mime-decoded data), you can enable only the fourth rule.  If you have all rules on, you'll get multiple decoding (probably two per packet).

Hey, listen up:  This is only designed to be run on pcap files with TCP data, where you have all the time in the world to read, parse and write data.  If you run this on a running sensor it will probably melt, so don't.  Also, it is what it is, so don't run it on anything important.

Each packet will start with a header as follows:

******************************  NEW PACKET  *****************************
Timestamp: 2009-08-27 18:08:29:16274
Src IP: 195.2.253.95:80
Dst IP: 10.11.250.196:1075
TCP Flags: ACK

The top line lets you know you have a new packet (easy to miss if you have a lot of data) and then you have a time-stamp (conveniently formatted in Wireshark format) and more IP/TCP header information.  If this is a pseudo packet rebuilt by the stream5 preprocessor, you see this instead of the NEW PACKET line above:

************************  NEW REASSEMBLED PACKET  ***********************

Then we start pulling apart the buffers.  First we check if there is data, and if there isn't any, we simply write:

[-No data in this packet-]

Otherwise we write the raw buffer out and check to see if the normalized buffer is different than the raw buffer.  If it isn't, you'll see the raw packet data and then:

[NORMALIZED/GZIP BUFFER DATA] (IDENTICAL TO RAW BUF)

If the data isn't the same, it will print the normalized data.

After this we get into specificly parsed buffers.  After the jump, we have two packets that are an example of a packet broken out.  It is a client request and server response over http, so you can see how we break things out:

******************************  NEW PACKET  *****************************
Timestamp: 2009-08-27 18:08:28:886026
Src IP: 10.11.250.196:1075
Dst IP: 10.2.253.95:80
TCP Flags: PSH ACK

******  BUFFER INFORMATION  ******
[RAW BUFFER DATA (0xacc1fe0)]:
0x0000  47 45 54 20 2f 41 14 41 41 41 41 2f 63 6f 6e 66  GET /AAAAAA/conf
0x0010  69 67 32 2e 62 69 6e 20 48 54 54 50 2f 31 2e 31  ig2.bin HTTP/1.1
0x0020  0d 0a 41 63 63 65 70 74 3a 20 2a 2f 2a 0d 0a 55  ..Accept: */*..U
0x0030  73 65 72 2d 41 67 65 6e 74 3a 20 4d 6f 7a 69 6c  ser-Agent: Mozil
0x0040  6c 61 2f 34 2e 30 20 28 63 6f 6d 70 61 74 69 62  la/4.0 (compatib
0x0050  6c 65 3b 20 4d 53 49 45 20 37 2e 30 3b 20 57 69  le; MSIE 7.0; Wi
0x0060  6e 64 6f 77 73 20 4e 54 20 35 2e 31 29 0d 0a 48  ndows NT 5.1)..H
0x0070  6f 73 74 3a 20 31 39 35 2e 32 2e 32 35 33 2e 39  ost: 195.2.253.9
0x0080  35 0d 0a 50 72 61 67 6d 61 3a 20 6e 6f 2d 63 61  5..Pragma: no-ca
0x0090  63 68 65 0d 0a 0d 0a                             che....
[NORMALIZED/GZIP BUFFER DATA] (IDENTICAL TO RAW BUF)

[HTTP_HEADER BUFFER DATA (0x8b1fb00)]:
0x0000  41 63 63 65 70 74 3a 20 2a 2f 2a 0d 0a 55 73 65  Accept: */*..Use
0x0010  72 2d 41 67 65 6e 74 3a 20 4d 6f 7a 69 6c 6c 61  r-Agent: Mozilla
0x0020  2f 34 2e 30 20 28 63 6f 6d 70 61 74 69 62 6c 65  /4.0 (compatible
0x0030  3b 20 4d 53 49 45 20 37 2e 30 3b 20 57 69 6e 64  ; MSIE 7.0; Wind
0x0040  6f 77 73 20 4e 54 20 35 2e 31 29 0d 0a 48 6f 73  ows NT 5.1)..Hos
0x0050  74 3a 20 31 39 35 2e 32 2e 32 35 33 2e 39 35 0d  t: 195.2.253.95.
0x0060  0a 50 72 61 67 6d 61 3a 20 6e 6f 2d 63 61 63 68  .Pragma: no-cach
0x0070  65 0d 0a 0d 0a                                   e....

[HTTP_HEADER_RAW BUFFER DATA (0xacc2002)]:
0x0000  41 63 63 65 70 74 3a 20 2a 2f 2a 0d 0a 55 73 65  Accept: */*..Use
0x0010  72 2d 41 67 65 6e 74 3a 20 4d 6f 7a 69 6c 6c 61  r-Agent: Mozilla
0x0020  2f 34 2e 30 20 28 63 6f 6d 70 61 74 69 62 6c 65  /4.0 (compatible
0x0030  3b 20 4d 53 49 45 20 37 2e 30 3b 20 57 69 6e 64  ; MSIE 7.0; Wind
0x0040  6f 77 73 20 4e 54 20 35 2e 31 29 0d 0a 48 6f 73  ows NT 5.1)..Hos
0x0050  74 3a 20 31 39 35 2e 32 2e 32 35 33 2e 39 35 0d  t: 195.2.253.95.
0x0060  0a 50 72 61 67 6d 61 3a 20 6e 6f 2d 63 61 63 68  .Pragma: no-cach
0x0070  65 0d 0a 0d 0a                                   e....

[HTTP_URI BUFFER DATA (0xacc1fe4)]:
0x0000  2f 41 41 41 41 41 41 2f 63 6f 6e 66 69 67 32 2e  /AAAAAA/config2.
0x0010  62 69 6e                                         bin

[HTTP_URI_RAW BUFFER DATA (0xacc1fe4)]:
0x0000  2f 41 41 41 41 41 41 2f 63 6f 6e 66 69 67 32 2e  /AAAAAA/config2.
0x0010  62 69 6e                                         bin
[HTTP_POST BUFFER DATA (NO DATA)]

[HTTP_METHOD BUFFER DATA (0xacc1fe0)]:
0x0000  47 45 54                                         GET

[HTTP_COOKIE BUFFER DATA (NO DATA)]

[HTTP_COOKIE_RAW BUFFER DATA] (NO DATA)]

**********  END PACKET  **********

****************************** NEW PACKET *****************************
Timestamp: 2009-08-27 18:08:29:16274
Src IP: 10.2.253.95:80
Dst IP: 10.11.250.196:1075
TCP Flags: ACK

****** BUFFER INFORMATION ******
[RAW BUFFER DATA (0xacc1fe0)]:

0x0000 48 54 54 50 2f 31 2e 31 20 32 30 30 20 4f 4b 0d HTTP/1.1 200 OK.
0x0010 0a 44 61 74 65 3a 20 54 68 75 2c 20 32 37 20 41 .Date: Thu, 27 A
0x0020 75 67 20 32 30 30 39 20 30 37 3a 34 39 3a 33 30 ug 2009 07:49:30
0x0030 20 47 4d 54 0d 0a 53 65 72 76 65 72 3a 20 41 70 GMT..Server: Ap
0x0040 61 63 68 65 2f 32 2e 32 2e 31 31 20 28 46 72 65 ache/2.2.11 (Fre
0x0050 65 42 53 44 29 20 6d 6f 64 5f 73 73 6c 2f 32 2e eBSD) mod_ssl/2.
0x0060 32 2e 31 31 20 4f 70 65 6e 53 53 4c 2f 30 2e 39 2.11 OpenSSL/0.9
0x0070 2e 37 65 2d 70 31 20 44 41 56 2f 32 20 50 48 50 .7e-p1 DAV/2 PHP
0x0080 2f 35 2e 32 2e 38 20 77 69 74 68 20 53 75 68 6f /5.2.8 with Suho
0x0090 73 69 6e 2d 50 61 74 63 68 0d 0a 4c 61 73 74 2d sin-Patch..Last-
0x00a0 4d 6f 64 69 66 69 65 64 3a 20 57 65 64 2c 20 32 Modified: Wed, 2
0x00b0 36 20 41 75 67 20 32 30 30 39 20 31 38 3a 30 39 6 Aug 2009 18:09
0x00c0 3a 34 33 20 47 4d 54 0d 0a 45 54 61 67 3a 20 22 :43 GMT..ETag: "
0x00d0 61 61 30 32 62 33 2d 63 37 63 34 2d 34 37 32 30 aa02b3-c7c4-4720
0x00e0 66 35 61 66 36 32 37 63 30 22 0d 0a 41 63 63 65 f5af627c0"..Acce
0x00f0 70 74 2d 52 61 6e 67 65 73 3a 20 62 79 74 65 73 pt-Ranges: bytes
0x0100 0d 0a 43 6f 6e 74 65 6e 74 2d 4c 65 6e 67 74 68 ..Content-Length
0x0110 3a 20 35 31 31 34 30 0d 0a 43 6f 6e 74 65 6e 74 : 51140..Content
0x0120 2d 54 79 70 65 3a 20 61 70 70 6c 69 63 61 74 69 -Type: applicati
0x0130 6f 6e 2f 6f 63 74 65 74 2d 73 74 72 65 61 6d 0d on/octet-stream.
0x0140 0a 0d 0a f2 2a 25 3f 37 50 e7 02 09 f5 10 cf 63 ....*%?7P......c
0x0150 47 4e 5f 2a b3 ac 05 b6 fe 42 cd fe c0 9a ec 6f GN_*.....B.....o
0x0160 bb 3c 98 d5 75 f7 6a 61 6c 30 88 6a 5c e5 20 65 .<..u.jal0.j\. e
0x0170 75 6f 51 ba 91 63 61 52 5a c8 91 cd 79 84 7e 96 uoQ..caRZ...y.~.
0x0180 96 58 e1 3e 20 f8 04 12 82 61 59 1e b6 18 d1 9b .X.> ....aY.....
0x0190 56 3b f3 e7 5b bb 12 66 10 19 92 8e f8 e1 d0 ea V;..[..f........
0x01a0 42 77 fd 8e a7 4e 0e 1f fa 83 32 f6 df 9c 91 79 Bw...N....2....y

[NORMALIZED/GZIP BUFFER DATA] (IDENTICAL TO RAW BUF)

[HTTP_HEADER BUFFER DATA (0x8b24b00)]:

0x0000 44 61 74 65 3a 20 54 68 75 2c 20 32 37 20 41 75 Date: Thu, 27 Au
0x0010 67 20 32 30 30 39 20 30 37 3a 34 39 3a 33 30 20 g 2009 07:49:30
0x0020 47 4d 54 0d 0a 53 65 72 76 65 72 3a 20 41 70 61 GMT..Server: Apa
0x0030 63 68 65 2f 32 2e 32 2e 31 31 20 28 46 72 65 65 che/2.2.11 (Free
0x0040 42 53 44 29 20 6d 6f 64 5f 73 73 6c 2f 32 2e 32 BSD) mod_ssl/2.2
0x0050 2e 31 31 20 4f 70 65 6e 53 53 4c 2f 30 2e 39 2e .11 OpenSSL/0.9.
0x0060 37 65 2d 70 31 20 44 41 56 2f 32 20 50 48 50 2f 7e-p1 DAV/2 PHP/
0x0070 35 2e 32 2e 38 20 77 69 74 68 20 53 75 68 6f 73 5.2.8 with Suhos
0x0080 69 6e 2d 50 61 74 63 68 0d 0a 4c 61 73 74 2d 4d in-Patch..Last-M
0x0090 6f 64 69 66 69 65 64 3a 20 57 65 64 2c 20 32 36 odified: Wed, 26
0x00a0 20 41 75 67 20 32 30 30 39 20 31 38 3a 30 39 3a Aug 2009 18:09:
0x00b0 34 33 20 47 4d 54 0d 0a 45 54 61 67 3a 20 22 61 43 GMT..ETag: "a
0x00c0 61 30 32 62 33 2d 63 37 63 34 2d 34 37 32 30 66 a02b3-c7c4-4720f
0x00d0 35 61 66 36 32 37 63 30 22 0d 0a 41 63 63 65 70 5af627c0"..Accep
0x00e0 74 2d 52 61 6e 67 65 73 3a 20 62 79 74 65 73 0d t-Ranges: bytes.
0x00f0 0a 43 6f 6e 74 65 6e 74 2d 4c 65 6e 67 74 68 3a .Content-Length:
0x0100 20 35 31 31 34 30 0d 0a 43 6f 6e 74 65 6e 74 2d 51140..Content-
0x0110 54 79 70 65 3a 20 61 70 70 6c 69 63 61 74 69 6f Type: applicatio
0x0120 6e 2f 6f 63 74 65 74 2d 73 74 72 65 61 6d 0d 0a n/octet-stream..
0x0130 0d 0a

[HTTP_HEADER_RAW BUFFER DATA (0xacc1ff1)]:
0x0000 44 61 74 65 3a 20 54 68 75 2c 20 32 37 20 41 75 Date: Thu, 27 Au
0x0010 67 20 32 30 30 39 20 30 37 3a 34 39 3a 33 30 20 g 2009 07:49:30
0x0020 47 4d 54 0d 0a 53 65 72 76 65 72 3a 20 41 70 61 GMT..Server: Apa
0x0030 63 68 65 2f 32 2e 32 2e 31 31 20 28 46 72 65 65 che/2.2.11 (Free
0x0040 42 53 44 29 20 6d 6f 64 5f 73 73 6c 2f 32 2e 32 BSD) mod_ssl/2.2
0x0050 2e 31 31 20 4f 70 65 6e 53 53 4c 2f 30 2e 39 2e .11 OpenSSL/0.9.
0x0060 37 65 2d 70 31 20 44 41 56 2f 32 20 50 48 50 2f 7e-p1 DAV/2 PHP/
0x0070 35 2e 32 2e 38 20 77 69 74 68 20 53 75 68 6f 73 5.2.8 with Suhos
0x0080 69 6e 2d 50 61 74 63 68 0d 0a 4c 61 73 74 2d 4d in-Patch..Last-M
0x0090 6f 64 69 66 69 65 64 3a 20 57 65 64 2c 20 32 36 odified: Wed, 26
0x00a0 20 41 75 67 20 32 30 30 39 20 31 38 3a 30 39 3a Aug 2009 18:09:
0x00b0 34 33 20 47 4d 54 0d 0a 45 54 61 67 3a 20 22 61 43 GMT..ETag: "a
0x00c0 61 30 32 62 33 2d 63 37 63 34 2d 34 37 32 30 66 a02b3-c7c4-4720f
0x00d0 35 61 66 36 32 37 63 30 22 0d 0a 41 63 63 65 70 5af627c0"..Accep
0x00e0 74 2d 52 61 6e 67 65 73 3a 20 62 79 74 65 73 0d t-Ranges: bytes.
0x00f0 0a 43 6f 6e 74 65 6e 74 2d 4c 65 6e 67 74 68 3a .Content-Length:
0x0100 20 35 31 31 34 30 0d 0a 43 6f 6e 74 65 6e 74 2d 51140..Content-
0x0110 54 79 70 65 3a 20 61 70 70 6c 69 63 61 74 69 6f Type: applicatio
0x0120 6e 2f 6f 63 74 65 74 2d 73 74 72 65 61 6d 0d 0a n/octet-stream..
0x0130 0d 0a ..

[HTTP_STAT_CODE BUFFER DATA (0xacc1fe9)]:
0x0000 32 30 30 200

[HTTP_STAT_MSG BUFFER DATA (0xacc1fed)]:
0x0000 4f 4b 0d 0a OK..

********** END PACKET **********


I have some things left to do on this. For example, raw buffers that are the same as their normalized buffers don't need to be printed. As we update things, we'll announce on @VRT_Sourcefire and @kpyke. Let us know how you're using this or if you notice any bugs at research@sourcefire.com.