Hi all, I've uploaded what should be RC1 for firestorm 0.5.4, if there are no major outstanding issues I shall release it as 0.5.4. The only issues standing in my way are build issues etc. I need to sort out my cross compiler to make RH9 rpms. (I'd rather not build binaries, I don't have the time). The URL is: http://www.scaramanga.co.uk/firestorm/firestorm-snapshot.tar.gz PowerPC debs are available from apt: deb http://www.scaramanga.co.uk/ scaramanga main deb-src http://www.scaramanga.co.uk/ scaramanga main Major new change is the addition of indexing and querying support in firecat. It works like this: 1. Build an index for the elog you want to search/filter in. This speeds up the queries, like in a database, only faster than most RDBMS ;) $ firecat -i large.elog 2. Take a look at the index, you can see which fields have been indexed and other information. If your file is foo.elog, the index will be called foo.elog.idx. $ firecat -l large.elog idx: large.elog.idx elog: large.elog status: up to date field: name=icmp.code num_keys=1 num_vals=272 type=ordered field: name=icmp.type num_keys=1 num_vals=272 type=ordered field: name=ip.dst num_keys=350 num_vals=49452 type=ordered field: name=ip.proto num_keys=2 num_vals=49452 type=ordered field: name=ip.src num_keys=1148 num_vals=49452 type=ordered field: name=ip.ttl num_keys=65 num_vals=49452 type=ordered field: name=prio num_keys=3 num_vals=49452 type=ordered field: name=rev num_keys=6 num_vals=49452 type=ordered field: name=sid num_keys=63 num_vals=49452 type=ordered field: name=tcp.dport num_keys=4487 num_vals=49180 type=ordered field: name=tcp.sport num_keys=26242 num_vals=49180 type=ordered field: name=udp.dport num_keys=0 num_vals=0 type=ordered field: name=udp.sport num_keys=0 num_vals=0 type=ordered 3. As you can see, not everything is indexed, but this is still a work in progress. You can query on any field mentioned there. The queries cannot be arbitrarily complex like SQL just yet. Here is an example. $ firecat large.elog -f log -q "tcp.dport<1024 prio=1" <... Results spew forth ...> 4. We can marvel at how fast it is in comparison to say, ethereal. The indexes help a lot: $ ls -alh large.elog large.elog.idx -rw-r----- 1 scara scara 36M 2003-07-02 11:27 large.elog -rw-r----- 1 scara scara 2.1M 2003-07-03 14:17 large.elog.idx $ time firecat large.elog -f log -q "tcp.dport=3128 ip.ttl>64" | wc -l 389 real 0m0.023s user 0m0.020s sys 0m0.000s $ firecat -f dump large.elog > large.cap $ time ethereal -nr large.cap -R "tcp.dstport == 3128 && ip.ttl > 64" \ | wc -l 389 real 0m23.013s user 0m22.610s sys 0m0.100s Firecat is almost exactly 1,000 times faster. Our firestorm test is too quick to be accurate anyway. I would need a query at least 10 times as complex. In future this should support much more complex queries, and other data types. For example, this currently treats IP addresses as numbers, and allows only standard comparison operators, it would need to be able to do CIDR and masks etc. The console will also allow construction of filters using a nice interface, and you will be able to save "favourite" queries, maybe some of them producing "canned" results in the background for even faster response times. -- // Gianni Tedesco (gianni at scaramanga dot co dot uk) lynx --source www.scaramanga.co.uk/gianni-at-ecsc.asc | gpg --import 8646BE7D: 6D9F 2287 870E A2C9 8F60 3A3C 91B5 7669 8646 BE7D
Attachment:
signature.asc
Description: This is a digitally signed message part