[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

0.5.4 release candidate 1



Hi all,

I've uploaded what should be RC1 for firestorm 0.5.4, if there are no
major outstanding issues I shall release it as 0.5.4. The only issues
standing in my way are build issues etc. I need to sort out my cross
compiler to make RH9 rpms. (I'd rather not build binaries, I don't have
the time).

The URL is:
 http://www.scaramanga.co.uk/firestorm/firestorm-snapshot.tar.gz

PowerPC debs are available from apt:
 deb http://www.scaramanga.co.uk/ scaramanga main
 deb-src http://www.scaramanga.co.uk/ scaramanga main

Major new change is the addition of indexing and querying support in
firecat. It works like this:

1. Build an index for the elog you want to search/filter in. This speeds
up the queries, like in a database, only faster than most RDBMS ;)

 $ firecat -i large.elog

2. Take a look at the index, you can see which fields have been indexed
and other information. If your file is foo.elog, the index will be
called foo.elog.idx.

 $ firecat -l large.elog
idx: large.elog.idx
elog: large.elog
status: up to date
field: name=icmp.code num_keys=1 num_vals=272 type=ordered
field: name=icmp.type num_keys=1 num_vals=272 type=ordered
field: name=ip.dst num_keys=350 num_vals=49452 type=ordered
field: name=ip.proto num_keys=2 num_vals=49452 type=ordered
field: name=ip.src num_keys=1148 num_vals=49452 type=ordered
field: name=ip.ttl num_keys=65 num_vals=49452 type=ordered
field: name=prio num_keys=3 num_vals=49452 type=ordered
field: name=rev num_keys=6 num_vals=49452 type=ordered
field: name=sid num_keys=63 num_vals=49452 type=ordered
field: name=tcp.dport num_keys=4487 num_vals=49180 type=ordered
field: name=tcp.sport num_keys=26242 num_vals=49180 type=ordered
field: name=udp.dport num_keys=0 num_vals=0 type=ordered
field: name=udp.sport num_keys=0 num_vals=0 type=ordered

3. As you can see, not everything is indexed, but this is still a work
in progress. You can query on any field mentioned there. The queries
cannot be arbitrarily complex like SQL just yet. Here is an example.

 $ firecat large.elog -f log -q "tcp.dport<1024 prio=1"
<... Results spew forth ...>

4. We can marvel at how fast it is in comparison to say, ethereal. The
indexes help a lot:

 $ ls -alh large.elog large.elog.idx
-rw-r-----  1 scara scara      36M 2003-07-02 11:27 large.elog
-rw-r-----  1 scara scara     2.1M 2003-07-03 14:17 large.elog.idx

 $ time firecat large.elog -f log -q "tcp.dport=3128 ip.ttl>64" | wc -l
    389
 
real    0m0.023s
user    0m0.020s
sys     0m0.000s

 $ firecat -f dump large.elog > large.cap
 $ time ethereal -nr large.cap -R "tcp.dstport == 3128 && ip.ttl > 64" \
	| wc -l
    389
 
real    0m23.013s
user    0m22.610s
sys     0m0.100s

Firecat is almost exactly 1,000 times faster. Our firestorm test is too
quick to be accurate anyway. I would need a query at least 10 times as
complex.

In future this should support much more complex queries, and other data
types. For example, this currently treats IP addresses as numbers, and
allows only standard comparison operators, it would need to be able to
do CIDR and masks etc. The console will also allow construction of
filters using a nice interface, and you will be able to save "favourite"
queries, maybe some of them producing "canned" results in the background
for even faster response times.

-- 
// Gianni Tedesco (gianni at scaramanga dot co dot uk)
lynx --source www.scaramanga.co.uk/gianni-at-ecsc.asc | gpg --import
8646BE7D: 6D9F 2287 870E A2C9 8F60 3A3C 91B5 7669 8646 BE7D

Attachment: signature.asc
Description: This is a digitally signed message part