Scribbles

At the back end of 2014 and start of 2015, I worked on high-frequency market-making of KOSPI200 futures contracts. Since I left that world, I spent a bit of time looking at real-time order-flow reconstruction from direct-market-access TAQ price feeds — the rough inverse of what a matching engine does: given new trades and quotes, infer the sequence of operations on the customer side. That's essentially data-cleaning for short-term predictions of taking the spread, a way of managing risk for automated market makers. Also useful for simulation as it tracks the position of your own orders within queues.

From around 2012-2014 I had a brief, lucky career as a blackjack player. I produced code for calculating optimal strategies and card-counting simulation, plus memorization aids — not yet posted, but with some interesting representations of the state space.

A number of technical challenges had to be overcome for this one. First, the protocol is SSL-encrypted — worked around with a debugger and a hex editor, patching the binary so the certificate check was skipped (the patch). Then built an SSL man-in-the-middle proxy — piggy — and used NAT to force the PokerStars client to connect to it instead of the real servers. That made it possible to watch the traffic going back and forth.

After the initial handshake, though, everything was compressed. Strings in the binary suggested LZHL; a search for LZHL lookup tables inside the binary confirmed it. The decompressor never fully worked — some data comes out fine, but there are "holes" of all-zero bytes for reasons that were never tracked down.

Looking back, they were probably fixed-sized zero-padded buffers and LZHL was just used to compress the padding away, and that's just how they handeled variable-length fields.

Example log files: pokerstars0.txt, pokerstars1.txt, pokerstars2.txt, pokerstars3.txt.

The Broadcom BCM94306 is the chip inside a Linksys Wireless-G adapter. The plan was to write a full specification for it — but the b43 driver came along first. The technique: run Windows 2000 Server inside QEMU, with the pciproxy patch forwarding register accesses through to the real card sitting in the host machine, and log every I/O, memory, and config-space access along the way while the Linksys driver (bcmwl5.inf) booted and ran.

That produced a full register-space dump and a complete access-log trace of the vendor driver initializing and running the card — exactly the kind of raw material a clean-room reimplementation would want. I have a suspicion the b43 developers used these logs to help extract the firmware blob.

Files: register dump (browsable), full pciproxy access log, an excised running-loop excerpt, driver variables.