Gianni Tedesco

Arbitrary Remote Code Execution is (Still) Arbitrary Remote Code Execution

2020-02-08T10:14:00+09:00

Long gone are the days when servers were like little pets. You logged in to them, you stroked them, you ran little commands on them. Probably some commands you copied and pasted out of stackoverflow or that you found in some google search results.

Then when the server died, you had a little funeral, buried it in a little pet cemetery, and then tried to remember how the hell you had set it up.

But that was fine, you were in grief, you didn't want another pet to be identical to your dead pet anyway.

That is, unless you were trying to deliver consistent, performant, and uninterrupted service to colleagues, customers, or the like.

Cattle, not Pets</h2>
In that case you'd want something a bit more repeatable. A bit more automated. A bit more like a production-line eviscerating pigs for the manufacture of sausage.
First there were configuration management systems, like:

1993: CFEngine</li>
2000: Spooner (proprietary, developed by yours truly) :)</li>
2005: Puppet</li>
2009: Chef</li>
2012: Ansible</li> </ul>
These tools allowed you to represent the configuration or desired state of a server as code or data but, in either case, as a version-controlled repository subject to the quality-oriented practices which began to be adopted by programmers in the early decades of the 21st century.
Later, in 2013, Docker came along. The idea behind docker was to use containers as a way to build or deploy software in an industrialised fashion, like the afore-mentioned sausage factory. The more inspirational, but frankly less tasty, metaphor here would be the containerisation of bulk cargo. Before this idea took hold, containers were thought of as more to do with the containment or confinement of running software. They were sometimes called jails and still widely viewed as being fancy chroots.
In combining the underlying technology of namespaces, control-groups, jails, containers with the philosophy and concepts of infrastructure-as-code, Docker, and systems like it, revolutionised how software services were delivered. By now these, and related, techniques are ubiquitous in the data-centre.
Which is great, because now all of the insecure and insane practices get written down and permanently archived in git repositories instead of being a secret burden of guilt and shame carried on the shoulders of an entire class of people: systems administrators.
Now the blame can be diffused among software developers of all hues and stripes and we can call it a cultural problem :)
Building and Bootstrapping Infrastructure Now.</h2>
Well, a container really is just a fancy chroot. Whereas chroot changes the process's view of the VFS namespace by setting the root to some directory, thereby restricting its view of the filesystem to a subtree of that directory, containers are processes which have their own private, possibly unique, set of mounts.
However, we still need to populate the mounted directories or filesystems with the software that's going to run!
There are any number of ways of doing this.
SnapCraft</h2>
Snapcraft</a> uses a YAML file which defines sources to be downloaded, which will then be compiled and installed on top of a base OS layer which contains compilers, a package manager, etc.
Snapcraft provides security measures, such as the ability to check source code downloads against checksums with the `source-checksum</code> feature and the ability to download code over https.`
`However, if you just download code from the internet over http, then you're just anettercap</a> or a dns poisoning away from being fed exploit code which will automatically be run as part of the snap build process.`
`The snapcraft tour, part of the getting started guide, suffered from this problem, which was reported in Oct 2016</a> and was fixed in May 2017</a>.`
`Which is great. We shouldn't be teaching a new generation of programmers to automatically trust and execute arbitrary remote code.`
LXC Templates : CVE-2017-18641</h2> LXC</a> shipped a series of templates which are scripts which initialise the container root filesystem. The centos and fedora templates relied on yum being installed on the host. The host's yum would be used to download RPMs and install them into the rootfs. The RPMs were being downloaded over http, which could be okay, since RPMs are signed. However yum was being invoked with --nogpgcheck</code> which disabled this feature. The upshot being that anyone using the template is also an ettercap or a hacked mirror or untrustworthy proxy away from having arbitrary code executed as root. This was reported</a> on Feb 3 2017. And it turned out after a quick investigation that the problem applied to around a dozen of the template scripts. Some were fixed in short order. But there were so many cases of this, all in different ad-hoc scripts, by different authors, that it became "a bit of a mess." By Feb 2 2018 the LXC team had started work on distrobuilder</a> which had support for https and gpg from the outset and would be difficult to get wrong. By LXC 3 the templates system had been removed to a historical repo and the status quo is now that images are built securely and signed on trusted infrastructure and then sent to users over https and with a signature which is checked by lxc-download</code>. This is a fine example of the concerns which go into building a software distribution mechanism in which trust can be established. Insecurely Downloaded Remote Code is, by Definition, Arbitrary.</h2> And If it's Arbitrary then You Have No Reason to Trust it.</h3> So Why Would You Want to Execute it?</h4> It's certainly tempting, when writing a Dockerfile, to imagine that everything is executing in some sort of secure isolation layer and that, because of this, security is irrelevant. For now, let's ignore the historical record of security vulnerabilities in container isolation, and the entire available attack surface of the kernel, and the fact that namespaces and control groups are not purpose-built security isolation systems. And the fact that even if they were such, then with all their flexibility and configurability there would be bound to be many possible insecure configurations. Let's ignore all of that for now. Usually, if you're creating a container, you want to be able to trust the code inside it with whatever else is in that container. So even if an attacker can't break out to the host, you won't want them to have full control over the container because they may be able to deface your website, defraud your customers, introduce backdoors into your software builds, etc. And if the attacker has corrupted the image at container build time, then you don't even have the defence that the container instances are cattle that can be shot in the head with a bolt-gun and replaced with a fresh new lump of meat, cloned from the same DNA. So are these practices common? From a quick search on github I was able to find 23,000 instances</a> of "wget http://</a>..." in Dockerfiles. By randomly clicking them you can see that most of these (100% of the ones I looked at anyway) absolutely are downloading arbitrary code and then running it. I was also able to find a further 4,000 instances</a> for curl, but some of these looked legitimate. With a bit of regex trickery one could probably find many more instances. GitHub really is a repository of endless trivial security vulnerabilities. A Brief Message from the Cast</h2> While software vendors take vulnerabilities seriously when they are discovered, and act to remediate them in a timely and responsible fashion, we probably need to be doing a little bit more, as a community, for the typical users of these things to drill into them the dangers involved. It's never going to be comfortable for purveyors of band-saws to be always pointing out that band-saws can irreparably alter the relation between a user's fingers and said user's hands. That's just not the fun stuff, compared to all the cool things you can make and do with power-tools. However, when we see how widespread some of these unsafe practices are, we might want to think about what sort of technical or social interventions might work to improve the situation.



Abusing the CPU's adder circuits
2019-06-15T10:14:00+09:00
Have you ever been asked the interview question "how do you count the number of
bits set in an integer?"</p>
A smart-ass like myself might answer something like:</p>
unsigned</span> int</span> count_set_bits</span></span>(</span></span></span>unsigned</span> int</span> x</span>)</span></span></span>
</span>{</span></span></span>
	return</span> __builtin_popcount</span>(</span></span></span>x</span></span>)</span></span></span>;</span>
</span></span>}</span></span></span>
</span></code></pre>
Which, on x86_64</code>, produces:</p>
	xor	eax, eax
	popcnt	eax, edi
	ret
</span></code></pre>
The interviewer, of course, doesn't like that. He wants to see you come up with
an algorithm. So you come up with something like this:</p>
unsigned</span> int</span> count_set_bits</span></span>(</span></span></span>uint32_t</span> x</span>)</span></span></span>
</span>{</span></span></span>
	unsigned</span> int</span> i,</span> ret;</span>
	for</span>(</span>ret =</span> i =</span> 0</span>;</span> i <</span> 32</span>;</span> i+</span>+</span>)</span></span> {</span>
		ret +=</span> !</span>!</span>(</span>x &</span> (</span>1U</span> <<</span> i)</span></span>)</span></span>;</span>
	}</span></span>
	return</span> ret;</span>
</span></span>}</span></span></span>
</span></code></pre>
Which gives you something like:</p>
	xor	r8d, r8d
	xor	eax, eax
	mov	ecx, 1
	.p2align 4,,10
	.p2align 3
.L4:
	shlx	edx, ecx, eax
	test	edx, edi
	setne	dl
	movzx	edx, dl
	inc	eax
	add	r8d, edx
	cmp	eax, 32
	jne	.L4
	mov	eax, r8d
	ret
</span></code></pre>
Which is, of course, dreadful. But the reason the interviewer likes this is
because it sets up the next question: "how can we optimize this?" Or if you
want to be more circumspect: "what if we only expect one or two bits to be
set?" or "what if our data is very sparse?"</p>
Realistically though, you either know about "Kernighan's trick", or, you don't.
It goes something like this:</p>
unsigned</span> int</span> count_set_bits</span></span>(</span></span></span>unsigned</span> int</span> x</span>)</span></span></span>
</span>{</span></span></span>
	unsigned</span> int</span> ret;</span>
	for</span>(</span>ret =</span> 0</span>;</span> x;</span> x &=</span> (</span>x -</span> 1</span>)</span></span>)</span></span>
		ret+</span>+</span>;</span>
	return</span> ret;</span>
</span></span>}</span></span></span>

</span></code></pre>
Which, at least for an Intel weenie such as myself, was a pointless exercise
because gcc compiles that to:</p>
	xor	eax, eax
	popcnt	eax, edi
	mov	edx, 0
	test	edi, edi
	cmove	eax, edx
	ret
</span></code></pre>
Which looks like a terrible code-generation bug in gcc. But hey, at least clang
does the right thing here:</p>
	popcnt	eax, edi
	ret
</span></code></pre>
Anyway, I digress. Did you notice the x &= (x - 1)</code> part? That is bitwise
woo-woo magic to unset the rightmost bit in x</code>. It's a neat trick, and if you
know it you'll ace this part of the job interview.</p>
But later, when you're at home, alone, contemplating the empty meaninglessness
of the universe, you might ask yourself "but why does it work?"</p>
Two's complement</h2>
Let's forget the &</code> part and just focus on what x - 1</code> is doing for now.</p>
One thing you're going to need to know about is
two's complement</a>
arithmetic.</p>
I won't bore you by recapitulating the details but we should do well to
remember the following equations:</p>
x - y = x + -y
-x    = ~x + 1
</span></code></pre>
Therefore:</p>
x - 1 = x + -1
      = x + 1111...
</span></code></pre>
So the magic rune is just adding all ones to our original value.</p>
Half adder</h2>
So here's the part where we get into how addition really works. You probably
have a good handle on this already from your school days, but let's refresh.</p>
The simplest case is if we're just adding together a couple of 1-bit numbers.</p>
Let's write out the truth table for that:</p>



x</th>
y</th>
result</th>
</tr>
</thead>


0</td>
0</td>
0</td>
</tr>

0</td>
1</td>
1</td>
</tr>

1</td>
0</td>
1</td>
</tr>

1</td>
1</td>
0... oh yeah, we need to carry...</td>
</tr>
</tbody>
</table>
Let's try again:</p>



x</th>
y</th>
result</th>
carry</th>
</tr>
</thead>


0</td>
0</td>
0</td>
0</td>
</tr>

0</td>
1</td>
1</td>
0</td>
</tr>

1</td>
0</td>
1</td>
0</td>
</tr>

1</td>
1</td>
0</td>
1</td>
</tr>
</tbody>
</table>
Right, if we're going to expand this to add numbers with more than one bit,
we're going to need to do something about this carry output.</p>
Full adder</h2>
Let's call the last adder with 2 inputs, and 2 outputs, the "half-adder."</p>
The full adder is going to have 3 inputs: the 2 operands, and then a carry-in
which is going to be connected to the next least significant bit's carry-out.</p>



x</th>
y</th>
carry-in</th>
result</th>
carry-out</th>
</tr>
</thead>


0</td>
0</td>
0</td>
0</td>
0</td>
</tr>

0</td>
1</td>
0</td>
1</td>
0</td>
</tr>

1</td>
0</td>
0</td>
1</td>
0</td>
</tr>

1</td>
1</td>
0</td>
0</td>
1</td>
</tr>

0</td>
0</td>
1</td>
1</td>
0</td>
</tr>

0</td>
1</td>
1</td>
0</td>
1</td>
</tr>

1</td>
0</td>
1</td>
0</td>
1</td>
</tr>

1</td>
1</td>
1</td>
1</td>
1</td>
</tr>
</tbody>
</table>
So you can see in this example of a 4-bit adder that there's this chain of
carry inputs going from right to left:</p>
</p>
Thanks to Colin M.L. Burnett</a>
for the excellent diagram.</p>
Putting it all together</h2>
The basic outline of the algorithm is that we start with an input, x</code>. And as
long as it isn't zero, we unset the least significant bit, and increment the
result.</p>
So if the input is 1000_1100</code>, the loop will iterate 3 times.</p>
x &= (x - 1)</code> is an operation which clears the least-significant set bit in
the input, x</code>.</p>
Since we're going to do x &= mask</code>, that must mean that mask</code> has a zero in
the position we want to zero out. And it can't have any zeroes where the input
has a one.</p>
So we abuse the carry-chain to create such a mask. By adding 1111_1111</code> to the
input, we're essentially ensuring that the carry signal is not asserted until
we hit the first one bit, but after that it will be asserted for all the rest
of the bits (travelling from right to left).</p>
If we extract the relevant parts of the adder truth table we can see how the
addition operation will produce the mask that we need:</p>



x</th>
y</th>
cin</th>
result</th>
cout</th>
explanation</th>
</tr>
</thead>


0</td>
1</td>
0</td>
1</td>
0</td>
Before the first bit, produce ones</td>
</tr>

1</td>
1</td>
0</td>
0</td>
1</td>
For the first bit, produce a zero and set the carry going</td>
</tr>

0</td>
1</td>
1</td>
0</td>
1</td>
now the result is zero if x was 0</td>
</tr>

1</td>
1</td>
1</td>
1</td>
1</td>
or 1 if x was one</td>
</tr>
</tbody>
</table>
What that gives us is a number which is (from right to left):</p>

all ones (but that's okay because corresponding input bits are zeroes)</li>
zero for the first one bit (which ensures that it will be cleared)</li>
and then equal to the input after that (leaving all higher bits unchanged)</li>
</ol>


Item-at-a-time Reduce Functions in Python
2019-03-09T23:28:47+09:00
If you need to write a
Reduce</a> function
in python, there's a number of ways of doing it. I'm going to assume you
already know what that is and leap right in. Suffice to say, you can think of
that as something like an aggregate function in an SQL query.</p>
Perhaps the most obvious would be to use
functools.reduce</a>.</p>
from</span></span> functools</span></span> import</span></span></span></span></span> reduce</span></span>
from</span></span> operator</span></span> import</span></span></span></span></span> add</span></span>

reduce</span></span>(</span>add</span></span>,</span> range</span></span>(</span>20</span></span>)</span></span></span>)</span></span>
</span></code></pre>
But in this case you would need the entire iterable to be ready at the time of
calling reduce</code>. What if you want to call the reduce</code> one step at a time?
This would be particularly important, for example, if you were receiving the
items over a network connection or reading them from a huge file.</p>
With Generators</h2>
Another way might be to use a generator, but that's not without its problems either:</p>
from</span></span> operator</span></span> import</span></span></span></span></span> add</span></span>

def</span> genreduce</span></span></span>(</span></span>init_val</span>,</span> func</span>)</span></span>:</span></span>
    val</span></span> =</span> init_val</span></span>
    yield</span> init_val</span></span>
    while</span> True</span>:</span></span>
        nxt</span></span> =</span> yield</span> val</span></span>
        val</span></span> =</span> func</span></span>(</span>val</span></span>,</span> nxt</span></span></span>)</span></span>

def</span> online_reduce</span></span></span>(</span></span>func</span>,</span> init_val</span>,</span> gen</span>)</span></span>:</span></span>
    state</span></span> =</span> genreduce</span></span>(</span>init_val</span></span>,</span> func</span></span></span>)</span></span>
    init_val</span></span> =</span> next</span></span>(</span>state</span></span></span>)</span></span>
    for</span> item</span> in</span></span></span> gen</span></span>:</span></span>
        init_val</span></span> =</span> state</span>.</span></span>send</span></span>(</span>item</span></span></span>)</span></span>
    return</span> init_val</span></span>
</span></code></pre>
So now things have got pretty ponderous. And the two things we want to specify
here are quite cumbersome to carry around - the function defining the reduction
and its initial value.</p>
With Function Attributes</h2>
What I'm really after is a concise way to specify the callees</em> - the reduce
functions themselves. And an intuitive pythonic protocol for a caller</em> to use
them.</p>
Luckily, in Python, we can just add attributes to functions.</p>
def</span> foldsum</span></span></span>(</span></span>prev_val</span>,</span> val</span>)</span></span>:</span></span>
    return</span> prev_val</span></span> +</span> val</span></span>
foldsum</span>.</span>init_val</span></span> =</span> 0</span>

def</span> online_reduce</span></span></span>(</span></span>func</span>,</span> gen</span>)</span></span>:</span></span>
    val</span></span> =</span> func</span>.</span>init_val</span></span>
    for</span> item</span> in</span></span></span> gen</span></span>:</span></span>
        val</span></span> =</span> func</span></span>(</span>val</span></span>,</span> item</span></span></span>)</span></span>
    return</span> val</span></span>
</span></code></pre>
This is nice, it lets us combine the function and the initial-value without
resorting to defining some new class, or just using a tuple like (add, 0)</code>
which is essentially untyped and can get confusing.</p>
It does make type checking blow up though.</p>
With a Decorator</h2>
Let's try again but make this a function decorator. We want to be able to
decorate our own functions as well as things like operator.add</code>, so we'll need
an extra level of indirection so we don't start adding attributes onto members
of the operator</code> module!</p>
When it comes to typing, we want the caller to be able to access
func.init_val</code> without causing a type error. We can define a custom protocol
for that using the
typing_extensions</a>
module.</p>
But also, our actual decorator needs to not blow up mypy</code> and that gets a
little bit tricky. I couldn't figure out how to avoid using a
cast</a>.</p>
With all that said, here's all the tedious boilerplate for the decorator:</p>
from</span></span> typing</span></span> import</span></span></span></span></span> Callable</span>,</span>Any</span>,</span>cast</span></span>
from</span></span> typing_extensions</span></span> import</span></span></span></span></span> Protocol</span></span>
from</span></span> functools</span></span> import</span></span></span></span></span> wraps</span></span>

class</span> ReduceFunc</span></span>(</span></span></span>Protocol</span>)</span></span>:</span></span>
    init_val</span></span> :</span> Any</span></span>
    def</span> __call__</span></span></span>(</span></span>prev_val</span>,</span> val</span>)</span></span>:</span></span>
        raise</span> NotImplementedError</span></span></span>

def</span> reducer</span></span></span>(</span></span>parm</span> </span>:</span> Any</span></span></span>)</span></span>:</span></span>
    def</span> decorator</span></span></span>(</span></span>func</span>)</span></span> </span>-></span> ReduceFunc</span></span></span>:</span></span>
        @</span></span></span>wraps</span></span>(</span>func</span></span></span>)</span></span>
        def</span> wrapper</span></span></span>(</span></span>a</span>,</span> b</span>)</span></span>:</span></span>
            return</span> func</span></span>(</span>a</span></span>,</span> b</span></span></span>)</span></span>
        rf</span></span> =</span> cast</span></span>(</span>ReduceFunc</span></span>,</span> wrapper</span></span></span>)</span></span>
        rf</span>.</span>init_val</span></span> =</span> parm</span></span>
        return</span> rf</span></span>
    return</span> decorator</span></span>

__all__</span></span> =</span> (</span>
    '</span></span></span>reducer'</span></span></span>
)</span></span>
</span></code></pre>
And here's two ways of defining a reduce</code> function. First for a user-defined
function, secondly when wrapping a function from somewhere else like the
operator</code> module.</p>
@</span></span></span>reducer</span></span>(</span>0</span></span>)</span></span>
def</span> foldsum</span></span></span>(</span></span>prev_val</span>,</span> val</span>)</span></span>:</span></span>
    return</span> prev_val</span></span> +</span> val</span></span>
</span></code></pre>
from</span></span> operator</span></span> import</span></span></span></span></span> add</span></span>
foldsum</span></span> =</span> reducer</span></span>(</span>0</span></span>)</span></span>(</span></span>add</span></span></span>)</span></span>
</span></code></pre>


Getting Bitmasks from SSE Vector Comparisons
2019-03-08T18:49:47+09:00
These days the single-threaded performance of CPUs just isn't advancing as
quickly as it did in my salad days of the late 90s and early 2000s. Personally
I think it's God's revenge for people putting pineapple on pizza. Regardless
of the causes, I find it to be increasingly the case that if you aren't using
SIMD in the performance critical areas of your software, then you are leaving a
lot of performance on the floor.</p>
Let's say that you have an array of uint16_t</code>'s and you want to go through
them all looking for all instances of a given number. In plain-old-C you would
do something like:</p>
void</span> my_pipeline_is_as_empty_as_my_soul</span></span>(</span></span></span>const</span> unsigned</span> count</span>,</span>
					const</span> uint16_t</span> vector</span>[</span>static</span> count]</span></span>)</span></span></span>
</span>{</span></span></span>
	unsigned</span> int</span> i;</span>
	for</span>(</span>i =</span> 0</span>;</span> i <</span> count;</span> i+</span>+</span>)</span></span> {</span>
		if</span> (</span> vector[</span>i]</span></span> ==</span> 0x1234</span> )</span></span> {</span>
			/*</span> do the thing */</span></span>
		}</span></span>
	}</span></span>
</span></span>}</span></span></span>
</span></code></pre>
But you've been chilling listening to "smooth" by Carlos Santana, and you've
heard of this hip new technology called SSE2. It allows you to do EIGHT of
these comparisons in a single bound!</p>
So you include the relevant header and you start working on an inner-loop which
handles 8 items at a time. We'll worry about any trailing data later. You'll
use _mm_load_si128</code> to just slurp an array of 8 elements into one of these
new wide-boy registers:</p>
#include</span> <</span>emmintrin.h></span></span>
</span>_Alignas</span>(</span></span></span>16</span></span>)</span></span></span> uint16_t</span> in16[</span>8</span>]</span></span> =</span> {</span>
	/*</span> 0 */</span></span> 0x1234</span>,</span>
	/*</span> 1 */</span></span> 0x4567</span>,</span>
	/*</span> 2 */</span></span> 0x1234</span>,</span>
	/*</span> 3 */</span></span> 0x1234</span>,</span>
	/*</span> 4 */</span></span> 0x1234</span>,</span>
	/*</span> 5 */</span></span> 0</span>,</span>
	/*</span> 6 */</span></span> 0x1212</span>,</span>
	/*</span> 7 */</span></span> 0x3434</span>,</span>

}</span></span>;</span>
__m128i h =</span> _mm_load_si128</span>(</span></span></span>(</span>__m128i *</span>)</span></span>in16</span></span>)</span></span></span>;</span>
//</span> 1234 4567 1234 1234 1234 0000 1212 3434
</span></span></code></pre>
And, as before, we are looking for all instances of 0x1234</code> so we do a
comparison using the _mm_cmpeq_epi16</code> intrinsic:</p>
__m128i n =</span> _mm_set1_epi16</span>(</span></span></span>0x1234</span></span></span>)</span></span></span>;</span>
//</span> 1234 1234 1234 1234 1234 1234 1234 1234
</span>__m128i r =</span> _mm_cmpeq_epi16</span>(</span></span></span>n,</span> h</span></span>)</span></span></span>;</span>
//</span> ffff 0000 ffff ffff ffff 0000 0000 0000
</span></span></code></pre>
Astute readers will notice that these equality tests can be achieved with
and</code>-ing or xor</code>-ing, and that's correct. But the topic of this post is the
comparison operators. These become really useful when you want to do range
comparisons using greater-than and/or less-than or things like that.</p>
Anyway, this is all wonderful but what am I going to do with this weird
two-byte boolean-like thing?</p>
I now have a vector of uint16_t</code>'s which are set to all 1s for matching items
and all 0s for non-matching items. We could just break these out in a loop but
that would waste all the effort we've put in to avoid branching and looping by
doing this with SIMD in the first place.</p>
It would be nice if we could get the results in a bitmask, so we can do other
set-wise operations on all 8 elements at once. Or use __builtin_ctz</code>
and __builtin_clz</code> respectively to find the first/last set bits.</p>
The solution</h2>
How we're going to do this is by using
_mm_packs_epi16</a>
to collapse the 16bit values down to 8bit values and then
_mm_movemask_epi8</a>
to extract the mask.</p>
__m128i p =</span> _mm_packs_epi16</span>(</span></span></span>r,</span> r</span></span>)</span></span></span>;</span>
//</span> ff 00 ff ff ff 00 00 00 (x2)
</span>int</span> mask =</span> _mm_movemask_epi8</span>(</span></span></span>p</span></span>)</span></span></span> &</span> 0xff</span>;</span>
//</span> 0x1d (or 00011101)
</span></span></code></pre>
Now, this might require a bit of unpacking - if you will excuse the pun. At
first glance _mm_packs_epi16</code> doesn't seem an obvious choice. Intel describes
it as:</p>

Convert packed 16-bit integers from a</code> and b</code> to packed 8-bit integers
using signed saturation, and store the results in dst</code>.</p>
</blockquote>
To put this in plain english, what happens is that each 16 bit integer in our
input vectors is treated as signed and squeezed down into a signed 8-bit
integer. But, instead of just chopping off the top bits, anything that doesn't
fit gets 'stuck' at the nearest end of the 8-bit range, 0x7f</code> (127) or 0x80</code>
(-128) (ie. saturation</a>).
The eight results from a</code> come first, followed by the eight from b</code>. Our
comparison results are either 0xffff</code>, which is -1 and fits as 0xff</code>, or
0</code>, so nothing ever actually saturates. A C implementation would look
something like this:</p>
int16_t</span> in[</span>16</span>]</span></span>;</span> //</span> this is the concatenation of the two 128bit operands, a then b
</span>int8_t</span> out[</span>16</span>]</span></span>;</span> //</span> output operand
</span>
for</span>(</span>i =</span> 0</span>;</span> i <</span> 16</span>;</span> i+</span>+</span>)</span></span> {</span>
	int</span> res;</span>

	/*</span> Load the input */</span></span>
	res =</span> in[</span>i]</span></span>;</span>

	/*</span> Signed saturation */</span></span>
	if</span> (</span> res ></span> 127</span> )</span></span> {</span>
		res =</span> 127</span>;</span>
	}</span></span> else</span> if</span> (</span> res <</span> -</span>128</span> )</span></span> {</span>
		res =</span> -</span>128</span>;</span>
	}</span></span>

	/*</span> Store the result */</span></span>
	out[</span>i]</span></span> =</span> res;</span>
}</span></span>
</span></code></pre>
To avoid adding a dependency on any other registers we use the same input
register for both operands. This leaves us with two identical copies of the
desired output in the upper and lower 8 lanes of the output register
respectively. That's why after converting to a bitmask with
_mm_movemask_epi8</code>, we mask out the upper 8 bits to ensure that they're always
zero.</p>
This is, of course, a little wasteful. If we're looping through a huge array
we would just do two lots of comparisons so we can fill two registers full of
results and then do a single _mm_packs_epi16</code> to generate a 16 bit mask.</p>
What about with 256 and 512 bit vectors?</h2>
The 256 bit version looks much the same. It produces a 16-bit bitmask at the
end and uses immintrin.h</code> and the corresponding AVX2 versions of each
operation. The one gotcha is that the AVX2 pack works on each 128-bit half of
the register separately, so the two halves of our result need stitching back
together:</p>
#include</span> <</span>immintrin.h></span></span>
</span>__m256i h =</span> _mm256_load_si256</span>(</span></span></span>(</span>__m256i *</span>)</span></span>ptr</span></span>)</span></span></span>;</span>
__m256i n =</span> _mm256_set1_epi16</span>(</span></span></span>0x1234</span></span></span>)</span></span></span>;</span>
__m256i r =</span> _mm256_cmpeq_epi16</span>(</span></span></span>n,</span> h</span></span>)</span></span></span>;</span>
__m256i p =</span> _mm256_packs_epi16</span>(</span></span></span>r,</span> r</span></span>)</span></span></span>;</span>
//</span> lanes 0-7 land in bits 0-7 (and 8-15), lanes 8-15 in bits 16-23 (and 24-31)
</span>unsigned</span> int</span> m =</span> _mm256_movemask_epi8</span>(</span></span></span>p</span></span>)</span></span></span>;</span>
int</span> mask =</span> (</span>m &</span> 0xff</span>)</span></span> |</span> (</span>(</span>m >></span> 8</span>)</span></span> &</span> 0xff00</span>)</span></span>;</span>
</span></code></pre>
If you have BMI2, _pext_u32(_mm256_movemask_epi8(r), 0x55555555)</code> gets the
same mask without the pack at all.</p>
For AVX-512 (specifically AVX-512BW, which is the part that handles 16-bit
elements) you needn't go through the rigmarole because of the in-built
support for mask registers.</p>
__m512i h =</span> _mm512_load_si512</span>(</span></span></span>(</span>const</span> __m512i *</span>)</span></span>ptr</span></span>)</span></span></span>;</span>
__m512i n =</span> _mm512_set1_epi16</span>(</span></span></span>0x1234</span></span></span>)</span></span></span>;</span>
__mmask32 mask  =</span> _mm512_cmpeq_epi16_mask</span>(</span></span></span>n,</span> h</span></span>)</span></span></span>;</span>
</span></code></pre>
Easy! And it's nice and fast. Looks like you can run comparisons on an entire
cache line, 32 uint16_t</code>'s at a time, with 3 or 4 cycles of latency (4 on
Skylake-X, 3 on Ice Lake and later) and a throughput of one per cycle. Pretty
neat.</p>
Corrections (October 2026)</h2>
The 2019 version of this post got a few things wrong:</p>

It described _mm_packs_epi16</code> as splitting each 16-bit value into two bytes
and adding them with saturation. It actually narrows each signed 16-bit
value to a signed 8-bit one, saturating at -128 and 127. The code worked
anyway, because -1 and 0 come out the same under either reading.</li>
The 128-bit mask was & 0x7f</code>, which throws away the result for lane 7. It
should be & 0xff</code>. The example only worked because lane 7 didn't match.</li>
The 256-bit instructions were described as SSE3. They're AVX2.</li>
The 256-bit code was broken: _mm256_packs_epi16</code> packs within each 128-bit
half, so & 0xffff</code> returned lanes 0-7 twice and lost lanes 8-15 entirely.</li>
The AVX-512 compare was claimed to have 1 cycle of latency and run two per
cycle. uops.info</a>
measures 4 cycles on Skylake-X and 3 on Ice Lake, at one per cycle (it only
runs on port 5).</li>
Smaller fixes to the code: in16</code> wasn't aligned for _mm_load_si128</code>, which
faults on unaligned addresses; a semicolon was missing; the comparison result
was called r</code> in one snippet and cmp_res</code> in the next; and
__builtin_ctzl</code> is now __builtin_ctz</code> to match the int</code> mask.</li>
</ul>


From Hash File to JIT: Building an Experimental OLAP Engine
2018-04-09T12:00:00Z
Despite sharing the same name as skunk</a>, a small cdb-inspired hash
file, this article is actually about my journey writing a full database system
intended for workloads where query speed is paramount and data is written to
infrequently in an append-only fashion. The system is currently experimental,
and not available, but here are some ruminations on the design considerations
and some of the salient technical details of how it got there.</p>
Table format (rec)</h2>
There are many options for how to store records. If records have primary keys
then, often, records are embedded into a b-tree. Other options are to use a
paged-format and then allocate space heap-style from there. Fancy items such as a
schema version, or a null bitmap might also be included.</p>
For our considerations, we don't care too much about primary-keyed tables, or
can implement them with a different format later. Also we don't support nulled
values or updates at all.</p>
For this reason we chose to round up record size to a power of two. And then
use a power-of-two of records inside each buffer, aiming for a page size of
1MiB. The trailing page is allowed to be shorter. This allows us to locate all
records with shifts and masks and means we write to the storage device in 1MiB
chunks which should be good for both spinning rust and SSD alike.</p>
Not much to say about this, it's the simplest imaginable approach. As we look
to support star-schemas and other primary-keyed tables later we will look into
a different table format for that.</p>
Originally we did support updates and deletes. Deletes were based on bitmaps
which were placed at the trailer of each page. But doing this caused locating a
record to include an idiv</code> (integer division) instruction that couldn't be
optimised into a multiply at compile-time. This led to a 10x slowdown in
locating records. In retrospect a better approach would have been a separate
roaring bitmap - which allows random access - that would cover the entire
table. Anyway, for my intended use-case, deletes are not important and updates
would be a staggering rarity.</p>
Insert Performance</h2>
A real-world dataset is used. It's 1 month of netflow records from a busy
network.</p>
Import 151,137,635 64-byte records (from binary) into a table. Final table
size is 9226 MiB. Note that space-overhead here is basically zero, this is
partly luck that our record size is a power-of-two already. Overhead could be
close to double in the worst-case.</p>
At the same time as importing we index src/dst ip's into a single index, and
protocol-id into another index. This results in 609 MiB (656,047 keys) and 77
MiB (39 keys) indices respectively. Note that the biggest index is only a 6.5%
overhead which is much better than with EWAH compressed bitmaps which almost
double the size. A postings-list or b-tree would be more inefficient still.</p>
Overall, the process takes 28.7s, or around 5.27M rows per second insert rate
with indexing. Without indexing, we can do this in 7.31s, which is about 20M
rows per second. There's still some room for optimisations here but this is a
nice start. Indexing performance seems pretty bad, but it makes sense because
we're not using write-optimised data structures since we care more about query
performance. Still, it might be a nice thing to add in future as a trade-off
that the user can make - much faster inserts in return for slightly slower
queries.</p>
Index Format (br64)</h2>
A big part of our intended use-case is using indices to speed up searches.
Eventually there will be specialised indices for various data types such as
text or multi-dimensional data. But for now we focus on integers of up to 64
bits in width.</p>
Since our indices are constructed in a single-pass during table construction we
have no reason to avoid b-trees whose worst-case performance is only
encountered when performing random inserts. If a b-tree is "bulk-loaded" then
we can obtain optimal packing and layout.</p>
We chose a fanout of 512. This works out to a page size of 4KiB which should be
good for mmapped access on both SSD and spinning rust - a random read anywhere
within the memory mapped index file should bring in exactly one node. Keys and
values are both 64bit integers. We lay out nodes in breadth-first-search (BFS)
order.</p>
For internal pages, the values are page-numbers pointing to a child page and
keys are splitting keys. The unused splitting key at the start of the page is
used to record the number of values in that page.</p>
For leaf nodes, the values are implementation-defined. The only implementation
right now is to point to a roaring-bitmap of records in the table which match
the associated key. Since there is no space in the pages to record node-type or
number of nodes, we store this information in the file-header. Specifically,
the number of keys, and the page number of the first leaf node is stored. From
this we can determine if a page is a leaf node by its page number (pgno ≥
first leaf) and how many keys are stored in the possibly not-full final leaf
(nr_keys</code> modulo fanout). Note that these header values also permit very
efficient index-scans, since our leaves are packed contiguously and can be
located instantly. A useful optimisation for queries ordered by the search key.</p>
Bitmaps are stored in something similar to roaring-bitmap style. This seems to
provide much better compression ratios than run-length encoded (RLE) bitmaps
while maintaining all the good properties (set operations can be performed
directly on compressed representation). But it also allows for fast random access</p>

unlike RLE.</li>
</ul>
These bitmaps work by dividing up the value-space and choosing the smallest
representation for each part of that space, based on its density. First, the
space of record-numbers is divided up into containers which cover 2^16
contiguous records, starting from the zeroth. For each container, we maintain a
count of the number of items which are present. For sparsely populated
containers we use a sorted array of 16-bit unsigned integers, for semi-dense
containers we use a bitmap, and for dense containers we use a sorted array of
16-bit integers mentioning all unset items (ie. an inverted-list
representation). The boundaries between sparse, semi-dense and dense are chosen
at the crossover points where each representation becomes the most
space-efficient.</p>
Unpacking and intersecting of the bitmaps can be performed pretty efficiently
with just a little bit of clever bit twiddling in C. Further, several of the
important cases, such as unpacking the bitmaps or intersecting bitmaps, can be
optimized with AVX-512. List containers aren't as easily optimized but the
silver lining there is that the list container is only used in very sparse (or
very dense) regions which are, by definition, the easiest cases.</p>
For 8 and 16 bit keys we can dispense with the b-tree and use a direct
table-lookup, avoiding the need to binary search. For larger composite keys we
will probably use a multi-level version of these schemes where each component
of a composite key is indexed in a separate b-tree. This should assist with
left-anchored queries and index-scans.</p>
Query Planner</h2>
Right now the query planner is unremarkable and uses heuristics to pick indices
to use and a join order that looks useful for hash joins. In future we will be
implementing a cost-based query planner.</p>
The write-once query-many architecture makes materialisation cheap and
efficient for us, so we may experiment with tweaking the query planner to make
more extensive use of materialisation, which is usually avoided in more
conventional RDBMS's. One anticipated problem with this is that without limiting
query plan searches only to left-deep trees then we increase our search space
which will increase the cost of planning. But for prepared-statements or
stored-procedures where the plan is generated only once, it might pay off,
especially if we introduce a parametrised planner.</p>
Another benefit of this architecture is that we can produce catalogs with
accurate selectivity information which will be useful when picking indices and
when planning equi-joins.</p>
Query Evaluator</h2>
Currently, the query evaluator follows the volcano model. It's a tree of nodes
which implement a generator function, iterating over the output of a previous
node. In reality the tree is more of a linked list since the planner is using
left-deep trees.</p>
Evaluation uses the "pull" method. Iteration over the results is driven from
the outside-in. That is to say that the front-end interface contains rewind</code>,
get-next-row</code> and is_eof?</code> methods. This avoids the need for consumers to
provide a callback function to receive the results of the queries.</p>
However, this is a pretty inefficient design since everything works one row at
a time. This means we need to spill our most performance critical loop state
into memory rather than giving it a chance to stay in registers. Consequently,
the overhead of the generators/iterators model starts to dominate the cost of
query evaluation.</p>
The inherent slowness of the volcano model has been worked around so far by
making use of batching, especially in tablescan, and indexseek node types. More
importantly the system is being moved towards more of a compiler-oriented
design where multiple stages may be inlined together into a well optimized
piece of machine-code.</p>
Bytecode Virtual Machine</h2>
Where possible, selection expressions are compiled into a bytecode for
efficient evaluation. Right now, only basic expressions are supported.  That
means column-to-column comparisons and column-to-literal comparisons joined by
conjunctions (AND) or disjunctions (OR). For more complex queries involving
arithmetic or variable assignments we fall back to other, more functional,
methods.</p>
The bytecode is designed to reduce the number of instructions as much as
possible. In my previous experiences with bytecode vm's, RISC-like approaches
based on separate loading, storing and control-flow instructions increase
overhead by making it necessary to perform more instruction dispatches. They
also require more state, such as registers and condition flags.</p>
In our system, the machine state is simply the instruction pointer, a halted
flag, and a matched flag. The program is a linear array of instructions,
contiguously following the machine state to reduce pointer chasing and boil the
main-loop down to a bunch of fast lea</code> and call</code> instructions.</p>
There are four instruction types:</p>

Halt without match</li>
Halt with match</li>
Compare a column to a column</li>
Compare a column to a literal</li>
</ol>
Each instruction contains a pointer to a function which implements the
instruction. A different function implements each of the instruction types. The
operand space in the instruction contains the information necessary to retrieve
the field (an index to find the relation, and an offset and length to locate
the column within the relation). And that is either doubled up for
field-to-field compares, or a literal is added in place of the second field's
information. Finally two addresses are included in each instruction, the first
is followed if the comparison evaluates false, and the second is followed if
the comparison evaluates true.</p>
The instruction at address zero halts without a match. The instruction at
address one halts with a match. This eliminates unnecessary condition checking
and branching in the inner loop. The entry point is always at address two.</p>
The compiler itself is only 317 lines of code, half of which is checking an
abstract-syntax tree representing the selection in order to determine if it
is representable in the bytecode. The main dispatch loop of the interpreter is 7
lines of code.</p>
On the dataset mentioned below, a series of un-indexed equality conditions over
4 columns using full table scan over ~151M records was reduced from 9.7s to
2.9s of run-time. That translates to an improvement in evaluation rate from 15M
rows per second to 50M rows per second.</p>
Contrary to popular belief, the cause of the improved performance is not
reduced branch-misses (which actually increased ten-fold) or cache misses
(which stayed roughly equal). But simply a drastic reduction in instruction
path-length (~99Bn down to ~27Bn), with a slight reduction in instructions
per-clock. About 20% of the removed instructions were branches, however.</p>
The main benefit of the machine is that it's pretty efficient - close to the
performance of native code in fact. But the downside is that only simple
operations are supported. Support for a much wider variety of selection
expressions could probably be achieved by adding new instructions and an
accumulator - maybe even a stack. But we would want to be careful not to slow
down the usual, simple, kinds of selections - the common case we've optimized
for - by adding too much complexity.</p>
JIT Compilation of Query Evaluators</h2>
In order to support more complex queries than the bytecode interpreter,
libgccjit</code> is used to compile selection expressions into native machine code.
This approach produces faster code at the expense of taking a bit more time and
resources during the preparation of the query. Compiling and optimizing is hard
work, after all. Certainly more so than the simple methods used for the bytecode
interpreter.</p>
A well known limitation of JIT compilation in query evaluation is that most of
the benefits to be had come from combining and inlining multiple stages of the
query, which can often be difficult.</p>
The architecture allows us to combine full table scans and selections into a
single JITed program. This is because it's easy to iterate over all records
with a simple for-loop without having to bring huge amounts of code to walk
through a complicated b-tree or record structure.</p>
Using this technique, the query used to test the bytecode evaluator above can
be executed in around 1.5s, which gives us an evaluation rate of nearly 105M
rows per second.</p>
One shortcoming right now is that no special-case code is generated to catch
exception conditions like undefined shifts or divides by zero.  Typically the
most difficult part about JITing evaluators is that as more internal functions
are inlined, more and more errors and exception cases need to be handled within
the JITd code which can greatly increase the complexity of this solution. After
all, if writing some code itself is difficult, then writing code which
generates an AST equivalent to that code is always going to be even more
difficult.</p>
Atomicity and Durability</h2>
Our append-only design makes atomicity and durability relatively easy to
achieve. There is code for WAL operations that we might resurrect in future.
Given our use cases, it's more likely that we will have a data-ingestion API
that provides special control over checkpointing and recovery.</p>
Final Thoughts and Future Directions</h2>
Overall, for a few weeks of part-time work, I'm quite happy with the results.
Certainly there are some interesting future directions still, especially in
query planning and evaluation.</p>
The most immediate task is rounding out the query language and implementing
joins, aggregates, and other expected features of an RDBMS. After that the
system will need a real cost-based query planner to make the best use of the
query evaluator.</p>
There are plans for a few exotic features to specifically optimize time-series
tables. Those are tables where timestamped rows are appended, either
periodically, or from a real-time event-stream. Also I'd be quite interested in
experimenting with column-store record organisation if I can ever find the free
time.</p>


Qemu, curses and caca
2010-06-21T14:00:00Z
I have recently discovered, and been very impressed with, the qemu ncurses
display driver. Basically it lets you run a guest OS inside of a regular
terminal emulator. Currently the curses display works almost perfectly when the
BIOS and guest OS have the VGA card programmed in text mode but when switched to
graphics mode all you get is a message telling you “Graphic mode 800 x 600″ for
example.</p>
Well, realising that there are the excellent aalib and libcaca ASCII-art text
rendering engines out there I began thinking about the possibilities.. Firstly
it is possible to set the (default) SDL display to use the libcaca video
driver. Then I realised libcaca has an ncurses back-end. So the following
command will let you run a full graphics mode OS inside your terminal emulator:</p>
$ env CACA_DRIVER=ncurses SDL_VIDEODRIVER=caca kvm ...
</span></code></pre>
The results look great but the only problem at the moment is mouse integration
in that button presses don’t get through. I think I have tracked this down to a
bug in the SDL libcaca wrapper so watch this space :)</p>
</p>

Gianni Tedesco

Arbitrary Remote Code Execution is (Still) Arbitrary Remote Code Execution

And If it's Arbitrary then You Have No Reason to Trust it.</h3>

Abusing the CPU's adder circuits

Item-at-a-time Reduce Functions in Python

With Function Attributes</h2> What I'm really after is a concise way to specify the callees</em> - the reduce functions themselves. And an intuitive pythonic protocol for a caller</em> to use them.</p> Luckily, in Python, we can just add attributes to functions.</p>

Getting Bitmasks from SSE Vector Comparisons

From Hash File to JIT: Building an Experimental OLAP Engine

Qemu, curses and caca