Security Hazards When Law is Code

As software continues to eat the world, there is an
increasing pressure to automate every aspect of society, from self-driving
cars, to algorithmic trading on the
stock market. As this pressure manifests into software
implementations of everything, there are obviously security concerns to be
addressed across many areas. But are there some domains and fields that are
distinctly susceptible to attacks, making them difficult to secure?

This thesis argues that one domain in particular—public policy and law—is
inherently difficult to automate securely using software-only implementations.
This is in large part because law and policy are written in a manner that
expects them to be flexibly interpreted to be fair or just.
Traditionally, this interpreting is done by judges and regulators who are capable
of understanding the intent of the laws they are enforcing. However, when these
laws are instead written in code, and interpreted by a machine, this capability
to understand goes away. Since they blindly follow written rules, computers can
be deterministically tricked to perform actions counter to their intended
behavior.

This thesis covers three case studies of law and policy being implemented in
code, and the security vulnerabilities they introduce in practice. The
first study is a security analysis of a previously deployed Internet voting
system, and shows how attackers could change the outcome of elections carried
out online. The second study looks at airport security, investigating how
full-body scanners can be defeated in practice, allowing attackers to conceal
contraband such as weapons or high explosives past airport checkpoints. Finally,
this thesis also studies how an Internet censorship system such as China's Great
Firewall can be circumvented by techniques that exploit the methods
employed by the censors themselves.

To address these concerns of securing software implementations of law, a hybrid
human-computer approach can be used. By combining the strengths of computers
speed and cost) and humans (ability to interpret and understand), systems can
be made more secure and more efficient than a method employing either alone.