Large-scale Security Analysis of HTTP Responses

Abstract

Ensuring the security of computer systems has never been more critical as our reliability on software is ever-increasing. It is not enough to create systems that work securely during their development. Instead, systems and software need continuous development and updates to stay secure.

Inspecting individual web applications for security vulnerabilities and concerns gives a lot of feedback for the specific web applications being checked. However, the impact of particular vulnerabilities can be more accurately observed by looking for vulnerabilities in a larger sample size of web applications.

In order to find vulnerabilities on a large scale, this thesis utilizes the crawl data from Common Crawl, an open repository of web crawl data. In this data, indicators of specific programs, software, or libraries are visible, allowing us to find vulnerable versions. Utilizing cloud computing on AWS and the crawl data, this thesis showcases how to scan 3 billion websites to find indicators of vulnerabilities on millions of domains. Using dynamic verification of the results, thousands of these indicators of vulnerabilities are then shown to be verifiable.