Maelstrom Crawler: A feedback-driven web vulnerability scanner - Improving web vulnerability scanning with dynamic strategies

Abstract

Web application vulnerability scanners rely on crawlers to explore applications and discover potential attack surfaces. The effectiveness of these scanners is therefore heavily influenced by the crawler’s ability to navigate diverse applications. Most existing crawlers use a fixed navigation strategy, despite different strategies often being more effective in different contexts. This thesis investigates whether a crawler that dynamically switches between navigation strategies based on feedback it receives at runtime can improve web application vulnerability scanning. To address this problem, we design and implement the Maelstrom Crawler, a feedback driven extension of the open-source Black Widow vulnerability scanner. Maelstrom dynamically alternates between randomised BFS and randomised DFS exploration using feedback channels that monitor the progress during execution and suggest when to switch strategy. An initial set of nine candidates for feedback channels was analysed using principal component analysis and an ablation study, resulting in five selected channels: growth, duplicate candidates, URL diversity, error rate, and novel code. These channels together guide the strategy-switching decisions through a majority voting based mechanism. To prevent excessive switching and thrashing, the design includes comparison and baseline windows together with a cooldown period. The final design was evaluated on three open-source applications: OsCommerce, WordPress, and Kanboard. Results show that Maelstrom Crawler achieved an average code coverage improvement of 9.63% compared to Black Widow, 91.68% compared to OWASP ZAP, and 70.74% compared to EvoCrawl. Internal experiments further demonstrated that feedback-driven strategy switching outperformed purely random switching. However, the improved code coverage did not consistently translate into increased vulnerability discovery, highlighting a gap between exploration effectiveness and vulnerability detection. The results indicate that feedback-driven strategy adaptation can improve crawler exploration efficiency and code coverage in web applications. Furthermore, they suggest that dynamically combining multiple navigation strategies is a promising direction for future web vulnerability scanners.