AI-Driven Threat Detection: Leveraging Big Data For Advanced Cybersecurity Compliance

Abstract

Every second of every hour, billions of Internet of Things-enabled devices are creating massive streams of data individually tailored to the intimate personal habits of their users. Simultaneously, sophisticated cybercriminal organizations, nation-state actors, and rapidly proliferating malware attacks ranging from hijacked personal tablets through Fortune 200 penetrated databases are impacting digital and thus physical assets across the entire political spectrum. This connectivity matrix is generating a massive and ever-expanding volume of network, system, and end-user security event data that combines with personal information from both the private sector and governments to fuel the artificial intelligence insights that we enjoy in our everyday lives. Yet, while the entire cybersecurity compliance lifecycle, including policy, network, system, enforcement, and incident response, generates and uses colossal data quantities, the proprietary, unstructured, and often classified nature of this data flow historically has limited our industry's adherence to AI-driven precepts.

In this paper, we introduce the principles of Threat Hooking, a Network Theory-driven approach to detecting and selectively blocking individual components within a collective logical threat. Our data science, Network Security Characterization Model detailed in this paper quantifies a specific element of Network Theory, which provides insight into both Network Health and individualized Threat Status. To demonstrate the innovation and theoretical underpinnings of Threat Hooking, we identify and analyze the massive datasets required from the network data immune system that we developed. After distilling relevant content from current cybersecurity research, we compiled an annotated dataset of live and emulated threat data and reported how AI-identified network artifacts that lead to human interpretable threat event detection can be verified, and if necessary, acted upon by cyber professionals.