SOAPA – A Matter of Scale

SOAPA – A Matter of Scale
Security Operations and Analytics Platform Architecture (SOAPA)


Intelligence-driven Security Operations Center (SOC)

The threat and risk environment has evolved rapidly in the past five years, with an increase in active threat actors and an escalation in the sophistication of their techniques – dictating an intelligence-driven Security Operations Center (SOC).

An intelligence-driven SOC goes beyond preventative technologies and the perimeter, and beyond events-based monitoring. It evolves and adapts because of the use of security intelligence that changes the scope and focus of security operations activities continuously. For rapid response, as much of the mundane work should be as automated as possible, and other human-augmented responses should be aided with decision support systems.

Cybersecurity Scale

In addition, the increasing growth of the Cloud, Internet of Things (IoT), Mobile and Digital Transformation are placing new demands on usability, scalability and enterprise-class features of cybersecurity analytics and operations products. Industry security experts believe that 2017 will be remembered as the year where cybersecurity analytics and operations encountered a wave of unprecedented scale.

Companies are seeking to refocus their SIEM investment migrating towards a modular, extensible, flexible, scalable architecture that leverages more of a holistic and hybrid Security as a Service (SaaS) also emerging as SIEM in the Cloud.

However the need for security scalability is nothing new. Leading SIEM vendors can all talk about how they re-architected their products over the past few years to scale from thousands to millions of events per second (EPS) and somehow make sense of all this activity. EPS growth will continue but cybersecurity scale is about to hit an exponential curve, driven by things such as:

  • Cloud utilization
  • Internet of Things (IoT)
  • Network growth
  • Digital transformation applications

These and other parallel trends are driving massive growth in the amount of data we need to collect, process, analyze, and store for cybersecurity analysis and operations.

Security Operations and Analytics Platform Architecture

Since it becomes impossible to centralize all security data, enterprise organizations will rely more and more on Security Operations and Analytics Platform Architecture (SOAPA) software to integrate distributed security data and analytics functions. In other words, some security analysis (i.e. threat intelligence research, EDR, malware analysis, etc.) will remain discrete, but SOAPA will act as an overall bridge for visibility across all the data for all analytics regardless of the data’s location.

Why SOAPA? Today many midsized and large enterprise customers have far too many disparate security point tools and simply can’t manage them effectively anymore. These organizations are consolidating to common platform architectures in two areas: information risk and protection and SOAPA.

SOAPA represents an opportunity to increase industry innovation and ultimately deliver a security architecture that allows organizations to increase productivity, accelerate actions streamlining day-to-day security operations.

In the past, most enterprises anchored their security analytics and operations with one common tool: Security Information and Event Management (SIEM) systems. But unlike the past, SIEM is one of several security tools within SOAPA, and these technologies must be designed for asynchronous cooperation so security analysts can quickly pivot across tools to find data and take action as they need to in real-time.

SOAPA is an architecture that sits “above and below the SIEM.” Things like probes and data collectors lie below the SIEM, while advanced analytics and security operations services like user behavior analytics (UBA) sit above and can help provide advanced SIEM functionality.

SOAPA Overview

Security Operations and Analytics Platform Architecture Overview

SOAPA is a dynamic, extensible, flexible, modular and, scalable architecture, meaning that new data sources and components will be added and integrated incrementally over time. It includes, for example:

  • Endpoint detection/response tools (EDR)
  • Incident response platforms (IRPs)
  • Network security analytics
  • UBA/machine learning algorithms
  • Vulnerability scanners and security asset managers
  • Anti-malware sandboxes
  • Threat intelligence

SOAPA drivers. Why are enterprise organizations moving toward SOAPA? Customers are stating that they cannot stay ahead in security operations using a collection of point tools when they are facing a dangerous threat landscape and a shortage of cybersecurity skills on their teams.

According to the recently published 2017 ESG IT spending intentions survey, 45 percent of organizations report a “problematic shortage” of cybersecurity skills.

SOAPA must supplement people with integration and intelligence. Customers don’t need more tools; they need their security technologies to add integration and intelligence so they can improve security efficacy, efficiency and productivity. Security analysts are pursuing numerous new investigations on a daily basis and simply can’t keep up with the volume.

Organizations are stating they cannot stay ahead in security operations using a collection of point tools when facing a dangerous threat landscape and a shortage of cybersecurity skills on their teams.

SOAPA Business Case

  • Cybersecurity analytics and operations are encountering a wave of unprecedented scale
    • Many midsized and large customers have far too many disparate security point tools
    • Cannot manage these disparate tools effectively
    • Seek consolidation/integration with a common platform
    • SOAPA is a scalabe architecture that sits “above and below the SIEM.”
  • Focuses on dynamic scalability, extensibility, flexibility, modularity, integration towards orchestration and automation
  • Scalability needed for growth across cloud, digital transformation applications, mobile, and so on
  • Drives massive growth in amount of data to collect, process, analyze, and store for cybersecurity analysis and operations
  • Modular components EDR, IRPs, Analytics, UEBA, Threat Intel, Vulnerability scanners, Asset managers, Anti-malware
  • Supplements some people with integration and intelligence, orchestration and, automation
  • Acts as a bridge for visibility across the data for all analytics regardless of the data’s location

Hybrid Security Operations Platform

Today’s enterprises can generate millions of security events every day and these events must be collected and analyzed around-the-clock to detect actual or pending attacks. Conventionally, organizations have staffed Security Operations Centers (SOCs) and deployed SIEM technology as the corner-stone of their security event monitoring programs. However, today many forward thinking enterprises are adopting hybrid plug-and-play models where some or all of these functions are outsourced to service providers.

Situational Awareness is critical, i.e., the ability to detect with precision and quickly respond to real threats. The ability to Detect with Precision (minimal false alerts) requires security operations process maturity and skilled resources.

SIEM in the Cloud … “or almost SaaS SIEM

  • Cloudiness refers to rapid provisioning, scaling, on-demand, multi-tenancy, added value add-on services, and so on
  • SIEMness refers to near-real time correlation, search, compliance, reports, security content, workflows, case management, and so on
  • Vendors are making a push for SIEM in the Cloud, e.g.,
    • IBM, FireEye, Splunk, AlertLogic, …
      • IBM QRadar on Cloud offers a complementary cloud-based service of professionals and managed infrastructure, while customer performs threat management tasks
      • Splunk Cloud (with Splunk Enterprise Security) offers cloud-based service of dashboards, reports, workflows, analytics, correlation searches, security indicators and, reports

Why Outsource Security Event Monitoring?

  1. Challenges in Hiring and Retaining Security Experts
  2. Threat Visibility
  3. 24×7 Vigilance
  4. Lack of SIEM Content
  5. More Effective SOC Analyst Investigations
  6. Rapid Response

Cyberattacks are constantly morphing as hackers exploit new vulnerabilities and create new variations of malware. Service providers are often the first to see new attack vectors and techniques as their customer base encompasses organizations in many different industries and locations. Compared to individual enterprises, users of a managed security service may also benefit from more sources of third-party threat intelligence feeds and advanced correlation analysis between threat intelligence data and other suspicious behavior. Overall, improved threat visibility increases the chance of detecting and preventing a cyber breach.

No SIEM can provide 100% accurate alerts. Security experts are needed to investigate suspicious alerts to determine the criticality of a threat. In a high performance SOC with a well-tuned SIEM, customers can expect the following:

  • Half of all high priority actionable alerts are the result of SOC analyst investigations
  • Of all the system alerts requiring analyst action, after investigation, about half turn out to be false positives

These data points underscore the importance of having sufficient human security experts available 24×7.

Service providers augment the existing team of SOC analysts and can often more effectively filter and correlate security events to present SOC analysts with better data. Outsourcing monitoring tasks also improves the morale of existing employees and allows them to focus on other priorities.

Organizations seek SIEM SaaS solutions to assist with staffing security teams on a 24/7 basis, they do not have a dedicated Security Operations Center (SOC) or the ability to staff three shifts of engineers year-round.

A hybrid security solution extends the customer’s security team by providing access to:

  • Security technical experts 24×7 monitoring
  • Security research team unit (Data Scientists and Analysts)
  • Thought leaders in the global security space

Overall a hybrid security solution prioritizes, organizes and offers comprehensive reporting on the customer environment, bringing situational awareness and manageability over the large volume of alerts and cases generated in the customer’s environment.

Analytics and Big Data = Scale and Modularity

  • Cybersecurity tactics and strategy are increasingly driven by data analytics
  • Enterprises are collecting, processing, analyzing and responding to more security data from a growing diversity of sources
  • Cybersecurity analytics and operations is in a state of innovative flux, organizations seek to refresh/revise SIEM investments
  • SIEM functionality extends to threat intelligence, analytics, network security analytics, EDR, UEBA, incident response automation and orchestration

ESG references/links

Thanks for your Interest!
Nige the Security Guy.


NG-OPS Advanced Defense – Part 1

NG-OPS Advanced Defense – Part 1: Identifying Defense Gaps & Improving Visibility

NG-OPS Strategy Guide
Advanced Defense Posture Assessment
NG-OPS Advanced Defense – Part 2

Today’s cyber security paradigm is a reactive cycle: when a threat is exposed, it is analyzed and a counter-solution is designed. The trouble is that attackers can easily reuse pieces of previous malware, modify them, and create a brand new threat, bypassing the newly updated security measures [1].

NG-OPS Advanced Defense

There are too many APT entry points in today’s virtual enterprise, too many individual endpoint security solutions triggering alerts, too much security threat intelligence to process in real-time. More importantly, there are too few trained personnel who can spot and respond to advanced threats. The pain points all cry out for a common holistic solution: NG-OPS Advanced Defense.

APT Prevalent Dangerous

The APT Conundrum and Challenges

Many organizations suffer from a lack of detection capability and precision, holistic situational awareness and behavioral anomaly detection, i.e., visibility. There is  too broad an attack surface, gaps in defense and, integration issues that together lead to reduction in the ability to detect, contain and respond to targeted attacks.

The typical challenges include, as follows:

  • Focus on prevention approach to address threat landscape
    • Fails to address increasing attack complexity and persistence with enough efficacy
  • Investments in protection model out of balance with today’s threat landscape
    • Technologies that don’t work together
  • Uncoordinated monitoring and compilation of security events & threats
    • Flood of unmanageable data = “Loss of visibility”
  • Organizations lack visibility into defense gaps, to enhance detection capability and precision
  • Organizations have not fully leveraged the kill chain life cycle approach
    • Reason why attackers are continuing to be so successful.
  • Common security architectures and compliance regimes are not prioritizing methods to address the kill chain

Reallocate Security Spend

Re-allocate Budget to Advanced Security Capabilities

The Changing Threat Environment

There is a growing need and urgency to evolve towards Advanced Security with a continually improving Detection, Containment and Response Capability. This is fundamentally due to cybercriminals doing their homework on organizations and waging a fierce, persistent campaign to find any possible way to get a foothold. Attackers have a fine tuned malware development process that is increasing in efficiency.

  • Evolving Malware Development Process
    • Create Malicious Tool (x 1)
    • Obfuscate Malware, Create Permutations (x 10,000)
    • Test against Detection Engines (OK)
    • Deploy Un-Detected Samples
  • Availability of Malware Tools
    • Results in high degree of Attack Automation
    • From systematic identification of targets to fully automated exploitation
  • Leads to increase in opportunistic attacks
    • Attacker no longer needs expertise or special skills

Malware Development

Malware Development Increases in Efficiency

Detection is the Weakest Link

Common intrusion detection methods are lacking in their ability to detect multi-step blended and targeted attacks.

Breach Detection Timespans

The Signature of an APT

A targeted attack aka advanced persistent threat (APT) is a targeted effort to obtain or change information by means that are difficult to discover, difficult to remove and difficult to attribute.

APT Attack Kill Chain 2

First – the bad guys get in. Always. It doesn’t matter if it’s social engineering, phishing, or some contractor organizations didn’t watch closely enough.  Sooner or later they find the weak spot and they exploit it – despite all of the best plans to keep them out. Target retail stores learned this the hard way. Who would have guessed that an HVAC system could be a point of weakness?

Case Study: The Target Attack Step-by-Step

In December 2013 – Target announced that it had been breached by attackers who had gotten away with 70M customers’ Personal Identifiable Information (PII) and 40M credit cards, financial damages currently stand at $148M, and are estimated to reach $1B. A high-level summary of the steps taken mapped to the kill chain are:

Target Kill Chain

  • Install malware to steal credentials from Target’s HVAC vendor.
  • Connect using stolen credentials, enables access to Target’s application dedicated to vendors.
  • Exploit a web application vulnerability on Target’s Web interface enables the attackers to execute code on Web application server.
  • Search relevant targets for propagation by LDAP querying Active Directory from the Web application’s server.
  • Steal access token from Domain Admin of the previously connected Domain Admin from the memory of application server.
  • Create new Domain Admin account using the stolen token in AD.
  • Propagate to computers using the new Domain Admin credentials
  • Steal 70M PII. Do not find credit cards, data is extracted using SQL
  • Steal 40M Credit Cards. The data is extracted by the Kaptoxa malware from the memory of the POS system.
  • Send stolen data to an FTP server in Target’s internal network.
  • Send stolen data via FTP to attackers-controlled FTP server.

Enabling Advanced Defense

Second – once they are in, organizations better figure out how to spot them. Developing, tuning, optimizing and evolving situational awareness and behavioral analysis allows network anomalies to be used to detect the different stages of APTs using various indicators.

  • Factors associated with APT attacks include the following:
    • Sudden increases in network traffic, outbound transfers
    • Unusual patterns of activity, such as large transfers of data outside normal office hours or to unusual locations
    • Repeated queries to dynamic DNS names
    • Unusual searches of directories and files of interest to an attacker, e.g., searches of source code repositories
    • Unrecognized, large outbound files that have been compressed, encrypted password-protected
    • Detection of communications to/from bogus IP addresses
    • External accesses that do not use local proxies or requests containing API calls
    • Unexplained changes in the configurations of platforms, routers or firewalls
    • Increased volume of IDS events/alerts

Attacker Defender View

Proactive Defensive Measures to Address Unknown Threats

Coming Soon

In NG-OPS Advanced Defense – Part 2  we will further develop the concept of developing and evolving an Advanced Defense security posture that identifies any gaps, improves detection capability and precision, enables proactive defensive measures to address unknown threats and — holistically integrates and operates continuous intelligence, detection and, response.

NG-OPS Ecosystem

In order to help organizations reduce operational overhead the NG-OPS Strategy Series will also include the following blog articles (although topics will be added as the theme develops and evolves):

  • NG-OPS SOC Version 2.0: Build a Next Generation Security Operations Center to protect critical assets
  • NG-OPS Evolving a SOC Team: Building, Nurturing and Evolving Security Operations Staff
  • NG-OPS Operational Maturity: Migrating from Reactive to Dynamic Defense and Cyber Resilience
  • NG-OPS Automation & Orchestration: The Next Big Thing: Automation and Orchestration
  • NG-OPS Threat Modeling: Manage complex systems using a structured methodical framework
  • NG-OPS Risk Management 2.0: Shifting focus from technical assets to critical business processes
  • NG-OPS Smart Telemetry: Leveraging 3 levels of telemetry: End-point, Gateway and, Infrastructure
  • NG-OPS Security DevOps: The critical convergence of security and IT operations with DevOps 

This new NG-OPS Strategy Guide builds upon and enhances the current APT Strategy Guide and Security Architecture Series Guide introducing a whole new set of topics into the framework.

Nige the Security Guy Bio


Nigel is a Chief Security Architect with 30 years of international experience in Security Operations, Management, Research, Development and Security Services. He started his career as an assembler programmer who was contracted by the US DoD to develop secure operating systems with multi-level security and preclude covert channels. Nigel is a passionate evangelist who loves working with organizations to share thought leadership and practical strategy to help defend against advanced targeted threats.


[1] Why cyber criminals are winning: The secret weapon of the black hats

[2] ISMG Advanced Persistent Threats Survey: New Strategies to Detect, Prevent, and Defend

Thanks for your interest!

Nige the Security Guy.

NG-OPS Strategy Guide

NG-OPS Strategy Guide: Navigating the Next Generation Security Operations Ecosystem

In today’s threat environment the only constant is change. In fact, everything is changing – the way our users work, the types of adversaries we face, and the techniques hackers use to infiltrate our networks. Such threats have become even more sophisticated than ever, bringing new risks and uncertainties that require more visibility in operations — thus a Next Generation Security Operations mindset.

NG-OPS Strategy Guide

This NG-OPS Strategy Guide introduces a new blog series on the Next Generation Security Operations Ecosystem to build upon and complement our prior blog series, they are as follows:

  • Security Architecture Series
  • Security Program Best-Practice Series
  • Security Assessment Series
  • APT Strategy Series

Disruptive Shifts and Converging Trends

The past few years have set the stage for some disruptive shifts in network security operations. These shifts are driven in part by the rise of BYOD, mobility, virtualization and the cloud, which have resulted in a new level of complexity and fragmentation with distributed systems.

Occurring in tandem, the proliferation of applications and infrastructure services inside the organization requires holistic organization into trust zones based upon risk and classification (see Adaptive Zone Defense) as well as greater policy orchestration, management and, visibility across access boundaries (inter-zone).

Next Generation Operations

The ability to translate complex business and organization goals into a set of automated data center workflows is critical to not slowing down the application delivery process. It is also an essential part of making compliance and security requirements a lot easier to manage in a very dynamic environment. Network security needs to transform into agile and adaptive end-to-end automated processes. This requires a systems approach when thinking about network security.

“The threat can be broken down into three components: intent, opportunity, and capability.
Organizations need to know, ‘What is the intent of adversaries? What are the opportunities available to them?
And what capabilities do they have to exploit the opportunities?”

Felix Mohan, Senior Vice President and
Chief Information Security Officer, Airtel

The delivery of an application can trigger a cascading series of actions to ensure that the application is delivered efficiently and in compliance with any regulatory requirements. Next-generation firewalls (NGFWs) now provide the ability to implement policies based on applications, users and content, and they can provide the appropriate hooks for automation and orchestration solutions.

These disruptive shifts and converging trends have fused application and network layer functions, causing a fundamental reset of the security operations function.

  • Organizations need to shift more security resources from preventing intrusion toward rapid detection and response
  • Improving detection and response requires an intelligence-driven context-aware security approach
  • Optimizing how security technologies, resources and process work together is pivotal to scaling security capabilities
  • Automation frees up analysts to focus more on higher priority risks affecting the most critical assets and data
  • SOCs need to build collaborative cross-disciplinary teams with highly specialized skill sets to combat advanced threats
  • Evolving security operations optimizes the interplay of people, processes and, technologies to enable rapid response
  • Orchestrated management of network infrastructure will be embraced as the next big thing
  • The rise of DevOps drives much needed convergence between security and IT operations to add security by design
  • Increases need to automate and optimize security operations to more effectively leverage resources/skills shortage

People Process Policy Technology

Reducing Operational Overhead

It’s a known fact that a lot of time is typically wasted on analyzing false positives generated by technology that is not correctly baselined, customized, tuned, optimized. Depending upon the environment, false positives can often be numerous and very difficult to verify, costing analysts valuable time determining whether or not something is an event the analyst should be worried about.

The tenets for this Next Generation Security Operations series are simple:

  • Increase visibility across the enterprise to identify active threats quickly
  • Understand the business impacts to better respond
  • Utilize resources to the fullest

“People in the SOC need ways to react faster and better — they need ways to improve the efficiency of what they do.
They need ways to reduce the amount of time between the onset of an attack and the time it’s stopped or remediated.”

Rich Mogull, founder of Securosis

NG-OPS Ecosystem

In order to help organizations reduce operational overhead the NG-OPS Strategy Series will currently include the following blog articles (although topics will be added as the theme develops and evolves):

  • NG-OPS SOC Version 2.0: Build a Next Generation Security Operations Center to protect critical assets
  • NG-OPS Evolving a SOC Team: Building, Nurturing and Evolving Security Operations Staff
  • NG-OPS Operational Maturity: Migrating from Reactive to Dynamic Defense and Cyber Resilience
  • NG-OPS Automation & Orchestration: The Next Big Thing: Automation and Orchestration
  • NG-OPS Threat Modeling: Manage complex systems using a structured methodical framework
  • NG-OPS Risk Management 2.0: Shifting focus from technical assets to critical business processes
  • NG-OPS Smart Telemetry: Leveraging 3 levels of telemetry: End-point, Gateway and, Infrastructure
  • NG-OPS Security DevOps: The critical convergence of security and IT operations with DevOps 

Please feel free to propose additional topics and/or vote for which topics should get published before the others.

This new NG-OPS Strategy Guide builds upon and enhances the current APT Strategy Guide introducing a whole new set of topics into the framework.

APT Strategy Maps

APT Strategy Guide Framework


The era of advanced threats calls for a new approach to information security. When dedicated cyber adversaries have the means and methods to elude commonly used defenses, such as signature based detection, it is clear that conventional approaches are no longer sufficient.

The need for a Next Generation Security Operations mindset is evident across the industry. Technologies will continue to improve but in parallel we do need to ensure that we also evolve and improve our security processes as well as invaluable resources and skills. Attackers are constantly evaluating their methods and improvising new techniques. Defenders must think in those same fluid terms to keep pace.

The value proposition for a Next Generation Security Operations program includes improved security, resource utilization and, cost-effectiveness. Together with increased visibility and vigilance defensive strategies can be precisely aimed at addressing the most significant threats and protecting the most critical assets and data. Leveraging automation and orchestration the security team will have the knowledge and the cycles it needs to make informed risk decisions and invest in the right security controls.


Orchestrating People and Process with Technology

Many enterprises are looking toward 3rd party security services to help them handle some elements of their defense. But that doesn’t mean the expertise of the SOC staff will become less important. In fact, most experts agree the next-generation security analyst will have to be smarter than ever. The security staff of the future is going to need expertise not only about the domain they’re defending, but also contextual expertise to determine what combinations of events might present a threat. On top of that, they’re going to need analytical expertise so that they can determine the source of the threat — and how to stop it

Thanks for your interest!

Nige the Security Guy.