Intrusion Detection Systems - Lecture #8: Correlation - Os3

Intrusion Detection Systems Matthijs Koot ([email protected]) Motivation

Intrusion Detection Systems Lecture #8: Correlation

What to correlate Information sources Signals, events and alerts Log and alert policy

How to correlate Purpose and general model Alert normalization

Matthijs Koot ([email protected])

Alert reduction (fusion, verification) Alert correlation

Summary

Faculteit van Natuurwetenschappen, Wiskunde en Informatica Universiteit van Amsterdam

2008-04-24 / SNE-IDS college ’07-’08

Outline

Intrusion Detection Systems Matthijs Koot ([email protected]) Motivation

Motivation

What to correlate Information sources Signals, events and alerts

What to correlate Information sources Signals, events and alerts Log and alert policy

Log and alert policy

How to correlate Purpose and general model Alert normalization Alert reduction (fusion, verification) Alert correlation

Summary

How to correlate Purpose and general model Alert normalization Alert reduction (fusion, verification) Alert correlation

Common problems with point-solution IDSs.

Intrusion Detection Systems Matthijs Koot ([email protected]) Motivation What to correlate

Common problems with point-solution IDSs: I Too many false positives I

I

Too many alerts for a single intrusion I

I

The rare true positive will be overlooked Causes brain damage to the (human) IDS operator

No high-level views I I I

What is the blackhat trying to do? Can we predict his next step? (probabilistically) Does the attack impact critical business systems?

Information sources Signals, events and alerts Log and alert policy

How to correlate Purpose and general model Alert normalization Alert reduction (fusion, verification) Alert correlation

Summary

Correlation.

Intrusion Detection Systems Matthijs Koot ([email protected])

Correlation provides (partial) solution:

Motivation What to correlate Information sources

Algorithms & External info

Signals, events and alerts Log and alert policy

How to correlate Purpose and general model Alert normalization Alert reduction (fusion, verification)

Correlation

Alert correlation

Summary

Intrusion alerts (uncorrelated) = Alert (unknown)

Intrusion alerts (correlated) = Alert cor. attack 1 = Alert cor. attack 2 = False positive

Outline

Intrusion Detection Systems Matthijs Koot ([email protected]) Motivation

Motivation

What to correlate Information sources Signals, events and alerts

What to correlate Information sources Signals, events and alerts Log and alert policy

Log and alert policy

How to correlate Purpose and general model Alert normalization Alert reduction (fusion, verification) Alert correlation

Summary

How to correlate Purpose and general model Alert normalization Alert reduction (fusion, verification) Alert correlation

Information sources.

Intrusion Detection Systems Matthijs Koot ([email protected])

Information sources: I IDSs (alerts) I I I

I

Logs (log entries) I I I

I

Host IDS (misuse/anomaly) Network IDS (misuse/anomaly) Application IDS (misuse/anomaly) Host: syslog, NT Eventlog Network: firewalls, routers, switches Application: Apache/IIS log, Oracle/MS-SQL log, SAP

Other (context knowledge) I I

I

Configuration Mgmt DB (CMDB) Network/service health monitors (Nagios, Solarwinds, etc) Vulnerability assessment systems

Motivation What to correlate Information sources Signals, events and alerts Log and alert policy

How to correlate Purpose and general model Alert normalization Alert reduction (fusion, verification) Alert correlation

Summary

Outline

Intrusion Detection Systems Matthijs Koot ([email protected]) Motivation

Motivation

What to correlate Information sources Signals, events and alerts

What to correlate Information sources Signals, events and alerts Log and alert policy

Log and alert policy

How to correlate Purpose and general model Alert normalization Alert reduction (fusion, verification) Alert correlation

Summary

How to correlate Purpose and general model Alert normalization Alert reduction (fusion, verification) Alert correlation

Signals, events and alerts.

Intrusion Detection Systems Matthijs Koot ([email protected])

Signals, events and alerts: Motivation What to correlate Information sources Signals, events and alerts

Signals

Log and alert policy

How to correlate

Events

Purpose and general model

Alerts Network traffic

Alert normalization Alert reduction (fusion, verification) Alert correlation

Summary

OS activity Application data Hardware sensors

Log policy and alert policy.

Intrusion Detection Systems Matthijs Koot ([email protected])

Log policy and alert policy:

Motivation What to correlate Information sources

Log policy

Alert policy

Signals, events and alerts Log and alert policy

How to correlate

Signals

Purpose and general model Alert normalization

Events

Alert reduction (fusion, verification)

Alerts

Alert correlation

Summary

Network traffic OS activity Application data Hardware sensors

Event correlation and alert correlation.

Intrusion Detection Systems Matthijs Koot ([email protected])

Event correlation and alert correlation:

Motivation What to correlate

Log policy

Alert policy

Information sources Signals, events and alerts Log and alert policy

How to correlate

Signals

Purpose and general model Alert normalization

Events Alerts

Alert reduction (fusion, verification) Alert correlation

Summary

Network traffic OS activity Application data Hardware sensors Alert correlation Event correlation

Definitions.

Intrusion Detection Systems Matthijs Koot ([email protected]) Motivation

Definition Intrusion event correlation refers to the interpretation, combination, and analysis of neutral events from all available sources, about target system activity for the purposes of intrusion detection and response.

What to correlate Information sources Signals, events and alerts Log and alert policy

How to correlate Purpose and general model Alert normalization Alert reduction (fusion, verification) Alert correlation

Definition Intrusion alert correlation refers to the interpretation, combination, and analysis of intrusion alerts, together with information external to the intrusion detection system, with the purpose of intrusion alert refinement and intrusion scenario building. Source: “Extending Intrusion Detection with Alert Correlation and Intrusion Tolerance", Dan Gorton, 2003

Summary

Example correlation table.

Intrusion Detection Systems Matthijs Koot ([email protected])

Today’s focus is alert correlation. But here is an example event correlation table:

Motivation What to correlate Information sources Signals, events and alerts Log and alert policy

How to correlate Purpose and general model Alert normalization Alert reduction (fusion, verification) Alert correlation

Summary

Source: “Log Correlation for Intrusion Detection - A Proof of Concept", Abad et al., 2003 (paper)

Outline

Intrusion Detection Systems Matthijs Koot ([email protected]) Motivation

Motivation

What to correlate Information sources Signals, events and alerts

What to correlate Information sources Signals, events and alerts Log and alert policy

Log and alert policy

How to correlate Purpose and general model Alert normalization Alert reduction (fusion, verification) Alert correlation

Summary

How to correlate Purpose and general model Alert normalization Alert reduction (fusion, verification) Alert correlation

Log policy.

Intrusion Detection Systems Matthijs Koot ([email protected])

Log policy: what should (not) be logged? I

Candidate-loggables (signals): I I I I

I

Application/data-level activity OS-level activity Network-level activity Hardware-level activity

What signals are (not) relevant to recognizing malicious activity? (non-trivial!) I I I

I I

Examine known attacks for (potential) log trails Deploy honeynet, get attacked, and learn Log everything, always (this might allow better recognition of 0-day attacks) Logging guidelines (NIST 800-92, NSA, vendors) Trade-off: logging detail vs. cost in performance/storage

Motivation What to correlate Information sources Signals, events and alerts Log and alert policy

How to correlate Purpose and general model Alert normalization Alert reduction (fusion, verification) Alert correlation

Summary

Alert policy. Alert policy: what events (don’t) yield an alert? I

Matthijs Koot ([email protected]) Motivation

Context I

Intrusion Detection Systems

Situation-specific understanding of threat and intrusion

What to correlate Information sources Signals, events and alerts Log and alert policy

I I I

I

I

CTO: attacks on technology CIO/CISO: attacks on information Compliancy officer: attack compromizing compliancy to law and regulations (e.g. attacks against audit trails (Sox), leakage of customer or patient data) CEO/CFO: attacks on high-value assets and business goals

What combination of events indicate an intrusion? I

I I

Examine known attack strategies for (potential) traces Deploy honeynet, get attacked, and learn Share/maintain expert rules in a common, public knowledgebase for security logs/events/alerts (CAPEC?)

How to correlate Purpose and general model Alert normalization Alert reduction (fusion, verification) Alert correlation

Summary

Outline

Intrusion Detection Systems Matthijs Koot ([email protected]) Motivation

Motivation

What to correlate Information sources Signals, events and alerts

What to correlate Information sources Signals, events and alerts Log and alert policy

Log and alert policy

How to correlate Purpose and general model Alert normalization Alert reduction (fusion, verification) Alert correlation

Summary

How to correlate Purpose and general model Alert normalization Alert reduction (fusion, verification) Alert correlation

Goals.

Intrusion Detection Systems Matthijs Koot ([email protected])

Goals of alert correlation: I Reduce the total number of alerts I I I I

I

Improve diagnostics I I I

I

Elimination Fusion Aggregation Synthesis Type of activity Relevance Verification

Track activity I I

Information leaked to attacker Information leaked from attacker

Motivation What to correlate Information sources Signals, events and alerts Log and alert policy

How to correlate Purpose and general model Alert normalization Alert reduction (fusion, verification) Alert correlation

Summary

Alert correlation process.

Intrusion Detection Systems Matthijs Koot ([email protected]) Motivation What to correlate Information sources Signals, events and alerts Log and alert policy

How to correlate Purpose and general model Alert normalization Alert reduction (fusion, verification) Alert correlation

Summary

Source: “Intrusion Detection and Correlation", Krügel, Valeur and Vigna, 2005 (book)

Alert correlation process.

Intrusion Detection Systems Matthijs Koot ([email protected]) Motivation What to correlate Information sources Signals, events and alerts Log and alert policy

How to correlate Purpose and general model Alert normalization Alert reduction (fusion, verification) Alert correlation

Summary

Source: “Intrusion Detection and Correlation", Krügel, Valeur and Vigna, 2005 (book)

Outline

Intrusion Detection Systems Matthijs Koot ([email protected]) Motivation

Motivation

What to correlate Information sources Signals, events and alerts

What to correlate Information sources Signals, events and alerts Log and alert policy

Log and alert policy

How to correlate Purpose and general model Alert normalization Alert reduction (fusion, verification) Alert correlation

Summary

How to correlate Purpose and general model Alert normalization Alert reduction (fusion, verification) Alert correlation

Alert normalization. Alert normalization: I I I

Intrusion Detection Systems Matthijs Koot ([email protected])

Normalize syntax and semantics

Motivation

Semantics: CVE, Bugtraq, intrusion alert ontology

What to correlate

Syntax: CIDF yielded IETF-IDWG, which yielded IDMEF/IDXP

IDMEF = Intrusion Detection Message Exchange Format

Information sources Signals, events and alerts Log and alert policy

How to correlate Purpose and general model Alert normalization Alert reduction (fusion, verification) Alert correlation

Summary

Alert normalization (2). Example IDMEF: brp-snort 1.2.3.4 9.8.7.5 ICMP PING NMAP

Adapted from: “A Comprehensive Approach to Intrusion Alert Correlation", Valeur et al., 2004 (paper)

Intrusion Detection Systems Matthijs Koot ([email protected]) Motivation What to correlate Information sources Signals, events and alerts Log and alert policy

How to correlate Purpose and general model Alert normalization Alert reduction (fusion, verification) Alert correlation

Summary

Alert normalization (3). IDXP is the transport model for IDMEF: I IDXP = Intrusion Detection Exchange Protocol I BEEP = Blocks Extensible Exchange Protocol (RFC 3080) I IDXP carries IDMEF messages and is implemented as a BEEP profile

Intrusion Detection Systems Matthijs Koot ([email protected]) Motivation What to correlate Information sources Signals, events and alerts Log and alert policy

How to correlate Purpose and general model Alert normalization Alert reduction (fusion, verification) Alert correlation

Summary

Outline

Intrusion Detection Systems Matthijs Koot ([email protected]) Motivation

Motivation

What to correlate Information sources Signals, events and alerts

What to correlate Information sources Signals, events and alerts Log and alert policy

Log and alert policy

How to correlate Purpose and general model Alert normalization Alert reduction (fusion, verification) Alert correlation

Summary

How to correlate Purpose and general model Alert normalization Alert reduction (fusion, verification) Alert correlation

Alert reduction (1): fusion.

Intrusion Detection Systems Matthijs Koot ([email protected]) Motivation

I

Alert reduction (1): fusion I

recognize and remove redundancy in alerts from different sensors

What to correlate Information sources Signals, events and alerts Log and alert policy

How to correlate Purpose and general model Alert normalization Alert reduction (fusion, verification) Alert correlation

Summary

Alert reduction (2): verification.

Intrusion Detection Systems Matthijs Koot ([email protected]) Motivation

I

Alert reduction (2): verification I

I

Passive I

I I

I

recognize and remove irrelevant and failed attacks Verify target’s (in)vulnerability in CMDB (e.g. ignore OS/2-Warp attacks on MINIX machines) Wait for post-intrusion activity Wait for post-intrusion INactivity (missing heartbeats?)

Active (perturbing) I I

Connect to target, check for rogue processes Connect to target, check (config) files against known-good hashes (e.g. Tripwire)

What to correlate Information sources Signals, events and alerts Log and alert policy

How to correlate Purpose and general model Alert normalization Alert reduction (fusion, verification) Alert correlation

Summary

Outline

Intrusion Detection Systems Matthijs Koot ([email protected]) Motivation

Motivation

What to correlate Information sources Signals, events and alerts

What to correlate Information sources Signals, events and alerts Log and alert policy

Log and alert policy

How to correlate Purpose and general model Alert normalization Alert reduction (fusion, verification) Alert correlation

Summary

How to correlate Purpose and general model Alert normalization Alert reduction (fusion, verification) Alert correlation

Intention recognition and alert clustering.

Intrusion Detection Systems Matthijs Koot ([email protected]) Motivation What to correlate Information sources

Two possible approaches to correlation: I Alert clustering I

I

I

“Don’t know what’s happening, but these alerts appear to be related." Rather like anomaly-detection (statistics and probabilism).

Intention recognition I I

“Aha! Alerts seem to match ." Rather like misuse-detection (predefined patterns).

Signals, events and alerts Log and alert policy

How to correlate Purpose and general model Alert normalization Alert reduction (fusion, verification) Alert correlation

Summary

Alert thread reconstruction.

Intrusion Detection Systems Matthijs Koot ([email protected]) Motivation

Alert thread reconstruction (alert clustering) I

I

Cluster alerts into threads based on spatial and temporal proximity Incoming alerts are added to their best-matching thread; one thread represents one attack (session) I

Which attributes should be compared? How? I I

I I

Exact match (a1 .srcIP = a2 .srcIP) Domain-specific non-exact (proximity, subnet)

What ‘weight’ is assigned to each attribute? Similarity matrices/expectations (require human knowledge, prone to human error).

What to correlate Information sources Signals, events and alerts Log and alert policy

How to correlate Purpose and general model Alert normalization Alert reduction (fusion, verification) Alert correlation

Summary

Alert thread reconstruction (2).

Intrusion Detection Systems Matthijs Koot ([email protected]) Motivation What to correlate

One view on correlativity between two alerts hi and hj :

Information sources Signals, events and alerts

Cor (hi , hj ) =

p X i,j=1

wij Cor (xi , yj )/

p X

Log and alert policy

wij

i,j=1

(1)

How to correlate Purpose and general model Alert normalization Alert reduction (fusion, verification) Alert correlation

I

Here, two alerts hi and hj have p attributes with values x1 ..xp resp. y1 ..yp . The weight wij is empirically fine-tuned.

I

Cor (xi , yj ) is evaluated using similarity matrices.

Source: “Automatic attack plan recognition from intrusion alerts", Li et al., 2007

Summary

Predefined attack scenarios.

Intrusion Detection Systems Matthijs Koot ([email protected])

Predefined attack scenarios (intention recognition):

Motivation What to correlate Information sources Signals, events and alerts Log and alert policy

How to correlate Purpose and general model Alert normalization Alert reduction (fusion, verification) Alert correlation

Summary

I

Specification of attack scenarios: I I I

Attack Scenario Language (Krügel) Chronicles formalism (Débar) LAMBA (Cuppens)

Prerequisite-consequence analysis.

Intrusion Detection Systems Matthijs Koot ([email protected]) Motivation

Prerequisite-consequence analysis (intention recognition):

What to correlate Information sources Signals, events and alerts Log and alert policy

I

Alert conditionality through hyper-alerts: (fact, prerequisite, consequence)

I

Fact specifies an alert (attributes)

I

Prerequisite specifies a necessary condition for an attack to be successful (predicate) Consequence specifies possible result (predicate)

I

I

I

If chronologics allow it, this may fulfill another prerequisite This yields a (may-)prepare-for relation

How to correlate Purpose and general model Alert normalization Alert reduction (fusion, verification) Alert correlation

Summary

Prerequisite-consequence analysis (2).

Intrusion Detection Systems Matthijs Koot ([email protected])

Example hyper-alert correlation graph:

Motivation What to correlate Information sources Signals, events and alerts Log and alert policy

How to correlate Purpose and general model Alert normalization Alert reduction (fusion, verification) Alert correlation

Summary

Source: “Constructing Attack Scenario’s through Correlation of Intrusion Alerts", Ning et al., 2002

Summary

Intrusion Detection Systems Matthijs Koot ([email protected]) Motivation What to correlate Information sources Signals, events and alerts

Discussed: I

Signals, events and alerts + policy

Log and alert policy

How to correlate Purpose and general model Alert normalization

I I

Reasons for doing event/alert correlation Normalization, reduction, correlation (ad 2005)

Not discussed (but should have): Bayes, Granger-Causality, EWMA control charts, visualization

Alert reduction (fusion, verification) Alert correlation

Summary

Feedback!

Intrusion Detection Systems Matthijs Koot ([email protected]) Motivation What to correlate Information sources Signals, events and alerts

Questions Questions regarding this lecture?

Log and alert policy

How to correlate Purpose and general model Alert normalization Alert reduction (fusion, verification)

Lab assignments: NONE. (focus on your project proposal) These slides will be uploaded here: http://os3.nl/2007-2008/courses/ids/

Alert correlation

Summary