QRadar Log Sources

QRadar Log Sources are displayed in Log Activity tab where each event information is in a form of record from that log source. An event is a record from a device that describes an action on a network or host. SIEM normalizes the varied information found in raw events.

QRadar SIEM supports many protocols, to receive raw events from log sources,such as syslog from operating systems, applications, firewalls, IPS/IDS, router, switches, SNMP and SOAP, data from a database table or view, such as JDBC or proprietary vendor-specific messaging protocols, such as OPSEC/LEA from Checkpoint , Snare, and Splunk Universal Forwarder or Adiscon Event Reporter licensed programs are also available.

QRadar SIEM  can collect events from Windows appliances too. For this purpose, it uses WinCollect program, which collects events by running as a service on a Windows system. The agent, can collect also events from other Windows servers, where the agent is not installed but where windows events are forwarded by other windows machines. WinCollect is centrally managed from the QRadar SIEM user interface.  The other method of collecting Windows events can be used the Microsoft Windows Management Instrumentation (WMI) licensed program, which is also administered through the QRadar SIEM user interface to collect events without an agent. However, WMI puts a major load on network and Windows servers. Domain controllers usually slow down when WMI is configured.

After raw events are normalized, it is easy to search, report, and correlate these normalized events

Normalizing means to map information to common field names, for example, information found by QRadar in other appliances log as SRC_IP or Source or IP and any other similar are mapped to QRadar’s Source IP.  The same process is created for user_name or username or login contained in a raw log which is normalized to QRadar field called User.

Normalized events are sorted by high-level and low-level categories to facilitate further processing.

Category	Category ID	Description
Recon	1000	Events that are related to scanning and other techniques that are used to identify network resources, for example, network or host port scans
DoS	2000	Events that are related to denial-of-service (DoS) or distributed denial-of-service (DDoS) attacks against services or hosts, for example, brute force network DoS attacks
Authentication	3000	Events that are related to authentication controls, group, or privilege change, for example, log in or log out.
Access	4000	Events resulting from an attempt to access network resources, for example, firewall accept or deny.
Exploit	5000	Events that are related to application exploits and buffer overflow attempts, for example, buffer overflow or web application exploits.
Malware	6000	Events that are related to viruses, trojans, back door attacks, or other forms of hostile software. Malware events might include a virus, trojan, malicious software, or spyware.
Suspicious Activity	7000	The nature of the threat is unknown but behavior is suspicious. The threat might include protocol anomalies that potentially indicate evasive techniques, for example, packet fragmentation or known intrusion detection system (IDS) evasion techniques
System	8000	Events that are related to system changes, software installation, or status messages.
Policy	9000	Events regarding corporate policy violations or misuse.
Unknown	10000	Events that are related to unknown activity on your system.
CRE	12000	Events that are generated from an offense or event rule.
Potential Exploit	13000	Events relate to potential application exploits and buffer overflow attempts.
Flow	14000	Events that are related to flow actions.
User Defined	15000	Events that are related to user-defined objects
SIM Audit	16000	Events that are related to user interaction with the Console and administrative functions.
VIS Host Discovery	17000	Events that are related to the host, ports, or vulnerabilities that the VIS component discovers.
Application	18000	Events that are related to application activity.
Audit	19000	Events that are related to audit activity.
Risk	20000	Events that are related to risk activity in IBM Security QRadar Risk Manager.
Risk Manager Audit	21000	Events that are related to audit activity in QRadar Risk Manager.
Control	22000	Events that are related to your hardware system.
Asset Profiler	23000	Events that are related to asset profiles.
Sense	24000	Events that are related to UBA.

How events are collected  and processed.

As mentioned before  log sources typically send syslog messages, but they can use other protocols also. These are received on incoming network interface of appliance, which is processed then  by ecs-ec.  You can check are there incoming events on interface using the following command:

# tcpdump -s 0 -A host "Device_IP_Address" and port 514

How to get events from packets per second on ethN (or any other network interface)?

# time tcpdump -c 500 -nn -i ethN dst port 514 | awk '{ print $3 }' | sort | uniq -c | sort -nr

Event Collector ecs-ec receives raw events as log messages from an external log source and using Device Support Modules (DSMs) in the event collectors parse and normalize raw events while raw log messages remain intact.  The Event Collector classifies them into low- and high-level categories. The Event Collector also bundles same events to conserve system usage through a process known as coalescing. Event collectors use traffic analysis to discover which kind of device a log source is if a Device Support Module (DSM) for that kind of device is installed.  In addition, the DSM for a device specifies how to map and normalize the device’s raw events. Otherwise those events are kept as “Stored“.

Event Processors receive the normalized events and raw events to analyze and store them. Each Event Processor processes events from the Event Collectors and flows data. Event processors correlate the information. The Event Processor examines information gathered by QRadar SIEM to show behavioural changes or policy violations. Rules are applied to the events to search for anomalies.

DSMs are set of regular expression rules which let extract and parse details from raw content and map them to correct field in the database. DSMs are provided by IBM in form of installable rpm packets, which can be automatically downloaded in process of QRadar autoupdates or installed by a customer after download it from FixCentral.

Check current version of DSM type

# rpm -qa|grep -i <name_of_source>


Install downloaded DSM
# yum install <packet_name>.rpm

Using rpm -qa, you can check from a command line is there DSM available in your system and in what version. After downloading you can install it using yum command.

Adding Log Sources

QRadar SIEM automatically discovers the log sources with autodiscovery enabled in your deployment that are sending syslog-only messages. When a source is automatically discovered, an associated data line item is displayed in the Log Sources window.

In case if autodiscovery causes problems it can be switched off using  script provided

# /opt/qradar/bin/tatoggle.pl

More than one Log source can listen for the same machine.  For example, on Linux machine, you can have a running database and both logs can be separated using different Log Source for the same appliance. While multiple event sources using the same protocol should be supported by one log source, you must configure QRadar SIEM, so that it can process the data in the correct order.

In some cases, we must manually define Log sources. In order to do this we must complete the following fields to create a log source. To add multiple log sources, if the log sources share the same protocol, you can use the Bulk Add feature under the menu Bulk Actions. You can add at most of 500 log sources.

Selecting the Coalescing Events check box causes QRadar SIEM to accumulate events with the same values for the following parameters:
• Log source
• Event name
• User name
• Source IP/Port
• Destination IP/Port
Within ten seconds of receiving the first event of the sequence, the first three events are stored individually. After that, all events in the sequence are accumulated into one single event and stored as such, thereby dropping the payload information of all accumulated events in the sequence.

Adding log source extensions

To overpass the default parser of the DSM you can use a log source extension. Any parsing rules in the log source extension take priority over the parsing rules in the default parser. Log source extensions immediately extend the parsing routines of specific devices. You must use a log source extension to detect an event that has missing or incorrect fields. A log source extension can also parse an event when the DSM is in use but fails to produce a correct result. You write log source extensions in XML (Extensible Markup Language) using most common word processing applications. For one Log Source you can create multiple log source extension and associate them with various log source types.

Since DSM editor feature has been added to QRadar, creating own log source extension is much more simple and there is no need write manually XML code anymore.

Log source parsing order

Defining the parsing order for log sources ensures that the required log sources are parsed in a specific order regardless of changes to the log source configuration. A parsing order prevents unnecessary parsing, ensuring that such changes to log source configuration do not affect system performance.
You can configure the order that you want each Event Collector in your deployment to use to apply DSMs to log sources. If a log source has multiple Log Source Types under the same IP address or host name, you can order the importance of these incoming log source events by defining the parsing order. Select an IP address to display the log sources at this IP. Use the Up, Down, Top, and Bottom options to rearrange the log sources and create the parsing order.