Malware and Security Analysis

# CHAPTER 16

Malware and Security Analysis #

1. Introduction #

Network administrators use Wireshark to ensure traffic is flowing smoothly. Cybersecurity analysts use Wireshark to catch the traffic that shouldn't be flowing at all. When malware infects a machine, it rarely sits silently; it actively reaches out across the network to steal data or receive commands from a hacker. By mastering Wireshark, a security analyst can spot the microscopic footprints left by these malicious connections. In this chapter, we will shift to an offensive/defensive mindset. We will learn how to identify automated Port Scans, recognize Command and Control (C2) beaconing, and detect advanced evasion techniques like DNS Tunneling.

2. Learning Objectives #

By the end of this chapter, you will be able to:

• Identify the visual signature of a TCP SYN Port Scan (e.g., Nmap).

• Analyze suspicious outbound connections from a compromised host.

• Detect "Beaconing" behavior associated with Malware Command and Control (C2).

• Understand the mechanics of DNS Tunneling for data exfiltration.

• Filter out legitimate traffic to isolate security anomalies.

3. Beginner-friendly Explanations #

The Door Rattler (Port Scanning): Imagine a thief walking down a long hotel hallway with 65,000 doors. The thief jiggles the handle of every single door. If it is locked, they move on. If it is unlocked, they write the room number down in a notebook to rob later. In networking, this is a Port Scan. A hacker sends a connection request to every single TCP port on your server (from Port 1 to Port 65535) just to see which "doors" are open and listening. Wireshark will catch the thief jiggling the handles.

4. Detecting Port Scans (Nmap) #

If an attacker uses a tool like Nmap to scan your network, Wireshark will light up like a Christmas tree. The Signature:

1. 1. You will see a massive, rapid burst of TCP SYN packets from one Source IP to one Destination IP.

1. 2. The Destination Port number will increment rapidly: Port 80, Port 81, Port 82, Port 83...

1. 3. If the port is closed (locked door), your server will reply with [RST, ACK] (Reset).

1. 4. If the port is open (unlocked door), your server will reply with [SYN, ACK].

*Filter:* tcp.flags.syn == 1 && tcp.flags.ack == 0. If you see hundreds of these per second targeting different ports, you are actively being scanned.

5. Malware Communication (C2 Beaconing) #

When a computer gets infected with Ransomware or a Trojan, the malware needs to phone home to the hacker's "Command and Control" (C2) server for instructions. It does this using Beaconing. A beacon is a small, rhythmic packet sent out to the internet (e.g., every exactly 60 seconds). *How to spot it in Wireshark:* Filter for HTTP or DNS. Look at the Time column. If you see a computer sending an HTTP GET request to a strange IP address perfectly every 60.000 seconds, continuously, all night long while the user is asleep, that computer is infected. Humans browse the web randomly; malware browses rhythmically.

6. Data Exfiltration: DNS Tunneling #

If a corporation has a strict firewall that blocks all outbound web traffic, how does the hacker steal the data? They use the one protocol that is never blocked: DNS. DNS Tunneling: The hacker takes a stolen password ("secret123"), encodes it, and makes a fake DNS request for it: Query: secret123.hackerdomain.com The corporate firewall allows the DNS request out to the internet. The hacker's DNS server receives the request, strips off the "secret123", and saves the stolen password. *How to spot it in Wireshark:* Filter for dns. If you see a computer making hundreds of DNS queries containing massive, random strings of gibberish text (e.g., A9F8B7C6...hacker.com), the computer is actively leaking data via DNS Tunneling.

7. Security Investigation Examples #

*Scenario: Suspicious File Download.* You suspect an employee downloaded malware over unencrypted HTTP.

1. 1. Filter: http.request.method == "GET".

1. 2. Look at the requested files. You see GET /update.exe.

1. 3. Clear the filter. Find that specific packet. Right-click -> Follow TCP Stream.

1. 4. Change the view format to "Raw". Save the output as suspect.exe.

1. 5. Upload the extracted file to an antivirus scanner like VirusTotal. You have just performed digital forensics!

8. Best Practices #

• Know Your Baseline: You cannot spot anomalies if you don't know what "normal" looks like. In a corporate environment, you should know exactly what servers your domain controllers normally talk to. If a domain controller suddenly starts opening HTTPS connections to an IP address in a foreign country, your baseline knowledge allows you to instantly flag it as a critical incident.

9. Common Mistakes #

• Assuming Encryption Hides the Threat: Beginners think they cannot analyze malware if it uses HTTPS. False! While you cannot read the payload, you can still read the IP addresses (Who are they talking to?), the SNI (What domain name did they request?), and the frequency of the packets (Are they beaconing?). Metadata is often just as valuable as the payload.

10. Mini Project: Hunt for the Nmap Scan (Theory) #

Imagine you are looking at a PCAP file containing a suspected cyberattack.

1. 1. Apply the filter: tcp.flags.reset == 1.

1. 2. Look at the Packet List. You see 1,000 consecutive RST packets sent from Server A to IP B in the span of 2 seconds.

1. 3. *Analysis:* Server A is aggressively hanging up the phone 1,000 times. This definitively proves that IP B just attempted to connect to 1,000 closed ports. You have successfully identified IP B as a hostile scanner and can now block them at the firewall.

11. Practice Exercises #

1. 1. Describe the visual "rhythm" of Command and Control (C2) beaconing in Wireshark. Why does it look different from normal human web browsing?

1. 2. Explain the mechanism of DNS Tunneling. Why do attackers use DNS specifically to steal data out of secure networks?

12. MCQs with Answers #

13. Interview Questions #

• Q: A corporate firewall blocks all outbound HTTP/HTTPS traffic from a secure database server. Explain how an attacker could theoretically exfiltrate stolen database records using DNS Tunneling, and how you would detect it in Wireshark.

• Q: Walk me through the exact Wireshark display filter syntax and packet behavior you would look for to confirm a machine is being actively port-scanned.

• Q: You capture traffic from a compromised host communicating via encrypted HTTPS to a hacker's C2 server. Since you cannot read the payload, what specific metadata in the capture can you analyze to build an indicator of compromise (IOC)?

14. FAQs #

Q: Can Wireshark stop the malware? A: No. Wireshark is a passive listening tool. It is the magnifying glass the detective uses to find the fingerprints. Once you identify the malicious IP address or behavior in Wireshark, you must log into your active security appliances (Firewalls, Intrusion Prevention Systems) to actually block the attack.

15. Summary #

In Chapter 16, we weaponized Wireshark for digital forensics. We learned to spot the aggressive, sequential signature of a TCP Port Scan, observing the flood of SYN requests and RST rejections. We trained our eyes to look for the unnatural, rhythmic timing of C2 malware beaconing amidst the chaos of standard web traffic. Finally, we exposed the clever evasion technique of DNS Tunneling, proving that even seemingly harmless protocols can be abused for data exfiltration. Wireshark is no longer just for troubleshooting; it is a critical instrument for threat hunting.

16. Next Chapter Recommendation #

You have found the malicious packets. Now you need to prove it to your boss. Proceed to Chapter 17: Exporting and Reporting Data.