Network Security Protocols Explained | TryHackMe Security Engineer Track

We covered network security protocols on the application layer, presentation layer and network layer of the OSI model. These protocols are used to exchange packets and data securely using either SSL/TLS or IPsec. This is necessary important to protect data against eavesdropping and man in the middle attacks (MITM). The most widely used mechanism for securing data over an insecure channel is to add an SSL/TLS wrapper, as in HTTPS, FTPS, POP3S, and SMTPS. This was part of TryHackMe Network Security Protocols.

Get Security+ Certificate Notes

The Complete Practical Web Application Penetration Testing Course

Challenge Description

Learn about secure network protocols at the different layers of the OSI model.

Video Highlights

SMTPS is not a proprietary protocol; instead, it wraps SMTP inside TLS. You can say that SMTPS is similar to SMTP on the application layer, with an extension of TLS encryption at the transport layer. For encryption, the STARTTLS command is used between the email client and the email server.

Port 587 and 465 are both frequently used for SMTPS traffic. Mails transmitted using SMTP are not encrypted, so they are prone to sniffing attacks. Therefore, SMTPS is used to encrypt emails through TLS before transmission. In addition, SMTPS also forbid attackers from sending spam messages from compromised/vulnerable domains, exfiltration sensitive information, and conducting phishing attacks.

POP3S Protocol

POP3 is considered weak from a security point of view. This requires an added layer of security; hence, POP3S comes into play. POP3S is an extension of POP3, which wraps the communications related to email messages within TLS. For this purpose, the client and server initiate the STARTTLS command, as shown in the figure below. After the EHLO, the POP3S server will trigger the switch to TLS. Note that EHLO stands for Extended HELO, where HELO is the command used to identify to the server.

DNSSEC

DNSSEC makes it possible to ensure that the DNS response we receive is from the domain owner. To achieve this, DNSSEC requires two main things:

1. The DNS zone owner should sign all DNS records using their private key.
2. The DNS zone publishes its public key so users can check the validity of the DNS records signatures.

In other words, the data to our DNS query is signed to ensure its integrity and authenticity; moreover, we can efficiently check the signature.

With signed records, DNSSEC provides the following:

• Authenticity: You can confirm that a certain DNS owner has authored and sent the record. Authenticity is possible because the received record is signed by the DNS owner’s private key.
• Integrity: You can ensure that no changes have been made to the record on its way. Any changes to the record will render its signature invalid.

SSH

The Secure Shell Protocol (SSH) provided the security requirements lacking in Telnet and remote login. With SSH, it is no longer feasible for the attacker to read the login credentials or modify the traffic.

SSL/TLS Protocol

Secure Socket Shell (SSL) and Transport Layer Security (TLS) are protocols used to encrypt data exchanged between a client, such as a web browser, and a server. Consider SSL/TLS as a wrapper that encrypts various communication protocols, such as HTTP and FTP, to create HTTPS and FTPS. SSL is not commonly used nowadays as TLS has been gradually replacing it.

SSL/TLS handshake is performed to encrypt the communication between client and server through the following steps:

1. Client Hello Message: The client sends a hello message to the server; it includes the client TLS version and the cypher suite that the client supports, in addition to random bytes.
2. Server Hello Message: The server responds with a hello message, highlighting its certificate, chosen cypher suite and random bytes.
3. Authentication: The client authenticates the server’s certificate through the certificate authority that issued it. For example, when we visit Google, Google shares its certificate. The received certificate is verified by our browser, which is pre-installed with the certificates of various certificate authorities.
4. Premaster Secret: The client encrypts random bytes with the server’s public key. (The client retrieves the public key from the server’s certificate.)
5. Decryption of Premaster: The server decrypts the premaster with its private key.
6. Session Keys Generated: The client and the server generate session keys based on client random bytes, random server bytes and premaster secret. Both will arrive at the same results; this session key is not transmitted, and encryption and decryption are based on this key.
7. Ready Messages: The client and server send a “finished” message using the session key to indicate that the session is ready for transmission. The client and server are now ready to exchange messages over SSL/TLS encrypted connection.

IPsec

IPsec stands for Internet Protocol Security. IPsec provides security by adding authentication and protecting the integrity and confidentiality of the network traffic. IPsec uses the following protocols:

1. Authentication Header (AH): Provides authentication and integrity.
2. Encapsulating Security Payload (ESP): Provides authentication, integrity, and confidentiality.
3. Security Association (SA): Is responsible for negotiating the encryption keys and algorithms. One example is Internet Key Exchange (IKE)

VPN

When the TCP/IP protocol was designed, security requirements such as confidentiality and integrity were not a design target. In contrast, availability was the priority as one of the purposes of the Internet is to withstand a nuclear attack, as is evident by the routing protocols adapting quickly when a link goes down. But we need to allow a corporation to use the existing Internet infrastructure to connect its offices securely. The answer lies in setting up a VPN.

A Virtual Private Network (VPN) makes it possible to establish a private connection over a public network. In other words, we can establish a secure connection over an insecure infrastructure.

For instance, in the figure below, we can see a remote office and a remote user connected over a VPN to the main office. A VPN connection requires a VPN client and a VPN server or concentrator. All the traffic between the VPN client and server is encrypted.

Room Answers

Video Walkthrough