Encrypted traffic has come to dominate network flows, which makes it difficult for traditional flow monitoring tools to maintain visibility. This is particularly true when the process to enable encryption occurs after an initial data exchange, causing the encryption attributes to be missed.

In this blog post we take a closer look at a new feature added to CERT’s Yet Another Flowmeter tool (YAF) to capture the attributes of encryption when it occurs after the start of the session. We call this mid-encryption. We explore what mid-encryption means, why it matters, how it works within YAF, and what benefits this brings to traffic analysis and network security teams.

From 2014 to 2024, we saw a steady increase in the percentage of traffic that is encrypted with more than 80 percent of pages loaded by Firefox and 96 percent of traffic across Google being encrypted.

CERT researchers developed Yet Another Flowmeter (YAF) 20 years ago to read network packets and create Internet Protocol Flow Information Export (IPFIX) network flow records—where each record summarizes a connection between two hosts (a network session. The infrequent use of encryption at that time meant YAF had full visibility into many of these records: YAF was able to capture the metadata of various connections, including: HTTP for web sites, Simple Mail Transport Protocol (SMTP), Internet Message Access Protocol (IMAP), and Post Office Protocol v3 (POP3).

For connections that started with an encryption request, YAF could capture attributes of the encrypted session (the Transport Layer Security (TLS) ClientHello and ServerHello) and the certificates used for encryption. Although the encrypted session itself was opaque, the captured attributes allowed network analysts to verify that certificates were legitimate, and the connection was properly encrypted.

What is Mid-Encryption?

Mid-encryption refers to a network session beginning in an unencrypted (usually text-based) state and transitioning to an encrypted state during the same session. This action is triggered using mechanisms such as STARTTLS, a command used in application-layered protocols (e.g., Simple Mail Transfer Protocol, Internet Message Access Protocol, Extensible Messaging and Presence Protocol) that begins encryption using TLS. Typically flow sensors label the session as encrypted or unencrypted by examining the beginning of the session. While this process usually helps with labeling the correct protocol and capturing the metadata, commands such as STARTTLS may lead to potential loss of visibility and metadata because they launch the encryption process during the session.

Why Mid-Encryption Support Matters

Today’s HTTP traffic is largely encrypted, but older protocols often use an opportunistic encryption model that is easier to implement and allows servers and clients to communicate when both parties do not support encryption. With opportunistic encryption, a session starts in plain text before negotiations for encryption occur via a STARTTLS or HTTPS upgrade. Early session metadata is available to the sensor, while the rest may be nontransparent.

Without mid-encryption support, YAF may miss the indicators of when encryption occurs and fail to label the session correctly. This scenario could lead to partial loss of visibility—we don’t know if encryption was successful—and incorrectly labeled flow records, which may lead to analysts needlessly investigating benign traffic.

Mid Encryption Capabilities

With the new feature, YAF can now track protocol negotiations in real time and identify encryption flags (like the STARTTLS command or TLS ClientHello). The Internet Protocol Flow Information Export (IPFIX) records it generates are enriched with encryption information: when the encryption began, what protocol was negotiated, and which portions of the flow are encrypted or clear text. The record also includes TLS ClientHello metadata: TLS version, cipher suites offered and selected, and server certificate details.

Mid encryption is useful with protocols that still allow clear text preludes before upgrading, such as SMTP, POP3, IMAP, Network News Transport Protocol (NNTP), Lightweight Directory Access Protocol (LDAP), XMPP, and IRC.

Example Use Case: STARTTLS in SMTP

A mail client connects to a mail server listening on port 25. The server replies with a greeting and a list of extensions that includes STARTTLS if supported. The client may issue SMTP commands, such as <code>EHLO, MAIL FROM</code>, and <code>RCPT TO</code>, that are transmitted in clear text. At this point the session is still unencrypted. The client at some point sends a <code>STARTTLS</code> command to which the server, if supported, replies with a message saying it is ready to start TLS communication (e.g., 220 Ready to start TLS). The client sends TLS ClientHello messages and TLS negotiation and encryption begins.

With the mid-encryption support, YAF is able to

• parse clear text for SMTP commands
• identify the STARTTLS command and replies
• identify the TLS ClientHello message
• identify when encryption starts and ends
• provide TLS deep packet inspection (DPI) data
• detect protocol nesting and record accurately

Figure 1: With mid encryption support, YAF captures plain text commands and encryption negotiation of a SMTP connection

YAF has the ability to label the flows correctly because it keeps track of the original protocol where the plain-text session started—SMTP for this use case. YAF would also maintain a sub-record labeling the TLS DPI data that provides network analysts a more complete picture of the protocols t upgrading to an encrypted session.

Figure 2: A YAF record containing DPI for SMTP text commands and TLS metadata

What Can an Analyst Do with Mid-Encryption?

Let’s take the SMTP use case as an example. Before adding mid-encryption, a record generated by YAF summarizing an SMTP connection using STARTTLS would not contain information regarding the quality of the encryption or the certificates used. It would only contain the server's welcome banner, the client's EHLO command, and a Boolean noting that STARTTLS was used.

With the support of mid-encryption, the records generated by YAF are augmented with service-specific TLS attributes and certificate information as seen in the diagram (Figure 2), which illustrates the IPFIX or JSON records. Within the original record for the SMTP protocol, a TLS DPI section (using the historical name SSL) will appear that would inform the analyst that the session was encrypted, the version of TLS, the encryption cipher, and certificate attributes such as the issuer, subject, key length and validity dates. A security analyst could identify the use of weak or revoked certificates or certificates issued by suspicious parties. The analyst would then be able to expand on their fingerprinting capabilities (e.g., JA3 or JA4+) and pivot from that information. This could be used to identify misconfigured machines or insider threats within an organization, or identify sources of unwelcome email that should be blocked.