Research from Okta’s Red Team has revealed a significant CVE (though currently unassigned) vulnerability dubbed “HollowByte” affecting OpenSSL. According to The Hacker News, the flaw allows an attacker to trigger a DDoS or local denial of service by sending a maliciously crafted 11-byte packet during the TLS handshake. This vulnerability is particularly dangerous because it bypasses many traditional security filters that look for large or suspicious traffic volumes.
Technical Analysis of the HollowByte Vulnerability
The core of the issue lies in how OpenSSL manages memory during the processing of specific TLS messages. When a server receives a truncated or malformed message of exactly 11 bytes, the OpenSSL internal memory management logic anticipates a larger payload. In response, it preemptively allocates a buffer of approximately 131 KB. If the expected data never arrives, the logic fails to properly deallocate or reuse this buffer.
OpenSSL memory leak on glibc systems
The severity of the HollowByte flaw is compounded on systems utilizing the glibc library. In testing, it was observed that once this memory is allocated by the OpenSSL process, it is not returned to the operating system’s global pool until the specific process is terminated or restarted. This behavior facilitates an asymmetric attack where a single threat actor, using minimal bandwidth, can systematically deplete a server’s available RAM. Over time, this leads to system instability or the invocation of the Out-of-Memory (OOM) killer, which may terminate critical security services or the web server itself.
Detecting HollowByte 11-byte TLS requests
Identifying this activity within a SOC environment requires specific SIEM logic. Traditional network monitoring often focuses on volumetric attacks or high-frequency requests. HollowByte is a “low-and-slow” resource exhaustion attack. To defend against this, security teams should focus on the state of the TLS state machine and the duration of open connections that fail to progress. Defenders should monitor for:
- A sudden spike in TLS handshake timeouts across internet-facing assets.
- High memory utilization in OpenSSL-dependent processes (such as Nginx, Apache, or HAProxy) that does not correlate with traffic spikes.
- Network patterns showing exactly 11 bytes sent in the initial application data phase followed by a connection stall.
Mitigation and Remediation Steps
The vulnerability was addressed in OpenSSL releases starting in June 2026. However, the lack of a formal CVSS score or public advisory at the time of the patch meant many organizations remained exposed.
OpenSSL HollowByte denial of service mitigation
The primary OpenSSL HollowByte denial of service mitigation is to update all OpenSSL installations to the latest stable versions. Because OpenSSL is a foundational library, this often requires updating the underlying operating system packages or rebuilding container images that bundle the library. If immediate patching is not feasible, organizations should consider implementing rate-limiting at the firewall or load balancer level for new TLS connections and decreasing the timeout for incomplete handshakes.
This flaw highlights a broader issue regarding Supply Chain Attack transparency and the risks of “silent patches.” When security-relevant bugs are fixed without a formal announcement, it leaves defenders blind to the risks. This vulnerability should be mapped against the MITRE ATT&CK framework under Resource Hijacking (T1496) and Endpoint Denial of Service (T1499).