There has been a lot of news around the recent Cisco IOS XE vulnerabilities CVE-2023-20198 and CVE-2023-2073. Information about this vulnerability was first published by Cisco on October 16th, 2023, and since then we have seen evidence of mass exploitation and implantation. In this post we share our technical insights so far into these vulnerabilities.

Cisco has a few different operating systems:

• Cisco IOS
• Cisco IOS XE (based on Linux)
• Cisco NX-OS (based on Linux)
• Cisco IOS XR (based on QNX)
• ASA OS (based on Linux)
• CatOS

Most prior vulnerability research is centered around Cisco IOS. Cisco IOS is a unique operating system that runs as basically one giant binary. In this post, we are focused on Cisco IOS XE which is based on Linux and quite different from IOS. After obtaining the filesystem for a patched and unpatched version of IOS XE we see that the vulnerable webui service is a combination of Nginx and the iosd binary.

Nginx + Lua

Cisco IOS XE uses a version of Nginx called OpenResty. OpenResty extends Nginx by adding support for Lua scripting. This is evident by examining the /usr/binos/conf/webui.conf file:

webui.conf

Here we can see the rewrite_by_lua_block and access_by_lua_block directives which run Lua code during the rewrite and access Nginx phases. Its clear now that the vulnerable webui service is using Nginx. We want to see the entire picture, so we start to look for the root Nginx config file. To our disappointment, we find that root Nginx config is a single line, include /tmp/nginx.conf. Since tmpfs filesystems are generally only available at runtime, we determine that the true Nginx config file must be dynamically generated. Searching for the /tmp/nginx.conf string, we are led to the iosd binary.

grep for nginx.conf

IOSD

iosd is a large monolithic binary. Presumably, when Cisco created IOS XE based on Linux, they wanted to repurpose as much code as possible from IOS. We think that might explain why there is so much functionality packed into iosd. We are unable to get Ghidra to ever finish analyzing iosd, but luckily IDA can handle the job. Examining the strings, of iosd we find that it is indeed be responsible for generating the overall Nginx config.

iosd strings

Based on further exploration of iosd we also see that it is responsible for most of the HTTP related processes in IOS XE. Here, in the cli, we can see a summary of the different HTTP modules managed by iosd.

http server session modules

There are quite a few differences we see comparing the patched and unpatched filesystems. We will first focus on the differences we think are related to CVE-2023-20198, the vulnerability used to create a user with level 15 privileges.

Proxy-Uri-Source Header

The first patch we are drawn to is in WSMAApiLib.lua which is used to interact with the backend wsma service listed above. We see that Cisco added a new header to an internal request they make in WSMASendCommand.

WSMASendCommand

Tracing the callers of WSMASendCommand we find that there are indeed ways to configure a new user with this function. However, all paths that we find require authentication so far.

We also notice that iosd is updated to set the same Proxy-Uri-Source header in the server handlers of the Nginx config.

iosd Proxy-Uri-Source header

We also notice that one of the HTTP handlers in iosd has added a check to make sure that the Proxy-Uri-Source is set to webui_internal.

iosd Proxy-Uri-Source check

Our Current Theory

Based on these changes, our theory is that Cisco is trying to enforce that all access to the wsma services go through WSMASendCommand which requires authentication. Based on the Niginx config changes in iosd, is seems like there might have been a way to hit the wsma service through another service in iosd and by setting the Proxy-Uri-Source header to global for all requests outside of WSMASendCommand, they are effectively implementing a kind of authentication.

There are quite a few HTTP services in iosd, some require authentication and some do not. However, we have not been able to find a way yet to redirect to the wsma services in an unauthenticated manner. The size of the iosd binary has prevented us so far from running bindiff.

Other Changes

There are a few other changes in the patch diff that we think are probably related to CVE-2023-20273, the vulnerability used to write the implant to disk. There is a change to how they handle IPv6 string validation that mentions the vulnerability in a comment specifically referencing the Cisco bug ID for these 0-days. It comments:

“As a cardinal rule for any validation or check, assume the input is invalid”.

The previous version of the function uses the less secure assumption that unless it violates some specific checks, that it is a valid IPv6 address. Now, the function assumes it is invalid unless passing a more thorough set of checks.

IPv6 validation change

This function is called from a few different places that all require authentication. It allows us to pass validation with invalid IPv6 strings.

Invalid IPv6 string

There is also a fix for what seems to be a logic bug in softwareUpdateUtils.lua.

softwareUpdateUtils.lua logic bug

There are two main observable changes:

1. A new custom header, “Proxy-Uri-Source”, is set in the Nginx configs and checked in the iosd binary before allowing access to the wsma endpoints
   1. The attacker has discovered a method to pass requests to the wsma endpoints without control of the forwarded headers
   2. The wsma endpoints expects an XML payload and will execute any arbitrary IOS CLI command in it
2. Updates to IPv6 address validation which is utilized in builtin software install functionality in endpoints like /webui/rest/upgrade or via wsma
   1. The attacker has discovered a way to write and execute arbitrary files by abusing this validation failure
   2. The software install vector also corresponds with the observed IOC in the Talos blog

      %WEBUI-6-INSTALL_OPERATION_INFO: User: username, Install Operation: ADD filename

Next steps would include reverse engineering the different HTTP endpoints in iosd in search of a way to hit the wsma service in an way that doesn’t require or bypasses authentication. Hopefully this information can help other researchers uncover and reproduce these vulnerabilities.