By Yarden Shafir

AI is becoming ubiquitous, as developers of widely used tools like GitHub and Photoshop are quickly implementing and iterating on AI-enabled features. With Microsoft’s recent integration of Copilot into Windows, AI is even on the old stalwart of computing—the desktop.

The integration of an AI assistant into an entire operating system is a significant development that warrants investigation. In this blog post, I’d like to share the results of my brief investigation into how Microsoft has integrated Copilot into its legacy desktop system. I’ll summarize some key features of the integration and explore some of the concerns and future considerations of the role of AI in desktop environments.

Some caveats

Before we get into the details, there are two important caveats to keep in mind.

First, and most importantly, Microsoft Copilot works only with a functioning internet connection. This tells us that the models in use are hosted, not local, and that by necessity, some data from your machine is sent to Microsoft whenever AI features are used.

Second, as with other AI-enabled tools, Copilot’s results aren’t always stable or reliable. The fact that Copilot can give you something unexpected takes some getting used to and requires an initial trial-and-error period to discover what works and what doesn’t. This implies that even well-resourced public deployments of generative AI have not sufficiently mitigated the hallucination problem.

Copilot in Windows

In the most recent Windows 11 release, Microsoft officially introduced Windows Copilot—an everyday AI companion that exists on the desktop and is ready to answer any question. According to Microsoft,

Copilot will uniquely incorporate the context and intelligence of the web, your work data and what you are doing in the moment on your PC to provide better assistance – with your privacy and security at the forefront.

On Windows builds that support Copilot, you’ll be able to see a new desktop icon that opens a side pane to the Copilot interface:

While this pane may look brand new, under the surface it is simply a view into Microsoft Edge running Bing AI inside an msedge.exe process. However, Copilot does include some new features and abilities beyond what “regular” Bing AI can do.

Just like Bing AI, Copilot does not have a local AI model. All queries and operations are done via a web interface to remote machines that process requests and return answers. Therefore, Copilot requires an active internet connection to function. Copilot will search its own knowledge base or access the web to give you answers to any questions you ask (and just like with any LLM, those answers may be confidently incorrect). By default, Copilot will perform only general web queries and won’t access any user data or data related to the current web session. However, even in that default state, Copilot does have access to metadata provided by the browser and operating system, such as the IP address, location (as provided by the browser), and preferred language.

An optional setting (which is disabled by default) allows Copilot to access the current browser session to collect information about the URLs and titles of the currently open web pages and the content of the active web page. It should not have access to any private data such as passwords or browser history.

Copilot comes with other capabilities beyond the ability to answer basic queries. The first is an integration with DALL-E to generate AI art. You can access this feature through general requests to Copilot or by typing #graphic_art(“prompt”). For example, typing #graphic_art(“tree”) will generate a picture of a tree.

Another interesting capability allows users to access hard-coded local operations through the #win_action(“command”) prompt. Each action results in a message from Copilot asking for user confirmation before performing the action. Here is the list of hard-coded #win_action options that seem to be available at the moment:

Currently, while all these actions are local, they cannot be used while the machine is offline. As Copilot matures, we look forward to seeing what new capabilities it can provide.

Even though Microsoft Copilot is in its early stages, it demonstrates significant capabilities. But as with any cloud-based AI application, it raises security and privacy concerns. These concerns center mainly around the fact that queries must be sent to a server for processing, and they might be stored, used to further train the AI model, or shared with other companies for various purposes (such as personalized advertising). Additionally, Copilot’s capacity to affect change on local systems is particularly noteworthy. This functionality introduces new concerns regarding the role of AI in desktop environments, a role that extends beyond the reach of most current AI-enabled products. For example, the ability to access local operations through Copilot could help attackers perform local actions on a machine without being detected; and if Microsoft expands the list of available operations in the future, this concern would only grow. Though the integration of AI into desktop environments is an exciting development, these concerns will have to be a critical focus of developers and researchers as Microsoft continues iterating on Copilot, and as more AI–operating system integrations inevitably enter the scene.

*** This is a Security Bloggers Network syndicated blog from Trail of Bits Blog authored by Trail of Bits. Read the original post at: https://blog.trailofbits.com/2023/12/27/ai-in-windows-investigating-windows-copilot/