The Federal Trade Commission is warning AI companies against secretly changing their security and privacy policies in hopes of leveraging the data they collect from customers to feed models they use to develop their products and services.

Surreptitiously amending terms of service without notifying customers is not unusual in the business world and AI companies’ insatiable need for data makes them vulnerable to looking at the massive amounts of information they collect from consumers and businesses to fuel their innovation, the FTC’s Office of Technology and The Division of Privacy and Identity Protection wrote in a column this week.

The agency equated these companies’ need for huge amounts of new data to the decades-long need to find new oil deposits.

“There is perhaps no data refinery as large-capacity and as data-hungry as AI,” agency officials wrote. “Companies developing AI products … possess a continuous appetite for more and newer data, and they may find that the readiest source of crude data are their own userbases. But many of these companies also have privacy and data security policies in place to protect users’ information.”

This Creates a Conundrum

“These companies now face a potential conflict of interest: they have powerful business incentives to turn the abundant flow of user data into more fuel for their AI products, but they also have existing commitments to protect their users’ privacy,” they wrote.

Changing the terms of service so they can use the data for their models might seem like a good answer to some of these organizations, but the FTC will crack down – and has in the past – on companies that do this without giving users the proper notice.

“To avoid backlash from users who are concerned about their privacy, companies may try to make these changes surreptitiously,” the officials wrote. “But market participants should be on notice that any firm that reneges on its user privacy commitments risks running afoul of the law.”

They added that “ultimately, there’s nothing intelligent about obtaining artificial consent.”

Ongoing Worries About Data Security

Concerns about the security and privacy of the data used to train large-language models (LLMs) and to use the rapidly expanding universe of tools like OpenAI’s ChatGPT and Google’s Gemini has been at the forefront over the past year as innovation of and the market around generative AI has exploded.

The worries have ranged from data leaking from AI models to threat groups using generative AI tools to improve their malicious activities. Menlo Security in a report this week outlined how common it’s become for people and companies using generative AI platforms to expose sensitive or proprietary corporate data. In addition, Microsoft and OpenAI detailed how state-sponsored cybercriminal gangs are leveraging such tools in their attacks.

There are numerous examples of data of AI technology users being exposed. An attack on OpenAI in March 2023 compromised the personal and payment information of 1.2% of ChatGPT Plus subscribers and cybersecurity firm Group-IB reported three months later it found as many as 100,000 compromised ChatGPT user accounts for sale on the dark web.

Wiz researchers in September reported that Microsoft’s AI team accidentally exposed 38 terabytes of private data while publishing open source training data on GitHub.

Making the Point

The need to protect such data is critical and the FTC wants to ensure that AI companies understand what’s expected of them.

“It may be unfair or deceptive for a company to adopt more permissive data practices – for example, to start sharing consumers’ data with third parties or to use that data for AI training – and to only inform consumers of this change through a surreptitious, retroactive amendment to its terms of service or privacy policy,” the agency wrote.

It noted enforcement actions in the past, including two decades ago when it charged Gateway Learning – the folks behind Hooked on Phonics – with changing its privacy policy without notifying customers or getting their consent so it could share consumer data with third parties.

Likewise, the agency last year accused genetic testing company 1Health of changing its privacy policy to expand the kinds of third parties that it could share users’ personal data with, again without notifying consumers or getting their consent.

“Even though the technological landscape has changed between 2004 and today, particularly with the advent of consumer-facing AI products, the facts remain the same: A business that collects user data based on one set of privacy commitments cannot then unilaterally renege on those commitments after collecting users’ data,” the officials wrote.

Keeping an Eye on AI

This isn’t the first time the agency has put AI companies on notice. Last month, the FTC noted model-as-a-service companies – those who develop and host AI models that become available to third parties through an API or end-user interface – face the same pressures of continuously ingesting new data that dog all AI organizations and that they needed to abide by their terms of service and privacy policies.

“For example, a company can train a large language model (LLM) and sell access to this model to businesses (online stores, hotels, banks, etc.) who apply it to customer service chatbots,” the agency wrote.

The incentive to develop new or customer-specific models or to refine existing ones by ingesting more new data can conflict with companies’ obligations to protect users’ data.

“This risk is particularly salient given that customers may reveal sensitive or confidential information when using a company’s models, such as internal documents and even their own users’ data,” the FTC wrote. “There’s also a risk that a model-as-a-service company may, through its APIs, infer a range of business data from the companies using its models, such as their scale and precise growth trajectories.”