Executive Summary

The rapid expansion of generative AI (GenAI) has led to a diverse set of web-based platforms offering capabilities such as code assistance, natural language generation, chatbot interaction and automated website creation. This article uses insights from our telemetry to show trends in how the GenAI web is evolving.

Because of its growing prevalence, GenAI also opens new vectors for threat actors to misuse. Adversaries are increasingly leveraging GenAI platforms to create realistic phishing content, clone trusted brands and automate large-scale deployment using services like low-code site builders. The threats are getting harder to detect.

We examine specific misuse scenarios including AI-generated phishing pages and malicious chatbots. We also provide indicators of these activities to support detection and response efforts.

Palo Alto Networks customers are better protected from the threats described in this article by the following products and services:

Advanced URL Filtering and Advanced DNS Security identify known domains and URLs associated with this activity as malicious.

If you think you might have been compromised or have an urgent matter, contact the Unit 42 Incident Response team.

Increase in the Use of GenAI

Introduction to Web-Based AI Services

GenAI has fueled a surge of new websites and platforms, from conversational assistants to multi-media creation tools. As the ecosystem evolves, we identify emerging patterns in how people are adopting different categories of AI services.

Writing assistants, meeting tools, code generators and website builders are streamlining tasks that previously demanded substantial manual effort, signaling a shift in how work is created and delivered across domains.These AI tools streamline workflows, reduce manual effort and enable new forms of content creation.

We are observing a significant upward trend in GenAI adoption across industries, particularly following the surge of public interest and innovation in the AI space. Within just six months, AI use has more than doubled and continues to grow steadily, as shown in Figure 1.

The blue line represents the number of AI website visits in billions, from April 2024-April 2025. The red bars indicate the number of new websites hosting AI services detected in millions. This overall trend of increased traffic to AI websites indicates a growing adoption of GenAI applications and services.

AI Service Trends and Industry Focus

We discuss how people are adopting AI-specific capabilities so we can better predict how those various functions can introduce new risks. A further review of our telemetry reveals:

• The top industries leading AI adoption include:
  • High tech
  • Education
  • Telecommunications
  • Professional and legal services
• The high tech sector dominates AI use, accounting for over 70% of total GenAI tool use shown in Figure 2
• Most of this activity is concentrated in text-generation applications (such as writing assistants and AI chatbots) and media-generation tools, as shown in Figure 3
• About 16% of AI service use is dedicated to data processing and workflow automation, such as email campaign generation

Overview of Phishing Attacks Misusing Various Types of AI Services

While GenAI tools offer powerful capabilities, they also introduce significant risks that threat actors can exploit for phishing and other types of cyberattacks.

• AI code assistants can significantly enhance software development. They provide real-time coding suggestions, automate code generation and can reduce errors. However, they can inadvertently expose proprietary code or sensitive intellectual property, creating entry points for targeted attacks.
• Text generation tools — such as conversational, writing and meeting assistants — can enhance productivity, content creation and customer interaction. However, attackers can manipulate them to generate convincing phishing content, spread misinformation or leak confidential data.
• AI model services simplify deployment, training and inference. However, they can expose sensitive models or data to unauthorized access. This increases the risk of model hijacking or misuse in malicious workflows.
• Attackers can use multi-media AI tools — including media generators and website builders — to rapidly create realistic-looking but fraudulent websites, deepfake content and deceptive phishing pages to mimic trusted brands.
• AI-powered data platforms and workflow automation tools optimize business processes, improve efficiency and support informed decision-making. When loosely governed, they can become vectors for data leakage, unauthorized access and automated exploitation across integrated systems.

Collectively, these risks underscore the potential of GenAI to amplify phishing campaigns and other social engineering threats. Therefore, stronger safeguards and threat detection are necessary.

Figure 4 shows the top three AI services misused for phishing include:

• Website generators (approximately 40%)
• Writing assistants (approximately 30%)
• Chatbots (almost 11%)

Misuse of Website Generation Services

Although these services are relatively new, attackers are already misusing AI-powered website builders for real-world phishing attacks.

We observed phishing websites that threat actors created using a popular AI-powered website builder, which is capable of producing websites within seconds. This platform allows someone to enter a prompt that can build and publish websites without any email or phone verification. The site uses AI to generate images and text based on this prompt, for creating a website.

We detected two real-world examples of AI-generated phishing landing pages in May 2025. Both of these phishing pages, shown in Figures 5 and 6, link to attacker-owned credential-stealing sites.

Some of the AI-assisted website builders we investigated appear to lack guardrails that would prevent someone from impersonating an existing business or organization. As a test, we used a known AI-assisted website builder to create a fake page to appear as if it’s for Palo Alto Networks.

The website builder only required a valid email address (not necessarily a Palo Alto Networks email address) to establish a trial account and publish a page impersonating our company. Since these pages are intended to quickly establish a web presence for a new company or organization, they lack the design elements that criminals would otherwise use to spoof a targeted brand.

In our test, the website builder promised to generate a free AI website in 60 seconds, which is an accurate statement. Our only input was a brief description of the company for an initial text prompt. Figure 7 shows a brief description of Palo Alto Networks we typed in the initial prompt before clicking on the “Enhance Prompt” button.

The Enhance Prompt button took our initial input and created a complete AI prompt for the page, as shown below in Figure 8. The finished prompt included an AI-generated paragraph about the company, a default design style that can easily be modified and a list of content to include on the site.

We then clicked on the arrow button noted in Figure 8, and the builder took approximately 5-10 seconds to create a staging environment for the site. From a hastily typed initial prompt of “Palo Alto Networks is a leading cybersecurity company that has next-generation firewalls and other security solutions,” the resulting page Figure 9 shows looks plausible for a cybersecurity company.

Scrolling through the index page generated by the site builder, we found a convincing AI-generated description of our company, as shown in Figure 10.

The index page included links to different pages that contain descriptions of next-generation firewalls, cloud security solutions and threat intelligence services. Figure 11 shows a link from the index page for threat intelligence services and the resulting page from that link. Like the company description, the description of these services mimics what most people would expect from an established cybersecurity company.

The website builder includes a button to publish the site. Pushing this button generated the dialogue window shown in Figure 12.

While we did not publish this fake Palo Alto Networks site, cybercriminals have misused this builder to publish phishing pages mimicking other brands, such as the two real-world examples we previously referenced.

Attackers can replicate similar attack vectors on other website builder platforms since many of these platforms have recently added AI-assisted features. Figure 13 shows an example of a fake gift card site spoofing popular vendors created through a popular website builder.

Currently, the real-world phishing attacks seen on AI-powered website builders appear relatively rudimentary and might not deceive most potential victims. However, in the medium to long term, we expect that these attacks will become more convincing as AI-powered website builders grow more powerful.

Misuse of Writing Assistant Services

In addition to website builders, we identified multiple real-world phishing URLs generated and hosted on third-party AI writing assistant platforms. In all these cases, the attacker used the app to host a phishing page. The phishing page displays a generic message like “You have new documents — click the button to view.” Clicking the button leads the victim to a secondary credential-stealing site, such as a fake Microsoft login page.

Despite being hosted on platforms that offer AI-powered content generation, these phishing pages are quite simple and show no clear signs of AI involvement (see Figures 14 and 15). This type of activity is similar to what we’ve seen in software as service (SaaS) platform misuse campaigns, such as phishing pages hosted on presentation builders or other legitimate content-sharing tools.

While attackers might leverage the AI functionality of these platforms in more powerful ways in the future, they are currently using these platforms primarily as a hosting service for malicious content.

Conclusion

In this article, we discussed web-based GenAI services and reviewed phishing attacks that misused them. We investigated examples of phishing pages from AI-powered website builders, and explored how criminals can use these builders to create phishing content more easily. Criminals have also misused AI-powered writing assistant services, but these platforms have been used primarily as a hosting service for malicious content with no clear signs of AI involvement.

Our telemetry reflects the growing adoption of GenAI applications and services, and we expect a corresponding increase in attacks that take advantage of GenAI as time passes.

Palo Alto Networks customers are better protected from the threats discussed in this article through the following products:

• Advanced URL Filtering and Advanced DNS Security identify known domains and URLs associated with this activity as malicious.

If you think you may have been compromised or have an urgent matter, get in touch with the Unit 42 Incident Response team or call:

• North America: Toll Free: +1 (866) 486-4842 (866.4.UNIT42)
• UK: +44.20.3743.3660
• Europe and Middle East: +31.20.299.3130
• Asia: +65.6983.8730
• Japan: +81.50.1790.0200
• Australia: +61.2.4062.7950
• India: 00080005045107

Palo Alto Networks has shared these findings with our fellow Cyber Threat Alliance (CTA) members. CTA members use this intelligence to rapidly deploy protections to their customers and to systematically disrupt malicious cyber actors. Learn more about the Cyber Threat Alliance.

Acknowledgments

Thanks to Peng Peng for his research into malicious chatbots, as well as Alex Starov and Jun Javier Wang for their suggestions.

Appendix

Additional screenshots of GenAI-related phishing pages we found in the wild are shown in Figures 16-22.