Security Insights

A lot of Canadian businesses exhaled when Bill C-27 died.

That exhale was a mistake.

For more than two years, Bill C-27 hung over Canadian privacy and technology law as the most significant reform effort in a generation. When it died on the Order Paper in early 2025, many businesses treated the news as a reprieve. The pressure to modernize, the thinking went, had been postponed indefinitely. Compliance could wait.

It could not. And the events of the past few weeks have made that clearer than ever.

How Bill C-27 Died

Bill C-27 died on January 6, 2025, when Parliament was prorogued following the resignation of Prime Minister Justin Trudeau. The bill had aimed to replace parts of Canada’s federal private sector privacy law, the Personal Information Protection and Electronic Documents Act (PIPEDA), and introduce the country’s first legislative framework for regulating high-impact artificial intelligence systems.

The bill bundled three distinct pieces of legislation together. The Consumer Privacy Protection Act would have replaced PIPEDA with stronger consent requirements, clearer rules around automated decision making, and significantly higher penalties for non-compliance. The Artificial Intelligence and Data Act represented Canada’s first attempt at comprehensive AI regulation, targeting systems used in commercial activity. A third act would have created an independent tribunal to hear appeals and enforce penalties, giving the Privacy Commissioner real enforcement teeth for the first time.

When Parliament prorogued, all three pieces of legislation died simultaneously. A snap federal election in the spring of 2025 pushed reform even further down the priority list, and there was no realistic path to reintroducing the bill in its original form.

The result was that Canada continued operating under PIPEDA, a law written in 2000 and largely unchanged since. For many businesses, that felt like the regulatory pressure had lifted. From experience working with Canadian organizations on privacy and security programs, that sense of relief was almost always misplaced. The obligations under PIPEDA never went away. What changed was that the law everyone expected to replace it simply was not there yet.

What the Government Just Launched

On June 4, 2026, Prime Minister Mark Carney launched AI for All, Canada’s new national AI strategy, at an event in Toronto alongside the Minister of Artificial Intelligence and Digital Innovation. The strategy is built around three priorities — trust, opportunity, and sovereignty — and commits the federal government to a five-year program of new legislation, investment, and public programs.

The trust component is the one that matters most for businesses thinking about privacy and compliance risk. The strategy explicitly commits to introducing new legislation, regulations, and standards to protect Canadians’ data, privacy, and children online. It specifically calls out concerns around surveillance pricing, deepfakes, and the inappropriate use of personal information, and commits to a child safety standard that the government intends to raise at this year’s G7 summit.

This is not a discussion paper or a consultation exercise. It is a stated government commitment to legislate, backed by a five-year, multi-billion dollar investment program targeting roughly $200 billion in additional economic growth and 250,000 new AI-related jobs.

For businesses that had quietly shelved their privacy modernization plans after Bill C-27 died, the timeline just moved back into view.

What the Incoming Legislation Is Expected to Include

The direction of upcoming legislation has already been signaled clearly by legal analysts tracking the file throughout 2025 and into 2026. A new federal private sector privacy statute, along with a companion bill establishing an enforcement tribunal, has been expected since the federal budget announcement late last year. That proposed statute is expected to include penalties of up to the greater of $25 million Canadian or five percent of global gross revenue — a dramatic increase over what PIPEDA currently allows.

Children’s privacy and AI-generated deepfakes have been identified publicly as priority areas for the new legislation. Data sovereignty — meaning where Canadian data is stored, processed, and controlled — is also expected to feature prominently.

For context, PIPEDA’s current enforcement mechanism leans heavily on the Privacy Commissioner issuing recommendations and public findings rather than binding penalties. The regime expected to follow AI for All is designed to function much more like the GDPR in Europe, with binding orders and financial penalties substantial enough to change executive behavior.

What Already Applies Right Now

While Parliament works through new legislation, two frameworks are already in force and already being actively enforced.

PIPEDA applies to virtually every private sector organization in Canada that collects, uses, or discloses personal information in the course of commercial activity. The Office of the Privacy Commissioner of Canada can launch investigations, issue public findings detailing non-compliance, and in serious cases refer matters to the Federal Court, which can order organizations to change their practices and award damages to affected individuals.

What has shifted recently is the Commissioner’s posture. Rather than relying primarily on recommendations and voluntary compliance, the Office of the Privacy Commissioner has become increasingly willing to pursue judicial remedies. A joint investigation into OpenAI, conducted alongside provincial privacy regulators in British Columbia, Alberta, and Quebec, concluded recently with findings that the company launched its product without having fully addressed known privacy issues. That investigation is a signal of where regulatory attention is heading, not an isolated case.

Then there is Quebec. Law 25 is fully in force, including the right to data portability that took effect in September 2024. It is enforced by Quebec’s Commission d’accès à l’information, with penalties of up to $10 million Canadian or two percent of global turnover, whichever is higher. Law 25 applies extraterritorially: any business that serves customers in Quebec or collects data from Quebec residents is in scope, regardless of where that business is headquartered.

Canada’s Privacy Commissioner, Philippe Dufresne, underscored this enforcement posture at the International Association of Privacy Professionals Canada Symposium in Toronto in May, where he announced new guidance on age assurance technologies and discussed the urgency of federal reform. The consistent message from regulators across the country has been that they are not waiting for new legislation before enforcing the obligations that already exist.

What This Signals for Canadian Businesses

The AI for All strategy reveals something beyond its specific legislative commitments. It signals that the federal government now treats AI governance as a matter of national sovereignty, not simply consumer protection. The language throughout the strategy around Canadian data, Canadian compute infrastructure, and Canadian oversight of AI systems is deliberate, and it reflects a broader anxiety about Canadian dependence on foreign cloud and AI infrastructure.

With no AI-specific statute currently in force following AIDA’s death alongside the rest of Bill C-27, organizations deploying AI in Canada in 2026 are navigating a patchwork of sector-specific guidance and general privacy law rather than a single clear standard. That patchwork is exactly what the incoming legislation is designed to replace. The practical question for any Canadian business using AI tools — whether that means a customer service chatbot, an internal productivity assistant, or a more sophisticated automated decision system — is whether governance work is happening now or will happen later under regulatory pressure.

From experience, the businesses that build privacy and AI governance structures proactively spend far less time and money than those that retrofit a program after an investigation or a customer complaint forces the issue.

What Canadian Businesses Should Do Now

The legislative landscape will keep shifting over the next year, and trying to predict the exact shape of the final bill is less useful than building the foundational practices that any version of the law will require.

Three actions matter most right now. First, every organization subject to PIPEDA needs a clearly designated individual accountable for privacy compliance. This is already a legal requirement, not a future one, and it is the first thing regulators ask about during any investigation.

Second, organizations using AI tools of any kind need a clear inventory of what personal data those tools touch, where that data is processed and stored, and whether vendor agreements reflect the organization’s obligations under Canadian privacy law. The OpenAI investigation is a useful illustration of how exposed even sophisticated technology companies can be when this groundwork has not been done.

Third, any business with customers or data subjects in Quebec should treat Law 25 as the practical floor for compliance today, not PIPEDA. Law 25 is stricter, fully in force, and being actively enforced with substantial penalties.

Bill C-27 may be dead, but the regulatory direction it represented is very much alive. The AI for All strategy makes that explicit. Businesses that spent the past year treating C-27’s death as a reason to wait are now working against a tighter timeline than they realize.

Related: What Shadow AI Is Doing to Canadian Businesses Right Now — how unsanctioned AI tools are creating PIPEDA exposure today, before any new legislation arrives.

Someone on your team is using an AI tool you did not approve.

They are pasting client emails into ChatGPT to get faster replies. Summarizing contracts with a free tool they found online. Running customer data through an app that saves them an hour every week. They are not doing it to cause problems. They are doing it because it works.

For small and medium-sized Canadian businesses without a dedicated IT or security team, this is happening right now, completely out of sight. That is Shadow AI, and it has become one of the fastest-growing data risks facing Canadian organizations today.

What Shadow AI Actually Means

Shadow AI refers to any artificial intelligence tool an employee uses without the knowledge or approval of the business. It follows the same pattern as Shadow IT — the long-standing problem of employees using personal apps and services for work purposes — but the scale and speed of AI adoption has made the risk significantly larger.

According to Verizon’s 2026 Data Breach Investigations Report, 15% of employees were classified as regular AI users on corporate devices in 2025. One year later, that figure reached 45%. That is a tripling in a single twelve-month window. A separate survey found that 66% of office professionals used AI tools despite believing it violated company policy.

For a business without anyone actively monitoring which tools are in use, that growth is completely invisible.

Where Business Data Is Actually Going

When an employee pastes client information into a free AI tool, that data leaves the organization’s environment. It travels to a server the business does not control, often located in another country, processed under terms of service the business owner has never read and almost certainly never agreed to on behalf of their clients.

The most common types of data employees feed into unsanctioned AI tools include customer information from sales and support teams, financial and legal documents including contracts, and internal business processes pasted in for help with a task.

For a law firm, that could be privileged client correspondence. For an accounting practice, it could be financial records. For a healthcare clinic, it could be patient information. In every one of those cases, the business remains legally responsible for what happens to that data, regardless of whether the owner knew it left the building.

The Canadian Compliance Exposure

Shadow AI is not just a security problem. For Canadian businesses, it is a legal one.

PIPEDA, Canada’s federal private sector privacy law, requires that personal information be protected regardless of where it is processed. If an employee sends client data to a US-based AI platform without the business’s knowledge, the organization may still bear legal responsibility if something goes wrong.

Quebec’s Law 25, which is fully in force and actively enforced, imposes stricter requirements. It mandates explicit consent for certain uses of personal data and places tighter restrictions around data transfers outside Quebec. Businesses that serve clients in Quebec are subject to these rules regardless of where the business itself is based.

IBM’s 2025 Cost of a Data Breach Report found that 20% of breached organizations were compromised through Shadow AI. When a breach involves a tool the business did not know existed, detection is slower and recovery is harder. For a small business without an incident response team, that gap can be the difference between surviving an incident and not.

This Is Not an Employee Problem

The instinct when reading this is often to frame it as a behaviour problem — employees using tools they should not be using, breaking the rules.

That framing misses the point.

From experience, employees turn to Shadow AI because the tools genuinely help them work faster and more effectively. If a business bans AI without offering an alternative, the problem does not go away. It just becomes invisible.

Research consistently shows a significant drop in unauthorized AI usage when approved alternatives are provided. The goal is not a crackdown. It is building a simple structure that gives employees a safe and sanctioned path to use these tools productively.

Three Steps Any Canadian Business Can Take This Week

These steps do not require a security team or a large budget. A business owner or office manager can action all three.

Start with a conversation, not a policy. Ask employees directly what AI tools they are using and what they are using them for. Frame it as a discovery exercise rather than a compliance audit. The answers will be more honest and the information more useful. Many business owners are surprised by what they find.

Draw a clear line around sensitive data. Before any formal policy exists, one simple rule closes a significant portion of the risk: client names, financial information, legal documents, and anything confidential should never go into an AI tool the business has not approved. Communicating that boundary clearly and consistently is a meaningful first step.

Write a one-page AI use policy. It does not need to be long or technical. It needs to answer three questions: which tools are approved, what types of data should never be entered into an AI prompt, and who to contact when someone is unsure. One page, shared with the team, is enough to establish accountability and reduce exposure.

Shadow AI is not a problem reserved for large organizations with complex systems. It is happening in Canadian businesses of every size, right now, through tools that may never appear in an IT audit. The organizations that get ahead of it are the ones that start the conversation before something goes wrong.

For enterprise security teams, the problem is larger in scale but identical in nature. The same starting point applies: find out what is in use, establish clear data boundaries, and give people a legitimate path forward.

Related: Canada’s Privacy and AI Reckoning — the regulatory landscape Shadow AI is operating in — and Your Firewall Is Fine. Your Employees Are the Problem. — why the human layer is always the first gap attackers find.

The security industry has a complicated relationship with AI right now. On one side, vendors are promising that machine learning will transform how organizations detect threats, prioritize vulnerabilities, and respond to incidents. On the other, practitioners who have lived through a few hype cycles are watching closely to see what actually holds up.

The honest answer is somewhere in the middle. AI has made meaningful contributions to both vulnerability management (VM) and incident response (IR). It has also given some organizations a false sense of progress while their foundational program gaps remain unaddressed.

Where AI Is Actually Delivering in Vulnerability Management and Incident Response

Vulnerability management used to be a largely periodic exercise. Scan, export, sort by CVSS score, patch what you can, repeat. The problem with that model is that it treats all critical vulnerabilities as equal and ignores the reality that exploitability depends heavily on context: network exposure, asset criticality, existing compensating controls, and active threat intelligence.

AI-driven VM platforms have made genuine progress here. Continuous attack surface monitoring means organizations now have dynamic visibility rather than a quarterly snapshot. Risk-based prioritization models factor in real-world exploitability, threat actor behavior, and business context, helping teams focus on the vulnerabilities that actually represent exposure rather than just high CVSS numbers. Correlation between internal asset data and external threat intelligence has also improved significantly, giving analysts context that used to take considerable manual effort to assemble.

On the IR side, the gains are equally concrete. AI-assisted triage has compressed the time between alert generation and meaningful investigation. Behavioral anomaly detection is catching lateral movement and credential abuse patterns that rule-based detection consistently missed. Automated containment on well-scoped playbooks can reduce dwell time in ways that simply were not possible with manual processes alone.

For security teams managing large environments with limited headcount, these capabilities are not optional enhancements. They are operationally significant.

Where the Fundamentals Still Carry the Weight

Here is where a more direct perspective is warranted: AI tools are force multipliers. That means they amplify what already exists. A well-run VM program with clean asset data, defined ownership, and consistent patch processes becomes faster and sharper with AI. A program built on incomplete inventory, unclear ownership, and inconsistent execution becomes faster at producing noise.

The fundamentals that AI cannot replace:

Asset inventory. This is the most unglamorous work in security and consistently the most neglected. If the CMDB is incomplete or stale, the attack surface is unknown regardless of what tool is scanning it. Every VM and IR program is only as good as the accuracy of what is in scope.

Defined severity tiers and escalation paths. During an active incident, ambiguity is expensive. Who declares a P1? Who calls legal? Who communicates to the board? These decisions should be made in advance, documented clearly, and practiced. No AI platform makes that decision under pressure.

Tabletop exercises. IR is a team capability, not an individual one. Runbooks only work if the people executing them have walked through the scenarios together. Teams that run regular tabletops respond measurably better when real incidents happen. The mechanics are familiar, the communication patterns are practiced, and the decision-making is faster.

Patch discipline. Risk-based prioritization is a meaningful improvement over blind CVSS sorting. But it requires organizational follow-through to mean anything. Reprioritization models fail when the underlying patch process is inconsistent or politically difficult to execute.

Runbook maintenance. Runbooks decay. Systems change, team members turn over, tools get replaced. A runbook that was accurate 18 months ago may contain broken escalation paths, outdated tool references, and contacts who no longer work at the organization. Regular review and validation is operational hygiene that AI-assisted detection cannot compensate for.

The Risk of Dashboard Confidence in AI-Powered Security Programs

The scenario that concerns practitioners most is what could be described as dashboard confidence: an organization that has modern, well-integrated security tooling, reasonable metric trends, and a general sense that the program is mature, but has never pressure-tested whether it would actually hold under real incident conditions.

AI platforms are good at producing dashboards. They are not good at flagging that the escalation chain has never been tested, that the regulatory notification timeline is unclear, or that the IR team has never executed a real containment together. Those gaps only surface when something breaks.

Regulatory obligations deserve particular attention. Depending on the sector and jurisdiction, a confirmed breach may trigger mandatory notification timelines measured in hours or days. AI can help detect and scope an incident faster. It cannot draft a regulatory communication, make legal determinations about reportability, or manage crisis communications with affected parties. Those require human judgment, defined process, and in many cases legal counsel involvement that should be scoped before an incident, not during one.

The Right Mental Model for AI in Security

The most useful framing for AI in security is this: think of it as a senior analyst operating at machine speed. It can surface patterns, triage signals, prioritize work, and accelerate response. What it cannot do is compensate for a program that was built on shaky foundations.

Organizations that get the most value from AI-powered VM and IR tools are the ones that already had reasonable fundamentals in place before they adopted them. They knew their assets. Their runbooks were current. Their escalation paths were practiced. AI made those programs meaningfully better.

The organizations struggling are the ones that adopted AI tooling as a substitute for program discipline rather than an extension of it.

A useful self-assessment: if the AI layer was removed from a security program tomorrow, how would the team perform? If that question is uncomfortable, the next investment priority is probably not another platform. It is the process work that was deferred to make room for it.

The basics are not new. They are not exciting. But they are consistently what separates organizations that manage incidents well from organizations that get managed by them.

Related: Why Your DR Plan May Fail When You Need It Most — the same discipline gaps that weaken IR programs also show up in disaster recovery — and Your Business Isn’t Too Small to Be Hacked for the SMB lens on these same risks.

Anthropic this week announced two significant security features for its Claude Code platform: a real-time security-guidance plugin and a self-hosted sandbox environment. The announcements, made at the company’s Code w/ Claude event in London, mark a meaningful expansion of AI-assisted security tooling directly into the developer workflow. For application security teams, the release raises both opportunities and important questions about where AI fits in a mature security program.

What Anthropic Announced: The Security Plugin and Self-Hosted Sandbox

The security-guidance plugin is available to all Claude Code users at no additional cost and can be installed through the built-in plugin marketplace. Once active, it operates automatically across three stages of development.

At the file edit stage, the plugin runs a fast, deterministic pattern match with no AI model call, flagging dangerous code constructs such as eval(), os.system(), child_process.exec(), and DOM injection vectors. Because this layer requires no inference, it adds no usage cost to the developer.

After AI-generated changes are made during a coding session, a second review pass examines the full diff and surrounding context to catch vulnerabilities introduced by the model itself. This is a notable design decision: the system audits its own output rather than assuming AI-generated code is safe.

At commit time, a deeper agentic review reads surrounding callers, sanitizers, and related files to reduce false positives before code is pushed. The combined approach targets approximately 25 high-risk vulnerability classes including:

• SQL injection
• Command injection
• Cross-site scripting (XSS)
• Hardcoded credentials and API keys
• Insecure deserialization
• Improper input validation

Anthropic reports that internal use of the plugin produced a 30 to 40 percent reduction in security-related pull request comments, suggesting meaningful improvement in code quality at the point of creation.

The Self-Hosted Sandbox

The self-hosted sandbox, currently in public beta, allows Claude Managed Agents to operate within a user-controlled environment connected to private MCP servers. Tool execution runs on the organization’s own infrastructure or through providers such as Cloudflare, Daytona, Modal, or Vercel, while Anthropic’s infrastructure handles orchestration and context management. The practical result is that sensitive files, repositories, and data remain within the organization’s network perimeter, subject to its own audit logging and security policies.

The Claude Code Security Initiative: Broader Context

The security-guidance plugin is part of Anthropic’s larger Claude Code Security initiative, which launched as a research preview in February 2026 and expanded to an Enterprise public beta in late April 2026. The broader platform goes beyond regex-based pattern matching, using advanced AI reasoning to conduct codebase-wide scans that trace data flows across files, run adversarial verification passes on findings, and propose targeted patches for human review. Anthropic has reported the system identified over 500 previously unknown high-severity vulnerabilities in open-source codebases during internal testing.

What This Means for Application Security Programs and AppSec Teams

The value of this tooling is real and should not be dismissed. Catching injection flaws, hardcoded secrets, and insecure API usage at the moment of creation is substantially better than discovering them during code review, penetration testing, or after a breach. Shifting security left has long been a goal of mature AppSec programs, and AI-assisted tooling at this layer is a credible step toward that goal.

However, security teams should be clear-eyed about what the plugin does not address.

Pattern-based vulnerability detection, however well implemented, operates on known categories of risk. It cannot perform threat modeling, assess architectural decisions, or reason about how a combination of individually minor issues might be chained into a significant attack path. It does not replace penetration testing, red team exercises, or the adversarial thinking that experienced security practitioners bring to a program.

Regulated environments will also need to evaluate how the plugin and sandbox fit within existing compliance requirements. The self-hosted sandbox addresses some data residency concerns, but organizations should assess the tooling against their specific control frameworks before broad deployment.

The most productive framing for security leaders is to treat the plugin as a force multiplier rather than a substitute. It raises the baseline quality of code entering the pipeline and reduces low-signal noise in security review. That creates capacity for security teams to focus on higher-judgment work: architecture reviews, threat modeling, incident response, and adversarial testing that no automated tool can replicate.

Getting Started

The security-guidance plugin is available now for all Claude Code users. Installation is handled through the /plugins command within Claude Code. The self-hosted sandbox is available in public beta. Organizations interested in the broader Claude Code Security platform, including deep codebase scanning and enterprise controls, can access it through Anthropic’s Enterprise tier.

The Bottom Line

Security tools reduce mistakes. Security professionals reduce risk. The organizations that understand the difference will benefit the most from AI-assisted development.

Related: The Rise of AI-Assisted Attacks: What Security Teams Need to Know — the threat context that makes tools like this necessary — and our Code Review service for teams that want a human expert behind the AI scan.

There is a technology myth that persists in boardrooms and budget meetings across every industry: that cybersecurity is fundamentally a technology problem. Buy the right firewall. Deploy the right antivirus. Subscribe to the right threat monitoring service. Problem solved.

It is not solved. Not even close.

The most sophisticated security stack in the world will not protect an organization whose employees do not know how to spot a phishing email. And the data is unambiguous on this point. Ninety-five percent of successful cyberattacks involve human error as a contributing factor. Not a software vulnerability. Not a misconfigured server. A person making a decision they were not equipped to make safely.

This is the gap that attackers exploit most reliably, most profitably, and most often.

The Human Layer Is the Weakest Link

Every organization has layers of defense. Firewalls, endpoint protection, email filtering, multi-factor authentication. Security teams invest significantly in these controls, and for good reason. They work.

But every one of those technical controls has a human being sitting behind it. And humans can be manipulated in ways that technology cannot.

A phishing email that bypasses every filter still needs a person to click it. A fraudulent wire transfer request still needs a person to authorize it. A credential harvesting page still needs a person to type their password into it. The attacker does not need to break through the technology. They need to convince one person to open a door.

What This Looks Like in Practice

Security awareness assessments conducted across organizations of varying sizes consistently surface the same patterns.

Employees receive phishing simulation emails and click them at rates that would surprise most executives. Not because those employees are careless or unintelligent, but because modern phishing attempts are sophisticated, personalized, and designed by people who study human psychology professionally.

Password hygiene remains a persistent problem. Employees reuse passwords across personal and professional accounts. They share credentials with colleagues for convenience. They use passwords that are technically compliant with policy but trivially guessable in practice.

Social engineering extends beyond email. Attackers call help desks impersonating employees and request password resets. They send text messages impersonating executives with urgent payment requests. They exploit the natural human instinct to be helpful, to avoid conflict, and to respond to authority.

In each of these scenarios, the technical controls are largely irrelevant. The human being is the attack surface.

Why Training Alone Is Not Enough

Many organizations respond to this reality by deploying annual security awareness training. A one-hour module, a completion certificate, a checkbox on the compliance audit.

This approach is better than nothing. It is not enough.

Research consistently shows that security awareness training produces measurable behavior change only when it is ongoing, scenario-based, and reinforced regularly. A single annual module does not build instinct. It builds familiarity with a module.

Effective security awareness programs treat the human layer with the same rigor applied to technical controls. They run simulated phishing campaigns throughout the year. They provide immediate, contextual feedback when an employee clicks something they should not have. They create a culture where reporting suspicious activity is encouraged and rewarded rather than stigmatized.

The goal is not compliance. The goal is behavior change at the individual level, sustained over time.

What Organizations Should Actually Do

Building a resilient human layer does not require an enormous budget. It requires consistency and the right approach.

• Run phishing simulations regularly. Not once a year. Quarterly at minimum, with varied scenarios that reflect current attack trends. Employees who are regularly tested develop sharper instincts than those who encounter a simulation once and forget about it.
• Make reporting easy and consequence-free. One of the most damaging dynamics in organizational security is the culture of blame. When employees fear consequences for clicking a phishing link, they stop reporting incidents. Unreported incidents become uncontained breaches. Create an environment where reporting is the expected and valued response.
• Tailor training to role and risk level. The threats facing a finance team are different from those facing a customer service team. Generic training produces generic awareness. Role-specific training produces relevant behavior change.
• Extend awareness beyond email. Train employees to recognize vishing, smishing, and in-person social engineering attempts. The attack surface is not limited to the inbox.
• Measure and iterate. Track click rates on phishing simulations over time. Measure reporting rates. Identify which teams or individuals need additional support. Treat human awareness as a metric, not a checkbox.

The Bottom Line

Technology will continue to improve. Attackers will continue to adapt. The one constant in this equation is the human being making decisions under pressure, with incomplete information, in an environment designed to manipulate them.

Organizations that invest in their human layer with the same seriousness they apply to their technical controls will be meaningfully more resilient than those that do not. The firewall matters. The person sitting behind it matters more.

Related: The Rise of AI-Assisted Attacks — why AI has made phishing dramatically harder to detect — and Identity Is No Longer an IT Problem for what strong MFA and access controls actually look like in practice.

AI adoption is accelerating faster than most organizations can govern it. And identity security is where that gap becomes dangerous.

That was not just a talking point at the Okta AI Identity Summit in Toronto last week. It was the thread running through every session, every panel, and every conversation in the room. The organizations leading on AI are not simply moving fast. They are building identity as the foundation first, and everything else on top of it.

The question is no longer whether AI will change how organizations operate. It already has. The question is whether identity infrastructure can keep pace with the speed at which AI is being adopted, and what happens to the organizations where it cannot.

The Three Questions Most Organizations Cannot Answer

Every organization deploying AI agents — whether for automation, decision support, or customer interaction — should be able to answer three questions clearly and immediately.

Where are your AI agents operating? This sounds straightforward. In practice, many organizations have AI tools running across multiple departments, deployed by different teams, with no centralized visibility into what those agents are doing or where they are active.

What systems and data can they access? AI agents that can query databases, send communications, process transactions, or interact with third-party systems carry real access risk. If that access has not been explicitly scoped and governed, the organization does not have an AI problem. It has an identity problem wearing an AI mask.

Who is accountable when they act autonomously? This is the question that tends to produce the longest silence. When an AI agent makes a decision that results in a data exposure, an unauthorized transaction, or a compliance violation, accountability cannot be an afterthought. It needs to be designed into the governance model from the beginning.

Most organizations cannot answer all three questions confidently. That is not a technology gap. It is a governance gap. And it does not get smaller as AI scales. It gets larger.

Why Identity Has Become a Strategic Control

For most of the history of enterprise IT, identity and access management lived in the back office. It was an IT function — necessary, largely invisible, and rarely discussed at the executive level unless something went wrong.

That positioning is no longer appropriate for the environment organizations are operating in today.

When AI agents are acting on behalf of the organization — making decisions, accessing systems, and interacting with data autonomously — identity is the mechanism that determines what they can do and what they cannot. It is the line between a well-governed AI deployment and an ungoverned one.

The organizations at the summit that are ahead of this problem share a common characteristic. They have elevated identity from an operational IT concern to a strategic business control. Their identity programs are not reactive. They are built to scale ahead of adoption, not scramble to catch up after it.

That shift in positioning — from back-office function to strategic foundation — is what separates organizations that will navigate AI governance well from those that will not.

What Zero Trust Has to Do With It

Zero Trust as a framework has been discussed extensively in security circles for years. In the context of AI agents, it takes on renewed practical relevance.

The core principle of Zero Trust is that no user, device, or system should be trusted by default, regardless of where it sits relative to the network perimeter. Every access request should be verified, every time, based on identity and context.

Applied to AI agents, this means every agent should have a defined identity. Every action that agent takes should be logged and attributable. Access should be granted on a least-privilege basis, scoped precisely to what the agent needs to perform its function and nothing beyond that.

This is not a theoretical framework for future consideration. Organizations deploying AI agents today without this governance model in place are accumulating risk with every deployment.

The Governance Gap Is the Real Risk

The conversation at the summit kept returning to the same underlying tension. AI adoption is being driven by business units, by product teams, by leadership mandates to move faster and do more with less. Identity governance is being driven by security and IT teams who are, in many cases, finding out about AI deployments after the fact.

That gap — between the speed of adoption and the maturity of governance — is where the real risk lives.

It is not that AI is inherently dangerous. It is that AI without identity governance is ungoverned. And ungoverned access, whether human or machine, is the foundation of most significant security incidents.

The organizations that will have a competitive advantage as AI matures are not necessarily the ones moving fastest. They are the ones building the governance infrastructure to move fast safely. Identity is that infrastructure.

Where to Start

For organizations looking to close the governance gap, the starting point is visibility. Before any controls can be implemented, there needs to be a clear picture of what AI agents are deployed, what access they have been granted, and what activity they are generating.

From there, the work is familiar to anyone who has run an identity program: define roles and access scopes, implement least privilege, establish logging and monitoring, and assign accountability clearly.

The difference in the AI context is the urgency. Every week that passes without this governance in place is a week of compounding exposure. The conversation is just getting started. The time to build the foundation is now.

Related: Zero Trust Is Not a Product — It’s a Strategy — identity is the control plane Zero Trust is built on — and The Rise of AI-Assisted Attacks for why identity is increasingly the primary attack surface.

There is a persistent and dangerous myth circulating among small business owners: that cybercriminals only target large organizations. Banks. Hospitals. Multinational corporations. The reality, as security professionals see it in practice, is far less forgiving.

Small and mid-sized businesses are not beneath the notice of attackers. In many cases, they are the preferred target, precisely because their defenses are weaker, their response capabilities are limited, and the path of least resistance runs straight through them.

The statistics bear this out. Nearly 60% of small businesses that experience a significant cyberattack close within six months. Not because the attack was uniquely sophisticated, but because the aftermath — the downtime, the recovery costs, the reputational damage, and the regulatory exposure — is more than most small operations can absorb.

The Assumption That Gets Businesses Breached

The most common gap that surfaces in security risk assessments is not a technical one. It is a mindset one.

Business owners who have never experienced a breach often operate under the assumption that their size makes them invisible. That assumption is wrong, and it is becoming more wrong every year.

Automated scanning tools — the same kind used by security researchers and malicious actors alike — sweep the internet continuously. They do not distinguish between a Fortune 500 company and a ten-person accounting firm. They look for open doors. And when they find one, they walk through it.

The businesses most at risk are not necessarily the least informed. They are often simply the ones who kept pushing the question of security to the next quarter, the next budget cycle, the next year.

What the Gaps Actually Look Like

Risk assessments conducted across small and mid-sized organizations consistently surface the same categories of exposure, regardless of industry.

• Access management is almost always underdeveloped. Employees who left the company months ago may still have active credentials. Shared passwords are common. There is rarely a clear picture of who has access to what and why.
• Patch management is another consistent weak point. Known vulnerabilities in operating systems, software, and network devices go unaddressed for months, sometimes years. These are not zero-day exploits. They are published, documented vulnerabilities with freely available exploit code.
• Backup and recovery processes, where they exist, are frequently untested. A backup that has never been restored is not a recovery plan. It is a file. Organizations discover this distinction at the worst possible moment.
• Human awareness remains the most exploited entry point across every sector. Staff who have not received regular, practical security training are significantly more likely to click a phishing link, hand over credentials, or authorize a fraudulent payment. This is not a failure of intelligence. It is a failure of preparation.

The Cost of Doing Nothing

Each of the gaps described above is addressable. None of them require enterprise budgets or dedicated security teams. What they require is awareness, prioritization, and a structured approach to identifying which risks matter most for a given organization.

The cost of not addressing them, however, can be terminal. Ransomware payments. Regulatory fines. Lost client contracts following a data breach notification. The reputational damage of being the business that got hacked. For many small organizations, any one of these outcomes is enough to force closure.

Where to Start

The most effective starting point for any small or mid-sized business is a structured risk assessment. Not a generic checklist downloaded from the internet, but an honest evaluation of the specific environment: the systems in use, the people who use them, the vendors with access, and the processes that hold it all together.

A proper risk assessment does not just identify problems. It prioritizes them, explains their business impact in plain language, and provides a clear path forward.

For business owners who have been putting this conversation off, the right time to have it is before an incident. Not after.

Related: Why Your DR Plan May Fail When You Need It Most — a deeper look at the backup and recovery gaps that leave businesses unable to recover — and take the free SMB Security Scorecard to see where you stand across 7 categories in 2 minutes.

The threat landscape has shifted — and most small businesses haven’t gotten the memo.

We’ve been conducting risk assessments across organizations of different sizes for several years. In the last 12 to 18 months, something has changed. The attacks we’re seeing evidence of aren’t just more frequent — they’re fundamentally smarter. And the reason is straightforward: attackers now have access to the same AI tools that the rest of us use every day.

This isn’t a distant, enterprise-level problem. It’s showing up in the assessments we run on small and mid-sized businesses right now.

What AI-Powered Attacks Actually Look Like

To be specific — “AI-powered attacks” can sound abstract.

Phishing has always been the most common entry point — it still is. But AI has removed the tells we used to rely on. Bad grammar, awkward phrasing, generic greetings — gone. Attackers now use large language models to generate personalized, perfectly written emails that reference your company name, your CEO’s writing style, even recent news about your business.

Then there’s voice cloning and deepfakes. This isn’t science fiction. Scammers are using AI-generated audio to impersonate executives and authorize fraudulent wire transfers. A CFO gets a voicemail that sounds exactly like the CEO asking them to process an urgent payment. That call never happened.

Finally, reconnaissance is now automated. Finding your exposed ports, checking for unpatched systems, identifying which of your employees reused passwords — attackers can run these scans in minutes. What used to take days of manual work now happens before your morning coffee.

Why SMBs Are Especially Vulnerable

There’s a common assumption that cybercriminals focus on large corporations. It’s wrong.

Small and mid-sized businesses are often seen as easier targets precisely because their defenses haven’t kept pace. Many are still running security playbooks built for threats from five years ago. And when a breach happens, the impact can be devastating — nearly 60% of small businesses that suffer a major cyberattack close within six months.

The resource gap is real. But the answer isn’t to spend more. It’s to spend smarter and close the right gaps first.

What You Should Actually Do

Based on what we see in risk assessments, here are the highest-impact steps that make a real difference:

• Train your people — consistently. Not a one-time onboarding module. Regular, scenario-based security awareness training. Ninety-five percent of breaches involve human error. AI makes those human moments harder to catch, so your team needs to be sharper.
• Move to phishing-resistant MFA. Standard SMS-based authentication is no longer sufficient. Passkeys and hardware security keys are significantly harder for attackers to bypass, even with AI-generated phishing pages.
• Build a financial verification protocol. Any request involving a wire transfer, payment change, or sensitive authorization should require a secondary verbal confirmation — a quick call to a known number, not a reply to the email. This single control has prevented countless business email compromise attacks.
• Know where your real gaps are. This is where a proper risk assessment becomes invaluable. Not a generic checklist, but an honest look at your specific environment — your systems, your people, your vendors, your processes. You can’t defend what you don’t know is exposed.

The Bottom Line

The same AI tools making attackers faster and more convincing are also available to defenders. But the window to close that gap doesn’t stay open forever.

If you’re a business owner or leader wondering whether your current defenses are still fit for purpose, that’s the right question to be asking — and the right time to get an answer.

Related: The Rise of AI-Assisted Attacks: What Security Teams Need to Know — an earlier deep dive on the same AI threat landscape — and Your Firewall Is Fine. Your Employees Are the Problem. for why phishing succeeds even when defenses are in place.

The security community has spent years talking about what the next generation of threats would look like. A lot of us assumed AI would help defenders more than attackers first. The reality is messier than that. From nation-state groups to ransomware operators, attackers are already using AI tools as part of how they work.

How Attackers Are Using AI Today

The most immediate impact is in social engineering. Phishing emails that used to have obvious grammar mistakes now look like they came from a real person in your company. AI lets attackers write convincing, personalized emails at scale, tailored to your industry, your role, even your writing style if they've scraped your LinkedIn.

Beyond phishing, AI is being used to:

• Accelerate reconnaissance — automated tools can ingest large amounts of OSINT data and surface actionable targets faster than any human analyst.
• Generate polymorphic malware — code that rewrites itself to evade signature-based detection, a task that once required significant skill but is now more accessible.
• Craft deepfake audio and video — used in BEC and executive impersonation attacks. Several high-profile cases in 2024 involved AI-generated voice calls impersonating CFOs.
• Automate vulnerability chaining — AI models can reason across known CVEs to identify exploit chains that human attackers might miss.

What Defenders Should Be Doing Right Now

• Retrain your phishing awareness programs. The old "look for spelling mistakes" heuristic is dead. Focus on security awareness training that teaches users to verify requests through out-of-band channels, especially for financial transactions or credential changes.
• Invest in behavioral detection. Signature-based controls alone won't catch AI-generated malware. EDR solutions with strong behavioral analytics become critical.
• Tighten identity controls. AI-assisted attacks often succeed at the initial access stage. MFA implementation, phishing-resistant credentials (FIDO2), and conditional access policies reduce the blast radius significantly.
• Treat AI as a defender's tool too. SIEM platforms and UEBA solutions are incorporating AI-driven anomaly detection. The NIST AI Risk Management Framework also provides guidance on managing AI risks across your organization. Make sure your team knows how to tune and trust these signals without alert fatigue.
• Conduct tabletop exercises that include AI scenarios. Test your incident response against a deepfake BEC or AI-assisted credential stuffing campaign before the real thing happens.

The Bottom Line

AI hasn't changed what attackers are after. They still want access, money, or disruption. What it has changed is the speed, scale, and quality with which they go after it. Security teams that update how they detect, train, and respond will be in a much better spot than those still using the same playbooks from five years ago.

Related: Hackers Are Using AI. Is Your Business Keeping Up? — the same threat landscape through an SMB lens, with the highest-impact steps to take right now.