Why AI needs new security concepts

Why Are We Talking About AI Security?

#

Artificial intelligence is not a new phenomenon. For many years, it has been used in areas such as finance and insurance, for example for forecasting, fraud detection, or risk assessment.

What we are currently experiencing, however, is a new phase: today, AI is available to almost everyone and is taking on tasks that for a long time were reserved exclusively for humans.

This was made possible primarily by advances in neural networks. Although they have existed for decades, it was only through significantly more powerful hardware, larger amounts of data, and new training methods that deep, large-scale neural networks could be made practically usable. As a result, the capabilities of AI systems have expanded considerably.

A crucial next step was the development of Large Language Models (LLMs)—large models that were specifically trained for language understanding and text generation. For the first time, they make it possible to interact with AI systems using natural language. Communication increasingly feels dialog-based, almost as if one were speaking with a human counterpart. Modern versions are capable of processing complex relationships and performing multi-step reasoning.

However, development continues. Current AI systems are no longer limited to listening and responding; they are beginning to actively take on tasks: they access tools, use external data sources, control processes, or make autonomous decisions within predefined boundaries.

At this point at the latest, the question of security becomes central.

• Whom do we entrust with our data?
• What happens in the event of an error?
• And how can such systems be misused or compromised?

These questions are at the heart of any serious discussion of AI security.

Different Systems, Different Risks

#

AI is not the same as AI—different systems come with different risks.

Classical AI and ML systems
Examples include regression or decision trees. They are primarily used for prediction and classification and operate within clearly defined inputs and outputs.

Modern AI systems (ML, LLMs, generative models)
These systems are based on neural networks, often using transformer architectures. They recognize patterns and generate text, images, or other types of content. Their flexibility and use of natural language give rise to new security requirements.

AI extensions

• Plug-ins / Function Calling: Extend models with external tools and functions.
• RAG (Retrieval-Augmented Generation): Integrates external documents or data sources into responses.
• Agents: Perform tasks partially autonomously and make decisions within predefined boundaries.

As complexity and autonomy increase, security risks rise as well.

Why Does AI Need Its Own Security Models?

#

Traditional software is usually operated via a graphical user interface or the command line. Inputs and outputs are clearly structured and can be validated and secured relatively easily.
In addition, traditional programs operate deterministically: the same input produces the same output. This makes them relatively easy to test and control from a security perspective.

With neural networks, and especially Large Language Models, the situation is fundamentally different.
Their internal processing is highly complex, non-transparent, and only partially understandable for humans. As a result, new attack surfaces emerge.

Inputs are no longer processed strictly through fixed program logic, but through natural language. Users “speak” to the model, and this language is processed statistically rather than interpreted in a rule-based manner. This makes traditional security mechanisms such as fixed validation rules much harder to apply.

In addition, modern AI systems often integrate external data sources and tools, such as documents, APIs, or other systems. This expands both their functionality and the potential risks.

AI systems behave fundamentally differently from traditional software. As a result, traditional security models alone are no longer sufficient — there is a need for dedicated, adapted security concepts for AI applications.

While traditional applications are deterministic and controllable, AI systems produce probabilistic results that vary depending on context.

What Risks Is GenAI Specifically Exposed To?

#

Security organizations such as NIST and OWASP publish overviews of current risks related to AI systems, and generative AI systems in particular. In doing so, several threats stand out as especially significant.

• Prompt Injection
  The processing of inputs represents one of the largest attack surfaces. Through carefully crafted inputs, models can be manipulated into producing outputs that were not intended. In certain architectures, this can also influence system-level instructions or rules within the application.

• RAG-specific risks
  Systems using Retrieval-Augmented Generation significantly expand the attack surface.
  Attacks can occur not only through user inputs, but also through the documents and data sources themselves.
  Manipulated, outdated, or instructive content can influence the system’s behavior without this being immediately visible.

• Insecure Output Handling
  Not only the input, but also the output of a model can pose a security risk. This is particularly critical when AI-generated content — such as code or commands — is further processed or executed without verification.
  Unvalidated AI outputs must never be directly integrated into production systems, decision-making processes, or automations.

• Sensitive Data Leakage
  Generative models can unintentionally disclose sensitive information. This includes internal information, configuration details, or confidential data that may be exposed through inputs or outputs.

• Excessive Agency
  Agent-based systems introduce additional risks. Overly broad permissions, missing constraints, or insufficient error handling can lead systems to perform actions that were not intended or can be abused.
  The more autonomously an AI system operates, the greater the risk that errors or manipulation will have real-world impacts on processes, data, or customers.

• Training Data Poisoning
  For freely available or reused models, there is a risk of manipulated training or fine-tuning data, for example from open model ecosystems such as Hugging Face. Such targeted modifications can influence model behavior in the long term or introduce hidden weaknesses.

• Model Denial of Service (DoS)
  Through deliberately crafted usage patterns, AI models can be heavily strained, for example by extremely long or complex inputs. This can impact availability or significantly increase resource and cost consumption.

These risks demonstrate that AI security does not begin at the model itself, but with architectural decisions, controlled exposure, and clearly defined responsibilities.

What Common Risks Arise From This?

#

From the GenAI-specific risks described above, several cross-cutting root causes can be identified that commonly occur across many AI systems:

• Excessive trust
  AI outputs are assumed to be correct or safe without being sufficiently questioned or verified.

• Lack of validation
  Inputs and outputs of AI systems are not consistently checked, filtered, or constrained.

• Insufficient access control
  AI systems or connected tools have overly broad permissions.

• Lack of transparency
  Decisions, data sources, and system boundaries are only partially understandable for users and operators.

• Lack of separation of responsibilities
  Responsibilities between humans, AI, the application, and connected systems are not clearly defined.

Many security issues do not arise from the AI itself, but from architectural, design, and governance decisions.

Security as a Fundamental Principle

#

For AI systems, security should not be an afterthought, but a core design principle. Proven security concepts and recognized standards help support this approach.

• Adhere to recognized security standards
  Aligning with established frameworks — such as those from BSI, NIST oder OWASP — provides a reliable foundation for the secure operation of applications, including in the AI context.

• Least Privilege
  Systems, components, and connected tools should be granted only the permissions they actually require.
  For example, if only read access is needed, no write or modification permissions should be assigned.

• Zero Trust
  No input and no output is trusted by default.
  All data—both inputs and AI-generated outputs—must be validated, checked, and filtered where necessary.

• Separation of roles
  Access should be clearly separated according to roles and responsibilities.
  This can be implemented using RBAC (Role-Based Access Control) or — for more fine-grained and dynamic scenarios — ABAC (Attribute-Based Access Control).

• Monitoring & Logging
  Comprehensive monitoring and logging are essential for detecting misbehavior at an early stage.
  Inputs, outputs, and relevant decisions must be traceable so that systems can be monitored, analyzed, and continuously improved.

AI Models Are Only One Building Block

#

In an AI agent system, the model is a central component, but it always operates in conjunction with additional dependencies. Security therefore does not concern the model alone, but the entire architecture.

Important building blocks include, among others:

• the LLM runtime (execution environment)
• external data access and data sources
• the platforms, frameworks, and libraries in use

The safety-first principle should be applied consistently to these components as well, since they significantly influence the overall system’s attack surface.

Equally important is an inventory and transparency concept.
All models, dependencies, and configuration changes should be clearly documented and traceable. Only in this way is it possible to:

• identify which components are in use
• track changes
• revert to previous states if necessary

The security of an AI system emerges from the interaction of all its components — not from the model alone.

Governance, Compliance, and Auditing

#

In addition to technical measures, governance and compliance play a central role in the security of AI systems.
Clear policies, defined responsibilities, and documented processes ensure that AI not only functions technically, but is also used in a controlled, transparent, and compliant manner.

An important component of this is red teaming. In this approach, AI systems are deliberately tested to identify weaknesses at an early stage — not with the goal of causing harm, but to make risks visible before they can be exploited.

It is important to note that security issues do not necessarily require highly specialized expert knowledge. Many types of attacks arise from design or configuration flaws and can be identified using commonly available testing and analysis tools. Precisely for this reason, structured review processes, regular testing, and clear governance rules are essential for identifying and mitigating risks at an early stage.

Conclusion

#

Artificial intelligence extends existing software architectures with new capabilities — and, as a result, with new risks. These risks arise less from the models themselves than from their integration, usage, and control within complex systems.

Using AI securely therefore requires more than protecting individual components.
What matters are clear responsibilities, controlled access models, consistent validation, and transparency across all involved building blocks. Combined with governance, auditing, and regular reviews, AI systems can be operated responsibly and sustainably.