How to Evaluate AI Tool Security: A Guide for Ops and IT Teams
AI tools create security and privacy risks that don't exist with traditional SaaS software — and most security review processes haven't caught up. When you buy a project management tool, your data is stored in a database. When you buy an AI tool, your data may be used to train models, processed through third-party foundation model APIs, stored in vector databases with different access controls, and used to generate outputs that combine your proprietary information with general knowledge.
Each of those differences creates risk. This guide covers how to assess it.
The Unique Security Questions for AI Tools
Traditional security questionnaires ask the right questions for traditional SaaS. For AI tools, you need to ask additional questions:
Is your data used to train models?
This is the most important question. Some AI vendors use customer data to fine-tune their models. If a model is trained on your proprietary documents, that information could in theory influence outputs for other customers.
Ask the vendor explicitly: "Is any data submitted through your product used to train, fine-tune, or improve your AI models?" Then read their terms of service and privacy policy to see if the written policies match the verbal answer. They often don't.
Which foundation model APIs does the tool use?
Most AI SaaS tools are built on top of foundation models — typically from OpenAI, Anthropic, Google, or Cohere. Your data is transmitted to these APIs as part of the tool's operation. Ask the vendor:
- Which AI providers do they use?
- Under what terms is your data processed by those providers?
- Are those providers covered under the vendor's DPA?
OpenAI, Anthropic, and Google all have enterprise agreements that include data processing terms and opt-out options for model training. Verify that the vendor has these agreements in place.
How is AI output cached or stored?
AI-generated content that references your data may be cached for performance reasons. Ask: Where is AI output stored? For how long? Who can access it? Is it associated with your account in a way that creates access control requirements?
What are the data retention policies for prompt history?
Every query sent to an AI tool is a prompt. Those prompts may contain sensitive information — customer names, financial data, proprietary business logic. Ask the vendor how long prompts are retained, who can access them, and whether they can be deleted on request.
Compliance Certifications: What Applies to AI Tools
SOC 2 Type II: Still the baseline. Required for any AI tool that processes sensitive business data. Note that SOC 2 was designed before generative AI existed — the controls may not specifically address AI model training data practices. Ask whether the AI-specific data handling practices are in scope for their most recent audit.
ISO 27001: More commonly required for European customers. Ask for the certificate and the scope statement to understand which systems and processes are covered.
GDPR compliance: AI tools present specific GDPR challenges. If personal data (customer names, contact info, employee data) flows through an AI tool's prompts or context, you need to understand the lawful basis for processing, the data minimization practices, and the right to erasure obligations.
HIPAA Business Associate Agreement (BAA): If your use case involves any protected health information, you need a signed BAA. Many AI vendors don't offer BAAs because HIPAA compliance for AI model processing is technically complex. Do not process PHI through any AI tool without a signed BAA.
FedRAMP: Required for US government customers. Very few AI vendors have FedRAMP authorization; this is a near-disqualifying requirement for most AI tool vendors selling to government agencies.
Data Handling Red Flags
These are signals that should trigger deeper scrutiny or disqualification:
- Vague terms of service language around "improving services": This often means data is used for model training. Ask for clarification in writing.
- No DPA available or available only on request with a long review process: Enterprise-grade vendors have DPAs ready. Difficulty obtaining one suggests immature privacy practices.
- Single-tenant architecture not available: For highly sensitive data, single-tenant deployments isolate your data from other customers. Many AI tools only offer multi-tenant SaaS. Assess whether that's acceptable for your data classification.
- No audit logs for AI queries: You should be able to see what data was sent to the AI and when. If the vendor can't provide this, your ability to investigate a data incident is severely limited.
- Foundation model provider not disclosed: Any vendor unwilling to disclose which AI models process your data is hiding information that's material to your security assessment.
The Model Training Opt-Out
Most major foundation model providers offer enterprise API agreements that include opt-out of data being used for model training. Verify that any AI vendor you work with has this opt-out in place with their underlying model provider. This is table stakes.
If the vendor uses OpenAI APIs under a standard (non-enterprise) API agreement, your data may be used to improve OpenAI's models by default. The enterprise agreement includes a contractual guarantee that data is not used for training. This is not a hypothetical concern — it's a documented difference in API terms.
Security Controls Specific to AI Features
For tools with significant AI features, assess:
Prompt injection protection: AI systems can be manipulated by malicious inputs to produce unintended outputs. Ask how the vendor protects against prompt injection attacks, especially if users can interact with AI in ways that might expose other customers' data.
Output filtering: Does the tool filter AI outputs to prevent generation of sensitive, harmful, or confidential content? This matters both for data security and for compliance in regulated industries.
Access controls on AI features: Can you restrict which users can access AI features? If AI features have access to sensitive documents or customer data, the access control model matters as much as the AI model itself.
Building a Fast AI Security Assessment
For each AI tool under evaluation:
- Pull the current terms of service and privacy policy — search for "model training," "improve," and "your data"
- Request the DPA and check the sub-processor list for foundation model providers
- Ask the vendor three direct questions in email: (a) Is customer data used for model training? (b) Which AI providers process my data? (c) Do you have a model training opt-out in place with those providers?
- Request the most recent SOC 2 report summary and note whether AI-specific practices are in scope
- Check their security page for documentation of data handling practices
This process takes 2-3 hours per vendor and should be a standard part of any AI tool evaluation. Teams using Trackr to evaluate AI tools can get a preliminary assessment of the vendor's AI sophistication, security posture signals, and integration depth as a starting point before the deeper security review.
Bottom Line
AI tool security isn't dramatically harder than traditional SaaS security — it requires asking different questions. The core issues are model training data practices, foundation model sub-processing disclosures, and prompt retention policies. Get answers to these in writing before you buy, and ensure they match the terms of service. Most AI vendors who are operating responsibly will have clear answers; those who don't are flagging a risk you should price in.
Trackr automates SaaS tool research. Submit any tool URL and get a scored 7-dimension report in under 2 minutes. Start free →