Asia AI Association

Article

How to evaluate an AI vendor: questions to ask

A practical guide to evaluating an AI vendor: data handling, security, fairness testing, contract terms and a simple trial, with questions to ask.

Published Updated 607 words

Short answer: Before buying an AI product, ask the vendor about six areas: what the product does and does not do, where your data goes, how it was tested, how it is secured, what the contract says about responsibility, and how you can leave. Ask for answers in writing, and have someone with relevant expertise check them. A polished demo shows what the tool can do at its best, and it is not evidence of how it behaves with your data.

What should we define before talking to vendors?

Write down the problem, the people affected, the data involved and what a good outcome looks like. Decide who in your organisation will own the tool. Without this, vendors define success for you.

What should we ask about the product?

What should we ask about data?

If personal data is involved, your obligations under the PDPA still apply when a vendor processes it for you, so your contract should address protection, retention and breach reporting. The PDPC's Guide to Managing Data Intermediaries explains the obligations when you outsource data processing, and its guidelines on the Transfer Limitation Obligation (section 26 of the PDPA) cover sending personal data outside Singapore.

What should we ask about security and assurance?

Ask what security controls and independent certifications the vendor holds, and what they cover. Certifications such as ISO/IEC 27001 or SOC 2 reports relate to security management and controls, not to whether an AI model is fair or accurate. Read each certificate's scope statement and ask which services and sites it covers. Ask how incidents are reported to customers and how quickly.

What should we ask about governance?

If you are a regulated financial institution, additional expectations on third-party arrangements and AI risk may apply. MAS's Guidelines on Outsourcing (effective 11 December 2024) and its Guidelines on Artificial Intelligence Risk Management (issued 7 October 2026) set out expectations, including for third-party AI.

What should the contract say?

Ask a lawyer to review: ownership of inputs and outputs, intellectual property indemnities, liability limits, service levels, audit rights, notice of material model changes, data return and deletion, and termination assistance. Terms vary widely, so use qualified legal counsel.

How do we run a fair trial?

Test on your own representative samples, include difficult cases, have users score outputs against a simple rubric, and compare at least two options. Record results.

Red flags

Keep learning

Subscribe to our free monthly newsletter at asia-ai.org, or look at membership at asia-ai.org/join.

This article is general information, not advice.

Sources

  1. IMDA, Model AI Governance Framework (including its sections on internal governance structures and on the level of human involvement in AI-augmented decision-making) and the Model AI Governance Framework for Agentic AI
  2. PDPC, Singapore's Approach to AI Governance
  3. PDPC, Guide to Managing Data Intermediaries
  4. MAS, Third-Party Risk Management and Guidelines on Outsourcing (effective 11 December 2024)
  5. MAS, Guidelines on Artificial Intelligence Risk Management for Financial Institutions (media release 7 October 2026)

Cite this page

Please cite as:

Asia AI Association. (2026, November 22). How to evaluate an AI vendor: questions to ask. Asia AI Association. https://asia-ai.org/insights/how-to-evaluate-an-ai-vendor

Content is current as of the updated date above; check the page for later changes.

More from Insights

© Asia AI Association · Home · All insights · RSS · Contact · Privacy