Back to ISTQB Labs
Traditional vs AI Testing
Understand the fundamental differences between testing traditional software and AI-based systems
Traditional Approach
DeterministicTesting a rule-based login system
| Input | Expected Output | Deterministic |
|---|---|---|
| Valid username + valid password | Login success | Yes |
| Valid username + invalid password | Error: Invalid password | Yes |
| Invalid username | Error: User not found | Yes |
| Empty fields | Error: Required fields | Yes |
| SQL injection attempt | Error: Invalid input | Yes |
Characteristics
- Deterministic outputs
- Clear pass/fail criteria
- Specification-based expected results
- Same input always gives same output
- Binary outcomes (success/failure)
AI Approach
ProbabilisticTesting an AI-based fraud detection login system
| Input | Expected Output | Deterministic |
|---|---|---|
| Valid credentials + normal behavior | Login success (95% confidence) | No |
| Valid credentials + unusual location | Additional verification (72% risk) | No |
| Valid credentials + new device | Maybe blocked (60% confidence) | No |
| Valid credentials + unusual time | Varies by risk model | No |
| Previously seen attack pattern | Likely blocked (85% confidence) | No |
Characteristics
- Probabilistic outputs
- No single "correct" answer
- Context-dependent behavior
- Output may vary between runs
- Confidence scores instead of binary
New Testing Challenges
- How do you define "correct" when output is probabilistic?
- What confidence threshold is acceptable?
- How do you handle different results for the same input?
- How do you test for bias in the risk assessment?
- How do you regression test when outputs naturally vary?