Test Phase
What is the Test Phase?
The Test phase is where the Test Agent systematically validates implementation against approved specifications by deriving test cases directly from acceptance criteria and requirements. This phase demonstrates how VibeSpec transforms specifications into comprehensive test validation without modifying implementation logic or writing production code.
In the Todo Application project, the Test Agent reads the approved product and architecture specifications to create test scenarios that validate every acceptance criterion, security requirement, and performance target. The agent translates specification requirements into test cases but never changes implementation or suggests code modifications.
The Test Agent operates within strict boundaries: it validates what exists against what was specified, reports failures objectively, and documents testing patterns in memory for future reuse. However, it never fixes failing tests by modifying implementation or changes specifications to match implementation behavior.
Understanding this phase is crucial because it demonstrates how systematic specification-based testing ensures implementation fidelity while maintaining clear separation between validation and development responsibilities.
Why This Matters
Problems It Solves
Specification-Implementation Gaps: Implementation often deviates from specifications in subtle ways that aren't caught during review. The Test Agent systematically validates every specification requirement to detect these gaps.
Incomplete Acceptance Criteria Validation: Manual testing may not cover all acceptance criteria comprehensively or consistently. The Test Agent ensures every criterion is validated with appropriate test scenarios.
Inconsistent Test Coverage: Different testers create different test scenarios for similar functionality, leading to gaps and redundancy. The Test Agent applies systematic test derivation from specifications for consistent coverage.
Test-Implementation Coupling: Tests written by implementers often validate what was built rather than what was specified. The Test Agent derives tests from specifications independently of implementation details.
Benefits You'll Gain
Specification Fidelity Validation: Every acceptance criterion and requirement is systematically tested to ensure implementation matches approved specifications exactly.
Objective Test Scenarios: Test cases are derived from specifications rather than implementation, providing unbiased validation of system behavior against requirements.
Systematic Coverage: All specification requirements receive appropriate test coverage based on their type, complexity, and risk level.
Failure Traceability: Test failures are directly traceable to specific specification requirements, enabling precise identification of compliance gaps.
Real-World Impact
Development teams using specification-derived testing catch 90% more requirement compliance issues and reduce post-deployment specification gaps by 85% compared to implementation-driven testing approaches. The systematic approach ensures specifications are validated rather than just implementation behavior.
How to Execute the Test Phase
How Test Cases are Derived from Specifications
Specification Analysis Process: The Test Agent systematically reads approved specifications to extract testable requirements:
- Acceptance Criteria Extraction: Identifies all "As a user, I want..." statements and success criteria
- Functional Requirement Mapping: Locates API contracts, data models, and business logic rules
- Security Requirement Identification: Finds authentication, authorization, and data protection requirements
- Performance Constraint Recognition: Extracts response time, scalability, and resource usage targets
Test Derivation Methodology:
Specification Statement → Test Scenario Derivation
─────────────────────────────────────────────────────
"Users can register with email and secure password" → Test user registration with valid/invalid inputs
"JWT tokens expire after 24 hours" → Test token expiration behavior at 24-hour boundary
"Todo CRUD operations with user authorization" → Test each operation with authorized/unauthorized users
"API responses within 500ms" → Test response time measurement for all endpoints
"Input validation prevents SQL injection" → Test malicious input handling and sanitization
Types of Tests Required
Specification-Driven Test Categories:
1. Functional Validation Tests
- Derived from acceptance criteria and user stories
- Validate that implemented features match specified behavior exactly
- Cover normal operation, edge cases, and error conditions
- Test data flows and business logic as specified
2. Security Compliance Tests
- Derived from safety governance rules and security requirements
- Validate authentication, authorization, and data protection
- Test input validation and attack prevention measures
- Verify secure configuration and secret management
3. Performance Validation Tests
- Derived from performance requirements and constraints
- Validate response times, throughput, and resource usage
- Test scalability and load handling as specified
- Measure against specific performance targets
4. Integration Contract Tests
- Derived from API specifications and integration requirements
- Validate external service integration behavior
- Test error handling and fallback mechanisms
- Verify data format and protocol compliance
Mapping of Acceptance Criteria to Tests
Systematic Acceptance Criteria Validation:
Product Specification Acceptance Criteria:
AC-001: "As a user, I want to register with email and password"
├── Test Scenario 1: Valid registration with proper email format
├── Test Scenario 2: Registration rejection with invalid email format
├── Test Scenario 3: Registration rejection with weak password
├── Test Scenario 4: Registration rejection with duplicate email
└── Test Scenario 5: Successful registration creates user record
AC-002: "As a user, I want to login with valid credentials"
├── Test Scenario 1: Successful login with correct credentials
├── Test Scenario 2: Login rejection with incorrect password
├── Test Scenario 3: Login rejection with non-existent email
├── Test Scenario 4: JWT token generation on successful login
└── Test Scenario 5: Rate limiting after multiple failed attempts
AC-003: "As a user, I want to create todos with title and priority"
├── Test Scenario 1: Todo creation with valid title and priority
├── Test Scenario 2: Todo creation rejection with empty title
├── Test Scenario 3: Todo creation rejection with invalid priority
├── Test Scenario 4: Todo creation requires user authentication
└── Test Scenario 5: Created todo belongs to authenticated user only
Architecture Specification Requirements:
AR-001: "JWT authentication with 24-hour expiration"
├── Test Scenario 1: Token generation includes 24-hour expiration
├── Test Scenario 2: Token validation succeeds within 24 hours
├── Test Scenario 3: Token validation fails after 24 hours
├── Test Scenario 4: Token refresh mechanism works correctly
└── Test Scenario 5: Expired tokens are rejected consistently
AR-002: "PostgreSQL database with User/Todo relationship"
├── Test Scenario 1: User creation stores data in PostgreSQL
├── Test Scenario 2: Todo creation links to user via foreign key
├── Test Scenario 3: User deletion cascades to associated todos
├── Test Scenario 4: Database constraints prevent orphaned todos
└── Test Scenario 5: Query performance meets specification targets
AR-003: "Input validation with Joi schemas"
├── Test Scenario 1: All API endpoints validate input format
├── Test Scenario 2: Invalid input returns 400 status with error details
├── Test Scenario 3: SQL injection attempts are blocked
├── Test Scenario 4: XSS attempts are sanitized or rejected
└── Test Scenario 5: Validation errors provide helpful user feedback
- Performance testing (response times, load handling, resource usage)
- AI integration security (data privacy, API key protection)
Validate all safety governance rules are properly enforced."
**What to expect**: Comprehensive security validation and performance benchmarking with specific metrics.
### Step 5: End-to-End Test Implementation
**E2E testing prompt:**
"Create end-to-end test suite covering complete user workflows:
- User registration and login flow
- Todo creation, editing, and management workflow
- AI suggestion generation and acceptance flow
- Mobile responsive behavior validation
- Error handling and recovery scenarios
- Data persistence and synchronization validation
Include realistic user scenarios and comprehensive workflow coverage."
**What this does**: Validates complete user experiences and system behavior from end-to-end perspective.
## What to Expect
### Example Test Scenarios
The Test Agent creates comprehensive test scenarios derived directly from specifications:
#### Authentication Test Scenarios
**Specification Requirement**: "Users can register with email and secure password"
**Test Scenario 1: Valid User Registration**
Given: A new user with valid email "user@example.com" and secure password "SecurePass123!" When: User submits registration form Then:
- User account is created in database
- Password is hashed with bcrypt
- Success response includes user ID and JWT token
- Token expires in 24 hours as specified
**Test Scenario 2: Invalid Email Registration**
Given: A new user with invalid email "invalid-email" and valid password When: User submits registration form Then:
- Registration is rejected with 400 status
- Error message indicates invalid email format
- No user account is created
- No JWT token is generated
**Test Scenario 3: Weak Password Registration**
Given: A new user with valid email and weak password "123" When: User submits registration form Then:
- Registration is rejected with 400 status
- Error message indicates password requirements
- Password complexity rules enforced as specified
- No user account is created
#### Todo Management Test Scenarios
**Specification Requirement**: "Users can create todos with title, priority, and due date"
**Test Scenario 1: Valid Todo Creation**
Given: Authenticated user with valid JWT token When: User creates todo with title "Complete project", priority "HIGH", due date "2026-02-01" Then:
- Todo is created in database linked to user
- Todo contains all specified fields
- Response includes todo ID and creation timestamp
- Todo is only accessible to creating user
**Test Scenario 2: Unauthorized Todo Creation**
Given: User without valid JWT token When: User attempts to create todo Then:
- Request is rejected with 401 status
- Error message indicates authentication required
- No todo is created in database
- User data remains protected
**Test Scenario 3: Invalid Todo Data**
Given: Authenticated user with valid JWT token When: User creates todo with empty title and invalid priority "URGENT" Then:
- Request is rejected with 400 status
- Error message indicates validation failures
- No todo is created in database
- Input validation rules enforced as specified
#### Security Test Scenarios
**Specification Requirement**: "Rate limiting prevents brute force attacks"
**Test Scenario 1: Rate Limiting Enforcement**
Given: User making repeated login attempts When: User makes 6 failed login attempts within 15 minutes Then:
- First 5 attempts return 401 status with "Invalid credentials"
- 6th attempt returns 429 status with "Too many attempts"
- Rate limiting window resets after 15 minutes
- Legitimate users are not affected
**Test Scenario 2: SQL Injection Prevention**
Given: Malicious user attempting SQL injection When: User submits todo title "'; DROP TABLE users; --" Then:
- Input is sanitized or rejected
- Database remains intact
- No SQL commands are executed
- Error response does not reveal database structure
### How Failures are Reported
**Systematic Failure Documentation:**
**Test Failure Report Format:**
TEST FAILURE REPORT
Test ID: AC-001-Scenario-2 Specification Reference: Product Spec AC-001 "User registration with email and password" Test Scenario: Invalid email registration rejection Expected Behavior: Registration rejected with 400 status and error message Actual Behavior: Registration accepted with invalid email format Failure Type: Specification Compliance Violation Impact Level: HIGH (Security and data integrity risk) Root Cause: Input validation middleware not enforcing email format requirements
**Failure Classification System:**
CRITICAL: Security violations or data corruption risks HIGH: Specification compliance failures affecting core functionality MEDIUM: Performance targets not met or edge case handling issues LOW: Minor usability issues or non-critical feature gaps
**Failure Traceability Matrix:**
Failed Test → Specification Requirement → Implementation Gap ───────────────────────────────────────────────────────────── AC-001-Scenario-2 → Email validation required → Missing Joi email validation AR-001-Scenario-3 → JWT 24-hour expiration → Token expiration set to 1 hour AC-003-Scenario-4 → User authorization required → Missing authentication middleware AR-003-Scenario-3 → SQL injection prevention → Input sanitization incomplete
### Memory Updates When Tests Fail
**Test Failure Pattern Documentation:**
When tests fail, the Test Agent documents patterns and insights in memory:
```markdown
# memory/patterns.md - Test Agent Entries
## Authentication Testing Pattern for Web Applications
**Problem**: Systematic validation of user authentication systems
**Test Categories**:
1. **Valid Credential Testing**: Successful registration and login scenarios
2. **Invalid Input Testing**: Malformed emails, weak passwords, missing fields
3. **Security Testing**: Rate limiting, brute force prevention, token security
4. **Authorization Testing**: Protected endpoint access, user data isolation
**Common Failure Patterns**:
- Input validation bypassed or incomplete (HIGH risk)
- Rate limiting not enforced consistently (MEDIUM risk)
- JWT expiration not matching specification (HIGH risk)
- Error messages revealing sensitive information (MEDIUM risk)
**Specification Mapping**:
- Email validation → Joi schema with email format requirement
- Password security → bcrypt hashing with specified salt rounds
- Rate limiting → Express middleware with specified attempt limits
- JWT expiration → Token generation with exact specification timing
**Usage Context**: Any web application requiring user authentication
**Failure Rate**: 23% of authentication implementations fail initial testing
Failure Learning Documentation:
# memory/decisions.md - Test Failure Analysis
## [2026-01-25 23:15] Todo Application Authentication Test Failures
**Context**: Initial testing of Todo application authentication system
**Failures Identified**:
1. Email validation not enforcing format requirements (AC-001 violation)
2. JWT expiration set to 1 hour instead of specified 24 hours (AR-001 violation)
3. Rate limiting not applied to registration endpoint (Security violation)
**Root Cause Analysis**:
- Input validation middleware incomplete implementation
- JWT configuration using wrong environment variable
- Rate limiting middleware not applied to all authentication routes
**Pattern Recognition**: Authentication implementations commonly fail on:
- Input validation completeness (67% of failures)
- Configuration parameter accuracy (45% of failures)
- Security middleware coverage (34% of failures)
**Prevention Measures**: Enhanced specification-to-test mapping for authentication features
**Status**: Failures documented, implementation returned to Coder Agent for fixes
Test Quality Improvement Tracking:
# memory/patterns.md - Test Quality Evolution
## Specification-Derived Testing Effectiveness Pattern
**Problem**: Ensuring test scenarios accurately validate specification requirements
**Effectiveness Metrics**:
- Specification coverage: 98% of acceptance criteria tested
- Failure detection rate: 89% of specification violations caught
- False positive rate: 3% of test failures due to test errors
- Test maintenance overhead: 15% reduction through pattern reuse
**Quality Improvements Applied**:
- Enhanced specification parsing for edge case identification
- Improved test scenario generation for security requirements
- Better failure reporting with specification traceability
- Automated test pattern application for similar features
**Usage Context**: Any application with detailed specifications and acceptance criteria
**Success Rate**: 94% of specification violations detected before deployment
Test Execution Results
Comprehensive Test Validation:
TEST EXECUTION SUMMARY
=====================
Total Test Scenarios: 127
├── Functional Tests: 78 scenarios
│ ├── Passed: 71 (91%)
│ └── Failed: 7 (9%)
├── Security Tests: 34 scenarios
│ ├── Passed: 31 (91%)
│ └── Failed: 3 (9%)
├── Performance Tests: 15 scenarios
│ ├── Passed: 13 (87%)
│ └── Failed: 2 (13%)
SPECIFICATION COMPLIANCE: 89% (113/127 scenarios passed)
CRITICAL FAILURES: 3 (Security violations requiring immediate fix)
HIGH FAILURES: 8 (Specification compliance issues)
MEDIUM FAILURES: 3 (Performance targets not met)
STATUS: TESTING FAILED - Implementation requires corrections
NEXT STEP: Return to Coder Agent with detailed failure report
├── unit/ # Isolated component testing │ ├── auth/ # Authentication logic tests │ ├── todos/ # Todo business logic tests │ ├── ai/ # AI integration tests │ └── utils/ # Utility function tests ├── integration/ # Component interaction testing │ ├── api/ # API endpoint tests │ ├── database/ # Database operation tests │ └── services/ # Service integration tests ├── security/ # Security validation tests │ ├── auth-security/ # Authentication security tests │ ├── input-validation/ # Input sanitization tests │ └── api-security/ # API security tests ├── performance/ # Performance and load tests │ ├── api-performance/ # API response time tests │ ├── database-performance/ # Database query tests │ └── frontend-performance/ # UI performance tests └── e2e/ # End-to-end workflow tests ├── user-workflows/ # Complete user journey tests ├── mobile-responsive/ # Mobile behavior tests └── error-scenarios/ # Error handling tests
## Unit Test Implementation
### Authentication Service Tests
```typescript
// tests/unit/auth/authService.test.ts
import { authService } from '../../../src/services/authService';
import { prismaMock } from '../../__mocks__/prisma';
import bcrypt from 'bcryptjs';
import jwt from 'jsonwebtoken';
describe('AuthService', () => {
beforeEach(() => {
jest.clearAllMocks();
});
describe('registerUser', () => {
it('should create user with hashed password', async () => {
const userData = {
email: 'test@example.com',
password: 'SecurePass123!'
};
const hashedPassword = 'hashed_password';
const mockUser = {
id: 1,
email: userData.email,
password: hashedPassword,
createdAt: new Date()
};
jest.spyOn(bcrypt, 'hash').mockResolvedValue(hashedPassword);
prismaMock.user.findUnique.mockResolvedValue(null);
prismaMock.user.create.mockResolvedValue(mockUser);
const result = await authService.registerUser(userData.email, userData.password);
expect(bcrypt.hash).toHaveBeenCalledWith(userData.password, 12);
expect(prismaMock.user.create).toHaveBeenCalledWith({
data: {
email: userData.email,
password: hashedPassword
},
select: {
id: true,
email: true,
createdAt: true
}
});
expect(result.user.email).toBe(userData.email);
expect(result.token).toBeDefined();
});
it('should throw error for duplicate email', async () => {
const userData = {
email: 'existing@example.com',
password: 'SecurePass123!'
};
prismaMock.user.findUnique.mockResolvedValue({
id: 1,
email: userData.email,
password: 'existing_hash',
createdAt: new Date(),
updatedAt: new Date()
});
await expect(authService.registerUser(userData.email, userData.password))
.rejects.toThrow('Email already registered');
});
it('should validate password complexity requirements', async () => {
const weakPasswords = [
'short',
'nouppercase123!',
'NOLOWERCASE123!',
'NoNumbers!',
'NoSpecialChars123'
];
for (const password of weakPasswords) {
await expect(authService.registerUser('test@example.com', password))
.rejects.toThrow('Password does not meet complexity requirements');
}
});
});
describe('authenticateUser', () => {
it('should return token for valid credentials', async () => {
const credentials = {
email: 'test@example.com',
password: 'SecurePass123!'
};
const mockUser = {
id: 1,
email: credentials.email,
password: 'hashed_password',
createdAt: new Date(),
updatedAt: new Date()
};
prismaMock.user.findUnique.mockResolvedValue(mockUser);
jest.spyOn(bcrypt, 'compare').mockResolvedValue(true);
const result = await authService.authenticateUser(credentials.email, credentials.password);
expect(result.user.email).toBe(credentials.email);
expect(result.token).toBeDefined();
expect(jwt.verify(result.token, process.env.JWT_SECRET!)).toBeTruthy();
});
it('should throw error for invalid credentials', async () => {
prismaMock.user.findUnique.mockResolvedValue(null);
await expect(authService.authenticateUser('invalid@example.com', 'wrongpassword'))
.rejects.toThrow('Invalid credentials');
});
});
});
Todo Business Logic Tests
// tests/unit/todos/todoService.test.ts
import { todoService } from '../../../src/services/todoService';
import { prismaMock } from '../../__mocks__/prisma';
import { Priority } from '@prisma/client';
describe('TodoService', () => {
const mockUserId = 1;
describe('createTodo', () => {
it('should create todo with valid data', async () => {
const todoData = {
title: 'Test Todo',
description: 'Test Description',
priority: Priority.HIGH,
dueDate: new Date('2026-02-01')
};
const mockTodo = {
id: 1,
...todoData,
userId: mockUserId,
completed: false,
createdAt: new Date(),
updatedAt: new Date()
};
prismaMock.todo.create.mockResolvedValue(mockTodo);
const result = await todoService.createTodo(mockUserId, todoData);
expect(result.title).toBe(todoData.title);
expect(result.priority).toBe(todoData.priority);
expect(prismaMock.todo.create).toHaveBeenCalledWith({
data: {
...todoData,
userId: mockUserId
},
select: expect.any(Object)
});
});
it('should validate title length requirements', async () => {
const invalidTodos = [
{ title: '', description: 'Valid description' },
{ title: 'a'.repeat(256), description: 'Valid description' }
];
for (const todoData of invalidTodos) {
await expect(todoService.createTodo(mockUserId, todoData))
.rejects.toThrow('Title must be between 1 and 255 characters');
}
});
it('should handle due date validation', async () => {
const pastDate = new Date('2020-01-01');
await expect(todoService.createTodo(mockUserId, {
title: 'Test Todo',
dueDate: pastDate
})).rejects.toThrow('Due date cannot be in the past');
});
});
describe('getUserTodos', () => {
it('should return filtered todos for user', async () => {
const mockTodos = [
{
id: 1,
title: 'High Priority Todo',
priority: Priority.HIGH,
completed: false,
userId: mockUserId,
createdAt: new Date(),
updatedAt: new Date()
},
{
id: 2,
title: 'Completed Todo',
priority: Priority.MEDIUM,
completed: true,
userId: mockUserId,
createdAt: new Date(),
updatedAt: new Date()
}
];
prismaMock.todo.findMany.mockResolvedValue(mockTodos);
const result = await todoService.getUserTodos(mockUserId, {
priority: Priority.HIGH,
completed: false
});
expect(prismaMock.todo.findMany).toHaveBeenCalledWith({
where: {
userId: mockUserId,
priority: Priority.HIGH,
completed: false
},
orderBy: { createdAt: 'desc' },
select: expect.any(Object)
});
});
});
});
Integration Test Implementation
API Endpoint Tests
// tests/integration/api/todos.test.ts
import request from 'supertest';
import { app } from '../../../src/app';
import { prisma } from '../../../src/database';
import { generateTestToken } from '../../helpers/auth';
describe('Todo API Endpoints', () => {
let authToken: string;
let userId: number;
beforeAll(async () => {
// Setup test user and authentication
const testUser = await prisma.user.create({
data: {
email: 'test@example.com',
password: 'hashed_password'
}
});
userId = testUser.id;
authToken = generateTestToken(userId);
});
afterAll(async () => {
// Cleanup test data
await prisma.todo.deleteMany({ where: { userId } });
await prisma.user.delete({ where: { id: userId } });
});
describe('POST /api/todos', () => {
it('should create todo with valid data', async () => {
const todoData = {
title: 'Integration Test Todo',
description: 'Test Description',
priority: 'HIGH',
dueDate: '2026-02-01T10:00:00Z'
};
const response = await request(app)
.post('/api/todos')
.set('Authorization', `Bearer ${authToken}`)
.send(todoData)
.expect(201);
expect(response.body.todo.title).toBe(todoData.title);
expect(response.body.todo.priority).toBe(todoData.priority);
expect(response.body.todo.userId).toBe(userId);
});
it('should reject todo without authentication', async () => {
const todoData = {
title: 'Unauthorized Todo',
description: 'Should fail'
};
await request(app)
.post('/api/todos')
.send(todoData)
.expect(401);
});
it('should validate input data', async () => {
const invalidTodos = [
{ title: '' }, // Empty title
{ title: 'a'.repeat(256) }, // Title too long
{ priority: 'INVALID' }, // Invalid priority
{ dueDate: 'invalid-date' } // Invalid date format
];
for (const todoData of invalidTodos) {
await request(app)
.post('/api/todos')
.set('Authorization', `Bearer ${authToken}`)
.send(todoData)
.expect(400);
}
});
});
describe('GET /api/todos', () => {
beforeEach(async () => {
// Create test todos
await prisma.todo.createMany({
data: [
{
title: 'High Priority Todo',
priority: 'HIGH',
completed: false,
userId
},
{
title: 'Completed Todo',
priority: 'MEDIUM',
completed: true,
userId
}
]
});
});
it('should return user todos with filtering', async () => {
const response = await request(app)
.get('/api/todos?priority=HIGH&completed=false')
.set('Authorization', `Bearer ${authToken}`)
.expect(200);
expect(response.body.todos).toHaveLength(1);
expect(response.body.todos[0].priority).toBe('HIGH');
expect(response.body.todos[0].completed).toBe(false);
});
it('should not return other users todos', async () => {
// Create another user and todo
const otherUser = await prisma.user.create({
data: {
email: 'other@example.com',
password: 'hashed_password'
}
});
await prisma.todo.create({
data: {
title: 'Other User Todo',
userId: otherUser.id
}
});
const response = await request(app)
.get('/api/todos')
.set('Authorization', `Bearer ${authToken}`)
.expect(200);
const otherUserTodos = response.body.todos.filter(
(todo: any) => todo.title === 'Other User Todo'
);
expect(otherUserTodos).toHaveLength(0);
// Cleanup
await prisma.todo.deleteMany({ where: { userId: otherUser.id } });
await prisma.user.delete({ where: { id: otherUser.id } });
});
});
});
Security Test Implementation
Authentication Security Tests
// tests/security/auth-security/authentication.test.ts
import request from 'supertest';
import { app } from '../../../src/app';
import { prisma } from '../../../src/database';
describe('Authentication Security', () => {
describe('Rate Limiting', () => {
it('should enforce rate limiting on login attempts', async () => {
const credentials = {
email: 'test@example.com',
password: 'wrongpassword'
};
// Make 5 failed attempts (rate limit)
for (let i = 0; i < 5; i++) {
await request(app)
.post('/api/auth/login')
.send(credentials)
.expect(401);
}
// 6th attempt should be rate limited
await request(app)
.post('/api/auth/login')
.send(credentials)
.expect(429);
});
});
describe('Password Security', () => {
it('should reject weak passwords', async () => {
const weakPasswords = [
'password',
'12345678',
'Password',
'Password123',
'password123!'
];
for (const password of weakPasswords) {
await request(app)
.post('/api/auth/register')
.send({
email: 'test@example.com',
password
})
.expect(400);
}
});
it('should accept strong passwords', async () => {
const strongPassword = 'SecurePass123!@#';
await request(app)
.post('/api/auth/register')
.send({
email: 'secure@example.com',
password: strongPassword
})
.expect(201);
// Cleanup
await prisma.user.delete({
where: { email: 'secure@example.com' }
});
});
});
describe('JWT Token Security', () => {
it('should reject expired tokens', async () => {
// Create token with past expiration
const expiredToken = jwt.sign(
{ userId: 1 },
process.env.JWT_SECRET!,
{ expiresIn: '-1h' }
);
await request(app)
.get('/api/todos')
.set('Authorization', `Bearer ${expiredToken}`)
.expect(401);
});
it('should reject malformed tokens', async () => {
const malformedTokens = [
'invalid.token.format',
'Bearer invalid',
'not-a-jwt-token'
];
for (const token of malformedTokens) {
await request(app)
.get('/api/todos')
.set('Authorization', `Bearer ${token}`)
.expect(401);
}
});
});
});
Performance Test Implementation
API Performance Tests
// tests/performance/api-performance/response-times.test.ts
import request from 'supertest';
import { app } from '../../../src/app';
import { generateTestToken } from '../../helpers/auth';
describe('API Performance', () => {
let authToken: string;
beforeAll(async () => {
authToken = generateTestToken(1);
});
describe('Response Time Requirements', () => {
it('should respond to GET /api/todos within 500ms', async () => {
const startTime = Date.now();
await request(app)
.get('/api/todos')
.set('Authorization', `Bearer ${authToken}`)
.expect(200);
const responseTime = Date.now() - startTime;
expect(responseTime).toBeLessThan(500);
});
it('should handle concurrent requests efficiently', async () => {
const concurrentRequests = 10;
const requests = Array(concurrentRequests).fill(null).map(() =>
request(app)
.get('/api/todos')
.set('Authorization', `Bearer ${authToken}`)
);
const startTime = Date.now();
const responses = await Promise.all(requests);
const totalTime = Date.now() - startTime;
// All requests should succeed
responses.forEach(response => {
expect(response.status).toBe(200);
});
// Average response time should be reasonable
const averageTime = totalTime / concurrentRequests;
expect(averageTime).toBeLessThan(1000);
});
});
describe('Load Testing', () => {
it('should handle high todo creation load', async () => {
const todoCount = 100;
const todos = Array(todoCount).fill(null).map((_, index) => ({
title: `Load Test Todo ${index}`,
description: `Description ${index}`,
priority: 'MEDIUM'
}));
const startTime = Date.now();
const requests = todos.map(todo =>
request(app)
.post('/api/todos')
.set('Authorization', `Bearer ${authToken}`)
.send(todo)
);
const responses = await Promise.all(requests);
const totalTime = Date.now() - startTime;
// All requests should succeed
responses.forEach(response => {
expect(response.status).toBe(201);
});
// Should complete within reasonable time
expect(totalTime).toBeLessThan(10000); // 10 seconds for 100 todos
});
});
});
Test Coverage and Quality Metrics
Coverage Analysis Results
## Test Coverage Report
### Unit Tests Coverage
- **Authentication Service**: 96% (48/50 lines)
- **Todo Service**: 94% (87/92 lines)
- **AI Service**: 91% (73/80 lines)
- **Validation Middleware**: 98% (49/50 lines)
- **Utility Functions**: 100% (25/25 lines)
**Overall Unit Test Coverage**: 95.2% ✅
### Integration Tests Coverage
- **API Endpoints**: 93% (112/120 endpoints)
- **Database Operations**: 96% (48/50 operations)
- **Service Integration**: 89% (67/75 integration points)
- **Authentication Flow**: 100% (15/15 scenarios)
**Overall Integration Test Coverage**: 93.1% ✅
### Security Tests Coverage
- **Authentication Security**: 100% (20/20 security rules)
- **Authorization Security**: 100% (15/15 authorization checks)
- **Input Validation**: 97% (29/30 validation rules)
- **API Security**: 95% (19/20 security measures)
**Overall Security Test Coverage**: 98.4% ✅
### Performance Tests Coverage
- **Response Time Tests**: 100% (10/10 endpoints)
- **Load Testing**: 90% (9/10 scenarios)
- **Concurrent Access**: 95% (19/20 scenarios)
- **Resource Usage**: 85% (17/20 metrics)
**Overall Performance Test Coverage**: 92.5% ✅
### End-to-End Tests Coverage
- **User Workflows**: 88% (22/25 workflows)
- **Error Scenarios**: 92% (23/25 scenarios)
- **Mobile Responsive**: 90% (18/20 breakpoints)
- **AI Integration**: 85% (17/20 AI scenarios)
**Overall E2E Test Coverage**: 88.8% ✅
## TOTAL PROJECT TEST COVERAGE: 93.6% ✅
Test Execution and Validation
Automated Test Pipeline
# .github/workflows/test.yml
name: Comprehensive Test Suite
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
services:
postgres:
image: postgres:14
env:
POSTGRES_PASSWORD: test_password
POSTGRES_DB: ai_todo_test
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
steps:
- uses: actions/checkout@v3
- name: Setup Node.js
uses: actions/setup-node@v3
with:
node-version: '18'
- name: Install dependencies
run: |
npm ci
cd backend && npm ci
- name: Run database migrations
run: cd backend && npm run db:migrate
- name: Run unit tests
run: npm run test:unit
- name: Run integration tests
run: npm run test:integration
- name: Run security tests
run: npm run test:security
- name: Run performance tests
run: npm run test:performance
- name: Run E2E tests
run: npm run test:e2e
- name: Generate coverage report
run: npm run test:coverage
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v3
Test Results Summary
✅ All Test Suites Passed
- Unit Tests: 287 tests passed, 0 failed
- Integration Tests: 156 tests passed, 0 failed
- Security Tests: 89 tests passed, 0 failed
- Performance Tests: 45 tests passed, 0 failed
- E2E Tests: 67 tests passed, 0 failed
Total: 644 tests passed, 0 failed ✅
Performance Benchmarks Met
- API Response Times: Average 180ms (target: under 500ms) ✅
- Database Query Performance: Average 45ms (target: under 100ms) ✅
- Frontend Load Time: 2.1s (target: under 3s) ✅
- Concurrent User Handling: 50 users (target: 25+ users) ✅
Security Validation Complete
- Authentication Security: All 20 security rules validated ✅
- Authorization Security: All 15 authorization checks passed ✅
- Input Validation: 29/30 validation rules enforced ✅
- API Security: 19/20 security measures implemented ✅
Security Status: FULLY COMPLIANT ✅
**Memory Evolution After Test Suite Creation:**
```markdown
# memory/patterns.md - NEW ENTRY
## Comprehensive Testing Pattern for AI-Enhanced Applications
**Problem**: Systematic validation of modern web applications with AI integration
**Test Categories**:
1. **Unit Tests**: Business logic isolation with proper mocking (95%+ coverage)
2. **Integration Tests**: Component interaction validation with real data
3. **Security Tests**: Absolute safety governance rule enforcement
4. **Performance Tests**: Response time and load handling validation
5. **E2E Tests**: Complete user workflow and AI integration validation
**Technology Stack**: Jest + Supertest + Playwright + Custom security validators
**Coverage Requirements**:
- Unit tests: 95%+ coverage for all business logic
- Integration tests: 90%+ coverage for API endpoints and services
- Security tests: 100% coverage for all safety governance rules
- Performance tests: All critical paths under 500ms response time
- E2E tests: 85%+ coverage for user workflows and AI scenarios
**AI-Specific Testing**:
- AI service integration with proper mocking and fallback testing
- AI response validation and error handling
- AI suggestion acceptance rate and quality metrics
- AI service failure and recovery scenario testing
**Quality Metrics**: 90%+ overall coverage, zero security violations, performance targets met
**Usage Context**: Modern web applications with AI integration and security requirements
# memory/decisions.md - NEW ENTRY
## [2026-01-25 22:30] AI Todo Application Testing Strategy
**Context**: Establishing comprehensive testing approach for AI-enhanced todo application
**Decision**: Implement 5-tier testing strategy with 90%+ coverage requirement
**Testing Tiers**:
1. Unit tests for business logic isolation and validation
2. Integration tests for component interaction and data flow
3. Security tests for absolute safety governance enforcement
4. Performance tests for response time and load validation
5. E2E tests for complete workflow and AI integration validation
**Quality Requirements**:
- 90%+ overall test coverage across all tiers
- 100% security rule validation and enforcement
- Performance targets: under 500ms API responses, under 3s page loads
- AI integration testing with proper error handling and fallbacks
**Test Automation**: Complete CI/CD pipeline with automated test execution
**Success Criteria**: All tests pass, coverage targets met, security compliance validated
**Status**: Implemented and all requirements met
Cycle 2: Pattern-Based Testing (Month 4)
Prompt:
"Create test suite for user notification feature using our established testing pattern."
Result:
[Test Agent - Pattern Recognition Active]
Testing Target: User notification feature
Applying Pattern: Comprehensive Testing Pattern for AI-Enhanced Applications
Coverage Requirements: 90%+ across all test tiers
## Pattern-Based Test Implementation (Accelerated)
### ✅ Test Suite Generated Using Established Pattern
- **Unit Tests**: Notification service logic and validation (96% coverage)
- **Integration Tests**: Email/SMS delivery and user preference handling (94% coverage)
- **Security Tests**: Notification content sanitization and privacy protection (100% coverage)
- **Performance Tests**: Notification delivery times and batch processing (92% coverage)
- **E2E Tests**: Complete notification workflow and user interaction (89% coverage)
### 🚀 Efficiency Gains Through Pattern Application
- **Test Creation Time**: 25 minutes (vs. 2+ hours for initial test suite)
- **Coverage Achievement**: 93.2% (exceeded 90% target)
- **Pattern Compliance**: 98% (notification-specific enhancements)
**STATUS**: All tests passing, coverage targets exceeded ✅
- Test suite created using established patterns
- Notification-specific testing scenarios included
- Performance and security requirements validated
Memory Evolution After Pattern-Based Testing:
# memory/patterns.md - UPDATED ENTRY
## Comprehensive Testing Pattern for AI-Enhanced Applications (UPDATED)
**Problem**: Systematic validation of modern web applications with AI integration
**Test Categories**:
1. **Unit Tests**: Business logic isolation with proper mocking (95%+ coverage)
2. **Integration Tests**: Component interaction validation with real data
3. **Security Tests**: Absolute safety governance rule enforcement
4. **Performance Tests**: Response time and load handling validation
5. **E2E Tests**: Complete user workflow and AI integration validation
**Feature-Specific Extensions**:
- **Notification Testing**: Email/SMS delivery, user preferences, content sanitization
- **Real-time Features**: WebSocket connections, live updates, connection handling
- **File Upload Testing**: Security validation, size limits, type restrictions
**Efficiency Improvements**:
- Pattern-based test generation reduces creation time by 85%
- Automated coverage validation ensures consistent quality
- Reusable test utilities and helpers for common scenarios
**Usage Context**: Modern web applications with AI integration and security requirements
**Success Rate**: 3/3 features tested successfully using this pattern
**Average Test Creation Time**: 25 minutes (down from 2+ hours)
Quality Indicators for Testing Cycles
Cycle 1 - Comprehensive Test Creation:
- ✅ Complete test suite covering all functionality with proper coverage metrics
- ✅ Security validation ensuring absolute safety governance rule enforcement
- ✅ Performance benchmarking with specific targets and validation
- ✅ Memory documentation of effective testing patterns and strategies
Cycle 2 - Pattern-Based Testing:
- ✅ Established patterns recognized and applied efficiently for new features
- ✅ Test creation time reduced significantly while maintaining quality standards
- ✅ Feature-specific testing scenarios included within established framework
- ✅ Pattern enhancement based on new testing requirements and scenarios
Memory-Driven Testing Efficiency:
- ✅ Testing patterns enable rapid, consistent test suite creation
- ✅ Quality standards maintained automatically through pattern application
- ✅ Coverage requirements achieved systematically across all features
- ✅ Continuous improvement through pattern evolution and optimization
Common Mistakes and Warnings
⚠️ Critical Warnings
-
Never Modify Implementation to Fix Failing Tests: The Test Agent validates specifications, not implementation. If tests fail, the implementation must be corrected by the Coder Agent, never by changing tests to match incorrect implementation.
-
Never Derive Tests from Implementation Instead of Specifications: Tests must validate specification requirements, not implementation behavior. Implementation-derived tests validate what was built rather than what was specified.
Common Mistakes
Mistake: Writing tests based on implementation behavior rather than specifications
Why it happens: Testers examine implementation and create tests that validate existing behavior
How to avoid: Derive all test scenarios from specifications before examining implementation
If it happens: Rewrite tests to validate specification requirements, not implementation details
Mistake: Modifying tests to make them pass instead of reporting failures
Why it happens: Pressure to show "green" test results leads to changing tests rather than fixing implementation
How to avoid: Treat test failures as specification compliance issues requiring implementation fixes
If it happens: Revert test changes and report failures to Coder Agent for implementation correction
Mistake: Accepting partial specification coverage as sufficient
Why it happens: Time pressure leads to testing only obvious scenarios while missing edge cases
How to avoid: Systematically validate every acceptance criterion and requirement from specifications
If it happens: Complete specification coverage before declaring testing complete
Mistake: Not documenting test failure patterns for future prevention
Why it happens: Focus on immediate test results without capturing learning for future improvement
How to avoid: Document all failure patterns and specification compliance issues in memory
If it happens: Retroactively analyze failures and update memory with prevention insights
Best Practices
- ✅ Derive Tests from Specifications Only: Use approved specifications as the sole source for test scenario creation
- ✅ Report Failures Objectively: Document test failures with specification references, never modify tests to pass
- ✅ Validate Complete Specification Coverage: Ensure every acceptance criterion and requirement is tested
- ✅ Document Testing Patterns: Capture effective test derivation techniques and failure patterns in memory
- ✅ Maintain Test-Implementation Independence: Keep test scenarios independent of implementation details and choices