fix: ChatBubble crash and DeepSeek API compatibility
- Fix ChatBubble to handle non-string content with String() wrapper - Fix API route to use generateText for non-streaming requests - Add @ai-sdk/openai-compatible for non-OpenAI providers (DeepSeek, etc.) - Use Chat Completions API instead of Responses API for compatible providers - Update ChatBubble tests and fix component exports to kebab-case - Remove stale PascalCase ChatBubble.tsx file
This commit is contained in:
273
_bmad/bmm/testarch/knowledge/burn-in.md
Normal file
273
_bmad/bmm/testarch/knowledge/burn-in.md
Normal file
@@ -0,0 +1,273 @@
|
||||
# Burn-in Test Runner
|
||||
|
||||
## Principle
|
||||
|
||||
Use smart test selection with git diff analysis to run only affected tests. Filter out irrelevant changes (configs, types, docs) and control test volume with percentage-based execution. Reduce unnecessary CI runs while maintaining reliability.
|
||||
|
||||
## Rationale
|
||||
|
||||
Playwright's `--only-changed` triggers all affected tests:
|
||||
|
||||
- Config file changes trigger hundreds of tests
|
||||
- Type definition changes cause full suite runs
|
||||
- No volume control (all or nothing)
|
||||
- Slow CI pipelines
|
||||
|
||||
The `burn-in` utility provides:
|
||||
|
||||
- **Smart filtering**: Skip patterns for irrelevant files (configs, types, docs)
|
||||
- **Volume control**: Run percentage of affected tests after filtering
|
||||
- **Custom dependency analysis**: More accurate than Playwright's built-in
|
||||
- **CI optimization**: Faster pipelines without sacrificing confidence
|
||||
- **Process of elimination**: Start with all → filter irrelevant → control volume
|
||||
|
||||
## Pattern Examples
|
||||
|
||||
### Example 1: Basic Burn-in Setup
|
||||
|
||||
**Context**: Run burn-in on changed files compared to main branch.
|
||||
|
||||
**Implementation**:
|
||||
|
||||
```typescript
|
||||
// Step 1: Create burn-in script
|
||||
// playwright/scripts/burn-in-changed.ts
|
||||
import { runBurnIn } from '@seontechnologies/playwright-utils/burn-in'
|
||||
|
||||
async function main() {
|
||||
await runBurnIn({
|
||||
configPath: 'playwright/config/.burn-in.config.ts',
|
||||
baseBranch: 'main'
|
||||
})
|
||||
}
|
||||
|
||||
main().catch(console.error)
|
||||
|
||||
// Step 2: Create config
|
||||
// playwright/config/.burn-in.config.ts
|
||||
import type { BurnInConfig } from '@seontechnologies/playwright-utils/burn-in'
|
||||
|
||||
const config: BurnInConfig = {
|
||||
// Files that never trigger tests (first filter)
|
||||
skipBurnInPatterns: [
|
||||
'**/config/**',
|
||||
'**/*constants*',
|
||||
'**/*types*',
|
||||
'**/*.md',
|
||||
'**/README*'
|
||||
],
|
||||
|
||||
// Run 30% of remaining tests after skip filter
|
||||
burnInTestPercentage: 0.3,
|
||||
|
||||
// Burn-in repetition
|
||||
burnIn: {
|
||||
repeatEach: 3, // Run each test 3 times
|
||||
retries: 1 // Allow 1 retry
|
||||
}
|
||||
}
|
||||
|
||||
export default config
|
||||
|
||||
// Step 3: Add package.json script
|
||||
{
|
||||
"scripts": {
|
||||
"test:pw:burn-in-changed": "tsx playwright/scripts/burn-in-changed.ts"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Key Points**:
|
||||
|
||||
- Two-stage filtering: skip patterns, then volume control
|
||||
- `skipBurnInPatterns` eliminates irrelevant files
|
||||
- `burnInTestPercentage` controls test volume (0.3 = 30%)
|
||||
- Custom dependency analysis finds actually affected tests
|
||||
|
||||
### Example 2: CI Integration
|
||||
|
||||
**Context**: Use burn-in in GitHub Actions for efficient CI runs.
|
||||
|
||||
**Implementation**:
|
||||
|
||||
```yaml
|
||||
# .github/workflows/burn-in.yml
|
||||
name: Burn-in Changed Tests
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
branches: [main]
|
||||
|
||||
jobs:
|
||||
burn-in:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
fetch-depth: 0 # Need git history
|
||||
|
||||
- name: Setup Node
|
||||
uses: actions/setup-node@v4
|
||||
|
||||
- name: Install dependencies
|
||||
run: npm ci
|
||||
|
||||
- name: Run burn-in on changed tests
|
||||
run: npm run test:pw:burn-in-changed -- --base-branch=origin/main
|
||||
|
||||
- name: Upload artifacts
|
||||
if: failure()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: burn-in-failures
|
||||
path: test-results/
|
||||
```
|
||||
|
||||
**Key Points**:
|
||||
|
||||
- `fetch-depth: 0` for full git history
|
||||
- Pass `--base-branch=origin/main` for PR comparison
|
||||
- Upload artifacts only on failure
|
||||
- Significantly faster than full suite
|
||||
|
||||
### Example 3: How It Works (Process of Elimination)
|
||||
|
||||
**Context**: Understanding the filtering pipeline.
|
||||
|
||||
**Scenario:**
|
||||
|
||||
```
|
||||
Git diff finds: 21 changed files
|
||||
├─ Step 1: Skip patterns filter
|
||||
│ Removed: 6 files (*.md, config/*, *types*)
|
||||
│ Remaining: 15 files
|
||||
│
|
||||
├─ Step 2: Dependency analysis
|
||||
│ Tests that import these 15 files: 45 tests
|
||||
│
|
||||
└─ Step 3: Volume control (30%)
|
||||
Final tests to run: 14 tests (30% of 45)
|
||||
|
||||
Result: Run 14 targeted tests instead of 147 with --only-changed!
|
||||
```
|
||||
|
||||
**Key Points**:
|
||||
|
||||
- Three-stage pipeline: skip → analyze → control
|
||||
- Custom dependency analysis (not just imports)
|
||||
- Percentage applies AFTER filtering
|
||||
- Dramatically reduces CI time
|
||||
|
||||
### Example 4: Environment-Specific Configuration
|
||||
|
||||
**Context**: Different settings for local vs CI environments.
|
||||
|
||||
**Implementation**:
|
||||
|
||||
```typescript
|
||||
import type { BurnInConfig } from '@seontechnologies/playwright-utils/burn-in';
|
||||
|
||||
const config: BurnInConfig = {
|
||||
skipBurnInPatterns: ['**/config/**', '**/*types*', '**/*.md'],
|
||||
|
||||
// CI runs fewer iterations, local runs more
|
||||
burnInTestPercentage: process.env.CI ? 0.2 : 0.3,
|
||||
|
||||
burnIn: {
|
||||
repeatEach: process.env.CI ? 2 : 3,
|
||||
retries: process.env.CI ? 0 : 1, // No retries in CI
|
||||
},
|
||||
};
|
||||
|
||||
export default config;
|
||||
```
|
||||
|
||||
**Key Points**:
|
||||
|
||||
- `process.env.CI` for environment detection
|
||||
- Lower percentage in CI (20% vs 30%)
|
||||
- Fewer iterations in CI (2 vs 3)
|
||||
- No retries in CI (fail fast)
|
||||
|
||||
### Example 5: Sharding Support
|
||||
|
||||
**Context**: Distribute burn-in tests across multiple CI workers.
|
||||
|
||||
**Implementation**:
|
||||
|
||||
```typescript
|
||||
// burn-in-changed.ts with sharding
|
||||
import { runBurnIn } from '@seontechnologies/playwright-utils/burn-in';
|
||||
|
||||
async function main() {
|
||||
const shardArg = process.argv.find((arg) => arg.startsWith('--shard='));
|
||||
|
||||
if (shardArg) {
|
||||
process.env.PW_SHARD = shardArg.split('=')[1];
|
||||
}
|
||||
|
||||
await runBurnIn({
|
||||
configPath: 'playwright/config/.burn-in.config.ts',
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
```yaml
|
||||
# GitHub Actions with sharding
|
||||
jobs:
|
||||
burn-in:
|
||||
strategy:
|
||||
matrix:
|
||||
shard: [1/3, 2/3, 3/3]
|
||||
steps:
|
||||
- run: npm run test:pw:burn-in-changed -- --shard=${{ matrix.shard }}
|
||||
```
|
||||
|
||||
**Key Points**:
|
||||
|
||||
- Pass `--shard=1/3` for parallel execution
|
||||
- Burn-in respects Playwright sharding
|
||||
- Distribute across multiple workers
|
||||
- Reduces total CI time further
|
||||
|
||||
## Integration with CI Workflow
|
||||
|
||||
When setting up CI with `*ci` workflow, recommend burn-in for:
|
||||
|
||||
- Pull request validation
|
||||
- Pre-merge checks
|
||||
- Nightly builds (subset runs)
|
||||
|
||||
## Related Fragments
|
||||
|
||||
- `ci-burn-in.md` - Traditional burn-in patterns (10-iteration loops)
|
||||
- `selective-testing.md` - Test selection strategies
|
||||
- `overview.md` - Installation
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
**❌ Over-aggressive skip patterns:**
|
||||
|
||||
```typescript
|
||||
skipBurnInPatterns: [
|
||||
'**/*', // Skips everything!
|
||||
];
|
||||
```
|
||||
|
||||
**✅ Targeted skip patterns:**
|
||||
|
||||
```typescript
|
||||
skipBurnInPatterns: ['**/config/**', '**/*types*', '**/*.md', '**/*constants*'];
|
||||
```
|
||||
|
||||
**❌ Too low percentage (false confidence):**
|
||||
|
||||
```typescript
|
||||
burnInTestPercentage: 0.05; // Only 5% - might miss issues
|
||||
```
|
||||
|
||||
**✅ Balanced percentage:**
|
||||
|
||||
```typescript
|
||||
burnInTestPercentage: 0.2; // 20% in CI, provides good coverage
|
||||
```
|
||||
Reference in New Issue
Block a user