OpenPets Scenario Documentation

This document explains how test scenarios are structured across the OpenPets plugin collection.

Overview

Test scenarios provide a standardized way to validate plugin functionality through executable test cases. There are two main approaches used across the plugins:

Sequential Query Arrays - Multi-step workflows using shared test runners
Command Strings - Single command scenarios for direct execution (slides-shiba)

Scenario Structure Types

Type 1: Sequential Query Arrays

This approach uses arrays of queries that execute sequentially, allowing context to flow between steps.

Structure

json

{
  "queries": [
    "simple query 1",
    "simple query 2"
  ],
  "scenarios": {
    "scenario-name": [
      "first query in workflow",
      "second query using context from first",
      "third query building on previous results"
    ],
    "another-scenario": [
      "query 1",
      "query 2"
    ]
  },
  "scripts": {
    "test:queries": "../../src/shared-utils/test-queries.sh",
    "test:scenarios": "../../src/shared-utils/run-scenarios.sh",
    "test:scenario-name": "../../src/shared-utils/run-scenarios.sh scenario-name"
  }
}

Key Features

Sequential Execution: Each query runs in order, with context preserved
Contextual Queries: Later queries can reference results from earlier ones (e.g., "now get the timezone for those coordinates")
Shared Test Runners: Uses centralized scripts for consistent execution
Simple Queries Field: Standalone queries for quick smoke tests

Running:

bash

# Run all scenarios
npm run test:scenarios

# Run specific scenario
npm run test:geocoding

# Run quick queries
npm run test:queries

Example: video-vikhan

json

{
  "queries": [
    "get the transcript from https://www.youtube.com/watch?v=y411m5DYBm8"
  ],
  "scenarios": {
    "video-analysis": [
      "get the transcript from https://www.youtube.com/watch?v=y411m5DYBm8",
      "summarize the main points from that transcript",
      "what topics were discussed in the video?"
    ],
    "subtitle-formats": [
      "get video info for https://www.youtube.com/watch?v=y411m5DYBm8",
      "now get the subtitles in SRT format",
      "get the same subtitles in VTT format instead"
    ],
    "multilingual-workflow": [
      "get spanish subtitles from https://www.youtube.com/watch?v=y411m5DYBm8",
      "now get french subtitles from the same video",
      "compare the word counts of both"
    ]
  },
  "scripts": {
    "test:scenarios": "../../src/shared-utils/run-scenarios.sh",
    "test:analysis": "../../src/shared-utils/run-scenarios.sh video-analysis",
    "test:multilingual": "../../src/shared-utils/run-scenarios.sh multilingual-workflow"
  }
}

Running:

bash

npm run test:scenarios          # All scenarios
npm run test:analysis           # Specific scenario
npm run test:multilingual       # Another scenario

Type 2: Command Strings

Used by: openpets/slides-shiba

This approach uses natural-language test queries as scenario values, suitable for standalone operations.

Structure

json

{
  "scenarios": {
    "scenario-name": "complete command here",
    "another-scenario": "another command"
  }
}

Key Features

Self-Contained: Each scenario is a complete command
Direct Execution: Can be run directly or via npm scripts
Tool-Specific: Commands explicitly reference tools
No Context Flow: Each scenario is independent

Example: slides-shiba

json

{
  "scenarios": {
    "create-basic": "use slides-create-markdown to create test-basic.md",
    "create-with-theme": "use slides-create-markdown to create test-gaia.md with gaia theme and beige background",
    "add-slide-end": "use slides-add-slide to add a slide titled \"New Feature\" to test-basic.md",
    "generate-pptx": "use slides-generate-pptx to create PowerPoint from test-basic.md",
    "batch-all-formats": "use slides-batch-generate to create pptx, pdf, and html from test-basic.md",
    "workflow-simple": "create a simple 3-slide presentation about AI and export to PPTX",
    "workflow-pitch": "create a 5-slide pitch deck with problem, solution, market, team, and CTA slides, then export all formats",
    "error-nonexistent-file": "try to add a slide to nonexistent.md and show the error",
    "integration-full": "create test-full.md with gaia theme, add 3 slides with different content types, add 2 images, preview it, then batch generate all formats"
  }
}

Running:

bash

# Direct execution
npm run scenario create-basic

# Or run the command directly
bun test:pet <pet-name> --query 'use slides-create-markdown to create test-basic.md'

Comparison

Aspect	Sequential Arrays	Command Strings
Context Flow	✅ Yes - queries reference previous results	❌ No - each command standalone
Multi-Step Workflows	✅ Native support	⚠️ Must be in single command
Test Runner	✅ Shared scripts	❌ Manual execution
Execution	Via shared scripts	Direct or npm scripts
Best For	Complex workflows, API testing	Tool validation, simple operations

Choosing an Approach

Use Sequential Query Arrays When:

✅ Plugin involves multi-step workflows
✅ Later steps depend on earlier results
✅ Testing API integrations with stateful operations
✅ Need automated test runner with reporting
✅ Want to validate context preservation

Examples:

Map APIs (geocode → get timezone → check street view)
Video analysis (get transcript → summarize → extract topics)
E-commerce (search product → add to cart → checkout)

Use Command Strings When:

✅ Plugin has independent tool operations
✅ Each scenario is self-contained
✅ Testing specific tool functionality
✅ Simple create/read/update operations
✅ Error handling validation

Examples:

File generators (create → modify → export)
Single-purpose tools
CRUD operations
Format converters

Scenario Naming Conventions

For Sequential Arrays

Use descriptive names indicating the workflow:

json

{
  "scenarios": {
    "geocoding-workflow": [...],      // Process-focused
    "planning-trip": [...],           // Use-case focused
    "multi-city-route": [...],        // Feature-focused
    "video-analysis": [...],          // Task-focused
    "subtitle-formats": [...],        // Format variation
    "multilingual-workflow": [...]    // Capability test
  }
}

For Command Strings

Use action-object pattern:

json

{
  "scenarios": {
    "create-basic": "...",            // action-adjective
    "create-with-theme": "...",       // action-with-modifier
    "add-slide-end": "...",           // action-noun-location
    "generate-pptx": "...",           // action-format
    "batch-all-formats": "...",       // action-scope
    "workflow-simple": "...",         // category-adjective
    "test-themes": "...",             // test-feature
    "error-nonexistent-file": "..."   // error-condition
  }
}

Running Scenarios

Sequential Query Arrays

bash

# All scenarios in plugin
npm run test:scenarios

# Specific scenario
npm run test:geocoding

# Quick smoke tests
npm run test:queries

# Everything
npm run test:all

Command Strings

bash

# Via npm (if configured)
cd pets/slides-shiba
npm run scenario create-basic
npm run scenario workflow-pitch

# Direct execution
bun test:pet <pet-name> --query 'use slides-create-markdown to create test.md'

# Batch execution (manual)
for scenario in create-basic add-slide-end generate-pptx; do
  npm run scenario $scenario
done

Creating New Scenarios

Sequential Array Scenario

Add to package.json:

json

{
  "scenarios": {
    "my-workflow": [
      "first query",
      "second query using context from first",
      "third query"
    ]
  }
}

Add npm script:

json

{
  "scripts": {
    "test:my-workflow": "../../src/shared-utils/run-scenarios.sh my-workflow"
  }
}

Run:

bash

npm run test:my-workflow

Command String Scenario

Add to package.json:

json

{
  "scenarios": {
    "my-test": "your command here"
  }
}

(Optional) Add npm script:

json

{
  "scripts": {
    "scenario": "npm run scenario:${SCENARIO}",
    "scenario:my-test": "your command here"
  }
}

Run:

bash

npm run scenario my-test
# or
bun test:pet <pet-name> --query 'your command here'

Scenario Categories

Functional Testing

Test core plugin capabilities:

json

{
  "scenarios": {
    "geocoding-workflow": [...],      // Core feature
    "create-basic": "...",            // Basic operation
    "add-slide-end": "..."            // Modification
  }
}

Integration Testing

Test multi-step workflows:

json

{
  "scenarios": {
    "planning-trip": [...],           // Real-world use case
    "video-analysis": [...],          // Complete workflow
    "workflow-conference": "..."      // End-to-end scenario
  }
}

Error Handling

Test failure cases:

json

{
  "scenarios": {
    "error-nonexistent-file": "...",
    "error-invalid-theme": "..."
  }
}

Format/Feature Variations

Test different configurations:

json

{
  "scenarios": {
    "subtitle-formats": [...],        // Format variations
    "test-themes": "...",             // Theme variations
    "test-aspect-ratios": "..."       // Configuration options
  }
}

Best Practices

General

Descriptive Names: Use clear, action-oriented names
Focused Scenarios: Each scenario tests one workflow or feature
Real-World Use Cases: Base scenarios on actual user workflows
Error Cases: Include failure scenarios
Documentation: Add comments or docs explaining complex scenarios

Sequential Arrays

Context Clarity: Make it clear when queries reference previous results
- ✅ "now get the timezone for those coordinates"
- ❌ "get timezone"
Logical Flow: Order queries in natural progression
- ✅ geocode → timezone → street view
- ❌ street view → geocode → timezone
Self-Contained: Each scenario should work independently
Reasonable Length: 3-5 queries per scenario optimal
Descriptive Queries: Write queries as natural language

Command Strings

Explicit Tools: Reference tools by name when testing specific functionality
- ✅ "use slides-create-markdown to create test.md"
- ⚠️ "create a presentation" (works but less specific)
Complete Commands: Each command should be runnable independently
Test Data: Use consistent test filenames (test-*.md)
Workflow Scenarios: For complex operations, describe full workflow in one command
Clean Up: Document any cleanup needed after scenarios

Example Structures

Minimal Plugin (Command Strings)

json

{
  "name": "openpets/my-plugin",
  "scenarios": {
    "basic": "basic operation",
    "advanced": "advanced operation",
    "error": "test error handling"
  }
}

Complex Plugin (Sequential Arrays)

json

{
  "name": "openpets/my-plugin",
  "queries": [
    "quick test 1",
    "quick test 2"
  ],
  "scenarios": {
    "workflow-1": [
      "step 1",
      "step 2 using context",
      "step 3 using previous results"
    ],
    "workflow-2": [
      "different step 1",
      "different step 2"
    ],
    "error-handling": [
      "trigger error condition",
      "verify error message"
    ]
  },
  "scripts": {
    "test:queries": "../../src/shared-utils/test-queries.sh",
    "test:scenarios": "../../src/shared-utils/run-scenarios.sh",
    "test:workflow-1": "../../src/shared-utils/run-scenarios.sh workflow-1",
    "test:all": "npm run test:queries && npm run test:scenarios"
  }
}

Hybrid Approach

json

{
  "name": "openpets/my-plugin",
  "queries": [
    "quick smoke test"
  ],
  "scenarios": {
    "multi-step": [
      "step 1",
      "step 2"
    ],
    "single-operation": "standalone command"
  },
  "scripts": {
    "test:multi": "../../src/shared-utils/run-scenarios.sh multi-step",
    "test:single": "standalone command"
  }
}

Scenario Metadata (Optional)

Enhanced scenario definitions can include metadata:

json

{
  "scenarios": {
    "geocoding-workflow": {
      "description": "Test complete geocoding workflow from address to street view",
      "requires": ["GOOGLE_MAPS_API_KEY"],
      "timeout": 30000,
      "queries": [
        "find coordinates",
        "get timezone",
        "check street view"
      ]
    }
  }
}

Note: Metadata support depends on test runner implementation.

package.json - Scenario definitions for each plugin
../../src/shared-utils/test-queries.sh - Shared query test runner
../../src/shared-utils/run-scenarios.sh - Shared scenario test runner
TESTING.md - Plugin-specific testing documentation
README.md - Plugin documentation with usage examples

Contributing

When adding scenarios to a new plugin:

Choose approach based on plugin characteristics (see "Choosing an Approach")
Follow naming conventions for consistency
Include both happy paths and error cases
Document complex scenarios in plugin README
Add npm scripts for easy execution
Test scenarios before committing

Future Enhancements

Potential improvements to scenario system:

[ ] Unified scenario runner supporting both types
[ ] Scenario validation/linting
[ ] Automatic scenario generation from tool definitions
[ ] Performance benchmarking integration
[ ] CI/CD scenario reporting
[ ] Visual scenario debugging
[ ] Scenario coverage analysis

Questions & Support

For questions about scenarios:

Check plugin-specific TESTING.md or README.md
Review existing scenarios in similar plugins
See shared-utils documentation for test runners
Open an issue in the repository

Last Updated: 2025-11-18

OpenPets Scenario Documentation ​

Overview ​

Scenario Structure Types ​

Type 1: Sequential Query Arrays ​

Structure ​

Key Features ​

Example: video-vikhan ​

Type 2: Command Strings ​

Structure ​

Key Features ​

Example: slides-shiba ​

Comparison ​

Choosing an Approach ​

Use Sequential Query Arrays When: ​

Use Command Strings When: ​

Scenario Naming Conventions ​

For Sequential Arrays ​

For Command Strings ​

Running Scenarios ​

Sequential Query Arrays ​

Command Strings ​

Creating New Scenarios ​

Sequential Array Scenario ​

Command String Scenario ​

Scenario Categories ​

Functional Testing ​

Integration Testing ​

Error Handling ​

Format/Feature Variations ​

Best Practices ​

General ​

Sequential Arrays ​

Command Strings ​

Example Structures ​

Minimal Plugin (Command Strings) ​

Complex Plugin (Sequential Arrays) ​

Hybrid Approach ​

Scenario Metadata (Optional) ​

Related Files ​

Contributing ​

Future Enhancements ​

Questions & Support ​

OpenPets Scenario Documentation

Overview

Scenario Structure Types

Type 1: Sequential Query Arrays

Structure

Key Features

Example: video-vikhan

Type 2: Command Strings

Structure

Key Features

Example: slides-shiba

Comparison

Choosing an Approach

Use Sequential Query Arrays When:

Use Command Strings When:

Scenario Naming Conventions

For Sequential Arrays

For Command Strings

Running Scenarios

Sequential Query Arrays

Command Strings

Creating New Scenarios

Sequential Array Scenario

Command String Scenario

Scenario Categories

Functional Testing

Integration Testing

Error Handling

Format/Feature Variations

Best Practices

General

Sequential Arrays

Command Strings

Example Structures

Minimal Plugin (Command Strings)

Complex Plugin (Sequential Arrays)

Hybrid Approach

Scenario Metadata (Optional)

Related Files

Contributing

Future Enhancements

Questions & Support