From f3d0a536aa4cf921b77a0635fe7c7b2a46ef1e73 Mon Sep 17 00:00:00 2001 From: Matthew O'Riordan Date: Sun, 27 Apr 2025 18:54:43 +1200 Subject: [PATCH 1/5] docs(rules): Enhance AI agent guidance and development workflow Refactors documentation and `.cursor` rules to improve clarity, consistency, and effectiveness for AI agent interactions, based on analysis of recent development history (`.specstory`) and the corresponding TODO item. Key changes include: - **Created `.cursor/rules/WORKFLOW.mdc`:** Centralizes the mandatory 4-step development workflow (Build, Lint, Test, Document) required for all changes. - **Refactored `.cursor/rules`:** Updated `development.mdc`, `ably.mdc`, and `project.mdc` to remove redundancy, improve focus, add specific guidance (linting philosophy, Dockerfile tips), and link to the new `WORKFLOW.mdc` and relevant docs. - **Enhanced `docs/Testing.md`:** Added clearer instructions for running and debugging different test types (Unit, Integration, E2E, Playwright), emphasizing E2E/Integration preference for core Ably features and addressing common pitfalls observed in test runs. - **Added `CONTRIBUTING.md`:** Formalized the contribution process, emphasizing the mandatory workflow (file moved to root directory). - **Added `docs/DEBUGGING.md`:** Provided initial tips for troubleshooting common test failures and local CLI issues. - **Updated `README.md`:** Added a link to the new `CONTRIBUTING.md` file. - **Updated `docs/TODO.md`:** Marked the documentation review task as complete. --- .cursor/rules/WORKFLOW.mdc | 35 +++++++ CONTRIBUTING.md | 40 ++++++++ README.md | 5 + docs/DEBUGGING.md | 65 ++++++++++++ docs/TODO.md | 5 +- docs/Testing.md | 204 +++++++++++++++++++++---------------- 6 files changed, 267 insertions(+), 87 deletions(-) create mode 100644 .cursor/rules/WORKFLOW.mdc create mode 100644 CONTRIBUTING.md create mode 100644 docs/DEBUGGING.md diff --git a/.cursor/rules/WORKFLOW.mdc b/.cursor/rules/WORKFLOW.mdc new file mode 100644 index 000000000..22209ac15 --- /dev/null +++ b/.cursor/rules/WORKFLOW.mdc @@ -0,0 +1,35 @@ +--- +description: +globs: +alwaysApply: false +--- +# Mandatory Development Workflow + +**IMPORTANT:** Before considering any task complete, you **MUST** perform and verify the following steps in order. Failure to complete these steps means the work is **not finished**. + +1. **Run Build:** + * Execute `pnpm prepare`. + * **Purpose:** Ensures TypeScript compiles, `oclif.manifest.json` is updated, and `README.md` reflects command changes. + * **Verification:** Check for build errors in the output. Ensure `oclif.manifest.json` and `README.md` changes (if any) are sensible. + +2. **Run Linter:** + * Execute `pnpm exec eslint .` (or `pnpm exec eslint -- [filepath]` for specific files). + * **Purpose:** Ensures code adheres to project style and best practices. Catches potential errors. + * **Verification:** Ensure the command exits with code 0 (no errors). **Do not** use workarounds like prefixing unused variables with `_`; address the root cause (e.g., remove unused code). See `development.mdc` for more details on linting philosophy. + +3. **Run Tests:** + * Execute relevant tests locally (e.g., `pnpm test:unit`, `pnpm test:integration`, `pnpm test:e2e`, `pnpm test:playwright`, or specific file paths like `pnpm test test/unit/commands/some-command.test.ts`). + * **Purpose:** Verifies changes haven't introduced regressions and that new features work as expected. + * **Verification:** Ensure all executed tests pass. If tests fail, debug them (see `docs/DEBUGGING.md`). Add or update tests (Unit, Integration, E2E) as appropriate for your changes. Refer to `docs/Testing.md` for the testing strategy. + +4. **Update Documentation & Rules:** + * Review and update all relevant documentation and rule files based on your changes. + * **Checklist:** + * `README.md`: Especially the command list if commands were added/changed (often handled by `pnpm prepare`). + * `docs/*.md`: Any file impacted by the changes (e.g., `Product-Requirements.md`, `Project-Structure.md`, `Testing.md`). + * `.cursor/rules/*.mdc`: Any rule file impacted by changes to development practices, Ably usage, project structure, etc. + * `docs/TODO.md`: Update any related tasks. + * **Purpose:** Keeps project knowledge current for both humans and AI agents. + * **Verification:** Manually confirm that documentation accurately reflects the implemented changes. + +**Only after successfully completing ALL four steps should you consider your task complete.** diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 000000000..36f2f6a83 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,40 @@ +# Contributing to the Ably CLI + +Thank you for your interest in contributing to the Ably CLI! + +## Development Workflow + +All code changes, whether features or bug fixes, **MUST** follow the mandatory workflow outlined in [.cursor/rules/WORKFLOW.mdc](mdc:.cursor/rules/WORKFLOW.mdc). + +In summary, this involves: + +1. **Build:** Run `pnpm prepare` to compile, update manifests, and update the README. +2. **Lint:** Run `pnpm exec eslint .` and fix all errors/warnings. +3. **Test:** Run relevant tests (`pnpm test:unit`, `pnpm test:integration`, `pnpm test:e2e`, `pnpm test:playwright`, or specific files) and ensure they pass. Add new tests or update existing ones as needed. +4. **Document:** Update all relevant documentation (`docs/`, `.cursor/rules/`, `README.md`) to reflect your changes. + +**Pull requests will not be merged unless all these steps are completed and verified.** + +## Key Documents + +Before starting work, please familiarize yourself with: + +* [Product Requirements](mdc:docs/Product-Requirements.md): Understand the goals and features. +* [Project Structure](mdc:docs/Project-Structure.md): Know where different code components live. +* [Testing Strategy](mdc:docs/Testing.md): Understand the different types of tests and how to run them. +* [Development Rules](mdc:.cursor/rules/development.mdc): Coding standards, linting, dependency management. +* [Ably Rules](mdc:.cursor/rules/ably.mdc): How to interact with Ably APIs/SDKs. + +## Reporting Issues + +Please report bugs or suggest features using GitHub Issues. + +## Submitting Pull Requests + +1. Fork the repository. +2. Create a new branch for your feature or bug fix. +3. Make your changes, ensuring you follow the **Mandatory Development Workflow** described above. +4. Commit your changes with clear messages. +5. Push your branch to your fork. +6. Create a Pull Request against the `main` branch of the `ably/cli` repository. +7. Ensure all CI checks pass. diff --git a/README.md b/README.md index def6f1e71..85efb69c6 100644 --- a/README.md +++ b/README.md @@ -4741,5 +4741,10 @@ The MCP server also provides the following resources: - `channel_presence` - Current presence members on a specific Ably channel # Terminal Server +## Contributing This repository includes a terminal server that provides a bash environment over WebSockets for the web based CLI. See [Server-Setup.md](./docs/Server-Setup.md) for more info on how it works and how to deploy it. + +# Contributing + +Please see [CONTRIBUTING.md](CONTRIBUTING.md) for details on how to contribute to this project. diff --git a/docs/DEBUGGING.md b/docs/DEBUGGING.md new file mode 100644 index 000000000..dcbe23289 --- /dev/null +++ b/docs/DEBUGGING.md @@ -0,0 +1,65 @@ +# Debugging Guide + +This guide provides tips for debugging common issues when developing the Ably CLI. + +## General Tips + +* **Check Logs:** Look for errors or relevant messages in the CLI output, test runner output, server logs (for Web CLI tests), or browser console (for Web CLI tests). +* **Isolate the Issue:** Try to reproduce the problem with the simplest possible command or test case. Comment out parts of the code or test to narrow down the source of the error. +* **Consult Documentation:** Review relevant project docs (`docs/`, `.cursor/rules/`) and Ably documentation (). + +## Debugging Tests + +Refer to [docs/Testing.md](mdc:docs/Testing.md) for how to run specific tests. + +### Mocha Tests (Unit, Integration, E2E) + +* **Verbose Output:** Run Mocha with increased verbosity if needed (though the default `spec` reporter is usually detailed). Check `scripts/run-tests.sh` for current settings. +* **Debugger:** Use the Node.js inspector: + ```bash + # Add --inspect-brk flag + node --inspect-brk --import 'data:text/javascript,...' ./node_modules/mocha/bin/mocha ... [your test path] + ``` + Then attach your debugger (e.g., Chrome DevTools, VS Code debugger). +* **`console.log`:** Add temporary `console.log` statements in the test or the code being tested. +* **Mocking Issues (Unit/Integration):** + * Verify mocks (`sinon`, `nock`) are correctly set up and restored (`beforeEach`/`afterEach`). + * Ensure stubs match the actual function signatures. + * Check that network requests are being intercepted as expected (e.g., using `nock.recorder`). +* **E2E Failures:** + * **Credentials:** Ensure `E2E_ABLY_API_KEY` (and any other required keys) are correctly set in your environment (`.env` file locally, secrets in CI). + * **Network:** Check for network connectivity issues to Ably services. + * **Resource Conflicts:** Ensure previous test runs cleaned up resources correctly (e.g., unique channel names per test run). + * **CI vs. Local:** If tests fail only in CI, suspect environment differences (Node version, dependencies, permissions, resources). Check CI logs carefully. + +### Playwright Tests (Web CLI) + +* **Test Output:** Playwright provides detailed error messages, including: + * The specific action that failed (e.g., `locator.waitFor`, `expect.toContainText`). + * The expected vs. received values. + * Call logs showing the sequence of actions. +* **Error Context:** Check the linked `error-context.md` file in `test-results/` for screenshots, DOM snapshots, and console logs at the point of failure. +* **Browser Console:** Add `page.on('console', ...)` listeners in your test (as shown in `.specstory` examples) to capture browser logs. +* **Debugging UI Mode:** Run Playwright with the UI for interactive debugging: + ```bash + pnpm exec playwright test test/e2e/web-cli/web-cli.test.ts --ui + ``` +* **Common Issues:** + * **Selector Timeouts:** The element didn't appear within the timeout. Check if the server/app started correctly, if there were errors, or if the selector is correct. + * **Incorrect Text/Assertions:** The expected text doesn't match the actual text in the terminal (check for subtle differences like whitespace, case sensitivity, or ANSI codes if not using `toContainText`). + * **Connection Errors:** Check browser console logs and terminal server logs for WebSocket connection issues (e.g., wrong port, server crash, `ERR_CONNECTION_REFUSED`). + * **Build Artifacts:** Ensure the Web CLI example (`examples/web-cli`) was built successfully (`pnpm --filter ably-web-cli-example build`) before the test runs, especially in CI. + +## Debugging the CLI Locally + +* **Run with Node Inspector:** + ```bash + node --inspect-brk bin/run.js [your command and flags] + ``` + Attach your debugger. +* **Verbose Flags:** Use the CLI's built-in `--verbose` flag if available for the command. +* **Oclif Debugging:** Set the `DEBUG` environment variable for oclif internal logs: + ```bash + DEBUG=oclif* bin/run.js [command] + ``` +* **Check Configuration:** Use `ably config` (opens the config file) to verify stored credentials or settings. diff --git a/docs/TODO.md b/docs/TODO.md index b3fcb1936..8ea9a06da 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -95,8 +95,9 @@ - [ ] Now that we have .editorconfig, ensure all files adhere in one commit - [ ] We are using a PNPM workspace, but I am not convinced that's a good thing. We should consider not letting the examples or React component dependencies affect the core CLI packaging. - [ ] Publish Docker image to Github registry and use a path such as `ghcr.io/ably/ably-cli-sandbox` for the published artefact. Building and publishing should use the locally built Ably CLI binary as opposed to the latest version so that local changes can be tested locally. -- [ ] Review the Cursor rules and Docs to ensure they are effective for prompting (use Claude's [prompt improver](https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/prompt-improver)) - - [ ] Ensure we have more specific rules around what is needed from test coverage based on what we've seen so far from LLMs. a) Linting and running tests is a must, b) docs and TODOs must be updated, c) output from tests should not be polluted with output from the CLI unless there are errors, d) have we learnt about unit tests being tricky/not adding enough value, so we should focos on integration & E2E test more, e) why are docs not being updated and linting always checked as a matter of course, f) frequently see erorrs like "Could not find file 'src/commands/channels/batch-publish.js' in the workspace" - should we recommend to LLMs to use .ts files, and also provide more guidance on how to run tests in verbose mode when executing directly and which commands to use, g) some prompts have tried to inject an API key as follows `ABLY_API_KEY=YOUR_KEY_HERE` into tests, but this is not necessary given API keys are in .env, tell the LLM where linting happens "Could not find file '.eslintrc.js' in the workspace.", h) working around linting issues by prefixing unused variables with a _ is a perfect example of the trying to solve the wrong problem. if a variable is not used and the linter is telling us that, then we should remove the varaible, not put a prefix on it so the acutal issue ignored, i) when vars aren't used, agent sometimes just comments them out or uses _ to hide the warnings instead of solving the problem (remove unused code) +- [x] Review the Cursor rules and Docs to ensure they are effective for prompting + - *Done: 2025-04-27* + - *Summary: Refactored `.cursor/rules`, updated `docs/Testing.md`, added `docs/CONTRIBUTING.md` and `docs/DEBUGGING.md` based on analysis of `TODO.md`, recent `.specstory` history, and Anthropic best practices. Created `WORKFLOW.mdc` to centralize mandatory steps.* ## Bugs diff --git a/docs/Testing.md b/docs/Testing.md index f9a2bbe11..d2010e0c9 100644 --- a/docs/Testing.md +++ b/docs/Testing.md @@ -1,99 +1,133 @@ -# Testing strategy & policy +# Testing Strategy & Policy + +See also: [Debugging Guide](mdc:docs/DEBUGGING.md) for troubleshooting tips. ## Testing Goals & Guiding Principles -1. Confidence: Ensure each command works as intended and avoid regressions. -2. Speed & Developer Experience: Most tests should be quick to run, easy to debug, and not require a live environment. -3. Real Integration Coverage (where needed): Some commands may need to be tested against real APIs (e.g. Ably's pub/sub product APIs and Control APIs) to verify end-to-end flows—especially for mission-critical commands. -4. Scalability: The test setup should scale as commands grow in complexity. -5. Mandatory Coverage: Adding or updating relevant tests is a required step for all feature additions or bug fixes. +1. **Confidence:** Ensure each command works as intended and avoid regressions. +2. **Speed & Developer Experience:** Most tests should be quick to run, easy to debug, and not require a live environment. +3. **Real Integration Coverage (where needed):** Some commands may need to be tested against real APIs (e.g., Ably's pub/sub product APIs and Control APIs) to verify end-to-end flows—especially for mission-critical commands. +4. **Scalability:** The test setup should scale as commands grow in complexity. +5. **Mandatory Coverage:** Adding or updating relevant tests is a **required** step for all feature additions or bug fixes. See [.cursor/rules/WORKFLOW.mdc](mdc:.cursor/rules/WORKFLOW.mdc). ## Testing Approach -### Unit Tests (Command-level, stubbed HTTP calls) - -- Primary Purpose: Quickly verify logic, parse user inputs, handle expected errors. -- Test Speed: Very fast; no external network dependency. - - Tools & Techniques: - - Mocha for testing framework along with @oclif/test - - Nock for stubbing HTTP requests. - - sinon for function/method stubs or spies. - - [nyc](mdc:https:/npm.im/nyc) for code coverage (bundled with oclif) -- Example: Testing the `ably apps create` command, ensuring that if you pass `--name myApp`, it calls the POST /apps endpoint with the correct payload. - -### Integration Tests (Higher-level, partial real environment) - -- Primary Purpose: Verify multiple commands or flows together, but still mostly stub out network calls. -- Key Differences from Unit: - - Might test a sequence of commands to ensure they produce expected output or state. - - Use a test user or ephemeral environment tokens, but still stub the heavy-lifting network calls. -- Tools & Techniques: - - execa or child_process.spawn to run the CLI in a subprocess, capturing stdout/stderr. - - Maintain a local mock server for REST opeations if some calls are required. - -### End-to-End Tests (Real environment) - -- Primary Purpose: Ensure that the entire command pipeline works with real Ably endpoints. -- Scope: Test critical flows (e.g., creating an app, listing channels, sending messages, etc.) with real Ably credentials. -- Challenges: Requires stable environment, correct credentials, potential costs if usage-based. -- Frequency: Run on merges to main and pull requests, but not typically locally -- Tools & Techniques: - - Same as integration tests (execa/spawn), but with real environment variables/tokens. - - Secret credetials to be stored GitHub Actions (e.g., via repository secrets). - -## Code Base Integration & Structure +### Unit Tests (`test/unit`) + +* **Primary Purpose:** Quickly verify command logic, flag parsing, input validation, error handling, and basic output formatting **in isolation**. Focus on testing individual functions or methods within a command class. +* **Dependencies:** **MUST** stub/mock all external dependencies (Ably SDK calls, Control API requests, filesystem access, `ConfigManager`, etc.). Use libraries like `sinon` and `nock`. +* **Speed:** Very fast; no network or filesystem dependency. +* **Value:** Useful for testing complex parsing, conditional logic, and edge cases within a command, but **less effective** at verifying core interactions with Ably services compared to Integration/E2E tests. +* **Tools:** Mocha, `@oclif/test`, `sinon`. +* **Example:** Testing `ably apps create --name myApp` ensures correct arguments are parsed and the *mocked* Control API call is made with the expected payload. + +### Integration Tests (`test/integration`) + +* **Primary Purpose:** Verify the interaction between multiple commands or components, including interactions with *mocked* Ably SDKs or Control API services. Test the CLI execution flow. +* **Dependencies:** Primarily stub/mock network calls (`nock` for Control API, `sinon` stubs for SDK methods), but may interact with the local filesystem for config management (ensure isolation). Use `ConfigManager` mocks. +* **Speed:** Relatively fast; generally avoids real network latency. +* **Value:** Good for testing command sequences (e.g., `config set` then `config get`), authentication flow logic (with mocked credentials), and ensuring different parts of the CLI work together correctly without relying on live Ably infrastructure. +* **Tools:** Mocha, `@oclif/test`, `nock`, `sinon`, `execa` (to run the CLI as a subprocess). +* **Example:** Testing `ably accounts login` (mocked browser flow) -> `ably apps list` (mocked Control API) -> `ably apps switch` -> `ably apps current` (mocked Control API). + +### End-to-End (E2E) Tests (`test/e2e`) + +* **Primary Purpose:** Verify critical user flows work correctly against **real Ably services** using actual credentials (provided via environment variables). +* **Dependencies:** Requires a live Ably account and network connectivity. Uses real Ably SDKs and Control API interactions. +* **Scope:** Focus on essential commands and common workflows (login, app/key management basics, channel publish/subscribe/presence/history, logs subscribe). +* **Speed:** Slowest test type due to network latency and real API interactions. +* **Value:** Provides the highest confidence that the CLI works correctly for end-users in a real environment. **Preferred** over unit tests for verifying core Ably interactions. +* **Tools:** Mocha, `@oclif/test`, `execa`, environment variables (`E2E_ABLY_API_KEY`, etc.). +* **Frequency:** Run automatically in CI (GitHub Actions) on PRs and merges. Can be run locally but may incur costs. +* **Example:** Running `ably channels publish test-channel hello` and then running `ably channels subscribe test-channel` in a separate process to verify the message is received. + +### Playwright Tests (`test/e2e/web-cli`) + +* **Primary Purpose:** Verify the functionality of the Web CLI example application (`examples/web-cli`) running in a real browser. +* **Dependencies:** Requires Docker, Node.js, a browser (installed via Playwright), and the Web CLI example app to be built. +* **Speed:** Slow; involves starting servers, Docker containers, and browser automation. +* **Value:** Ensures the embeddable React component, terminal server, and containerized CLI work together as expected. +* **Tools:** Playwright Test runner (`@playwright/test`), Docker. +* **Frequency:** Run automatically in CI, separate from Mocha tests. + +## Running Tests + +Refer to [.cursor/rules/WORKFLOW.mdc](mdc:.cursor/rules/WORKFLOW.mdc) for the mandatory requirement to run tests. + +* **Run All (Unit, Integration, E2E - excluding Playwright):** + ```bash + pnpm test + ``` +* **Run Only Unit Tests:** + ```bash + pnpm test:unit + ``` +* **Run Only Integration Tests:** + ```bash + pnpm test:integration + ``` +* **Run Only E2E Tests (Mocha-based, excluding Playwright):** + ```bash + pnpm test:e2e + ``` +* **Run Only Playwright Web CLI Tests:** + ```bash + pnpm test:playwright + ``` +* **Run Specific File(s):** (Uses Mocha runner by default) + ```bash + # Example: Run a specific unit test file + pnpm test test/unit/commands/apps/create.test.ts + + # Example: Run all tests in the integration/core directory + pnpm test test/integration/core/**/*.test.ts + + # Example: Run a specific Playwright test file (uses Playwright runner) + pnpm test test/e2e/web-cli/web-cli.test.ts + ``` +* **Debugging Tests:** See [docs/DEBUGGING.md](mdc:docs/DEBUGGING.md). + +## Test Coverage and Considerations + +* **Adding/Updating Tests:** When adding features or fixing bugs, add or update tests in the appropriate category (Unit, Integration, E2E, Playwright). +* **Focus:** Prioritize **Integration and E2E tests** for verifying core functionality involving Ably APIs/SDKs, as unit tests with extensive mocking provide less confidence in these areas. +* **Output Modes:** Tests should cover different output modes where relevant: + * Default (Human-readable) + * JSON (`--json`) + * Pretty JSON (`--pretty-json`) +* **Web CLI Mode:** Integration/E2E tests for commands with different behavior in Web CLI mode should simulate this using `ABLY_WEB_CLI_MODE=true` environment variable. The Playwright tests cover the actual Web CLI environment. +* **Test Output:** Test output (stdout/stderr) should be clean. Avoid polluting test logs with unnecessary debug output from the CLI itself. Failures should provide clear error messages. +* **Asynchronous Operations:** Use `async/await` properly. Avoid brittle `setTimeout` calls where possible; use event listeners or promise-based waits. +* **Resource Cleanup:** Ensure tests clean up resources (e.g., close Ably clients, kill subprocesses, delete temp files). Use the `afterEach` or `afterAll` hooks and helpers like `trackAblyClient`. +* **Realtime SDK Stubbing:** For Unit/Integration tests involving the Realtime SDK, stub the SDK methods directly (`sinon.stub(ably.channels.get('...'), 'subscribe')`) rather than trying to mock the underlying WebSocket, which is complex and brittle. +* **Credentials:** E2E tests rely on `E2E_ABLY_API_KEY` (and potentially others) being set in the environment (locally via `.env` or in CI via secrets). **Never** hardcode credentials in tests. + +## Codebase Integration & Structure ### Folder Structure -The CLI will use common folder structures for tests as follows: - +``` . ├── src -│ ├── commands -│ │ ├── apps -│ │ │ └── create.ts -│ │ └── realtime -│ │ └── connect.ts -│ └── ... -├── test -│ ├── unit -│ │ ├── apps -│ │ │ └── create.test.ts -│ │ └── realtime -│ │ └── connect.test.ts -│ ├── integration -│ │ └── cli-flows.test.ts -│ └── e2e -│ └── e2e.test.ts +│ └── commands/ +├── test/ +│ ├── e2e/ # End-to-End tests (runs against real Ably) +│ │ ├── core/ # Core CLI functionality E2E tests +│ │ ├── channels/ # Channel-specific E2E tests +│ │ └── web-cli/ # Playwright tests for the Web CLI example +│ │ └── web-cli.test.ts +│ ├── helpers/ # Test helper functions (e.g., e2e-test-helper.ts) +│ ├── integration/ # Integration tests (mocked external services) +│ │ └── core/ +│ ├── unit/ # Unit tests (isolated logic, heavy mocking) +│ │ ├── base/ +│ │ ├── commands/ +│ │ └── services/ +│ ├── setup.ts # Full setup for E2E tests (runs in Mocha context) +│ └── mini-setup.ts # Minimal setup for Unit/Integration tests └── ... +``` -- test/unit mirrors the command structure, focusing on command-level logic. -- test/integration contains broader tests of multiple commands or realistic user flows, but generally mocking out network. -- test/e2e has real environment-dependent tests, possibly fewer in number, but ensuring real interactions. - -### E2E Test Organization for Channel Commands - -For end-to-end tests, especially for complex command groups like the `channels` commands, we use the following structure: - -- Root level commands (e.g., `channels list`, `channels publish`) are tested in a file named `[command-group]-e2e.test.ts` (e.g., `channels-e2e.test.ts`). -- Sub-topic commands (e.g., `channels presence`, `channels occupancy`) are tested in dedicated files named `[command-group]-[sub-topic]-e2e.test.ts` (e.g., `channel-presence-e2e.test.ts`, `channel-occupancy-e2e.test.ts`). - -This organization: -1. Makes it easier to find and maintain tests for specific features -2. Separates concerns for better readability and debugging -3. Allows for more efficient test execution (run only the tests related to a particular feature) -4. Prevents test files from growing too large - -The e2e tests leverage shared helper functions from the `test/helpers/e2e-test-helper.js` file to reduce code duplication and ensure consistent test behavior. - -## Testing coverage and considerations +### E2E Test Organization -- Each command's code should be isolated code by stubbing out all network requests and external dependencies to ensure we have coverage over expected inputs and outputs -- All tests should provide coverage for the following modes: - - Local CLI human readable - primary use case for the CLI is users executing the CLI locally and expecting human readable responses - - Local JSON mode (`--json`) - this use case is for developers using the CLI in their scripts, such as bash commands with `jq` to parse responses. It is important the only output from commands with `--json` is JSON without any superfluous information such as state change events unnecessary for the command result. - - Local Pretty JSON mode (`--pretty-json`) - this use case is for developers using the CLI locally but preferring to see raw JSON output. We need to ensure JSON is colour coded. - - Web CLI mode - in this mode the CLI is invoked from a Docker container via the web terminal we provide. Tests should simulate the Web CLI mode by injecting the `ABLY_WEB_CLI_MODE=true` environment variable when runnin commands. In addition, we should have a few end to end tests that execute commands directly to the Docker container itself, and thne some tests should be run to spin up the example web server and ensure the basic features of the Web CLI works in the browser. -- For tests that subscribe or run on indefinitely waiting for events, the test suite will have to send a SIGINT to stop the command. -- Testing commands where Realtime Ably clients are used, it's not practical to stub the network requests as the underlying websocket protocol is complicated which would result in this test suite being too coupled to the underlying protocol. Stubbing should be done on the SDK interfaces themselves instead. -- Some end to end tests will exist that concurrent perform actions like publishing and subscribing to messages to ensure the CLI works with real network requests. +E2E tests are organized by feature/topic (e.g., `channels-e2e.test.ts`, `presence-e2e.test.ts`) to improve maintainability and allow targeted runs. They use shared helpers from `test/helpers/e2e-test-helper.ts`. From 4f5b6103616d9ea7c21b4cc1a4dbe4b23793a866 Mon Sep 17 00:00:00 2001 From: Matthew O'Riordan Date: Sun, 27 Apr 2025 19:06:25 +1200 Subject: [PATCH 2/5] docs: improve documentation and AI assistance guidelines - Create agent-agnostic .cursor/rules/AI_ASSISTANCE.mdc with common patterns and code examples - Add detailed TROUBLESHOOTING.md with solutions for common errors - Enhance Testing.md with better visual hierarchy and progressive disclosure - Move CONTRIBUTING.md to root directory following standard conventions - Update README.md with link to CONTRIBUTING.md --- .cursor/rules/AI_ASSISTANCE.mdc | 169 ++++++++++++++++++++++++ README.md | 2 +- docs/TODO.md | 2 +- docs/TROUBLESHOOTING.md | 214 +++++++++++++++++++++++++++++++ docs/Testing.md | 219 +++++++++++++++++++++++++------- 5 files changed, 556 insertions(+), 50 deletions(-) create mode 100644 .cursor/rules/AI_ASSISTANCE.mdc create mode 100644 docs/TROUBLESHOOTING.md diff --git a/.cursor/rules/AI_ASSISTANCE.mdc b/.cursor/rules/AI_ASSISTANCE.mdc new file mode 100644 index 000000000..0e93f58ee --- /dev/null +++ b/.cursor/rules/AI_ASSISTANCE.mdc @@ -0,0 +1,169 @@ +--- +description: +globs: +alwaysApply: false +--- +# AI Assistance Guidelines + +This document provides guidance for AI assistants working with the Ably CLI codebase. These guidelines are agent-agnostic and designed to help any AI assistant provide high-quality contributions. + +## Approaching This Codebase + +### Understanding the Project + +1. **Start with Key Documentation**: + - [Product Requirements](mdc:docs/Product-Requirements.md) + - [Project Structure](mdc:docs/Project-Structure.md) + - [Testing Strategy](mdc:docs/Testing.md) + - [Mandatory Workflow](mdc:.cursor/rules/WORKFLOW.mdc) + +2. **Recognize Context Boundaries**: + - The CLI is focused on Ably services - do not suggest features outside this scope + - Always check if a feature already exists before suggesting implementation + +### Common Patterns + +1. **Command Structure**: + ```typescript + // Commands should follow the oclif structure + export default class MyCommand extends Command { + static description = 'Clear description of what the command does' + + static examples = [ + '<%= config.bin %> commandName', + '<%= config.bin %> commandName --some-flag', + ] + + static flags = { + // Flag definitions + } + + static args = { + // Argument definitions + } + + async run() { + const {args, flags} = await this.parse(MyCommand) + // Implementation + } + } + ``` + +2. **Authorization Pattern**: + ```typescript + // Command should support auth via the standard auth helper + import { getAuth } from '../../helpers/auth' + + // Then in the run method: + const auth = await getAuth(flags) + const client = new Ably.Rest(auth) + ``` + +3. **Error Handling**: + ```typescript + try { + // Operation that might fail + } catch (error) { + if (error.code === 40100) { + this.error('Authentication failed. Check your API key or token.') + } else { + this.error(`Failed: ${error.message}`) + } + } + ``` + +### Anti-Patterns to Avoid + +1. **❌ Direct HTTP Requests for Data Plane APIs** + ```typescript + // WRONG: Using fetch directly for data plane operations + const response = await fetch('https://rest.ably.io/channels/...') + ``` + + **✅ Correct Approach** + ```typescript + // CORRECT: Using the Ably SDK + const client = new Ably.Rest(auth) + const channel = client.channels.get('channel-name') + ``` + +2. **❌ Inconsistent Command Structure** + ```typescript + // WRONG: Non-standard command structure + export default class { + async execute(args) { + // Implementation + } + } + ``` + +3. **❌ Hardcoded Endpoints** + ```typescript + // WRONG: Hardcoded endpoint URLs + const url = 'https://control.ably.net/v1/apps' + ``` + + **✅ Correct Approach** + ```typescript + // CORRECT: Using the config + const controlHost = flags.controlHost || config.controlHost || 'control.ably.net' + const url = `https://${controlHost}/v1/apps` + ``` + +## Effective Testing + +1. **Mocking Ably SDK Example**: + ```typescript + // Example of proper Ably SDK mocking + import * as sinon from 'sinon' + + describe('my command', () => { + let mockChannel: any + let mockClient: any + + beforeEach(() => { + mockChannel = { + publish: sinon.stub().resolves(), + presence: { + get: sinon.stub().resolves([]), + enter: sinon.stub().resolves(), + }, + } + + mockClient = { + channels: { + get: sinon.stub().returns(mockChannel), + }, + close: sinon.stub().resolves(), + } + + // Important: stub the constructor, not just methods + sinon.stub(Ably, 'Rest').returns(mockClient) + }) + + afterEach(() => { + sinon.restore() // Don't forget cleanup! + }) + }) + ``` + +2. **Testing Resource Cleanup**: + Always ensure resources are properly closed/cleaned up, especially in tests: + ```typescript + // Example of proper resource cleanup + afterEach(async () => { + await client.close() + sinon.restore() + }) + ``` + +## Contributing Workflow + +All contributions must follow the [Mandatory Workflow](mdc:.cursor/rules/WORKFLOW.mdc): + +1. **Build**: Run `pnpm prepare` +2. **Lint**: Run `pnpm exec eslint .` +3. **Test**: Run appropriate tests +4. **Document**: Update relevant docs + +See [CONTRIBUTING.md](mdc:CONTRIBUTING.md) for details. diff --git a/README.md b/README.md index 85efb69c6..24cffdd98 100644 --- a/README.md +++ b/README.md @@ -4747,4 +4747,4 @@ This repository includes a terminal server that provides a bash environment over # Contributing -Please see [CONTRIBUTING.md](CONTRIBUTING.md) for details on how to contribute to this project. +Please see [CONTRIBUTING.md](CONTRIBUTING.md) for details on how to contribute to this project, including our mandatory development workflow, testing requirements, and code quality standards. diff --git a/docs/TODO.md b/docs/TODO.md index 8ea9a06da..a93adac6f 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -97,7 +97,7 @@ - [ ] Publish Docker image to Github registry and use a path such as `ghcr.io/ably/ably-cli-sandbox` for the published artefact. Building and publishing should use the locally built Ably CLI binary as opposed to the latest version so that local changes can be tested locally. - [x] Review the Cursor rules and Docs to ensure they are effective for prompting - *Done: 2025-04-27* - - *Summary: Refactored `.cursor/rules`, updated `docs/Testing.md`, added `docs/CONTRIBUTING.md` and `docs/DEBUGGING.md` based on analysis of `TODO.md`, recent `.specstory` history, and Anthropic best practices. Created `WORKFLOW.mdc` to centralize mandatory steps.* + - *Summary: Refactored `.cursor/rules`, updated `docs/Testing.md`, added `CONTRIBUTING.md` and `docs/DEBUGGING.md` based on analysis of the `.specstory` files and Anthropic best practices. Added code examples, troubleshooting guide, and agent-agnostic AI assistance rules. Enhanced visual hierarchy and progressive disclosure in documentation. Created `WORKFLOW.mdc` to centralize mandatory steps.* ## Bugs diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md new file mode 100644 index 000000000..f80424845 --- /dev/null +++ b/docs/TROUBLESHOOTING.md @@ -0,0 +1,214 @@ +# Troubleshooting Guide + +This document provides solutions for common issues encountered when developing or testing the Ably CLI. + +--- + +## Common Build and Testing Errors + +### `.js` vs `.ts` Extension Issues + +**Problem**: Tests failing with errors about modules not being found or incorrect paths. + +**Example Error**: +``` +Error: Cannot find module '../commands/publish' +``` + +**Solution**: +- Check import statements and ensure they reference `.ts` files, not `.js` files. +- When running tests, remember that imports in test files should use the `.ts` extension. + +```typescript +// ❌ INCORRECT +import MyCommand from '../../src/commands/my-command' + +// ✅ CORRECT +import MyCommand from '../../src/commands/my-command.ts' +``` + +--- + +### Memory Leaks in Tests + +**Problem**: Tests fail with memory errors or hang indefinitely. + +**Example Error**: +``` +FATAL ERROR: JavaScript heap out of memory +``` + +**Solution**: +- Ensure all resources are properly closed, especially Ably clients: + +```typescript +// Always close Ably clients in tests +afterEach(async () => { + await client.close() +}) + +// For multiple clients, ensure all are closed +afterEach(async () => { + await Promise.all([ + client1.close(), + client2.close(), + ]) +}) +``` + +- Check for unclosed connections or long-running promises that never resolve +- Verify that `sinon.restore()` is called in `afterEach` blocks to clean up stubs/mocks + +--- + +### WebSocket Mocking Challenges + +**Problem**: Tests involving WebSocket connections fail or hang. + +**Example Error**: +``` +Timeout of 2000ms exceeded +``` + +**Solution**: +- For tests involving Realtime connections, be sure to mock the WebSocket properly: + +```typescript +// Example of properly mocking a WebSocket connection +beforeEach(() => { + // Create a fake WebSocket implementation + const fakeWebSocketInstance = { + addEventListener: sinon.stub(), + removeEventListener: sinon.stub(), + send: sinon.stub(), + close: sinon.stub() + } + + // Mock the WebSocket constructor + global.WebSocket = sinon.stub().returns(fakeWebSocketInstance) as any +}) + +afterEach(() => { + // Clean up + delete (global as any).WebSocket +}) +``` + +- Always trigger the appropriate events to simulate connection success/failure +- Don't forget to restore the original WebSocket constructor after tests + +--- + +### HTTP Request Mocking Issues + +**Problem**: Tests involving HTTP requests fail with network errors. + +**Example Error**: +``` +Error: connect ECONNREFUSED +``` + +**Solution**: +- Use `nock` or `sinon` to properly mock HTTP requests: + +```typescript +// Using nock for HTTP mocking +import * as nock from 'nock' + +beforeEach(() => { + nock('https://control.ably.net') + .get('/v1/apps') + .reply(200, { apps: [] }) +}) + +afterEach(() => { + nock.cleanAll() +}) +``` + +- Ensure all expected HTTP requests are mocked +- For control API requests, check the host being used matches the mocked domain + +--- + +## Running the CLI Locally + +### Command Not Found + +**Problem**: Unable to run the CLI locally with the `ably` command. + +**Solution**: +- Link the CLI locally: + ```bash + pnpm link --global + ``` +- Make sure the CLI is built before linking: + ```bash + pnpm prepare && pnpm link --global + ``` + +### Environment Variables + +**Problem**: CLI not using the expected configuration. + +**Solution**: +- Check your local configuration with: + ```bash + ably config + ``` +- Use environment variables to override config for testing: + ```bash + ABLY_API_KEY=your_key ably channels:list + ``` + +--- + +## Linting and Formatting Issues + +### ESLint Errors + +**Problem**: ESLint reporting errors that don't make sense. + +**Solution**: +- Clear ESLint cache and try again: + ```bash + pnpm exec eslint --cache --cache-location .eslintcache . + ``` +- For specific files: + ```bash + pnpm exec eslint -- path/to/file.ts + ``` + +### TypeScript Type Errors + +**Problem**: TypeScript compilation errors. + +**Example Error**: +``` +Property 'x' does not exist on type 'Y' +``` + +**Solution**: +- Check that type definitions are up to date: + ```bash + pnpm install @types/node@latest @types/mocha@latest + ``` +- Use proper type assertions when necessary: + ```typescript + const result = (response as any).items as Item[] + ``` +- Add missing type definitions: + ```typescript + interface MyConfig { + apiKey?: string + controlHost?: string + } + ``` + +--- + +## Documentation Issues + +If you find errors in documentation or rules, please update them using the proper workflow and submit a pull request. + +See [CONTRIBUTING.md](mdc:CONTRIBUTING.md) for more details on the development workflow. diff --git a/docs/Testing.md b/docs/Testing.md index d2010e0c9..53ac19ae5 100644 --- a/docs/Testing.md +++ b/docs/Testing.md @@ -1,36 +1,136 @@ # Testing Strategy & Policy -See also: [Debugging Guide](mdc:docs/DEBUGGING.md) for troubleshooting tips. +
+

📘 ESSENTIALS FIRST 📘

+
-## Testing Goals & Guiding Principles +> **💡 QUICK START:** Run `pnpm test` for all tests or `pnpm test:unit` for faster unit tests. +> **📋 MANDATORY:** All code changes require related tests. See [WORKFLOW.mdc](mdc:.cursor/rules/WORKFLOW.mdc). +> **🐛 DEBUGGING:** See [Debugging Guide](mdc:docs/DEBUGGING.md) for troubleshooting tips. +> **🔍 TROUBLESHOOTING:** See [Troubleshooting Guide](mdc:docs/TROUBLESHOOTING.md) for common errors. + +--- + +## 🚀 Testing Goals & Guiding Principles 1. **Confidence:** Ensure each command works as intended and avoid regressions. 2. **Speed & Developer Experience:** Most tests should be quick to run, easy to debug, and not require a live environment. 3. **Real Integration Coverage (where needed):** Some commands may need to be tested against real APIs (e.g., Ably's pub/sub product APIs and Control APIs) to verify end-to-end flows—especially for mission-critical commands. 4. **Scalability:** The test setup should scale as commands grow in complexity. -5. **Mandatory Coverage:** Adding or updating relevant tests is a **required** step for all feature additions or bug fixes. See [.cursor/rules/WORKFLOW.mdc](mdc:.cursor/rules/WORKFLOW.mdc). +5. **Mandatory Coverage:** Adding or updating relevant tests is a **required** step for all feature additions or bug fixes. + +--- + +## 🏃‍♂️ Running Tests + +Refer to [.cursor/rules/WORKFLOW.mdc](mdc:.cursor/rules/WORKFLOW.mdc) for the mandatory requirement to run tests. + +| Test Type | Command | Description | +|-----------|---------|-------------| +| **All Tests** | `pnpm test` | Run all test types except Playwright | +| **Unit Tests** | `pnpm test:unit` | Fast tests with mocked dependencies | +| **Integration Tests** | `pnpm test:integration` | Tests with mocked Ably services | +| **E2E Tests** | `pnpm test:e2e` | Tests against real Ably services | +| **Playwright Tests** | `pnpm test:playwright` | Web CLI browser tests | + +**Run Specific Files:** +```bash +# Run a specific test file +pnpm test test/unit/commands/apps/create.test.ts + +# Run all tests in a directory +pnpm test test/integration/core/**/*.test.ts +``` -## Testing Approach +--- -### Unit Tests (`test/unit`) +
+

📊 Testing Approach - Expand for Details

 + +### 🧪 Unit Tests (`test/unit`) * **Primary Purpose:** Quickly verify command logic, flag parsing, input validation, error handling, and basic output formatting **in isolation**. Focus on testing individual functions or methods within a command class. * **Dependencies:** **MUST** stub/mock all external dependencies (Ably SDK calls, Control API requests, filesystem access, `ConfigManager`, etc.). Use libraries like `sinon` and `nock`. * **Speed:** Very fast; no network or filesystem dependency. * **Value:** Useful for testing complex parsing, conditional logic, and edge cases within a command, but **less effective** at verifying core interactions with Ably services compared to Integration/E2E tests. * **Tools:** Mocha, `@oclif/test`, `sinon`. -* **Example:** Testing `ably apps create --name myApp` ensures correct arguments are parsed and the *mocked* Control API call is made with the expected payload. -### Integration Tests (`test/integration`) +**Example:** +```typescript +// Example unit test with proper mocking +import {expect} from '@oclif/test' +import * as sinon from 'sinon' +import {AblyCommand} from '../../src/base/ably-command' + +describe('MyCommand', () => { + let mockClient: any + + beforeEach(() => { + // Set up mocks + mockClient = { + channels: { + get: sinon.stub().returns({ + publish: sinon.stub().resolves() + }) + }, + close: sinon.stub().resolves() + } + sinon.stub(AblyCommand.prototype, 'getAblyClient').resolves(mockClient) + }) + + afterEach(() => { + sinon.restore() + }) + + it('publishes a message to the specified channel', async () => { + // Test implementation + }) +}) +``` + +### 🔄 Integration Tests (`test/integration`) * **Primary Purpose:** Verify the interaction between multiple commands or components, including interactions with *mocked* Ably SDKs or Control API services. Test the CLI execution flow. * **Dependencies:** Primarily stub/mock network calls (`nock` for Control API, `sinon` stubs for SDK methods), but may interact with the local filesystem for config management (ensure isolation). Use `ConfigManager` mocks. * **Speed:** Relatively fast; generally avoids real network latency. * **Value:** Good for testing command sequences (e.g., `config set` then `config get`), authentication flow logic (with mocked credentials), and ensuring different parts of the CLI work together correctly without relying on live Ably infrastructure. * **Tools:** Mocha, `@oclif/test`, `nock`, `sinon`, `execa` (to run the CLI as a subprocess). -* **Example:** Testing `ably accounts login` (mocked browser flow) -> `ably apps list` (mocked Control API) -> `ably apps switch` -> `ably apps current` (mocked Control API). -### End-to-End (E2E) Tests (`test/e2e`) +**Example:** +```typescript +// Example integration test with proper subprocess execution +import {expect, test} from '@oclif/test' +import * as nock from 'nock' + +describe('config commands', () => { + beforeEach(() => { + // Set up mocks + nock('https://control.ably.net') + .get('/v1/apps') + .reply(200, { apps: [{ id: 'app123', name: 'Test App' }] }) + }) + + afterEach(() => { + nock.cleanAll() + }) + + test + .stdout() + .command(['config', 'set', 'apiKey', 'abc:def']) + .it('sets the API key', ctx => { + expect(ctx.stdout).to.contain('API key saved') + }) + + test + .stdout() + .command(['apps', 'list']) + .it('lists apps using the saved API key', ctx => { + expect(ctx.stdout).to.contain('Test App') + }) +}) +``` + +### 🌐 End-to-End (E2E) Tests (`test/e2e`) * **Primary Purpose:** Verify critical user flows work correctly against **real Ably services** using actual credentials (provided via environment variables). * **Dependencies:** Requires a live Ably account and network connectivity. Uses real Ably SDKs and Control API interactions. @@ -39,9 +139,40 @@ See also: [Debugging Guide](mdc:docs/DEBUGGING.md) for troubleshooting tips. * **Value:** Provides the highest confidence that the CLI works correctly for end-users in a real environment. **Preferred** over unit tests for verifying core Ably interactions. * **Tools:** Mocha, `@oclif/test`, `execa`, environment variables (`E2E_ABLY_API_KEY`, etc.). * **Frequency:** Run automatically in CI (GitHub Actions) on PRs and merges. Can be run locally but may incur costs. -* **Example:** Running `ably channels publish test-channel hello` and then running `ably channels subscribe test-channel` in a separate process to verify the message is received. -### Playwright Tests (`test/e2e/web-cli`) +**Example:** +```typescript +// Example E2E test with real services +import {expect, test} from '@oclif/test' +import {execSync} from 'child_process' + +describe('channels commands', function() { + // Longer timeout for E2E tests + this.timeout(10000) + + const testChannel = `test-${Date.now()}` + const testMessage = 'Hello E2E test' + + it('can publish and then retrieve history from a channel', async () => { + // Publish a message + execSync(`ABLY_API_KEY=${process.env.E2E_ABLY_API_KEY} ably channels publish ${testChannel} "${testMessage}"`) + + // Wait a moment for message to be stored + await new Promise(resolve => setTimeout(resolve, 1000)) + + // Get message from history + const result = execSync( + `ABLY_API_KEY=${process.env.E2E_ABLY_API_KEY} ably channels history ${testChannel} --json` + ).toString() + + const history = JSON.parse(result) + expect(history).to.be.an('array').with.lengthOf.at.least(1) + expect(history[0].data).to.equal(testMessage) + }) +}) +``` + +### 🎭 Playwright Tests (`test/e2e/web-cli`) * **Primary Purpose:** Verify the functionality of the Web CLI example application (`examples/web-cli`) running in a real browser. * **Dependencies:** Requires Docker, Node.js, a browser (installed via Playwright), and the Web CLI example app to be built. @@ -50,44 +181,14 @@ See also: [Debugging Guide](mdc:docs/DEBUGGING.md) for troubleshooting tips. * **Tools:** Playwright Test runner (`@playwright/test`), Docker. * **Frequency:** Run automatically in CI, separate from Mocha tests. -## Running Tests +
-Refer to [.cursor/rules/WORKFLOW.mdc](mdc:.cursor/rules/WORKFLOW.mdc) for the mandatory requirement to run tests. +--- + +
+

🔧 Advanced Testing Guidance - Expand for Details

 -* **Run All (Unit, Integration, E2E - excluding Playwright):** - ```bash - pnpm test - ``` -* **Run Only Unit Tests:** - ```bash - pnpm test:unit - ``` -* **Run Only Integration Tests:** - ```bash - pnpm test:integration - ``` -* **Run Only E2E Tests (Mocha-based, excluding Playwright):** - ```bash - pnpm test:e2e - ``` -* **Run Only Playwright Web CLI Tests:** - ```bash - pnpm test:playwright - ``` -* **Run Specific File(s):** (Uses Mocha runner by default) - ```bash - # Example: Run a specific unit test file - pnpm test test/unit/commands/apps/create.test.ts - - # Example: Run all tests in the integration/core directory - pnpm test test/integration/core/**/*.test.ts - - # Example: Run a specific Playwright test file (uses Playwright runner) - pnpm test test/e2e/web-cli/web-cli.test.ts - ``` -* **Debugging Tests:** See [docs/DEBUGGING.md](mdc:docs/DEBUGGING.md). - -## Test Coverage and Considerations +## 📝 Test Coverage and Considerations * **Adding/Updating Tests:** When adding features or fixing bugs, add or update tests in the appropriate category (Unit, Integration, E2E, Playwright). * **Focus:** Prioritize **Integration and E2E tests** for verifying core functionality involving Ably APIs/SDKs, as unit tests with extensive mocking provide less confidence in these areas. @@ -102,7 +203,7 @@ Refer to [.cursor/rules/WORKFLOW.mdc](mdc:.cursor/rules/WORKFLOW.mdc) for the ma * **Realtime SDK Stubbing:** For Unit/Integration tests involving the Realtime SDK, stub the SDK methods directly (`sinon.stub(ably.channels.get('...'), 'subscribe')`) rather than trying to mock the underlying WebSocket, which is complex and brittle. * **Credentials:** E2E tests rely on `E2E_ABLY_API_KEY` (and potentially others) being set in the environment (locally via `.env` or in CI via secrets). **Never** hardcode credentials in tests. -## Codebase Integration & Structure +## 🗂️ Codebase Integration & Structure ### Folder Structure @@ -131,3 +232,25 @@ Refer to [.cursor/rules/WORKFLOW.mdc](mdc:.cursor/rules/WORKFLOW.mdc) for the ma ### E2E Test Organization E2E tests are organized by feature/topic (e.g., `channels-e2e.test.ts`, `presence-e2e.test.ts`) to improve maintainability and allow targeted runs. They use shared helpers from `test/helpers/e2e-test-helper.ts`. + +
+ +--- + +## 🎯 Best Practices Quick Reference + +1. **✅ DO** prioritize Integration and E2E tests for core Ably functionality +2. **✅ DO** clean up all resources in tests (clients, connections, mocks) +3. **✅ DO** use proper mocking (`sinon`, `nock`) for Unit/Integration tests +4. **✅ DO** avoid testing implementation details when possible (test behavior) + +5. **❌ DON'T** rely solely on unit tests for Ably API interactions +6. **❌ DON'T** leave resources unclosed (memory leaks) +7. **❌ DON'T** use brittle `setTimeout` when avoidable +8. **❌ DON'T** hardcode credentials or API keys in tests + +--- + +
+🔍 For detailed troubleshooting help, see the Troubleshooting Guide. +
From 826f8db5604aae2b29accc3f8bf978fb2c31b5f3 Mon Sep 17 00:00:00 2001 From: Matthew O'Riordan Date: Sun, 27 Apr 2025 21:34:22 +1200 Subject: [PATCH 3/5] chore: standardize file naming to title-case-with-hyphens MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Standardized file naming conventions across: .cursor/rules: ABLY.mdc → Ably.mdc .cursor/rules: DEVELOPMENT.mdc → Development.mdc .cursor/rules: PROJECT.mdc → Project.mdc .cursor/rules: WORKFLOW.mdc → Workflow.mdc docs: DEBUGGING.md → Debugging.md Updated all cross-references to maintain consistency. --- .cursor/rules/{AI_ASSISTANCE.mdc => AI-Assistance.mdc} | 6 +++--- .cursor/rules/{ably.mdc => Ably.mdc} | 0 .cursor/rules/{development.mdc => Development.mdc} | 0 .cursor/rules/{project.mdc => Project.mdc} | 0 .cursor/rules/{WORKFLOW.mdc => Workflow.mdc} | 2 +- CONTRIBUTING.md | 2 +- README.md | 2 +- docs/{DEBUGGING.md => Debugging.md} | 0 docs/Testing.md | 10 +++++----- docs/{TROUBLESHOOTING.md => Troubleshooting.md} | 2 +- 10 files changed, 12 insertions(+), 12 deletions(-) rename .cursor/rules/{AI_ASSISTANCE.mdc => AI-Assistance.mdc} (96%) rename .cursor/rules/{ably.mdc => Ably.mdc} (100%) rename .cursor/rules/{development.mdc => Development.mdc} (100%) rename .cursor/rules/{project.mdc => Project.mdc} (100%) rename .cursor/rules/{WORKFLOW.mdc => Workflow.mdc} (97%) rename docs/{DEBUGGING.md => Debugging.md} (100%) rename docs/{TROUBLESHOOTING.md => Troubleshooting.md} (97%) diff --git a/.cursor/rules/AI_ASSISTANCE.mdc b/.cursor/rules/AI-Assistance.mdc similarity index 96% rename from .cursor/rules/AI_ASSISTANCE.mdc rename to .cursor/rules/AI-Assistance.mdc index 0e93f58ee..a962dbe07 100644 --- a/.cursor/rules/AI_ASSISTANCE.mdc +++ b/.cursor/rules/AI-Assistance.mdc @@ -15,7 +15,7 @@ This document provides guidance for AI assistants working with the Ably CLI code - [Product Requirements](mdc:docs/Product-Requirements.md) - [Project Structure](mdc:docs/Project-Structure.md) - [Testing Strategy](mdc:docs/Testing.md) - - [Mandatory Workflow](mdc:.cursor/rules/WORKFLOW.mdc) + - [Mandatory Workflow](mdc:.cursor/rules/Workflow.mdc) 2. **Recognize Context Boundaries**: - The CLI is focused on Ably services - do not suggest features outside this scope @@ -159,11 +159,11 @@ This document provides guidance for AI assistants working with the Ably CLI code ## Contributing Workflow -All contributions must follow the [Mandatory Workflow](mdc:.cursor/rules/WORKFLOW.mdc): +All contributions must follow the [Mandatory Workflow](mdc:.cursor/rules/Workflow.mdc): 1. **Build**: Run `pnpm prepare` 2. **Lint**: Run `pnpm exec eslint .` 3. **Test**: Run appropriate tests 4. **Document**: Update relevant docs -See [CONTRIBUTING.md](mdc:CONTRIBUTING.md) for details. +See documentation in `.cursor/rules/Workflow.mdc` for details. diff --git a/.cursor/rules/ably.mdc b/.cursor/rules/Ably.mdc similarity index 100% rename from .cursor/rules/ably.mdc rename to .cursor/rules/Ably.mdc diff --git a/.cursor/rules/development.mdc b/.cursor/rules/Development.mdc similarity index 100% rename from .cursor/rules/development.mdc rename to .cursor/rules/Development.mdc diff --git a/.cursor/rules/project.mdc b/.cursor/rules/Project.mdc similarity index 100% rename from .cursor/rules/project.mdc rename to .cursor/rules/Project.mdc diff --git a/.cursor/rules/WORKFLOW.mdc b/.cursor/rules/Workflow.mdc similarity index 97% rename from .cursor/rules/WORKFLOW.mdc rename to .cursor/rules/Workflow.mdc index 22209ac15..1997ad727 100644 --- a/.cursor/rules/WORKFLOW.mdc +++ b/.cursor/rules/Workflow.mdc @@ -20,7 +20,7 @@ alwaysApply: false 3. **Run Tests:** * Execute relevant tests locally (e.g., `pnpm test:unit`, `pnpm test:integration`, `pnpm test:e2e`, `pnpm test:playwright`, or specific file paths like `pnpm test test/unit/commands/some-command.test.ts`). * **Purpose:** Verifies changes haven't introduced regressions and that new features work as expected. - * **Verification:** Ensure all executed tests pass. If tests fail, debug them (see `docs/DEBUGGING.md`). Add or update tests (Unit, Integration, E2E) as appropriate for your changes. Refer to `docs/Testing.md` for the testing strategy. + * **Verification:** Ensure all executed tests pass. If tests fail, debug them (see `docs/Debugging.md`). Add or update tests (Unit, Integration, E2E) as appropriate for your changes. Refer to `docs/Testing.md` for the testing strategy. 4. **Update Documentation & Rules:** * Review and update all relevant documentation and rule files based on your changes. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 36f2f6a83..c1abbd6f4 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -4,7 +4,7 @@ Thank you for your interest in contributing to the Ably CLI! ## Development Workflow -All code changes, whether features or bug fixes, **MUST** follow the mandatory workflow outlined in [.cursor/rules/WORKFLOW.mdc](mdc:.cursor/rules/WORKFLOW.mdc). +All code changes, whether features or bug fixes, **MUST** follow the mandatory workflow outlined in [.cursor/rules/Workflow.mdc](mdc:.cursor/rules/Workflow.mdc). In summary, this involves: diff --git a/README.md b/README.md index 24cffdd98..c96448a4a 100644 --- a/README.md +++ b/README.md @@ -4747,4 +4747,4 @@ This repository includes a terminal server that provides a bash environment over # Contributing -Please see [CONTRIBUTING.md](CONTRIBUTING.md) for details on how to contribute to this project, including our mandatory development workflow, testing requirements, and code quality standards. +Please see the documentation in `.cursor/rules/Workflow.mdc` for details on how to contribute to this project, including our mandatory development workflow, testing requirements, and code quality standards. diff --git a/docs/DEBUGGING.md b/docs/Debugging.md similarity index 100% rename from docs/DEBUGGING.md rename to docs/Debugging.md diff --git a/docs/Testing.md b/docs/Testing.md index 53ac19ae5..62af32d8d 100644 --- a/docs/Testing.md +++ b/docs/Testing.md @@ -5,9 +5,9 @@ > **💡 QUICK START:** Run `pnpm test` for all tests or `pnpm test:unit` for faster unit tests. -> **📋 MANDATORY:** All code changes require related tests. See [WORKFLOW.mdc](mdc:.cursor/rules/WORKFLOW.mdc). -> **🐛 DEBUGGING:** See [Debugging Guide](mdc:docs/DEBUGGING.md) for troubleshooting tips. -> **🔍 TROUBLESHOOTING:** See [Troubleshooting Guide](mdc:docs/TROUBLESHOOTING.md) for common errors. +> **📋 MANDATORY:** All code changes require related tests. See [Workflow.mdc](mdc:.cursor/rules/Workflow.mdc). +> **🐛 DEBUGGING:** See [Debugging Guide](mdc:docs/Debugging.md) for troubleshooting tips. +> **🔍 TROUBLESHOOTING:** See [Troubleshooting Guide](mdc:docs/Troubleshooting.md) for common errors. --- @@ -23,7 +23,7 @@ ## 🏃‍♂️ Running Tests -Refer to [.cursor/rules/WORKFLOW.mdc](mdc:.cursor/rules/WORKFLOW.mdc) for the mandatory requirement to run tests. +Refer to [.cursor/rules/Workflow.mdc](mdc:.cursor/rules/Workflow.mdc) for the mandatory requirement to run tests. | Test Type | Command | Description | |-----------|---------|-------------| @@ -252,5 +252,5 @@ E2E tests are organized by feature/topic (e.g., `channels-e2e.test.ts`, `presenc ---
-🔍 For detailed troubleshooting help, see the Troubleshooting Guide. +🔍 For detailed troubleshooting help, see the Troubleshooting Guide.
diff --git a/docs/TROUBLESHOOTING.md b/docs/Troubleshooting.md similarity index 97% rename from docs/TROUBLESHOOTING.md rename to docs/Troubleshooting.md index f80424845..5bd0ca4e3 100644 --- a/docs/TROUBLESHOOTING.md +++ b/docs/Troubleshooting.md @@ -211,4 +211,4 @@ Property 'x' does not exist on type 'Y' If you find errors in documentation or rules, please update them using the proper workflow and submit a pull request. -See [CONTRIBUTING.md](mdc:CONTRIBUTING.md) for more details on the development workflow. +See documentation in `.cursor/rules/Workflow.mdc` for more details on the development workflow. From 1eded5074e6d5f6909be51ab2a7f6332353d224d Mon Sep 17 00:00:00 2001 From: Matthew O'Riordan Date: Mon, 28 Apr 2025 07:42:15 +1200 Subject: [PATCH 4/5] Various small PR feedback improvements See https://github.com/ably/cli/pull/10 - Always attach AI and Workflow cursor rules - Fix README typos --- .cursor/rules/.cursorindexingignore | 2 ++ .cursor/rules/AI-Assistance.mdc | 2 +- .cursor/rules/Workflow.mdc | 2 +- CONTRIBUTING.md | 10 +++++----- README.md | 4 ++-- 5 files changed, 11 insertions(+), 9 deletions(-) create mode 100644 .cursor/rules/.cursorindexingignore diff --git a/.cursor/rules/.cursorindexingignore b/.cursor/rules/.cursorindexingignore new file mode 100644 index 000000000..68347b34e --- /dev/null +++ b/.cursor/rules/.cursorindexingignore @@ -0,0 +1,2 @@ +# Don't index SpecStory auto-save files, but allow explicit context inclusion via @ references +.specstory/** diff --git a/.cursor/rules/AI-Assistance.mdc b/.cursor/rules/AI-Assistance.mdc index a962dbe07..acb7b11a1 100644 --- a/.cursor/rules/AI-Assistance.mdc +++ b/.cursor/rules/AI-Assistance.mdc @@ -1,7 +1,7 @@ --- description: globs: -alwaysApply: false +alwaysApply: true --- # AI Assistance Guidelines diff --git a/.cursor/rules/Workflow.mdc b/.cursor/rules/Workflow.mdc index 1997ad727..563773a81 100644 --- a/.cursor/rules/Workflow.mdc +++ b/.cursor/rules/Workflow.mdc @@ -1,7 +1,7 @@ --- description: globs: -alwaysApply: false +alwaysApply: true --- # Mandatory Development Workflow diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c1abbd6f4..5f4a66cde 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -19,11 +19,11 @@ In summary, this involves: Before starting work, please familiarize yourself with: -* [Product Requirements](mdc:docs/Product-Requirements.md): Understand the goals and features. -* [Project Structure](mdc:docs/Project-Structure.md): Know where different code components live. -* [Testing Strategy](mdc:docs/Testing.md): Understand the different types of tests and how to run them. -* [Development Rules](mdc:.cursor/rules/development.mdc): Coding standards, linting, dependency management. -* [Ably Rules](mdc:.cursor/rules/ably.mdc): How to interact with Ably APIs/SDKs. +* [Product Requirements](./docs/Product-Requirements.md): Understand the goals and features. +* [Project Structure](./docs/Project-Structure.md): Know where different code components live. +* [Testing Strategy](./docs/Testing.md): Understand the different types of tests and how to run them. +* [Development Rules](mdc:.cursor/rules/Development.mdc): Coding standards, linting, dependency management. +* [Ably Rules](mdc:.cursor/rules/Ably.mdc): How to interact with Ably APIs/SDKs. ## Reporting Issues diff --git a/README.md b/README.md index c96448a4a..d57750915 100644 --- a/README.md +++ b/README.md @@ -17,6 +17,7 @@ * [Commands](#commands) * [MCP Server](#mcp-server) * [Terminal Server](#terminal-server) +* [Contributing](#contributing) # CLI Usage @@ -4741,10 +4742,9 @@ The MCP server also provides the following resources: - `channel_presence` - Current presence members on a specific Ably channel # Terminal Server -## Contributing This repository includes a terminal server that provides a bash environment over WebSockets for the web based CLI. See [Server-Setup.md](./docs/Server-Setup.md) for more info on how it works and how to deploy it. # Contributing -Please see the documentation in `.cursor/rules/Workflow.mdc` for details on how to contribute to this project, including our mandatory development workflow, testing requirements, and code quality standards. +Please see the documentation in [`.cursor/rules/Workflow.mdc`](.cursor/rules/Workflow.mdc) for details on how to contribute to this project, including our mandatory development workflow, testing requirements, and code quality standards. From 37556703d9b864d826091c00624060fea7862ab1 Mon Sep 17 00:00:00 2001 From: Matthew O'Riordan Date: Mon, 28 Apr 2025 08:09:39 +1200 Subject: [PATCH 5/5] Simplify Product Requirements Rewritten for simplification --- docker/README.md | 2 + docs/Product-Requirements.md | 789 +++++++++++++++-------------------- 2 files changed, 339 insertions(+), 452 deletions(-) diff --git a/docker/README.md b/docker/README.md index 5b0a45547..897d2d993 100644 --- a/docker/README.md +++ b/docker/README.md @@ -47,5 +47,7 @@ The security tests are also integrated into the CI pipeline using GitHub Actions For more information about the security hardening measures implemented, see: +- [Container-Security.md](../docs/Container-Security.md) - [Security-Hardening.md](../docs/Security-Hardening.md) - [Security-Testing-Auditing.md](../docs/Security-Testing-Auditing.md) +- [User-Namespace-Remapping.md](../docs/User-Namespace-Remapping.md) diff --git a/docs/Product-Requirements.md b/docs/Product-Requirements.md index 171fa4a3e..bac5bc83b 100644 --- a/docs/Product-Requirements.md +++ b/docs/Product-Requirements.md @@ -1,472 +1,357 @@ # Product Requirements Document: Ably CLI -## Overview - -The Ably CLI will enable developers to easily integrate, test and debug Ably's product features directly from the command line. -In addition, via the Control API, it will enable developers to provision and configure their apps and accounts. - -## Objective - -Provide an intuitive CLI that helps developers interact with Ably product APIs and SDKs. -Developers should be able to discover features of the product through the CLI, and use the CLI as part of their getting started experience, development of their initial apps and later in production. -The CLI should be straightforward and easily extensible to ensure product teams can add features to the CLI as soon as they available in the relevant product's Javascript SDK. -The CLI should persist authentication tokens and API keys so that commands can easily be executed without explicitly passing in API keys to every call. - -## Target Audience - -- Software developers using Ably to build realtime features -- DevOps engineers managing Ably resources and deployments - -## CLI for both the control and data planes - -The CLI is intended for us with both our control plane as well as our data plane. We define these as follows: - -_Control Plane_ is used to inspect and manage the configuration, administration, setup, and lifecycle of apps, resources and accounts. -The Control Plane is available as part of the Ably logged in dashboards and also via our _Control API_. -The Control Plane is shared between all products and is considered a platform capability as opposed to a specific product capability. -The documentation for this API is available at and . - -_Data Plane_ handles the core product functionality, runtime operations, data processing, and user interactions—typically exposed through our SDKs but also through raw REST APIs. -Each of our products has its own unique data plane, be that in the form of an SDK or additionally unique REST endpoints. -The documentation for each product is available at . - -## Key Features & Functional Requirements - -### CLI command hierarchy - -We are committed to delivering a great developer experience, and our CLI is no exception. -As such, the interface and information architecture of the topics and commands offered by the CLI are critical to delivering an intuitive, simple and delightful experience for developers. -In the proposed CLI interface below, we use topics to group or nest commands that are conceptuallyu related together, whilst -ensuring each command has an intuitive name that ensures a developer can discover the features of the CLI without reverting to documentation. - -Below is the proposed information architecture / interface for the CLI, which maps to the conceptual concepts of the Ably platform and products. - -#### The following commands are scoped to the account and depend on the control API access token for authentication - -_Note: The `topicSeparator` configured for oclif is not a `:` but instead a space (` `). All commands must respect this configuration_ - -- `$ ably login` -> proxy convenience to $ ably accounts login -- `$ ably accounts login` -> this command will take the user through the authentication process, and then persist the access token in the local config which will be used by default for all subsequent commands that depend on the control API. When logging in, if no alias is provided, the default account token is overwritten. If an alias is provided, an additional token is stored in the config indexed by that alias name. -- `$ ably accounts list` -> list all the accounts already configured in the local config indexed by their alias name (unless `default`), which includes the the token ID, capabilities, user email, and account ID and name. -- `$ ably accounts logout` -> will warn the user that this is descructive and all configuration for this account will be deleted. If executed, this will remove the access token and any associated data from the local config file. -- `$ ably accounts current` -> confirms which account is currently selected (including the assigned alias name unless `default`) and makes a call to the control API endpoint to confirm the token is valid and confirm the account details -- `$ ably accounts stats` -> view account stats, with a --live option which polls every 6 seconds and returns key metrics such as peak connections, channels, message throughput, and cumulative messages sent etc. -- `$ ably accounts switch` -> switch to the provided account ID or account alias - -- `$ ably auth` -> Topic that houses all product authentication commands (those used to access the data planes of those products). - -#### The following commands are all scoped to an app, so if there is not a default app in the config, the --app [name OR ID] argument is required, or a key or token must be provided as an argument - -- `$ ably auth issue-jwt-token` -> Creates an Ably JWT, which is the preferred approach for tokens. Capabilities and other token options can be specified. (see and ) -- `$ ably auth issue-ably-token` -> Creates an Ably Token. Capabilities and other token options can be specified (see and ) -- `$ ably auth revoke-token [token]` -> Revokes the token provided (see ) -- `$ ably auth keys` -> This is a sub-topic for key management -- `$ ably auth keys list` -> list all keys in the app (uses the Control API) -- `$ ably auth keys revoke [key identifier or complete key]` -> revoke a key in the app (uses the Control API) -- `$ ably auth keys get [key identifier or complete key]` -> view all the details for a key (uses the Control API) -- `$ ably auth keys update [key identifier or complete key]` -> update properties of the key such as capabilities and the name -- `$ ably auth keys switch [key identifier or complete key]` -> switch to this key for all client requests for this app. This API key will be stored in the config for the currently selected app. -- `$ ably auth keys current` -> Show the current key if one is configured for this app in the config. Note that for each app only one key can be configured, but switching between apps will remember the last configured key for the app if used before. - -- `$ ably apps` -> Topic for manage apps -- `$ ably apps list` -> list all apps available -- `$ ably apps create` -> create a new app -- `$ ably apps update` -> update the current app or named app from the --app argument -- `$ ably apps delete` -> delete the app following confirmation by the user in the CLI -- `$ ably apps set-apns-p12` -> allow a user to upload a P12 cert for use with APNS -- `$ ably apps stats` -> view app stats, with a --live option which polls every 6 seconds and returns key metrics such as peak connections, channels, message throughput, and cumulative messages sent etc. -- `$ ably apps switch` -> switch to this app for all subsequent requests -- `$ ably apps current` -> Show the current app if one is configured for the account. -- `$ ably apps logs` subscribe -> proxy for ably logs apps subscribe -- `$ ably apps logs` history -> proxy for ably logs apps history -- `$ ably apps channel-rules` -> Manage Ably channel rules (named namespaces in the Control API) -- `$ ably apps channel-rules list` -> list all channel rules -- `$ ably apps channel-rules create` -> create a channel rule -- `$ ably apps channel-rules update` -> update a channel rule -- `$ ably apps channel-rules delete` -> delete a channel rule (prompt for interactive confirmation) - -- `$ ably channels` -> Topic for Ably Pub/Sub channels -- `$ ably channels list` -> use the channel enumeration API to return a list of live channels -- `$ ably channels publish` -> publish a message to a channel with optional support for encryption. Support --count and --delay arguments to send multiple messages, with interpolation for `{{.Count}}` and `{{.Timestamp}}` in the message argument. -- `$ ably channels batch-publish` -> use the REST batch publish API to publish a batch of messages -- `$ ably channels subscribe` -> subscribe to messages published on one or more provided channels. Support for encryption, deltas, and rewind are available as arguments. Stay subscribed until terminated. -- `$ ably channels occupancy get` -> returns the current occupancy on a channel -- `$ ably channels occupancy subscribe` -> subscribes to ongoing occupancy changes using the meta occupancy channel -- `$ ably channels history` -> provides access to historical messages on a channel -- `$ ably channels presence` -> this is a topic that groups presence functionality together -- `$ ably channels presence enter` -> enter presence and remain present until the script is terminated. Show who is entering and leaving whilst present. -- `$ ably channels presence subscribe` -> show the complete list of members present, then show events of who is entering or leaving, until the script is terminated. -- `$ ably channels logs channel-lifecycle subscribe` -> alias for `ably logs channel-lifecycle subscribe` - -- `$ ably config` -> opens the config TOML file with the default text editor - -- `$ ably connections logs` connections-lifecycle -> set up as an alias to ably logs connection-lifecycle -- `$ ably connections logs connection-lifecycle subscribe` -> alias for `ably logs connection-lifecycle subscribe` -- `$ ably connections stats` -> this is largely a duplication of ably app stats, however it is focussed on connection stats only. This should also support the --live option. -- `$ ably connections test` -> simply connects to Ably, confirms that the conenction was established, and closes the connection. Allows transport params options so that WebSockets can be disabled for example, and only HTTP is used, or only WebSockets is used without HTTP support. - -- `$ ably rooms` -> Topic for Ably Chat -- `$ ably rooms list` -> list chat rooms using the channel enumeration API, filtering out those that are not chat -- `$ ably rooms messages` send -> send a chat message. Support --count and --delay arguments to send multiple messages, with interpolation for `{{.Count}}` and `{{.Timestamp}}` in the message argument. -- `$ ably rooms messages` subscribe -> subscribe to chat messages -- `$ ably rooms messages` get` -> get historical messages -- `$ ably rooms messages get` -> get historical messages -- `$ ably rooms occupancy get` -> returns the current occupancy on a room -- `$ ably rooms occupancy subscribe` -> subscribes to ongoing occupancy changes using the meta occupancy channel -- `$ ably rooms presence subscribe` -> show the complete list of members present, then show events of who is entering or leaving, until the script is terminated. -- `$ ably rooms presence enter` -> enter a room and remain present until the script is terminated. Show who is entering and leaving whilst present. -- `$ ably rooms reactions subscribe` -> subscribe to room reactions -- `$ ably rooms reactions send` -> send a room reaction -- `$ ably rooms typing subscribe` -> subscribe to typing indicators and show who is typing and stops typing in realtime -- `$ ably rooms typing start` -> start typing, and remain typing until the CLI is terminated - -- `$ ably spaces` -> Topic for Ably Spaces -- `$ ably spaces list` -> list Spaces using the channel enumeration API, filtering out those that are not chat -- `$ ably spaces members subscribe` -> show the complete list of members present, then show events of who is entering or leaving, until the script is terminated. -- `$ ably spaces members enter` -> enter a space and remain present until the script is terminated. Show who is entering and leaving whilst present. -- `$ ably spaces locations set` -> allow a location to be set, and remain at this location until the script is terminated. Show other location changes whilst the script is running. -- `$ ably spaces locations subscribe` -> subcribe to the set of locations, show all locations, and then show location changes as they happen -- `$ ably spaces locations get-all` -> show all current locations for a space and then exit -- `$ ably spaces cursors set` -> allows a specific x and y location to be set, but if not position is set, then the client will simulate movements of a mouse moving periodically. -- `$ ably spaces cursors subscribe` -> show a table of all cursors and their positions, and show changes as they happen. Use colour coding to show when changes have been made, and after 2s revert back to the default colour. -- `$ ably spaces cursors get-all` -> retrieve all current cursors, show the details, and exit. -- `$ ably spaces locks acquire` -> acquire a lock for the provided ID, and keep that lock until the script is terminated. Show any other changes to locks whilst the script is running. -- `$ ably spaces locks subscribe` -> show a table of all locks and show live updates. Use colour coding to show when changes have been made, and after 2s revert back to the default colour. -- `$ ably spaces locks get` -> a lock ID argument is required, confirm if a lock exists or not and show the lock if one does exist -- `$ ably spaces locks get-all` -> show a list of all locks and exit - -- `$ ably logs` -> Topic for streaming and retrieving logs -- `$ ably logs channel-lifecycle subscribe` -> Stream logs from the `[meta]channel.lifecycle` meta channel. See Ably docs for event types. -- `$ ably logs connection-lifecycle subscribe` -> Stream logs from the `[meta]connection.lifecycle` meta channel. -- `$ ably logs connection-lifecycle history` -> View historical connection lifecycle logs. -- `$ ably logs connection subscribe` -> Stream logs from the `[meta]connection` meta channel. -- `$ ably logs app subscribe` -> Stream logs from the app-wide `[meta]log` meta channel. Supports `--rewind`. -- `$ ably logs app history` -> View historical app logs from `[meta]log`. -- `$ ably logs push subscribe` -> Stream logs from the app push notifications `[meta]log:push` meta channel. Supports `--rewind`. -- `$ ably logs push history` -> View historical push logs from `[meta]log:push`. - -- `$ ably integrations` -> Manage Ably integrations (named rules in the Control API) -- `$ ably integrations list` -> list all integrations in the app -- `$ ably integrations get` -> get an integration by ID -- `$ ably integrations create` -> create an integration rule -- `$ ably integrations update` -> update an integration rule -- `$ ably integrations delete` -> delete an integration rule (prompt for interactive confirmation) - -- `$ ably queues` -> Manage Ably Queues -- `$ ably queues list` -> list all queues in the app -- `$ ably queues create` -> create a queue -- `$ ably queues delete` -> delete a queue (prompt for interactive confirmation) - -- `$ ably bench` -> Topic for benchmark tests -- `$ ably bench publisher` -> allow a publisher to start publishing messages on a specified channel at a frequency of at most 20 messages per second allowing the total number of messages published to be 10,000. Before the publisher starts publishing though, it will check if there are other subscriber bench clients present on the channel and it will also become present and announce it is a publisher and the details of the test. If there are none, it will ask the user if we should wait for subscribers to be present, or continue with a prompt. Once completed, the client will conrim how many messages were published, if there were any errors, and in the case of REST, what the average latency was for the request to complete, and in the case of Realtime, what the average latency was for a message to be received back. Whilst running the test, the message rate and trailing latency for the last 5 seconds shoudl be shown ,along with the cumulative messages sent and percentage of test execute. This should all be presented in a console UI with a progress bar for example. Support for publishing via REST or Realtime is needed. If a test is already running and another one starts, and error will be shown and the CLI should exit. Each publisher can show others that the test is runing using presence data. -- `$ ably bench subscriber` -> will attach to a channel and be present with data in the presence set indicating it's a subscriber and waiting. Once a publisher bercomes present, the subscriber will indicate a test is starting, and using a UI control, will show running metrics whilst the test is running. Once the test is complete, the summary will be printed out with the test ID, summary of the test paramters, confirmation of how many messages were received or failed, average latencies (based on the publish time assuming clocks are in sync, as the published message will have a latency in the data, and the latency it arrives at Ably is coded into the timestamp field) for end to end, and for it to be received by Ably, with p50, p90, p95 shown. Once the test is done, the subscriber will wait patiently for the next test to start. If a test is running and another one starts, and error will be shown and the CLI should exit. - -#### The following commands are mostly not scoped to accounts or apps, but may use auth for identification - -- `$ ably help` -> Topic to get help from Ably -- `$ ably help ask` -> Ask a question to the Ably AI agent for help. This is done by using the Control API and sending a request to the /v1/help endpoint. Notes on how this works below. -- `$ ably help contact` -> Contact us -> open a browser to -- `$ ably help support` -> Get support from Ably -> open a browser to -- `$ ably help status` -> Check the status of the Ably service using the endpoint, and if {status:true} then Ably is up and there are no open incidents, and if {status:false}, then there are open incidents. Either way, tell the user to go to to get the Ably status. - -- `$ ably --version` -> returns the version of the CLI - -### Authentication - -There are two distinct types of authentication needs for the CLI. - -1. User level (Control Plane) -> an Ably registered user can generate access tokens for use with the Control API. - Access tokens are used to access the Control API and are bound to a single account. - All control API operations performed with access tokens are made on behalf of the user who issued the token and have the same rights that that user has for that account. - -2. App level (Data Plane) -> every call to the Ably product data plane APIs, that is the APIs offered by the products such as Ably Pub/Sub and Ably Chat, requires authentication with an API key or a token generated from an API key. - API keys are issued for apps and can only belong to one app. Apps are sandboxed ensuring API keys cannot be shared across apps and data cannot traverse the boundaries of the app. - - 1. As a result, users of the CLI, when using commands that depend on the data plane, must provide an API key, a token, or have a default app and API key selected from within the CLI. - 2. As a convenience, if a user has logged in using `ably accounts login`, but has not yet selected an app and is not explicitly providing auth credentials (like an API key or token), then: - - The user will be told that no app is in use and if they press Enter, we will list the apps and let them select one (using a nice interactive CLI UI). - - Once they have selected an app, they will then be shown a list of API keys, and they will too select one. - - Once that is done, the default app ID will be stored in the config for the current access token, and the API key will be stored alongside the list of known apps so that subsequent requests for that app can be automatically authenticated. - -_Note:_ - -- The expected behaviour is that users will obtain an access token, log into the CLI with this token, and this in turn will be used to browse apps and select an API key for all data plane operations. -- However, it is also perfectly valid for a user to use the CLI and simply pass in an explicit API key, Ably JWT token or Control API access token with an argument, and the CLI will operate without any local config needed. -- We provide conveniences in the `ably auth` commands so that users can issue Ably JWT Tokens or Ably Tokens for testing purposes. Whilst we support CLI data plane commands being issued with Ably JWT Tokens or Ably native Tokens, we expect this is unlikely and API keys will be used instead given they are not short-lived. -- When users run the command `ably login`, we should delight with some colourful ASCII art derived from the Ably.com logo. Then ask the user to Press Enter to open the browser to obtain a token, which in turn will take the user to . They will then be asked to enter the token in the CLI, once entered, they will be offered the option of entering an alias name for the account or leave empty for the default account. - -### Debugging & Logging - -- Flags for enabling debugging of the CLI itself -- Log command which has options to subscribe to named log channels and show live logs. Log channel types are described in , but must include [meta]connection.lifecycle, [meta]channel.lifecycle, [meta]clientEvents:connections, [meta]clientEvents:apiRequests, [meta]log, [meta]log:push. -- Some log channels like [meta]log and [meta]log:push support rewind, so offer the option to rewind or view historical logs with the history feature - -### Developer Experience - -- Helpful inline documentation and examples (`--help`). -- Autocompletion for bash/zsh shells. -- Friendly, descriptive error handling and troubleshooting suggestions. -- Interactive commands where applicable. - -### Configuration - -- The CLI will follow popular conventions for storing sensitive configuration information and store the config in the user's home directory using the path `~/.ably/config` - - We need ensure we treat the Ably config file as sensitive (secure file permissions, etc.). - - Examples from other CLIs include ~/.aws/credentials, ~/.kube/config, ~/.config/gh/hosts.yml, etc. -- It is expected that the TOML format will be used for the config file as this is human readable and easy to use. -- The configuration file will mostly be used to store credentials for control plane and data plane access. - - In practice, most users will only have one login and one account at Ably. - - However, we do need to support users who have access to multiple accounts, either because their organisation has multiple accounts or because perhaps they have a free account and a company paid account. - - Each Control API token is scoped to only one account, so if we a user has access to multiple accounts, then they will need one local access token per account. - - When a user logs in with `ably accounts login`, if they do not choose to create an alias for the account, then the access token will be stored in the default account. - - If a user does provide an alias when logging in, then the access token and all other related credentials for the apps, will be stored in a separate section in the configuration file. This will allow a user to switch between accounts and retain the credentials. -- When users switch accounts, the config must store which account is current. -- When users switch apps, the config must store which app is current. Multiple app configs are stored with each account so that users can switch between apps and the last configured key will be used automatically. -- When users switch keys, the config should replace the existing app key with the new switched one. Each app has only one associated key. - -### Global arguments - -- `--host`: Override the host endpoint for all product API calls. In the product SDKs, this is typically the `host` option. Note that specifying the host will override both the Realtime and REST host for the Realtime client. -- `--env`: Override the the environment for all product API calls. In the product SDKs, this is typically the `env` option. -- `--control-host`: Override the host endpoint for the control API, which defaults to control.ably.net. -- `--access-token`: Overrides any configured access token used for the Control API -- `--api-key`: Overrides any configured API key used for the Control API -- `--client-id`: Overrides any default client ID when using API authentication. By default, if API key authentication is being used, then the client ID is automatically defined as `ably-cli-` with a random set of 8 characters generated. This ensures that operations like presence will work, and other clients can easily see that events are being generated from the CLI. -- `--json`: Instead of regular human readable output from the CLI, only valid JSON will be outputted for consumption by other services such as `jq` -- `--pretty-json`: Much like the `--json` argument, only JSON will be outputted, however the JSON will be formatted and colorized to make it more human readable. -- `--verbose`: Includes additional log events that a user is likely to be interested in if they want to understand the state changes of the command, such as the connection state being connected, or disconnected, when subscribing to a channel. Each CLI command needs to decide if there are any material events that should be emitted, and output those as log events. The format of these log events must be standardise across all CLI commands, and output as JSON when in one of the two JSON modes, `--pretty-json` or `--json`. - -#### JSON and Pretty JSON modes +## 1. Overview -1. CLI commands by default provide user-friendly readable outputs, with colour coding, and status updates where useful can be included in the response. -2. When `--json` argument is used, the assumption is that this is designed as a machine readable format, i.e. when piped into a jq command, or when made available to MCP servers. The JSON output should only include the output expected from the command and exclude any status updates or meta information. For example, subscribing to messages, should only output the messages received and not emit any updates on connection state. The exception to this is when there is a terminal/critical error where the CLI needs to output the error state and stop. In JSON mode, oclif's regular `log` commands suppress logging so that output is limited to intended data as opposed to logging and progress information. In addition, when JSON mode is enabled, all error events emitted from the Ably client will be captured and emitted as JSON errors, as the default logLevel of 1 emits errors directly to the console, which will break any clients that depend on JSON output. -3. When `--pretty-json` argument is used, this follows the same assumptions as `--json`, with the difference being that the CLI can now assume a human is reading the JSON, and as such, we prettify the JSON output and colour code it to make it more legible. -4. If the additional argument`--verbose` is used with `--json` and `--pretty-json`, additional log events that the user is likely to be interested in if they want to understand the state changes of the command, such as the connection state being connected, or disconnected, when subscribing to a channel, will be shown. +The Ably CLI enables developers to integrate, test, and debug Ably's product features directly from the command line. It also allows managing Ably apps and accounts via the Control API. -### Usability +## 2. Objective -- Where applicable, use visually compelling console UI elements to make the experience for a developer delightful +Provide an intuitive CLI for developers to interact with Ably APIs and SDKs, aiding discovery, initial development, and production use. The CLI should be easily extensible for new product features and persist authentication credentials securely. -#### Structure of commands +## 3. Target Audience -- When a user enters a command that does not exist that is similar to an actual command, the CLI should ask the user if they meant the command that exists and will give the user the option to run that command. -- oclif supports two styles of commands, either with spaces delimiting the commands such as `ably account stats` or colons delimiting the commands such as `ably account:stats`. This CLI uses spaces only and all documentation and helpers in the command files must use spaces consistently. -- If a user issues a command to any topic (commmands with sub-commands), as a convenience the CLI should show the list of commands available with simple descriptions and simple examples where applicable. For example, calling `ably accounts` should show the list of all sub-commands. +- Software developers using Ably. +- DevOps engineers managing Ably resources. + +## 4. Control Plane vs. Data Plane -### Documentation +- **Control Plane**: Manages configuration, administration, setup, and lifecycle of apps, resources, and accounts via the Ably dashboard and Control API (). +- **Data Plane**: Handles core product functionality (Pub/Sub, Chat, Spaces) via SDKs and REST APIs (). -- You need to generate Markdown documentation in this repository that explains clearly how users can install, configure, and use the CLI. See good examples of docs at , and - -### Embeddable web example and React component for CLI - -This repository should include an example of the CLI running inside of a web browser using direct command execution. This will be achieved by: - -- Create a React Ably CLI component that can be used by other web applications to provide this CLI as an interactive terminal inside a browser. -- The React Ably CLI component will open a WebSocket connetion to a terminal server. -- This repository will include a new terminal server role which will listen for new WebSocket connections, and when a new connection comes in, it will open a new docker container, and connect the WebSocket to that container. If the WebSocket connection drops, the docker session will be closed and all artefacts deleted. Sessions older than 30 minutes are terminated, and there will be a configurable number of max concurrent sessions. -- The docker container will be based on an node:22-alpine base image, with the Ably CLI installed using `npm intall @ably/cli`. -- The docker container will prevent the user from doing anything apart from running the `ably` command. Every other command should be rejected to prevent any abuse of these containers running when new websocket connections are established. -- The React Ably CLI will use Xterm JS () to display a fully functional terminal inside the browser. Note may be a useful library to consider as well. -- The React component should require two arguments, a control access token and an API key. This will map to the ABLY_ACCESS_TOKEN env var OR --access-token global argument, and ABLY_API_KEY env var or --api-key global argument. -- The demo web example should be built with Vite and kept very simple. It assumes the user is running the CLI in the context of a single app, account and key, and chosing a different key, app or account is handlded from within the UI of the web interface in real use case examples. -- The environment variables ABLY_ACCESS_TOKEN and ABLY_API_KEY are defined in the environment or in a .env file in that folder. The demo should exist in a folder /example/web-cli. -- Please include an example for how developers can use the React Web CLI component from installing it to embedding it in the README.md file. -- The web terminal that is run from Docker should exit immediately with a suitable error message is ABLY_ACCESS_TOKEN and ABLY_API_KEY are not valid. ABLY_API_KEY has three parts, app ID, key ID, and key secret in the format [APP_ID].[KEY_ID]:[KEY_SECRET]. Ensure all are present and exit with a suitable error message if not. -- When the web terminal is running, the CLI should be able to detect that it is running in web mode via an environment variable set by the Docker terminal scripts. When running in this mode, the Ably CLI will have some intentionally reduced functionality to reflect the fact that switching accounts, apps, or keys is done via the UI, not the terminal, and the need for a local configuration is thus not needed. The CLI code should introduce a mechanism to clearly manage the rules of which commands are not runnable in Web CLI mode along with suitable error messages shown to the user. The commands not allowed at present are: - - ably accounts login -> should tell the user they are already logged in and cannot log in again via the web CLI - - ably accounts list -> should tell the user this feature is not available in the web CLI, please use the web dashboard at - - ably accounts current -> should show the current account information by using the me control API request, but needs to assume a local config does not exist - - ably accounts logout -> should tell the user they cannot log out via the web CLI - - ably accounts switch -> should tell the user they cannot change accounts in the web CLI, and should use the dashboard at to switch accounts - - ably apps switch -> should tell the user they cannot switch apps from within the web CLI and should use the web dashboard at - - ably apps current -> should determine the app info by using the app ID extacted from the ABLY_API_KEY, adn then using the control API to get the app information,. - - ably auth keys switch -> should tell the user they cannot switch apps from within the web CLI and should use the web interface to change keys - - ably config -> should not be visible in the list of comamnds, but if called explicitly, should tell the user that a local config is not supported in the web CLI version - -## Technical Requirements - -### Supported Platforms - -- Linux -- macOS -- Windows - -### Technology - -- Built with Node.JS -- Uses the oclif framework, see and -- Built with TypeScript - -### Distribution - -- Must be publishable to popular package managers Homebrew and npm -- Standalone executables are available for direct download - -### Integration & Extensibility - -- Supports both JSON output and human-readable more friendly formats typical in CLIs -- JSON output format for easy scripting and integration. -- Ideally potential plugins or extensions for future integrations, but I believe we can lean on the oclif framework for this. - -### Model Context Protocol (MCP) Server - -This CLI can also act as an MCP server for AI use cases, specifically focussed on IDE tools and AI desktop tools like Claude that support MCP. -Given MCP passes in ENV vars, we need to ensure the following environmnet variables are supported so that the MCP server can handle authentication with Ably data and control planes. -Where we currently generate a random client ID with the cli prefix, we should now use an mcp prefix to make it clear the origination of the requests. -The CLI offers interactive mode in some places. When MCP is being used, we need to ensure interactive features are not enabled so that AI tools can use MCP without any blocking. -All MCP requests should have a max time of 15s per request, so that if we are, for example, subscribing to messages, we will capture the subscription for 15 seconds, and then gracefully exit the application, indicating we terminated the response from continuing indefinitely. -Only a subset of commands from the CLI will be available, all of them listed explicitly below. The commands should support relevant arguments and options where available. -The Node MCP Typescript SDK library should be used to deliver the MCP server, see . -The MCP server should run in file mode i.e. it will be run locally with the MCP client. - -#### MCP commands available - -The following commands from the CLI should be made available in the MCP server: - -```sh -ably apps list -ably apps stats - -ably auth keys list - -ably channels history -ably channels publish -ably channels list -ably channels presence get -``` - -#### Environment variables - -- ABLY_ACCESS_TOKEN - overrides the default access token used for the control API -- ABLY_API_KEY - overrides the default API key used for the data plane -- ABLY_CLIENT_ID - overrides the default client ID assigned -- ABLY_CONTROL_HOST - overrides the default control API host -- ABLY_HOST - overrides the default data plane host -- ABLY_ENVIRONMENT - overrides the default data plane environment - -### Test coverage - -- As features are added to the CLI, suitable tests must be added. -- End to end tests are preferred for all data plane operations, whereas API mocking is acceptable for the Control API given it has an OpenAPI spec you can depend on. - -### APIs the CLI will depend on - -#### Ably Control API - -For all control plane management of apps and their associated resources, the Control API will be used. -The control API is documented in full at and has an OpenAPI specification at . -All authentication with the Control API requires a Control API token that is configurable by a registered Ably user at . -An access token is associated with a single user and a single account, and all API endpoints, apart from one, can be called without knowing which account the access token belongs to. -The exception is the endpoint, which requires an account ID. The CLI can however determine the account ID for the current token by calling the , and reading the account.id JSON value. - -#### Ably Pub/Sub - -For all CLI commands that depend on the Pub/Sub product, or lower level platform features (mostly underpinned by the Pub/Sub product), the Ably Javascript Realtime SDK must be used. See . -Whilst there are raw REST API endpoints available for Pub/Sub, this CLI should not use them as the Realtime SDK exposes a `request` method that enables REST API calls to the pub/sub service, but the SDK handles authentication, request retries, fallback behaviour, encoding etc. As such, using the SDK `request` method for all REST requests is required when a suitable method in the SDK does not already exist. - -#### Ably Chat - -For all CLI commands that depend on the Ably Chat product, Ably Javascript Chat SDK must be used. See [TBC] -Whilst there are raw REST API endpoints available for Ably Chat, this CLI should not use them. -If the CLI requires access to a REST API endpoint, and this endpoint is not made available directly in the Chat Javascript SDK, then it should use the underlying Pub/Sub SDK's `request` method to call the API. - -#### JWT - -To generate JWT tokens, the CLI will need to depend on a JWT library to generate the Ably JWT tokens. - -## Usage Examples +## 5. Key Features & Functional Requirements + +### 5.1. Command Hierarchy + +The CLI uses topics (space-separated) to group related commands logically. + +*Note: The `topicSeparator` configured for oclif is a space (` `). All commands must respect this configuration.* + +**Account Management (`ably accounts`)** +*(Requires Control API access token)* + +- `$ ably login [TOKEN]`: Alias for `ably accounts login`. Logs in or prompts for token. +- `$ ably accounts login [TOKEN]`: Authenticates the user with a Control API access token, storing it in the local config. Supports `--alias` for multiple accounts. +- `$ ably accounts list`: Lists locally configured accounts (aliases, token details, user info, account details). +- `$ ably accounts logout [ALIAS]`: Removes the access token and associated data for the specified (or default) account after warning. +- `$ ably accounts current`: Shows the currently selected account (alias or `default`) and verifies the token against the Control API `/me` endpoint. +- `$ ably accounts stats`: Views account stats. Supports `--live` for polling key metrics (peak connections, channels, message throughput, etc.). +- `$ ably accounts switch [ALIAS]`: Switches the active account configuration to the specified alias or prompts for selection. + +**App Management (`ably apps`)** +*(Requires Control API access token; operations scoped to the current or specified app)* + +- `$ ably apps list`: Lists all apps available in the current account. +- `$ ably apps create`: Creates a new app. Requires `--name`. Supports `--tls-only`. +- `$ ably apps update ID`: Updates the app specified by `ID`. Supports `--name`, `--tls-only`. +- `$ ably apps delete [ID]`: Deletes the specified (or current) app after confirmation. Supports `--force`. +- `$ ably apps set-apns-p12 ID`: Uploads an APNS P12 certificate for the specified app `ID`. Requires `--certificate`. Supports `--password`, `--use-for-sandbox`. +- `$ ably apps stats [ID]`: Views app stats for the specified (or current) app. Supports `--live` polling, `--unit`, `--start`, `--end`, `--limit`. +- `$ ably apps switch [APPID]`: Switches the active app configuration to the specified App ID or prompts for selection. +- `$ ably apps current`: Shows the currently selected app configuration. +- `$ ably apps logs subscribe`: Alias for `ably logs app subscribe`. +- `$ ably apps logs history`: Alias for `ably logs app history`. + +**Channel Rules (`ably apps channel-rules`)** +*(Manage Ably channel rules/namespaces via Control API)* + +- `$ ably apps channel-rules list`: Lists all channel rules for the current/specified app. +- `$ ably apps channel-rules create`: Creates a channel rule. Requires `--name`. Supports various flags like `--persisted`, `--push-enabled`, etc. +- `$ ably apps channel-rules update NAMEORID`: Updates a channel rule specified by name or ID. Supports various flags. +- `$ ably apps channel-rules delete NAMEORID`: Deletes a channel rule specified by name or ID after confirmation. Supports `--force`. + +**Authentication & Authorization (`ably auth`)** +*(Manage data plane auth: API Keys, Tokens)* + +- `$ ably auth issue-jwt-token`: Creates an Ably JWT token. Supports `--capability`, `--ttl`, `--client-id`, `--token-only`. +- `$ ably auth issue-ably-token`: Creates an Ably Token. Supports `--capability`, `--ttl`, `--client-id`, `--token-only`. +- `$ ably auth revoke-token TOKEN`: Revokes a specific Ably Token or JWT. Supports revoking by `--client-id`. + +**API Key Management (`ably auth keys`)** +*(Manage API keys via Control API)* + +- `$ ably auth keys list`: Lists all API keys for the current/specified app. +- `$ ably auth keys create`: Creates a new API key. Requires `--name`. Supports `--capabilities`. +- `$ ably auth keys get KEYNAMEORVALUE`: Shows details for a specific API key (using `APP_ID.KEY_ID` format or full key value). +- `$ ably auth keys update KEYNAME`: Updates properties (name, capabilities) of an API key (using `APP_ID.KEY_ID` format). +- `$ ably auth keys revoke KEYNAME`: Revokes an API key (using `APP_ID.KEY_ID` format) after confirmation. Supports `--force`. +- `$ ably auth keys switch [KEYNAMEORVALUE]`: Sets the default API key for the current app in the local config. Prompts if no key specified. +- `$ ably auth keys current`: Shows the currently configured API key for the selected app. + +**Pub/Sub Channels (`ably channels`)** +*(Interact with Ably Pub/Sub using the Realtime SDK)* + +- `$ ably channels list`: Lists active channels via channel enumeration API. Supports `--prefix`, `--limit`. +- `$ ably channels publish CHANNEL MESSAGE`: Publishes a message. Supports `--name`, `--encoding`, `--count`, `--delay`, JSON/text message, message interpolation (`{{.Count}}`, `{{.Timestamp}}`), `--transport rest|realtime`. +- `$ ably channels batch-publish [MESSAGE]`: Publishes multiple messages via REST batch API. Supports `--channels`, `--channels-json`, `--spec`. +- `$ ably channels subscribe CHANNELS...`: Subscribes to messages on one or more channels. Supports `--rewind`, `--delta`, `--cipher-*` flags for decryption. Runs until terminated. +- `$ ably channels history CHANNEL`: Retrieves message history. Supports `--start`, `--end`, `--limit`, `--direction`, `--cipher` flags. +- `$ ably channels logs [TOPIC]`: Alias for `ably logs channel-lifecycle subscribe`. (Currently only supports `channel-lifecycle`). +- `$ ably channels occupancy get CHANNEL`: Gets current occupancy metrics for a channel. +- `$ ably channels occupancy subscribe CHANNEL`: Subscribes to live occupancy metrics using the meta channel. Runs until terminated. +- `$ ably channels presence enter CHANNEL`: Enters presence and stays present. Supports `--data`, `--show-others`. Runs until terminated. +- `$ ably channels presence subscribe CHANNEL`: Subscribes to presence events (joins, leaves, updates). Runs until terminated. + +**Connections (`ably connections`)** +*(Interact with Pub/Sub connections)* + +- `$ ably connections stats`: Shows connection stats (similar to `ably apps stats` but connection-focused). Supports `--live`. +- `$ ably connections test`: Performs a simple connection test. Supports transport options. +- `$ ably connections logs [TOPIC]`: Alias for `ably logs connection-lifecycle subscribe`. (Currently only supports `connection-lifecycle`). + +**Chat Rooms (`ably rooms`)** +*(Interact with Ably Chat using the Chat SDK)* + +- `$ ably rooms list`: Lists chat rooms (filters channel enumeration). +- `$ ably rooms messages send ROOMID TEXT`: Sends a chat message. Supports `--count`, `--delay`, interpolation. +- `$ ably rooms messages subscribe ROOMID`: Subscribes to chat messages. Runs until terminated. +- `$ ably rooms messages get ROOMID`: Gets historical chat messages. +- `$ ably rooms occupancy get ROOMID`: Gets current occupancy for a room. +- `$ ably rooms occupancy subscribe ROOMID`: Subscribes to live room occupancy. Runs until terminated. +- `$ ably rooms presence enter ROOMID`: Enters presence in a room and stays present. Runs until terminated. +- `$ ably rooms presence subscribe ROOMID`: Subscribes to room presence events. Runs until terminated. +- `$ ably rooms reactions send ROOMID MESSAGEID EMOJI`: Sends a message reaction. +- `$ ably rooms reactions subscribe ROOMID`: Subscribes to message reactions. Runs until terminated. +- `$ ably rooms typing start ROOMID`: Starts sending typing indicators. Runs until terminated. +- `$ ably rooms typing subscribe ROOMID`: Subscribes to typing indicators. Runs until terminated. + +**Spaces (`ably spaces`)** +*(Interact with Ably Spaces using the Spaces SDK)* + +- `$ ably spaces list`: Lists Spaces (filters channel enumeration). +- `$ ably spaces members enter SPACEID`: Enters a Space and stays present. Runs until terminated. +- `$ ably spaces members subscribe SPACEID`: Subscribes to Space member events (enter, leave, update). Runs until terminated. +- `$ ably spaces locations set SPACEID`: Sets own location and stays updated. Shows other location changes. Runs until terminated. Supports position arguments. +- `$ ably spaces locations subscribe SPACEID`: Subscribes to location updates for all members. Runs until terminated. +- `$ ably spaces locations get-all SPACEID`: Gets current locations for all members and exits. +- `$ ably spaces cursors set SPACEID`: Sets own cursor position. Simulates movement if no position given. Runs until terminated. Supports position arguments. +- `$ ably spaces cursors subscribe SPACEID`: Subscribes to cursor position updates. Uses color coding for changes. Runs until terminated. +- `$ ably spaces cursors get-all SPACEID`: Gets current cursor positions for all members and exits. +- `$ ably spaces locks acquire SPACEID LOCKID`: Acquires a lock and holds it. Shows other lock changes. Runs until terminated. +- `$ ably spaces locks get SPACEID LOCKID`: Checks if a specific lock exists and shows details. +- `$ ably spaces locks get-all SPACEID`: Gets all current locks and exits. +- `$ ably spaces locks subscribe SPACEID`: Subscribes to lock status updates. Uses color coding for changes. Runs until terminated. + +**Logging (`ably logs`)** +*(Stream and retrieve logs from meta channels)* + +- `$ ably logs app subscribe`: Streams logs from `[meta]log`. Supports `--rewind`. +- `$ ably logs app history`: Retrieves historical logs from `[meta]log`. Supports `--limit`, `--direction`. +- `$ ably logs channel-lifecycle subscribe`: Streams logs from `[meta]channel.lifecycle`. +- `$ ably logs connection-lifecycle subscribe`: Streams logs from `[meta]connection.lifecycle`. +- `$ ably logs connection-lifecycle history`: Retrieves historical connection logs. Supports `--limit`, `--direction`. +- `$ ably logs connection subscribe`: Streams logs from `[meta]connection`. Supports `--rewind`. +- `$ ably logs push subscribe`: Streams logs from `[meta]log:push`. Supports `--rewind`. +- `$ ably logs push history`: Retrieves historical push logs from `[meta]log:push`. Supports `--limit`, `--direction`. + +**Integrations (`ably integrations`)** +*(Manage Ably integrations/rules via Control API)* + +- `$ ably integrations list`: Lists all integration rules for the current/specified app. +- `$ ably integrations create`: Creates an integration rule. Requires various flags depending on rule type. +- `$ ably integrations get RULEID`: Shows details for a specific integration rule. +- `$ ably integrations update RULEID`: Updates an integration rule. Supports various flags. +- `$ ably integrations delete RULEID`: Deletes an integration rule after confirmation. Supports `--force`. + +**Queues (`ably queues`)** +*(Manage Ably Queues via Control API)* + +- `$ ably queues list`: Lists all queues for the current/specified app. +- `$ ably queues create`: Creates a queue. Requires various flags. +- `$ ably queues delete QUEUENAME`: Deletes a queue after confirmation. Supports `--force`. + +**Benchmarking (`ably bench`)** +*(Run benchmark tests)* + +- `$ ably bench publisher CHANNEL`: Starts a publisher test. Measures latency, throughput. Waits for subscribers if `--wait-for-subscribers` is used. Shows progress UI. Supports `--messages`, `--rate`, `--message-size`, `--transport rest|realtime`. +- `$ ably bench subscriber CHANNEL`: Starts a subscriber test. Waits for publisher, measures latency, message count. Shows progress UI. Runs until publisher finishes, then waits for next test. + +**Help & Info (`ably help`)** + +- `$ ably help ask QUESTION`: Sends a question to the Ably AI Help agent via Control API. Shows "Thinking..." animation. Displays answer and links. Supports `--continue` for follow-up questions, storing conversation context locally. +- `$ ably help contact`: Opens in the browser. +- `$ ably help support`: Opens in the browser. +- `$ ably help status`: Checks Ably service status via and directs user to . +- `$ ably --version`: Shows the CLI version. +- `$ ably [topic]`: If a topic is called without a command, it shows available sub-commands, descriptions, and examples. +- `$ ably [command] --help`: Shows detailed help for a specific command. + +**Configuration (`ably config`)** + +- `$ ably config`: Opens the local config file (`~/.ably/config`) in the default text editor. Supports `--editor`. + +**MCP Server (`ably mcp`)** + +- `$ ably mcp start-server`: Starts the CLI in Model Context Protocol (MCP) server mode. + +### 5.2. Authentication + +Two types: +1. **Control Plane (User Level)**: Uses Control API access tokens obtained from . Scoped to one user and one account. Managed via `ably accounts` commands. +2. **Data Plane (App Level)**: Uses API keys or Ably Tokens/JWTs. Scoped to a single app. Managed via `ably auth` commands. The CLI prioritizes credentials in this order: + * Command-line flags (`--api-key`, `--token`, `--access-token`). + * Environment variables (`ABLY_API_KEY`, `ABLY_TOKEN`, `ABLY_ACCESS_TOKEN`). + * Locally stored configuration (`~/.ably/config`) for the current app/account. + +**Convenience Workflow:** +If a data plane command is run without explicit auth flags/env vars and no app/key is configured locally: +- If logged in (`ably accounts login` done), the CLI prompts the user to select an app from their account. +- Then, it prompts the user to select an API key for that app. +- The selected app ID and API key are stored in the local config for future use with that app under the current account profile. + +**Login Flow (`ably accounts login`):** +- Displays Ably ASCII art. +- Prompts user to press Enter to open in browser (unless `--no-browser`). +- Prompts user to paste the obtained token. +- Prompts for an optional alias (`--alias`) to store the account credentials. If no alias, updates the `default` account profile. + +### 5.3. Configuration (`~/.ably/config`) + +- Stored in TOML format at `~/.ably/config`. +- File permissions should be secured (e.g., `600`). +- Stores Control API access tokens (potentially multiple, identified by `default` or alias). +- Stores the currently active account alias. +- For each account profile, stores the currently active app ID. +- For each account profile, stores a mapping of app IDs to their last used API key. +- `ably config` command opens this file. + +### 5.4. Global Arguments + +- `--host `: Overrides the default REST/Realtime host for Data Plane SDK calls. +- `--env `: Overrides the default environment (e.g., `production`) for Data Plane SDK calls. +- `--control-host `: Overrides the default Control API host (`control.ably.net`). +- `--access-token `: Overrides any configured Control API access token. +- `--api-key `: Overrides any configured Data Plane API key. +- `--token `: Authenticates Data Plane calls using an Ably Token or JWT instead of an API key. +- `--client-id `: Overrides the default client ID (`ably-cli-<8_random_chars>`) for Data Plane operations when using API key auth. Use `"none"` to disable sending a client ID. Not applicable for token auth. +- `--json`: Outputs results as raw, machine-readable JSON. Suppresses status messages and non-essential logs. Errors are output as JSON. +- `--pretty-json`: Outputs results as formatted, colorized JSON for human readability. Suppresses status messages. Errors are output as JSON. +- `--verbose` (`-v`): Outputs additional status/log events (e.g., connection state changes). Formatted as JSON when used with `--json` or `--pretty-json`. + +### 5.5. Usability & Developer Experience + +- Helpful inline documentation (`--help`). +- Autocompletion support for bash/zsh. +- Friendly, descriptive error messages with troubleshooting hints. +- Interactive prompts for confirmations (e.g., delete) and selections (e.g., switching accounts/apps without args). +- Use of console UI elements for better visualization (e.g., progress bars in `bench`, tables, status indicators). +- Did-you-mean suggestions for mistyped commands. +- Topic commands (`ably accounts`, `ably apps`, etc.) list sub-commands when called directly. + +### 5.6. Embeddable Web Example & React Component + +*(This section describes a feature allowing the CLI to run in a browser)* + +- **React Component (`@ably/react-cli-terminal` - hypothetical name)**: + - Embeddable component using Xterm.js (). + - Connects via WebSocket to a dedicated Terminal Server. + - Requires `controlAccessToken` and `apiKey` props passed during initialization. These map to `ABLY_ACCESS_TOKEN` and `ABLY_API_KEY` environment variables within the container. +- **Terminal Server**: + - Listens for WebSocket connections. + - On connection, spawns a new Docker container running the Ably CLI. + - Proxies WebSocket stream to the container's stdin/stdout. + - Terminates Docker container on WebSocket disconnect or after timeout (e.g., 30 mins). + - Limits concurrent sessions. + - Includes security measures (see [docker/README.md](../docker/README.md)). +- **Docker Container**: + - Based on `node:22-alpine` (or similar). + - Installs `@ably/cli`. + - Environment variables `ABLY_ACCESS_TOKEN` and `ABLY_API_KEY` must be set and validated on startup. + - Restricts execution to only the `ably` command for security. + - Sets an environment variable (e.g., `ABLY_CLI_MODE=web`) to indicate web context. +- **Web CLI Mode Restrictions**: + - The CLI detects `ABLY_CLI_MODE=web`. + - Disables commands related to local configuration and switching credentials: + - `ably accounts login`: Error message (already logged in). + - `ably accounts list`: Error message (use dashboard). + - `ably accounts logout`: Error message (cannot log out). + - `ably accounts switch`: Error message (use dashboard). + - `ably apps switch`: Error message (use dashboard). + - `ably auth keys switch`: Error message (use web interface). + - `ably config`: Error message (no local config). + - Adapts commands relying on local config: + - `ably accounts current`: Uses `/me` Control API endpoint based on provided token. + - `ably apps current`: Uses app ID from `ABLY_API_KEY` and calls Control API for details. +- **Demo Web Example (`/example/web-cli`)**: + - Simple Vite-based example showing how to use the React component. + - Reads `ABLY_ACCESS_TOKEN` and `ABLY_API_KEY` from `.env` file or environment. + - Includes usage instructions in `README.md`. + +### 5.7. Model Context Protocol (MCP) Server + +*(This section describes running the CLI as an MCP server for AI tools)* + +- **Activation**: Run `$ ably mcp start-server`. +- **Technology**: Uses `@modelcontextprotocol/typescript-sdk` in file mode. +- **Authentication**: Relies on environment variables: + - `ABLY_ACCESS_TOKEN`: Control API token. + - `ABLY_API_KEY`: Data Plane key. + - `ABLY_CLIENT_ID`: Optional client ID override (defaults to `ably-mcp-`). + - `ABLY_CONTROL_HOST`: Optional Control API host override. + - `ABLY_HOST`: Optional Data Plane host override. + - `ABLY_ENVIRONMENT`: Optional Data Plane environment override. +- **Behavior**: + - Disables all interactive prompts. + - Sets a maximum execution time per request (e.g., 15 seconds). Long-running commands like `subscribe` will run for this duration then exit gracefully. + - Uses a distinct client ID prefix (`ably-mcp-`). +- **Available MCP Commands (Subset of CLI)**: + - `ably apps list` + - `ably apps stats` + - `ably auth keys list` + - `ably channels history` + - `ably channels publish` + - `ably channels list` + - `ably channels presence get` (*Note: README has `ably channels presence subscribe` and `enter`, but not `get`. PRD lists `get` but not here. Assuming `get` is intended for MCP based on non-streaming nature.*) + +## 6. Technical Requirements + +- **Platforms**: Linux, macOS, Windows. +- **Technology**: Node.js, TypeScript, oclif framework (). +- **Distribution**: npm (`@ably/cli`), Homebrew, standalone executables. +- **Dependencies**: Use `pnpm`. Use SDKs (`ably-js`, `ably-chat-js`, `ably-spaces-js`) for data plane; direct HTTPS for Control API. Use JWT library for token generation. + +## 7. Testing Requirements + +- Unit, integration, and end-to-end tests are required. +- Data plane operations should ideally have E2E tests. +- Control API interactions can be mocked (based on OpenAPI spec). +- See [Testing Strategy](Testing.md) for strategy. + +## 8. API Dependencies + +- **Control API**: Used for account/app management. See and OpenAPI spec. Requires Control API access token. +- **Ably SDKs**: + - **Pub/Sub**: Use `ably-js` (). Use `request()` method for REST calls not covered by SDK methods. + - **Chat**: Use `ably-chat-js` (link TBC). Use underlying `ably-js` `request()` if needed. + - **Spaces**: Use `ably-spaces-js` (link TBC). Use underlying `ably-js` `request()` if needed. +- **JWT Library**: For generating Ably JWTs (`ably auth issue-jwt-token`). + +## 9. Usage Examples (Illustrative) ```sh +# Log in (interactive or provide token) ably login -ably channels list --app appId -ably channels publish channelName '{"name":"update","data":"Hello, World"}' -ably subscribe my-channel --app "App name" -ably apps list -ably accounts switch account-alias -``` - -### Ably AI Agent - -The Ably AI agent is available as an endpoint within the control API under /v1/help. -The request made to that endpont is a POST request with body {"question":} -An example response is below. When waiting for a response, we will show a "Thinking..." message with an animation to indicate the system is preparing an answer. Once the answer comes back, we should output the response, along with the list of links from the array at the end of the response. - -Once an answer is provided: - -- Suggest to the user that they can do a follow up question with syntax `ably help ask --continue "Question"` -- We should store the last question and answer locally so that we can provide the context if there is a --continue command -- When the --continue flag is used, inject the previous conversation thread, including all previous questions and answers -- Any subsequent question without --continue, resets the locally stored info and stores the most recent question and answer - -```json -{ - "answer": " - -Here's how to get started with Ably: - -1. Create an Ably account and get your API key [(1)](https://ably.com/docs/getting-started/quickstart#step-2) - -2. Add the Ably Client Library SDK to your project. For JavaScript, you can either: - -```html - -``` -[(1)](https://ably.com/docs/getting-started/quickstart#step-2) - -Or install via NPM: - -``` -npm install ably -``` - -[(1)](https://ably.com/docs/getting-started/quickstart#step-2) - -3. Connect to Ably: - -```javascript -const ably = new Ably.Realtime("YOUR-API-KEY"); -await ably.connection.once("connected"); -console.log("Connected to Ably!"); -``` - -[(1)](https://ably.com/docs/getting-started/quickstart#step-2) - -4. Subscribe to a channel: +# List apps in the current account +ably apps list -```javascript -const channel = ably.channels.get("quickstart"); -await channel.subscribe("greeting", (message) => { - console.log("Received a greeting message in realtime: " + message.data); -}); -``` +# Publish a message to a channel for the current/specified app +ably channels publish my-channel '{"greeting":"Hello"}' --name update -[(1)](https://ably.com/docs/getting-started/quickstart#step-2) +# Subscribe to messages (runs until stopped) +ably channels subscribe my-channel --app "My Specific App" -5. Publish a message: +# Switch active account config +ably accounts switch my-work-alias -```javascript -await channel.publish("greeting", "hello!"); +# Ask the AI helper a question +ably help ask "How do I use channel history?" ``` -[(1)](https://ably.com/docs/getting-started/quickstart#step-2) - -Note: For production environments, you should use token authentication instead of basic authentication with an API key to avoid exposing it client-side [(1)](https://ably.com/docs/getting-started/quickstart#step-2)", -"links": [ -{ -"label": "1", -"type": "documentation", -"url": "https://ably.com/docs/getting-started/quickstart", -"title": "Ably Pub/Sub | Quickstart guide", -"breadcrumbs": [ -"Docs", -"Ably Pub/Sub | Quickstart guide" -], -"description": null -} -] -} +## 10. Ably AI Agent (`ably help ask`) -``` - -``` +- Uses Control API endpoint `/v1/help`. +- POST request with `{"question": "user question"}`. +- If `--continue` flag is used, sends previous Q&A context along with the new question. +- Displays "Thinking..." animation while waiting. +- Renders Markdown answer and lists source links. +- Stores last Q&A locally for `--continue`. New question without flag resets context.