UI Tests for the extension

[DonJayamanne]

Create simple framework for running smoke tests for the extension with the following requirements/benefits:

• Update WIKI
• UI Testing/System Testing
  • Developers can focus on writing unit tests
  • Most of the integration/system Tests, etc can all be replaced with these UI Tests (slowly over time)
  • Hopefully this will result in fewer regressions
  • Bug bashes can focus on testing new features
• Ubiquitous language (BDD)
  • Anyone can write new tests.
  • Even if we were to refactor the entire extension, the tests will not require any changes, unlike other tests.
  • Current implemented in Python, if we use nodejs, the tests are all still
    usable.
• Test Results
  • The test results can contain screenshots for manual verification
  • Currently when ever a test fails, we capture a screenshot of VSCode
    • This is useful to identify why tests failed
  • Capture all VS Code (Extension/Debugger) logs as part of the test result
• CI Integration
  • Can fail CI based on test results
  • Test results can be integrated with Azdo (though I see now value in integrating this with Azdo), i.e. using the CLI output is easier..
  • Test results can be uploaded into Azdo (as artifacts) - for manual verification or when identifying why tests fail (to view error logs, and screenshots)
• Run against multiple versions of Python
• Run against VS Code Stable and Insiders

Notes

• These tests are slow when using Python (if its running on CI, this should not be a problem)
• Some coding will be necessary (i.e. nothing comes for free).
• Output in Terminal cannot be tested when using Python (we are limited to a few tests)

[DonJayamanne]

UI Tests for debugging Python Tests using the Test Explorer

• Ensure debugging works
• Ensure the right function/suite is debugged
• Ensure breakpoints work
• Ensure stack trace is correct
• Ensure we do not break/nor debug other functions/suites...
• Same set of tests run for ALL test frameworks/runners (Unit Test, pytest & nose)

[DonJayamanne]

Sample Report

[DonJayamanne]

Sample Test

[DonJayamanne]

Azdo Integartion