Skip to content

Only run test harness tests in misc shard instead of every shard#101218

Merged
jmagman merged 1 commit into
flutter:masterfrom
jmagman:smoketests
Apr 5, 2022
Merged

Only run test harness tests in misc shard instead of every shard#101218
jmagman merged 1 commit into
flutter:masterfrom
jmagman:smoketests

Conversation

@jmagman

@jmagman jmagman commented Apr 2, 2022

Copy link
Copy Markdown
Member

Every test run shard first runs a 30 second "smoke test" suite to prove the test harness works. This suite runs 64 times per commit (once per shard/subshard/platform). Rename it to a "test harness tests" and only run it in the misc shard (which still runs 4 times per commit).

Running and passing in misc:
https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8818037355707764481/+/u/run_test.dart_for_framework_tests_shard_and_subshard_misc/test_stdout

Running test harness tests...
▌01:03:31▐ RUNNING: cd .; bin/cache/artifacts/engine/linux-x64/flutter_tester --help
▌01:03:31▐ ELAPSED TIME: 0.050s for bin/cache/artifacts/engine/linux-x64/flutter_tester --help in .
▌01:03:31▐ RUNNING: cd dev/automated_tests; ../../bin/flutter test --test-randomize-ordering-seed=20220401 --fatal-warnings test_smoke_test/pass_test.dart
▌01:03:44▐ ELAPSED TIME: 13.296s for ../../bin/flutter test --test-randomize-ordering-seed=20220401 --fatal-warnings test_smoke_test/pass_test.dart in dev/automated_tests
▌01:03:44▐ RUNNING: cd dev/automated_tests; ../../bin/flutter test --test-randomize-ordering-seed=20220401 --fatal-warnings test_smoke_test/fail_test.dart
▌01:03:48▐ ELAPSED TIME: 3.625s for ../../bin/flutter test --test-randomize-ordering-seed=20220401 --fatal-warnings test_smoke_test/fail_test.dart in dev/automated_tests
▌01:03:48▐ RUNNING: cd dev/automated_tests; ../../bin/flutter test --test-randomize-ordering-seed=20220401 --fatal-warnings test_smoke_test/pending_timer_fail_test.dart
▌01:03:52▐ ELAPSED TIME: 3.723s for ../../bin/flutter test --test-randomize-ordering-seed=20220401 --fatal-warnings test_smoke_test/pending_timer_fail_test.dart in dev/automated_tests
▌01:03:52▐ RUNNING: cd dev/automated_tests; ../../bin/flutter test --test-randomize-ordering-seed=20220401 --fatal-warnings test_smoke_test/crash1_test.dart
▌01:03:55▐ ELAPSED TIME: 3.491s for ../../bin/flutter test --test-randomize-ordering-seed=20220401 --fatal-warnings test_smoke_test/crash1_test.dart in dev/automated_tests
▌01:03:55▐ RUNNING: cd dev/automated_tests; ../../bin/flutter test --test-randomize-ordering-seed=20220401 --fatal-warnings test_smoke_test/crash2_test.dart
▌01:03:59▐ ELAPSED TIME: 3.357s for ../../bin/flutter test --test-randomize-ordering-seed=20220401 --fatal-warnings test_smoke_test/crash2_test.dart in dev/automated_tests
▌01:03:59▐ RUNNING: cd dev/automated_tests; ../../bin/flutter test --test-randomize-ordering-seed=20220401 --fatal-warnings test_smoke_test/syntax_error_test.broken_dart
▌01:04:01▐ ELAPSED TIME: 2.873s for ../../bin/flutter test --test-randomize-ordering-seed=20220401 --fatal-warnings test_smoke_test/syntax_error_test.broken_dart in dev/automated_tests
▌01:04:01▐ RUNNING: cd dev/automated_tests; ../../bin/flutter test --test-randomize-ordering-seed=20220401 --fatal-warnings test_smoke_test/missing_import_test.broken_dart
▌01:04:04▐ ELAPSED TIME: 2.831s for ../../bin/flutter test --test-randomize-ordering-seed=20220401 --fatal-warnings test_smoke_test/missing_import_test.broken_dart in dev/automated_tests
▌01:04:04▐ RUNNING: cd dev/automated_tests; ../../bin/flutter test --test-randomize-ordering-seed=20220401 --fatal-warnings test_smoke_test/disallow_error_reporter_modification_test.dart
▌01:04:08▐ ELAPSED TIME: 3.885s for ../../bin/flutter test --test-randomize-ordering-seed=20220401 --fatal-warnings test_smoke_test/disallow_error_reporter_modification_test.dart in dev/automated_tests
▌01:04:08▐ RUNNING: cd examples/api; ../../bin/flutter test --test-randomize-ordering-seed=20220401 --fatal-warnings --null-assertions --sound-null-safety

Pre-launch Checklist

  • I read the Contributor Guide and followed the process outlined there for submitting PRs.
  • I read the Tree Hygiene wiki page, which explains my responsibilities.
  • I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
  • I signed the CLA.
  • I listed at least one issue that this PR fixes in the description above.
  • I updated/added relevant documentation (doc comments with ///).
  • I added new tests to check the change I am making, or this PR is test-exempt.
  • All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

@jmagman jmagman self-assigned this Apr 2, 2022
@flutter-dashboard flutter-dashboard Bot added the c: contributor-productivity Team-specific productivity, code health, technical debt. label Apr 2, 2022
@jmagman jmagman changed the title Only run test harness tests once instead of every shard Only run test harness tests in misc shard instead of every shard Apr 2, 2022

@zanderso zanderso left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@fluttergithubbot

Copy link
Copy Markdown
Contributor

This pull request is not suitable for automatic merging in its current state.

@jmagman jmagman merged commit bc0ec85 into flutter:master Apr 5, 2022
@jmagman jmagman deleted the smoketests branch April 5, 2022 00:27
engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Apr 5, 2022
engine-flutter-autoroll added a commit to engine-flutter-autoroll/plugins that referenced this pull request Apr 5, 2022
engine-flutter-autoroll added a commit to engine-flutter-autoroll/plugins that referenced this pull request Apr 5, 2022
engine-flutter-autoroll added a commit to engine-flutter-autoroll/plugins that referenced this pull request Apr 5, 2022
engine-flutter-autoroll added a commit to engine-flutter-autoroll/plugins that referenced this pull request Apr 5, 2022
engine-flutter-autoroll added a commit to engine-flutter-autoroll/plugins that referenced this pull request Apr 5, 2022
engine-flutter-autoroll added a commit to engine-flutter-autoroll/plugins that referenced this pull request Apr 5, 2022
engine-flutter-autoroll added a commit to engine-flutter-autoroll/plugins that referenced this pull request Apr 5, 2022
engine-flutter-autoroll added a commit to engine-flutter-autoroll/plugins that referenced this pull request Apr 5, 2022
engine-flutter-autoroll added a commit to engine-flutter-autoroll/plugins that referenced this pull request Apr 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

c: contributor-productivity Team-specific productivity, code health, technical debt.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants