Record elapsed time for synthetic test results - #423
Merged
Conversation
bslobodin
marked this pull request as ready for review
August 12, 2026 12:10
bslobodin
force-pushed
the
fix/synthetic-test-duration
branch
from
August 12, 2026 12:23
975c5bb to
142e9ee
Compare
bslobodin
changed the title
bslobodin
force-pushed
the
fix/synthetic-test-duration
branch
from
August 12, 2026 12:24
142e9ee to
362323e
Compare
bslobodin
marked this pull request as draft
August 12, 2026 12:24
bslobodin
force-pushed
the
fix/synthetic-test-duration
branch
from
August 12, 2026 12:28
362323e to
394a435
Compare
bslobodin
marked this pull request as ready for review
August 12, 2026 12:28
A result built by LazySingleExample#build_error_result or
#build_stale_skip_result does not run through Minitest's normal timing path,
so its #time stays nil. TestData passes that straight through as
test_duration, which the shopify_build_test_results Monorail schema declares
as a non-nullable double.
The exporter therefore rejects the whole event
("at .test_duration expected type double, got null") and the test result is
dropped silently: the producing build only sees an HTTP 400 inside a 207 and
carries on. One unresolvable test class costs every affected shard's results,
so a single stale queue entry loses tens of rows at a time.
Populate missing result times from LazySingleExample's existing start and
finish timestamps. This records class resolution and file loading work while
preserving the duration Minitest assigns when the runnable executes.
Co-authored-by: Boris Slobodin <boris.slobodin@shopify.com>
Assisted-By: devx/d2514402-869f-427f-956d-3ae0f06cb475
bslobodin
force-pushed
the
fix/synthetic-test-duration
branch
from
August 12, 2026 12:33
394a435 to
be59f49
Compare
daiaventureira
approved these changes
Aug 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
LazySingleExampledurations from its elapsed start and finish timestamps, including class resolution and file loading time0.0exporter fallback so any remaining timeless result cannot invalidate the batchLazySingleExamplesynthesizes results for load errors and stale queue entries. Those results bypass Minitest's timing path, leavingtimenil; Monorail requirestest_durationto be a non-nullable double, so one such result can reject the exported batch.Test plan
shadowenv exec -- bundle exec rake test TEST_FILES='test/minitest/queue/lazy_single_example_test.rb test/minitest/queue/test_data_test.rb'0.12duration, and the0.0exporter fallback