This repository was archived by the owner on Nov 6, 2023. It is now read-only.
Repository navigation
This repository was archived by the owner on Nov 6, 2023. It is now read-only.
Cultivate a checklist to prepare for next full-fetch-test run #13829
Description
Type: other
There was a fair amount of discussion and work to be done after the last run of the full fetch test, which disables rulesets if we observe problems on any of their hosts - see #10538.
A list of tasks were formulated to perform before the next run of the test. These take the form of both one-time tasks that will generically help our workflow, as well as tasks that should be performed before each run. I'm sure I'm missing something, please help me cultivate this list.
One-Time Tasks
- Instead of disabling the entire ruleset, remove individual hosts (Update https-everywhere-checker to disable single target hosts rather than entire rules #1998)
- The report that is generated when a ruleset is disabled (beside the
Disabled by https-everywhere-checker becausetext) should include a timestamp - Run comment-removal script (NodeJS modules to remove obsolete https-everywhere-checker messages #11963)
Before Each Run
- Update the cipher list in with ciphers from an updated version of Firefox ESR
- Create an issue (or send an email) announcing that there will be a forthcoming run of the full fetch test
- Before merging, from
HEADonmasterdo agit diff --name-only <base commit>(see Disabling rulesets that fail the comprehensive fetch test across mult… #10538 (comment))
Reacted by Hugo Gruson and pipboy96
Activity
cschanaj commentedon Dec 3, 2017
We should clean up the obsolete comments (if any) after each run. See also #11963, which has been pending review for some time. Currently, round 200+ rulesets are having obsolete comments.
P.S. If we are going to have a new checker which remove individual hosts instead of disabling the entire ruleset, #11963 need not to be merged. However, please run the script in #11963 before the next fetch-test. thanks!
cschanaj commentedon Dec 15, 2017
In #10516 (comment), it is suggested that
Any chance that an official such list will be available after the next
full-fetch-test?Bisaloo commentedon Jan 31, 2018
Didn't you also update the
ruleset-whitelist.csvformat to be able to use it infull-fetch-test?#10227
Hainish commentedon Jan 31, 2018
@cschanaj currently there is no automated mechanism for deleting rulesets that have been disabled for the same reason after subsequent runs. The deletion tool has to be created before we submit the results of those deletions for review.
Hainish commentedon Jan 31, 2018
@Bisaloo with the creation of
ruleset-whitelist.csv, we are now whitelisting rulesets from the fetch test the that are expected to fail the fetch test, and whitelisting rulesets from the coverage test that are expected to fail the coverage test.This should help future runs of the
full-fetch-testtool, since we will no longer be whitelisting rulesets from fetches that are only expected to fail the coverage test.Either way, the
full-fetch-testtool should be able to use the new whitelist format OOTB, since it is not touching that whitelist directly and is relying on internal tools included in this repo to do that work.Does this answer your question?
Bisaloo commentedon Feb 1, 2018
Yes, I'm sorry, I should have read the code first. I didn't realise that
full-fetch-testwas using regular tests from this repo.pipboy96 commentedon Sep 17, 2019
@zoracon @andresbase @Hainish What's the current status of this?
zoracon commentedon Sep 18, 2019
@pipboy96 I'm planning on running a fetch test this fall.