Allow easier input of domains requested to be reviewed

@jeremyn

Credits for this idea goes to @jeremyn

The idea was originally posted in #6307
Moving it here to keep the other issue clean from off-topic discussion.

The discussion left off with me saying:

users could tweet domains using some hashtag:

I like the idea as I am a Twitter user, however, not everyone have a Twitter account and I think it will be difficult to list and mark which are done.

What about using Google Forms and having it being saved in a Spreadsheet? Contributors could mark domains with its status and the spreadsheet would be public for read-access. Thoughts?

@Foorack

Thanks for making this issue @Foorack.

Important questions are, how many sites do we expect people to enter? Who are the different categories of users and what are their levels of trustworthiness and technical skill? Who is going to officially own and be responsible for this tool?

Anything that allows unrestricted write access to the general internet is a challenge, and anything that allows the general internet to write something and have it automatically appear somewhere else like a status page is even worse. At a minimum the tool needs one free-form text field to submit a domain, and another field to submit optional notes about that domain ("This site behaves differently if you access it from outside the U.S....", that sort of thing.) So, the back-end needs to:

make sure the submitted domain is a valid domain in the RFC meaning of "valid"
do a basic check that the domain is actually a use case for HTTPS Everywhere (supports both HTTP and HTTPS)
do basic automated spam/troll filtering
do basic de-duplication

I don't think there should be a status page, but if we do have one, then we also need to put the entries into a hidden queue for a trusted person to approve before they appear on that page.

So most of the work is back-end data processing. I'm not experienced with Google Forms but my guess is it would save us a little front-end work in creating a form but would not help at all on the back-end and might even complicate things.

With the Twitter approach, the problem becomes almost entirely back-end since there is no form.

My feeling is this tool should be a small but complete and standalone web application.

Anything that allows unrestricted write access to the general internet is a challenge, and anything that allows the general internet to write something and have it automatically appear somewhere else like a status page is even worse.

Eh, we do have an unrestricted internet. Fortunately. At least in most places. At least when we exclude net neutrality issu... I get too far. So what do you mean by "internet"? You should specify this term a bit more.
Also we can't write to the "general internet". Don't know what a "specific internet" would be, but, well... I think these sentences confuse me.

At a minimum the tool needs one free-form text field to submit a domain, and another field to submit optional notes about that domain ("This site behaves differently if you access it from outside the U.S....", that sort of thing.)

Okay, getting back to sentences I understand. So yes, that's it. Although a check-boxes for common issues (mixed content, ...) would be useful. In this case, however, we would have to explain users what mixed content is. That's quite difficult...
However this could be circumvented by letting HTTPSE check by themself whether it is Mixed Content or by doing a temporary ruleset and ask the user again after - say - one week: "Could you use XY.com successfully?" Basically this would be some interactive way to submit domains and they are usable for non tech-savvy users.

make sure the submitted domain is a valid domain in the RFC meaning of "valid"

I'm sure some kind of RegExp will do this. However we could also solve this issue on the client site (when we use the "interactive submission" idea again): Users can only submit the domain they are currently using. And if browsers can connect to these ones - and HTTPS could e.g. somehow check the IP to make sure it is no internal LAN ip - they are allowed to submit it.

I think all the other things you mentioned can also be checked on the client side with the interactive submission.

I'm not experienced with Google Forms

I am sure we don't need this. It's a simple UI... I am also sure EFF does not want to "expose" their users to Google. 😄

My feeling is this tool should be a small but complete and standalone web application.

My feeling is it should be included in the HTTPSE addon and maybe it could ask the user when they visit a HTTPS: "Hey, you visited example.com for more than 100 times. We have not included this into our HTTPS list yet. Do you want to help us by submitting this domain?"

BTW there is already a backend, which does some mixed content testing and so on: #1192

Also https://github.com/EFForg/httpse-ruleset-tests could be somehow integrated into HTTPSE. And I thought there was an issue for this task, but I cannot find it right now.

I like the idea of integrating it into the extension and allowing users to report/submit domains.

and ask the user again after - say - one week: "Could you use XY.com successfully?"

Though I'm not that big fan of HTTPSE breaking websites, especially for non-techy users as they will just remove the extension completely. Even if we implement it into the extension instead of having a web UI there is still the need for a backend system collecting and testing the domains. I would be really good if the backend created issues automatically. :)

@jeremyn

HTTPSE breaking websites, especially for non-techy users as they will just remove the extension completely.

Yeah, but in my idea the user has to opt-in explicitly and there could even be a "This site is broken!" button or something shown, which reminds the user that something is not working. Aso - in an ideal case - HTTPE could detect Mixed content issues by itself and stop the experiment or at least ask the user whether the site is okay.

Even if we implement it into the extension instead of having a web UI there is still the need for a backend system collecting and testing the domains.

Yeah, at least for the bad guys using bots to submit invalid domains and so on...

I would be really good if the backend created issues automatically.

No, it would be good if it opens PRs automatically. 😃

As suggested by @jeremyn in #3069 (comment) the bot could also parse the Certificate Transparency list, but in this case the domain needs to be checked on the server side again.

I'm sorry my general/specific internet phrasing was confusing. "General" means anonymous users, "specific" means people with some EFF trust.

I meant that we want to protect EFF from being DOS'd or embarrassed by malicious domain submitters. We need to ask how a hostile government or criminal organization with a large botnet could use this tool to overwhelm EFF's systems or volunteer resources. If it allows for people to submit endless variations of dc38cea7-cff6-40d0-85a0-2876e08d9259.com then we need to plan for that. If public users can browse the status of all submitted domains in a list, then we want to prevent users from submitting endless variations of fake "www.eff-sucksssssss.com" domains or fake domains with curse words, slurs and so on. Unfortunately I think some sort of captcha would be required.

Client-side validation is fine in addition to server-side validation but shouldn't replace server-side validation.

We should assume non-malicious users of this tool have no idea how to express what's wrong with the site other than "both http and https work, help!" The optional note field is for the user to tell us anything weird about the site they happen to know about.

I like the idea of putting this tool directly into the add-on and/or allowing people to submit the site they're currently browsing.

My hope for this issue is to let users easily report a site so some technical volunteer can look into it as time permits. Automating creating rulesets, issues or pull requests is a whole extra layer of work and maintenance that I think most people involved would eventually regret doing.

Unfortunately I think some sort of captcha would be required.

That's a good idea. This would also make it harder for people wanting to spam and would certainly discourage bots.

Also the HTTPSE server could require a submission of a current state for seven days. This may be the time the user has to test the domain (& the temporary ruleset) by themself before the request is even published in the status page/add-on page.

dc38cea7-cff6-40d0-85a0-2876e08d9259.com

A server could just test whether it is pingable/curl succeeds. And again: Users might only be able to submit a domain when they are currently browsing it. This also means that "www.eff-sucksssssss.com" is impossible unless they really register this domain. (in which case it could again be - technically - valid for HTTPSE inclusion)

Client-side validation is fine in addition to server-side validation but shouldn't replace server-side validation.

Yeah, of course. I just think the client side "validation" (with the "test period" mentioned) could filter out many broken websites and so on. It only matters to non-malicious users and prevents them from submitting bad entries.

Automating creating rulesets, issues or pull requests is a whole extra layer of work and maintenance that I think most people involved would eventually regret doing.

I think compared with what we already proposed here, it is not that difficult. It just needs a bot user interacting with GitHubs API.

Sorry, what does this mean? "Also the HTTPSE server could require a submission of a current state for seven days."

I don't want this to require, encourage, or even allow users to provide a point of contact. People might feel uncomfortable if they think it's not anonymous. They might be nervous about some programmer contacting them with technical questions. They might be nervous about talking with someone in English. They might just prefer not to be bothered. On the other end of it, they might expect personal follow-ups. They might send emails to EFF asking for a list of domains they submitted, like they have some kind of account with EFF. Law enforcement might contact EFF to get the email address of someone who submitted a controversial domain.

I agree that client-side validation to reduce user error and ease work on EFF server-side is fine.

I'm inclined to leave automated issues/pull requests as a possible "phase 2" for this project, after the basic infrastructure of collecting domains and making them visible to volunteers is figured out.

Sorry, what does this mean? "Also the HTTPSE server could require a submission of a current state for seven days."

Okay, imagine this:

User starts submission of site in client (addon).
Client informs user that a test phase begins and provides some information on how to cancel, etc.
For each day of the test phase, which passes, the client sends the current state (domain.example: 0 mixed content, 0 times user clicked on "this does not work", 12 sites visited) or it just sends the domain to the HTTPSE server (domain.example: test phase day 1 of 7). The server keeps track of the days and will only allow a final submission once 7 days are passed. Additionally it could verify whether the data of the days, which has been send, is logically. If not it rejects the request.
Also the server could further require additional tests and informs the client to make the test phase longer. E.g. less than 100 sites visited --> user needs to test more sites.
When everything works, the site is finally submitted (possibly after asking the user again to confirm the site worked in the last 7 days.) with a lot of test stats already collected, to this repo and (e.g.) a PR is automatically made. The user could be informed of the PR URL, so they can track the case further and may be able to respond to further questions.
(The client could also allow users to enter their GitHub name to be @mentioned, but this is another details, which does not matter for the general concept here.)

The requirement to submit the data each day and to make the data looking realistic and to react on different replies of the server, could make it potentially harder to fake these submissions.

Of course this whole thing is only imagination. I have not taken into account how it is exactly possible or how difficult it is to implement this. I am just throwing ideas into this issue. 😆

As for your contact thing: As explained in the lines above, I think this is a good thing. But of course it should be optional. With all these tests involved, there (hopefully) should not be the need to contact the author of the domain submission, so it can be anonymous for those who want to say anonymous. In this case all users fearing to get technical questions, might just not enter their GitHub username. (They would not even be registered on GitHub, so that's fine.)

On the other end of it, they might expect personal follow-ups. They might send emails to EFF asking for a list of domains they submitted

In my example an optional GitHub username submission would make this obsolete.

Law enforcement might contact EFF to get the email address of someone who submitted a controversial domain.

This would be obsolete too as all information is public anyway. It's on GitHub. (They might contact GitHub to get IPs, but this does not matter for HTTPSE here.)

I'm inclined to leave automated issues/pull requests as a possible "phase 2" for this project, after the basic infrastructure of collecting domains and making them visible to volunteers is figured out.

Structure it as you want. Split the project as you want. Implement it or do not implement it. As I said I am just submitting some ideas here, which would be great both from a user and developer (= here: ruleset maintainer) perspective.
I'd just say one thing: If this would be there already, I think it would be an awesome thing.

@rugk

I'm just throwing out ideas too. I don't know if EFF even likes this idea or would sponsor it. I may be willing to do some coding for it if the EFF says they will use it and can provide specifications.

Imagine your least technical friend. This person notices that sometimes there's a lock when they visit their (e.g.) hometown bank's website and sometimes there isn't. Their genius friend @rugk put this "S" thing into their internet program that's supposed to make sure the lock is always there. In my view, ideally this user can somehow submit their bank's website for review through this tool we're discussing without being intimidated by the entry form. If it mentions GitHub credentials or there's any hint that some stranger from the internet might contact them, they won't do it.

Also, allowing for the possibility of unexpected contact from EFF opens the door to phishing attacks, for example "You recently submitted $POPULAR_BANK.com to the EFF for review. We're having trouble with it. Please provide your account id and password." etc.

Yet another danger is maliciously submitting a website in someone else's name. I submit $DISGUSTING_SITE.com under your name and then you get an email asking about. Basically if we take email addresses then we need to verify that the person submitting it actually controls it.

Basically, I don't think we should accept user contact info. In fact we probably shouldn't accept anything but the domain field, not even optional notes. Maybe we only let them submit the domain they are currently browsing with no typing involved.

I really don't like the idea of collecting browsing history in the add-on for any reason. I think we can verify the domain is legitimate by a few server-side automated requests and maybe a DNS lookup, in addition to any client-side regex-type checking we think is worth doing.

I really don't like the idea of collecting browsing history in the add-on for any reason.

It is not collected. It is only done locally of course.

Our current problem is not currently lack of coverage (although clearly the more coverage the better), but rather scalability. Unfortunately, the desirability for better coverage is at ends with the scalability issue.

The number of domains that have deployed HTTPS has increased, and as a result we're seeing a greater number of ruleset PRs. These PRs generally need a human to review them. We can generally assume that for trivial rulesets (ones for which the rule is just to redirect from http to https) there is no malicious intent. Even PRs without malicious intent, though, have to be spot-checked by ruleset maintainers to make sure there is no subtle way that functionality is lost. So there's a bottleneck between the point of PR submission and inclusion.

There is also the problem of scalability in terms of memory consumption and download size. We did some memory profiling and saw that a long-running HTTPSE instance on Chrome takes up about 53mb, and the extension download for Firefox is 2.7mb, for Chrome 1.7. This will also get higher as more rulesets come rolling in. This probably won't be a huge issue unless we have some automated submission & inclusion system, but we do want to keep both the memory profile and download size low.

So in short: Too much domains also would not be good for HTTPSE.

Hmm, maybe HTTPS by default is coming nearer. 😃

Regarding scalability, one could combine it with an "alexa ranking test" so we secure not every private blog. Would keep memory low.

Mabye not entirely automated and with a small entry barrier to only go to the slightly more advanced people, one could do a default-off #2718 , no automated submission.
We wouldn't be overwhelmed with "hey check this" but people that are intrested and notice something could still help.
If a site really is broken and they care , my missing $ on heise.de for example, people will come over here.
I do not think we need an automated complain management ;)

More important is growing the test bench here i would say. Find Rules that can be purged or are broken
or some that are disabled and could be deleted or re-instated because the cert is valid again or so.
Maintenance is difficult already

@Hainish

Hey, sorry I haven't replied to this yet. I have a pretty good idea of how this should look.

Re: closing, let's punt on it for now. It's super easy to change later anyway. My guess is that if we add a note in the issue template explicitly saying, "don't change the formatting, this is read by a bot," people will do a better job. We can also improve the bot so it gets better at dealing with common formatting problems. I can also write in functionality where you can ping the bot to reparse an issue once formatting is fixed.

So, I think the next action item is to create a repo, right? @Hainish could you make that happen, and give me admin (or just write) access to it? Also, do we want the bot to have a separate repo? I vote yes; I think one of the biggest benefits to splitting out the rules is that you can clone the repo and get just the rules with no code. (Unless we wanted the bot to sit in this repo.)

@strugee

@strugee You can get started developing the bot code in a separate repository that you own. When it's ready or at least near completion, the code can be transferred to the EFF if they think that's appropriate.

For development you might want to automate creating a small issue repository with each issue as a single test case. Also, test mocks for the GitHub API seems to be a well-traveled path though I can't give personal recommendations. In any case running the bot against an EFF-owned repository should come at the end of the development, and the bot should probably be owned by either @Hainish , or @Hainish can create a new bot-only user and run the bot under that.

@jeremyn

I was going to respond basically with what @jeremyn said exactly. When it's ready to be handed off, we'll simply have to create a new GitHub API key, repo, and issue template for that repo.

@strugee one thing I've been doing recently for these standalone tools is putting them in the utils/ folder within HTTPS Everywhere. This keeps all the associated utilities for the project in one place, which I consider a bit neater. Also, I've been creating Dockerfiles for easy deployment. If you're familiar with docker and are so inclined, by all means follow suit. Otherwise I'll probably just dockerize it after you hand it off.

As far as deployment goes, I'll most likely just stand this up on the same server as we host the labeller.

@strugee

In any case running the bot against an EFF-owned repository should come at the end of the development

Oh yeah, that was always the plan. Wouldn't want to spam the tracker :)

@strugee one thing I've been doing recently for these standalone tools is putting them in the utils/ folder within HTTPS Everywhere. This keeps all the associated utilities for the project in one place, which I consider a bit neater.

Cool. So I'll develop this in utils/ in a branch and send a PR then?

Also, I've been creating Dockerfiles for easy deployment. If you're familiar with docker and are so inclined, by all means follow suit.

I'm not but I've been meaning to learn. So I might try my hand at writing a Dockerfile anyway.

@Hainish

Ping @Hainish

@strugee

@strugee yes, develop in utils/ and send a PR please

Just as an FYI, I'm still actively working on this :)

Taking a while because of a bunch of mocks and stuff that need to be written for the tests. But after all that's done it should be really easy to develop and test without setting up GitHub and stuff.

@strugee

Thanks @strugee

@jeremyn

I'm in favor of this approach, but @jeremyn had strong opinions against auto-closing so I'll let him argue the point.

I've already argued that position in this issue.

@koops76 In my suggested instructions in #11615 (comment), my plan is that reporters put the highest common domain in the Domain: field and specific subdomains in the free-form description part. An issue might look like this:

Domain: example.com

Specifically one.example.com and two.example.com.

If a casual reporter notices that one.example.com should be covered, it generally means that the entire example.com ruleset needs to be created or updated.

Navigation Menu

Repository navigation

Allow easier input of domains requested to be reviewed #6322

24 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Allow easier input of domains requested to be reviewed #6322

Description

Activity

jeremyn commented on Aug 15, 2016

rugk commented on Aug 15, 2016

rugk commented on Aug 15, 2016

rugk commented on Aug 15, 2016

Foorack commented on Aug 15, 2016

rugk commented on Aug 15, 2016

jeremyn commented on Aug 15, 2016

rugk commented on Aug 15, 2016

jeremyn commented on Aug 15, 2016

rugk commented on Aug 15, 2016

jeremyn commented on Aug 15, 2016

rugk commented on Aug 17, 2016

Hainish commented on Aug 17, 2016

rugk commented on Aug 18, 2016

numismatika commented on Aug 30, 2016

24 remaining items

strugee commented on May 29, 2017

jeremyn commented on May 30, 2017

Hainish commented on May 30, 2017

strugee commented on May 31, 2017

strugee commented on Jun 6, 2017

Hainish commented on Jun 8, 2017

strugee commented on Jul 17, 2017

Hainish commented on Jul 18, 2017

Hainish commented on Sep 19, 2017

jeremyn commented on Sep 19, 2017

jeremyn commented on Sep 21, 2017

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions