Remove Disconnect references from Shavar services

[say-yawn]

About this Issue

After merging the changes from #110 I saw that the staging S3 showed that there were files updated when we expected to updates:

Diffs of the logs before and after merging #110 shows that the order of the domains have changed causing the list to be "changed" (look at the diff between the .log files attached).
ads-track-digest256-BEFORE.log
ads-track-digest256.log
We should wait to apply #110 to versioned branches so that the domains are added alphabetically to prevent not-real update like this one. We should also remove all the Disconnect category references in the services and configs.

Acceptance Criteria

• We should wait to apply removes Disconnect category and remaps to appropriate categories #110 to versioned branches so that we deploy meaningful updates and not a simple re-ordering of the items in the list. Issue filed Remove Disconnect category in all the versioned branches #112
• The domains added in the list should be alphanumerical to prevent not-real update like this one. Issue filed Add domains in alphanumeric order shavar-list-creation#132
• We should also remove all the Disconnect category references in the services and configs. See Remove Disconnect and Google mapping references shavar-list-creation#118 for more details

[boolean5]

Hi @skim1102, in continuation of mozilla-services/shavar-list-creation#121, I was thinking about sending a PR that removes references to the Disconnect category and to disconnect_mapping.json from the shavar-prod-lists repository. This would address the 3rd A/C of this issue. Should I go ahead or have you started working on this already?

[say-yawn]

@boolean5 I went ahead and added the issue from shavar-list-creation repo in the description of the issue. Let's wait to get all the acceptance criteria fulfilled before we check the third acceptance criteria. Alternatively, the second criteria:

The domains added in the list should be alphabetical to prevent not-real update like this one

has not been worked on yet and would be a good extension for the work you have been doing to remove Disconnect category and JSON file references. Would you like to pick it up as an extended work you have been doing?

[boolean5]

Yes, working on the 2nd A/C sounds good. Thanks for suggesting this. I'll get back to you if I have any questions.

[boolean5]

@skim1102, some more context on this would be very helpful.

It is not clear to me how the application of #110 caused the order of domains to change. Could you please explain this a little?

Was there a domain that was added out of alphabetical order in #110? Does it have to do with the way the Disconnect list parser parses disconnect-blacklist.json?

[say-yawn]

@boolean5 certainly! I apologize for the delayed response.

It is not clear to me how the application of #110 caused the order of domains to change. Could you please explain this a little?

Yes, as you have mentioned, the Disconnect list parser uses .add as seen here which does not impose a specific order (since Set is an unordered collection). However, when the remapping of the Disconnect category domains into the respective categories happened here, you can see that the companies and their domains were added in alphanumerical order. This resulted in the list being "changed" even though the contents/domains in the list were not changed. While we can't enforce the ordering in trackingprotection-tools package, we can do so in the shavar-list-creation repo and prevent a list from being "updated" due to changes in the order of the domains written to the file/list.

[say-yawn]

Also, if you intend to keep working on this issue, please assign yourself to mozilla-services/shavar-list-creation#132. Thank you!

[boolean5]

Thanks for explaining this!

I can't self-assign mozilla-services/shavar-list-creation#132 because I don't have write permissions for that repository. I still intend to work on it though, so feel free to assign it to me.