Following that article, I'm trying to understand how one can go from https://dxr.mozilla.org/mozilla-cent...ilterAdult.jsm to https://github.com/matthewruttley/co...ter/sites.json.

It seems base domains are first md5 encrypted and then base64 encoded but I cannot verify that with any provided example...

Let's take "bet365.com":
...but the last result string doesn't match anything in https://dxr.mozilla.org/mozilla-cent...ilterAdult.jsm!

I've certainly missed something because my last base64 string:

• ends with 1 "=" instead of 2,
• contains 45 characters instead of 25...

...but I don't know what/where.
Any hint please?
Thanks

They my well contain a salt? I'm certainly wishing I had an alternative to Firefox if they are really so patheic as to censor with a "naughty list" -- I didn't realise Firefox was programmeed by babies.

Have you considered that some Firefox users are parents? Parents take care of these little creatures called "children", who are incompetent, at least in law, to make decisions for themselves, thus parents have the option of a content filter. If you do not use it, it does not affect you.

Actually one of the required competencies for a parent is the ability to supervise their children and not rely on some third party. The filter should be unnecessary.

You want to supervise your child 24/7 like she's a kid in the goddamn Panopticon or something? Why does it matter to you? It's not your kid. Nobody's making you turn the filter on.

Personally, the first thing I do with Firefox is turn off ALL of that new tab stuff. I configure it to give me the search page for DDG.

So, I can't say if it's even possible to turn of the referenced filtering. Discussion of whether or not the filter is seems, to me, to be off-topic.

That said, l0f4r0, I don't think I understand your question, either.
Is bet365.com being filtered from the new tab?
Do you want it to be and it's not?

Please clarify.

Actually, it was my kids.

If you're not willing or able to supervise 24/7 you shouldn't consider being a parent.

Edit: "supervise" means watching for problems, not necessarily "forbidding".

Humm very clever, didn't think about that
It's a possibility indeed.

No no, actually my thread is not about the filter relevance (so people try to keep on topic please) nor content.
I'm just trying to verify technically by myself how plain text base domain entries have been encrypted&encoded so they appear that way in the filter source code.
bet365.com was just an example to go through the whole process (I hoped to fall on my feet but apparently not...)

It's not really a content filter, it just blocks sites from showing on the "new tab screen". You (or your children) can still visit as much as you want...


First note that md5sum has some extra output apart from the has itself:
We can get rid of that with tr, but the base64 result still doesn't look right:

The problem is that md5sum prints the hash in base16, so we're getting the base64 of the base16 ASCII encoding. We can use xxd to convert to binary:

Now we have something that looks like it should be in the list, but you won't find it! That's because the list is updated over time, but you can find it in the original: https://hg.mozilla.org/mozilla-centr...b096219c#l1.45

^ Awesome, very well done ntubski
I didn't realize that MD5 is hex/base16. For me it was just ASCII with [a-f0-9] range...
Question: why not encode base16 into base64 directly? Would there be any drawback?

You are responsible for your prodgeny and nobody else. It does affect me -- for example I had yo tell my ISP "I want to see porn'" just to get a slightly less filtered internet. What is wrong with you people? Parent your children properly! Stop making everyone else a victim of your inability to parent!

Not sure what you mean. If you meant the command pipeline I posted above, it's just that I'm not aware of any base16->base64 program. If you meant the Firefox code, then it already encodes directly to base64 (without going to base16 at all).

PS Can't you guys take your off-topic "Censorship!" vs "Think of the children!" arguments to another thread?

what a perfect way of showing a computer-literate kid where the naughty stuff is!
oh i see. probably makes sense, but way too much work to keep a list like that updated...
also it sort of implies that FF users go to "bad" sites and want to hide that from other users...
______________________
QFT!
definitely true for the first few years; it gets easier after that.

Not sure if it's a criticism towards me but, just in case, do you really think computer-literate kids need that kind of listing to locate naughty stuff nowadays? ^^

I was just referring to your following sentence:
That's why you suggested to convert base16 to binary via xxd and then to base64.
My question is: would there be any drawback to convert from base16 to base64 directly? I don't mean a program which would do that without explicitely going to temporary binary form, but I really mean a base64 of a base16 string. In other words I mean:
versus
+1 definitely. As I said previously this is not the goal here. This thread is only technical (md5/base64). Thank you for your understanding.

Yes, the drawback is that the former gives the wrong answer