LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > General
User Name
Password
General This forum is for non-technical general discussion which can include both Linux and non-Linux topics. Have fun!

Notices


Reply
  Search this Thread
Old 02-19-2019, 12:28 PM   #1
l0f4r0
Member
 
Registered: Jul 2018
Location: Paris
Distribution: macOS, Slackware
Posts: 803

Rep: Reputation: 281Reputation: 281Reputation: 281
Firefox adult/drug/gambling/alcohol site filter: md5 encryption/deciphering & base64 encoding/decoding


Following that article, I'm trying to understand how one can go from https://dxr.mozilla.org/mozilla-cent...ilterAdult.jsm to https://github.com/matthewruttley/co...ter/sites.json.

It seems base domains are first md5 encrypted and then base64 encoded but I cannot verify that with any provided example...

Let's take "bet365.com":
Code:
$ echo -n "bet365.com" | md5sum
aa90cd229c5a87c1548aa5e6e4845a52
$ echo -n "aa90cd229c5a87c1548aa5e6e4845a52" | base64
YWE5MGNkMjI5YzVhODdjMTU0OGFhNWU2ZTQ4NDVhNTI=
...but the last result string doesn't match anything in https://dxr.mozilla.org/mozilla-cent...ilterAdult.jsm!

I've certainly missed something because my last base64 string:
  • ends with 1 "=" instead of 2,
  • contains 45 characters instead of 25...

...but I don't know what/where.
Any hint please?
Thanks
 
Old 02-19-2019, 01:41 PM   #2
273
LQ Addict
 
Registered: Dec 2011
Location: UK
Distribution: Debian Sid AMD64, Raspbian Wheezy, various VMs
Posts: 7,161

Rep: Reputation: 2156Reputation: 2156Reputation: 2156Reputation: 2156Reputation: 2156Reputation: 2156Reputation: 2156Reputation: 2156Reputation: 2156Reputation: 2156Reputation: 2156
They my well contain a salt? I'm certainly wishing I had an alternative to Firefox if they are really so patheic as to censor with a "naughty list" -- I didn't realise Firefox was programmeed by babies.
 
Old 02-19-2019, 03:03 PM   #3
Woolie Wool
Member
 
Registered: Jul 2017
Location: Memphis, TN
Distribution: Arch
Posts: 39

Rep: Reputation: Disabled
Have you considered that some Firefox users are parents? Parents take care of these little creatures called "children", who are incompetent, at least in law, to make decisions for themselves, thus parents have the option of a content filter. If you do not use it, it does not affect you.

Last edited by Woolie Wool; 02-19-2019 at 03:08 PM.
 
Old 02-19-2019, 04:43 PM   #4
fido_dogstoyevsky
Member
 
Registered: Feb 2015
Location: Victoria, Australia
Distribution: Slackware 14.2
Posts: 293
Blog Entries: 2

Rep: Reputation: 302Reputation: 302Reputation: 302Reputation: 302
Quote:
Originally Posted by Woolie Wool View Post
Have you considered that some Firefox users are parents? Parents take care of these little creatures called "children", who are incompetent, at least in law, to make decisions for themselves, thus parents have the option of a content filter. If you do not use it, it does not affect you.
Actually one of the required competencies for a parent is the ability to supervise their children and not rely on some third party. The filter should be unnecessary.
 
Old 02-19-2019, 04:45 PM   #5
Woolie Wool
Member
 
Registered: Jul 2017
Location: Memphis, TN
Distribution: Arch
Posts: 39

Rep: Reputation: Disabled
You want to supervise your child 24/7 like she's a kid in the goddamn Panopticon or something? Why does it matter to you? It's not your kid. Nobody's making you turn the filter on.
 
Old 02-19-2019, 04:54 PM   #6
scasey
Senior Member
 
Registered: Feb 2013
Location: Tucson, AZ, USA
Distribution: CentOS 7.5
Posts: 2,579

Rep: Reputation: 878Reputation: 878Reputation: 878Reputation: 878Reputation: 878Reputation: 878Reputation: 878
Personally, the first thing I do with Firefox is turn off ALL of that new tab stuff. I configure it to give me the search page for DDG.

So, I can't say if it's even possible to turn of the referenced filtering. Discussion of whether or not the filter is seems, to me, to be off-topic.

That said, l0f4r0, I don't think I understand your question, either.
Is bet365.com being filtered from the new tab?
Do you want it to be and it's not?

Please clarify.
 
Old 02-19-2019, 04:59 PM   #7
fido_dogstoyevsky
Member
 
Registered: Feb 2015
Location: Victoria, Australia
Distribution: Slackware 14.2
Posts: 293
Blog Entries: 2

Rep: Reputation: 302Reputation: 302Reputation: 302Reputation: 302
Quote:
Originally Posted by Woolie Wool View Post
You want to supervise your child 24/7 like she's a kid in the goddamn Panopticon or something? Why does it matter to you? It's not your kid. Nobody's making you turn the filter on.
Actually, it was my kids.

If you're not willing or able to supervise 24/7 you shouldn't consider being a parent.

Edit: "supervise" means watching for problems, not necessarily "forbidding".

Last edited by fido_dogstoyevsky; 02-19-2019 at 05:03 PM.
 
Old 02-20-2019, 07:02 AM   #8
l0f4r0
Member
 
Registered: Jul 2018
Location: Paris
Distribution: macOS, Slackware
Posts: 803

Original Poster
Rep: Reputation: 281Reputation: 281Reputation: 281
Quote:
Originally Posted by 273 View Post
They my well contain a salt?
Humm very clever, didn't think about that
It's a possibility indeed.

Quote:
Originally Posted by scasey View Post
That said, l0f4r0, I don't think I understand your question, either.
Is bet365.com being filtered from the new tab?
Do you want it to be and it's not?
Please clarify.
No no, actually my thread is not about the filter relevance (so people try to keep on topic please) nor content.
I'm just trying to verify technically by myself how plain text base domain entries have been encrypted&encoded so they appear that way in the filter source code.
bet365.com was just an example to go through the whole process (I hoped to fall on my feet but apparently not...)
 
Old 02-20-2019, 07:07 AM   #9
ntubski
Senior Member
 
Registered: Nov 2005
Distribution: Debian, Arch
Posts: 3,413

Rep: Reputation: 1600Reputation: 1600Reputation: 1600Reputation: 1600Reputation: 1600Reputation: 1600Reputation: 1600Reputation: 1600Reputation: 1600Reputation: 1600Reputation: 1600
Quote:
Originally Posted by Woolie Wool View Post
Have you considered that some Firefox users are parents? Parents take care of these little creatures called "children", who are incompetent, at least in law, to make decisions for themselves, thus parents have the option of a content filter. If you do not use it, it does not affect you.
It's not really a content filter, it just blocks sites from showing on the "new tab screen". You (or your children) can still visit as much as you want...


Quote:
Originally Posted by l0f4r0 View Post
It seems base domains are first md5 encrypted and then base64 encoded but I cannot verify that with any provided example...

Let's take "bet365.com":
Code:
$ echo -n "bet365.com" | md5sum
aa90cd229c5a87c1548aa5e6e4845a52
$ echo -n "aa90cd229c5a87c1548aa5e6e4845a52" | base64
YWE5MGNkMjI5YzVhODdjMTU0OGFhNWU2ZTQ4NDVhNTI=
...but the last result string doesn't match anything in https://dxr.mozilla.org/mozilla-cent...ilterAdult.jsm!
First note that md5sum has some extra output apart from the has itself:
Code:
$ echo -n "bet365.com" | md5sum
aa90cd229c5a87c1548aa5e6e4845a52  -
We can get rid of that with tr, but the base64 result still doesn't look right:

Code:
$ echo -n "bet365.com" | md5sum | tr -dc 'a-f0-9' 
aa90cd229c5a87c1548aa5e6e4845a52$ echo -n "bet365.com" | md5sum | tr -dc 'a-f0-9' | base64
YWE5MGNkMjI5YzVhODdjMTU0OGFhNWU2ZTQ4NDVhNTI=
The problem is that md5sum prints the hash in base16, so we're getting the base64 of the base16 ASCII encoding. We can use xxd to convert to binary:

Code:
$ echo -n "bet365.com" | md5sum | tr -dc 'a-f0-9' | xxd -r -p | base64
qpDNIpxah8FUiqXm5IRaUg==
Now we have something that looks like it should be in the list, but you won't find it! That's because the list is updated over time, but you can find it in the original: https://hg.mozilla.org/mozilla-centr...b096219c#l1.45
 
Old 02-20-2019, 11:29 AM   #10
l0f4r0
Member
 
Registered: Jul 2018
Location: Paris
Distribution: macOS, Slackware
Posts: 803

Original Poster
Rep: Reputation: 281Reputation: 281Reputation: 281
^ Awesome, very well done ntubski
I didn't realize that MD5 is hex/base16. For me it was just ASCII with [a-f0-9] range...
Question: why not encode base16 into base64 directly? Would there be any drawback?
 
Old 02-20-2019, 01:37 PM   #11
273
LQ Addict
 
Registered: Dec 2011
Location: UK
Distribution: Debian Sid AMD64, Raspbian Wheezy, various VMs
Posts: 7,161

Rep: Reputation: 2156Reputation: 2156Reputation: 2156Reputation: 2156Reputation: 2156Reputation: 2156Reputation: 2156Reputation: 2156Reputation: 2156Reputation: 2156Reputation: 2156
Quote:
Originally Posted by Woolie Wool View Post
Have you considered that some Firefox users are parents? Parents take care of these little creatures called "children", who are incompetent, at least in law, to make decisions for themselves, thus parents have the option of a content filter. If you do not use it, it does not affect you.
You are responsible for your prodgeny and nobody else. It does affect me -- for example I had yo tell my ISP "I want to see porn'" just to get a slightly less filtered internet. What is wrong with you people? Parent your children properly! Stop making everyone else a victim of your inability to parent!
 
Old 02-20-2019, 04:16 PM   #12
ntubski
Senior Member
 
Registered: Nov 2005
Distribution: Debian, Arch
Posts: 3,413

Rep: Reputation: 1600Reputation: 1600Reputation: 1600Reputation: 1600Reputation: 1600Reputation: 1600Reputation: 1600Reputation: 1600Reputation: 1600Reputation: 1600Reputation: 1600
Quote:
Originally Posted by l0f4r0 View Post
Question: why not encode base16 into base64 directly? Would there be any drawback?
Not sure what you mean. If you meant the command pipeline I posted above, it's just that I'm not aware of any base16->base64 program. If you meant the Firefox code, then it already encodes directly to base64 (without going to base16 at all).

PS Can't you guys take your off-topic "Censorship!" vs "Think of the children!" arguments to another thread?
 
Old 02-21-2019, 05:12 AM   #13
ondoho
LQ Addict
 
Registered: Dec 2013
Posts: 11,062
Blog Entries: 7

Rep: Reputation: 2793Reputation: 2793Reputation: 2793Reputation: 2793Reputation: 2793Reputation: 2793Reputation: 2793Reputation: 2793Reputation: 2793Reputation: 2793Reputation: 2793
Quote:
Originally Posted by l0f4r0 View Post
what a perfect way of showing a computer-literate kid where the naughty stuff is!
Quote:
Originally Posted by ntubski View Post
It's not really a content filter, it just blocks sites from showing on the "new tab screen". You (or your children) can still visit as much as you want...
oh i see. probably makes sense, but way too much work to keep a list like that updated...
also it sort of implies that FF users go to "bad" sites and want to hide that from other users...
______________________
Quote:
Originally Posted by fido_dogstoyevsky View Post
Actually one of the required competencies for a parent is the ability to supervise their children and not rely on some third party.
Quote:
Originally Posted by fido_dogstoyevsky View Post
If you're not willing or able to supervise 24/7 you shouldn't consider being a parent.
Edit: "supervise" means watching for problems, not necessarily "forbidding".
QFT!
definitely true for the first few years; it gets easier after that.

Last edited by ondoho; 02-21-2019 at 05:13 AM.
 
Old 02-24-2019, 12:35 PM   #14
l0f4r0
Member
 
Registered: Jul 2018
Location: Paris
Distribution: macOS, Slackware
Posts: 803

Original Poster
Rep: Reputation: 281Reputation: 281Reputation: 281
Quote:
Originally Posted by ondoho View Post
what a perfect way of showing a computer-literate kid where the naughty stuff is!
Not sure if it's a criticism towards me but, just in case, do you really think computer-literate kids need that kind of listing to locate naughty stuff nowadays? ^^

Quote:
Originally Posted by ntubski View Post
Not sure what you mean. If you meant the command pipeline I posted above, it's just that I'm not aware of any base16->base64 program. If you meant the Firefox code, then it already encodes directly to base64 (without going to base16 at all).
I was just referring to your following sentence:
Quote:
The problem is that md5sum prints the hash in base16, so we're getting the base64 of the base16 ASCII encoding.
That's why you suggested to convert base16 to binary via xxd and then to base64.
My question is: would there be any drawback to convert from base16 to base64 directly? I don't mean a program which would do that without explicitely going to temporary binary form, but I really mean a base64 of a base16 string. In other words I mean:
Code:
echo -n "bet365.com" | md5sum | tr -dc 'a-f0-9' | base64
versus
Code:
echo -n "bet365.com" | md5sum | tr -dc 'a-f0-9' | xxd -r -p | base64
Quote:
Originally Posted by ntubski View Post
PS Can't you guys take your off-topic "Censorship!" vs "Think of the children!" arguments to another thread?
+1 definitely. As I said previously this is not the goal here. This thread is only technical (md5/base64). Thank you for your understanding.
 
Old 02-24-2019, 08:49 PM   #15
ntubski
Senior Member
 
Registered: Nov 2005
Distribution: Debian, Arch
Posts: 3,413

Rep: Reputation: 1600Reputation: 1600Reputation: 1600Reputation: 1600Reputation: 1600Reputation: 1600Reputation: 1600Reputation: 1600Reputation: 1600Reputation: 1600Reputation: 1600
Quote:
Originally Posted by l0f4r0 View Post
My question is: would there be any drawback to convert from base16 to base64 directly? I don't mean a program which would do that without explicitely going to temporary binary form, but I really mean a base64 of a base16 string. In other words I mean:
Code:
echo -n "bet365.com" | md5sum | tr -dc 'a-f0-9' | base64
versus
Code:
echo -n "bet365.com" | md5sum | tr -dc 'a-f0-9' | xxd -r -p | base64
Yes, the drawback is that the former gives the wrong answer
 
  


Reply

Tags
base64, firefox, md5


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] base64 encoding in program vs piping to base64 command-line utility 0x53h Linux - General 3 08-05-2016 10:39 AM
Help deciphering/decoding/debugging abrt report scarf Linux - Security 1 06-12-2016 09:04 PM
Decoding Base64 encoded PHP scripts on Linux Server charu Linux - Newbie 2 01-18-2011 11:57 AM
Howto convert base64 to md5 celeron Linux - Security 3 03-01-2007 02:19 AM
Recommended XML / Base64 encoding libraries? overbored Programming 2 10-14-2004 11:21 AM

LinuxQuestions.org > Forums > Non-*NIX Forums > General

All times are GMT -5. The time now is 11:23 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration