Polipo cache to disk, cannot activate disk caching, different browsers redownload con

Trauma · 11-23-2013, 12:11 PM

Hi all.

I've just installed Polipo via Yum; running Fedora 19-64 on an AMD 8g ram SSD system.

I have Polipo up and running with browsers set to use it.

Only problem is, it doesn't use much ram and it won't write files to disk. So, I don't think it is working properly.

Here is my /etc/polipo/config file:

Quote:

[root@localhost ~]# cat /etc/polipo/config
# Sample configuration file for Polipo. -*-sh-*-

# You should not need to use a configuration file; all configuration
# variables have reasonable defaults. If you want to use one, you
# can copy this to /etc/polipo/config or to ~/.polipo and modify.

# This file only contains some of the configuration variables; see the
# list given by ``polipo -v'' and the manual for more.

### Configuration from Fedora RPM
### *****************************
daemonise = true
pidFile = /var/run/polipo/polipo.pid
logFile = /var/log/polipo/polipo.log

### Basic configuration
### *******************

# Uncomment one of these if you want to allow remote clients to
# connect:

# proxyAddress = "::0" # both IPv4 and IPv6
# proxyAddress = "0.0.0.0" # IPv4 only

# If you do that, you'll want to restrict the set of hosts allowed to
# connect:

allowedClients = 127.0.0.1, 192.168.0.69
allowedPorts = 1-65535
# allowedClients = "127.0.0.1, 134.157.168.57"
# allowedClients = "127.0.0.1, 134.157.168.0/24"

# Uncomment this if you want your Polipo to identify itself by
# something else than the host name:

proxyName = "localhost"

# Uncomment this if there's only one user using this instance of Polipo:

cacheIsShared = false

# Uncomment this if you want to use a parent proxy:

# parentProxy = "squid.example.org:3128"

# Uncomment this if you want to use a parent SOCKS proxy:

# socksParentProxy = "localhost:9050"
# socksProxyType = socks5

### Memory
### ******

# Uncomment this if you want Polipo to use a ridiculously small amount
# of memory (a hundred C-64 worth or so):

# chunkHighMark = 819200
# objectHighMark = 128

# Uncomment this if you've got plenty of memory:

chunkHighMark = 267108864
objectHighMark = 16384

### On-disk data
### ************

# Uncomment this if you want to disable the on-disk cache:

diskCacheRoot = /var/cache/polipo

# Uncomment this if you want to put the on-disk cache in a
# non-standard location:

diskCacheRoot = "/home/spinifex/.polipo-cache"

# Uncomment this if you want to disable the local web server:

# localDocumentRoot = ""

# Uncomment this if you want to enable the pages under /polipo/index?
# and /polipo/servers?. This is a serious privacy leak if your proxy
# is shared.

disableIndexing = false
disableServersList = false

### Domain Name System
### ******************

# Uncomment this if you want to contact IPv4 hosts only (and make DNS
# queries somewhat faster):

dnsQueryIPv6 = no

# Uncomment this if you want Polipo to prefer IPv4 to IPv6 for
# double-stack hosts:

# dnsQueryIPv6 = reluctantly

# Uncomment this to disable Polipo's DNS resolver and use the system's
# default resolver instead. If you do that, Polipo will freeze during
# every DNS query:

# dnsUseGethostbyname = yes

### HTTP
### ****

# Uncomment this if you want to enable detection of proxy loops.
# This will cause your hostname (or whatever you put into proxyName
# above) to be included in every request:

# disableVia=false

# Uncomment this if you want to slightly reduce the amount of
# information that you leak about yourself:

censoredHeaders = from, accept-language
censorReferer = maybe

# Uncomment this if you're paranoid. This will break a lot of sites,
# though:

# censoredHeaders = set-cookie, cookie, cookie2, from, accept-language
# censorReferer = true

# Uncomment this if you want to use Poor Man's Multiplexing; increase
# the sizes if you're on a fast line. They should each amount to a few
# seconds' worth of transfer; if pmmSize is small, you'll want
# pmmFirstSize to be larger.

# Note that PMM is somewhat unreliable.

pmmFirstSize = 16384
pmmSize = 8192

# Uncomment this if your user-agent does something reasonable with
# Warning headers (most don't):

# relaxTransparency = maybe

# Uncomment this if you never want to revalidate instances for which
# data is available (this is not a good idea):

# relaxTransparency = yes

# Uncomment this if you have no network:

# proxyOffline = yes

# Uncomment this if you want to avoid revalidating instances with a
# Vary header (this is not a good idea):

# mindlesslyCacheVary = true
[root@localhost ~]#

It seems long winded but I think I've tried just about every variable I can in order to get Polipo to cache to disk.

When I access http://localhost:8123/polipo/index? (the Polipo interface via web browser) and see if Disk Caching is working I get

Quote:

Index

No diskCacheRoot.

back

So, it isn't caching do disk. That means that every web browser that accesses the web via Polipo will redownload content.

For what it's worth I use primary browsers Opera and Firefox and the both access the same webpages at different times (I know it sounds crazy but my wife uses one browser and won't use the other, and ditto likewise for me). Those browsers will redownload the same content 3, 4, sometimes 5 times a day (sometimes 10-20 tabs each) -(don't ask why, that is just the way it is) and, as you can guess, this is a very slow, tedious process. Hence the reason for installing Polipo in the first place.

Only problem is, Polipo won't cache. So basically it isn't doing its job.

I've read most of the documentation and "Googled" 'polipo enable disk cache' polipo "diskcacheroot"' but that hasn't proved fruitful. Am I missing something? I must be but I can't figure out what.

Suggestions appreciated I've been looking at this for two days and am not getting anywhere.

I've a post over at FedoraForums.org which includes this issue and whilst there have been some kind and helpful suggestions, I'm not getting much joy from there.

Thank you.

unSpawn · 11-25-2013, 01:16 PM

As per the other thread (dunno if Polipo would b0rk over it) but you defined "diskCacheRoot" twice too? So pick one. What does the polipo.log file say? Is /var/cache/polipo owned and writable by the user Polipo runs as?

Trauma · 11-25-2013, 01:30 PM

Hi UnSpawn.

Yes. I had the cache directory specified twice; as a matter of irrational desperation.

This is what I am currently using:

Quote:

[root@localhost /]# cat /etc/polipo/config
# Sample configuration file for Polipo. -*-sh-*-

# You should not need to use a configuration file; all configuration
# variables have reasonable defaults. If you want to use one, you
# can copy this to /etc/polipo/config or to ~/.polipo and modify.

# This file only contains some of the configuration variables; see the
# list given by ``polipo -v'' and the manual for more.

### Configuration from Fedora RPM
### *****************************
daemonise = true
pidFile = /var/run/polipo/polipo.pid
logFile = /var/log/polipo/polipo.log

### Basic configuration
### *******************

# Uncomment one of these if you want to allow remote clients to
# connect:

# proxyAddress = "::0" # both IPv4 and IPv6
# proxyAddress = "0.0.0.0" # IPv4 only

# If you do that, you'll want to restrict the set of hosts allowed to
# connect:

allowedClients = 127.0.0.1, 192.168.0.69
allowedPorts = 1-65535
# allowedClients = "127.0.0.1, 134.157.168.57"
# allowedClients = "127.0.0.1, 134.157.168.0/24"

# Uncomment this if you want your Polipo to identify itself by
# something else than the host name:

proxyName = "localhost"

# Uncomment this if there's only one user using this instance of Polipo:

cacheIsShared = false

# Uncomment this if you want to use a parent proxy:

# parentProxy = "squid.example.org:3128"

# Uncomment this if you want to use a parent SOCKS proxy:

# socksParentProxy = "localhost:9050"
# socksProxyType = socks5

### Memory
### ******

# Uncomment this if you want Polipo to use a ridiculously small amount
# of memory (a hundred C-64 worth or so):

# chunkHighMark = 819200
# objectHighMark = 128

# Uncomment this if you've got plenty of memory:

chunkHighMark = 267108864
objectHighMark = 16384

### On-disk data
### ************

# Uncomment this if you want to disable the on-disk cache:

# diskCacheRoot = ""

# Uncomment this if you want to put the on-disk cache in a
# non-standard location:

diskCacheRoot = "/home/spinifex/.polipo-cache"

# Uncomment this if you want to disable the local web server:

# localDocumentRoot = ""

# Uncomment this if you want to enable the pages under /polipo/index?
# and /polipo/servers?. This is a serious privacy leak if your proxy
# is shared.

disableIndexing = false
disableServersList = false

### Domain Name System
### ******************

# Uncomment this if you want to contact IPv4 hosts only (and make DNS
# queries somewhat faster):

dnsQueryIPv6 = no

# Uncomment this if you want Polipo to prefer IPv4 to IPv6 for
# double-stack hosts:

# dnsQueryIPv6 = reluctantly

# Uncomment this to disable Polipo's DNS resolver and use the system's
# default resolver instead. If you do that, Polipo will freeze during
# every DNS query:

# dnsUseGethostbyname = yes

### HTTP
### ****

# Uncomment this if you want to enable detection of proxy loops.
# This will cause your hostname (or whatever you put into proxyName
# above) to be included in every request:

# disableVia=false

# Uncomment this if you want to slightly reduce the amount of
# information that you leak about yourself:

censoredHeaders = from, accept-language
censorReferer = maybe

# Uncomment this if you're paranoid. This will break a lot of sites,
# though:

# censoredHeaders = set-cookie, cookie, cookie2, from, accept-language
# censorReferer = true

# Uncomment this if you want to use Poor Man's Multiplexing; increase
# the sizes if you're on a fast line. They should each amount to a few
# seconds' worth of transfer; if pmmSize is small, you'll want
# pmmFirstSize to be larger.

# Note that PMM is somewhat unreliable.

pmmFirstSize = 16384
pmmSize = 8192

# Uncomment this if your user-agent does something reasonable with
# Warning headers (most don't):

# relaxTransparency = maybe

# Uncomment this if you never want to revalidate instances for which
# data is available (this is not a good idea):

# relaxTransparency = yes

# Uncomment this if you have no network:

# proxyOffline = yes

# Uncomment this if you want to avoid revalidating instances with a
# Vary header (this is not a good idea):

# mindlesslyCacheVary = true

unSpawn · 11-25-2013, 04:38 PM

Quote:

Originally Posted by Trauma

This is what I am currently using:

Code:

diskCacheRoot = "/home/spinifex/.polipo-cache"

This implies Polipo runs as user "spinifex" or else it won't have write access to the directory. Usually Polipo will run as user "polipo" and commonly the cache directory will be /var/cache/polipo with read, write and execute rights for user polipo and read and execute rights for group polipo. *BTW do note I asked three questions.

Trauma · 11-28-2013, 01:46 PM

Quote:

Originally Posted by unSpawn

This implies Polipo runs as user "spinifex" or else it won't have write access to the directory. Usually Polipo will run as user "polipo" and commonly the cache directory will be /var/cache/polipo with read, write and execute rights for user polipo and read and execute rights for group polipo. *BTW do note I asked three questions.

Hi. Sorry for taking so long. I appreciate your replies.

I checked /var/log/polipo/polipo.log and your appear to be right. I think I've got screwy permissions all over the place. You are right about the polipo-cache directory I made up in /home/spinifex/ That is my home folder. I thought if I placed it there then I'd be able to view the cached content Polipo is using to serve to the different web browsers I use. It was a mistake obviously, unless I go and change file permissions but I think I screwed that up too. I need some help with file permissions please. I'm a bit out of my depth.

The /var/log/polipo/polipo.log relevant sections are:

Quote:

... CUT ...

Unsupported Cache-Control directive post-check -- ignored.
Unsupported Cache-Control directive pre-check -- ignored.
Unsupported Cache-Control directive post-check -- ignored.
Unsupported Cache-Control directive pre-check -- ignored.
Unsupported Cache-Control directive post-check -- ignored.
Unsupported Cache-Control directive pre-check -- ignored.
Unsupported Cache-Control directive cache -- ignored.
Unsupported Cache-Control directive store -- ignored.
Unsupported Cache-Control directive post-check -- ignored.
Unsupported Cache-Control directive pre-check -- ignored.
Unsupported Cache-Control directive post-check -- ignored.
Unsupported Cache-Control directive pre-check -- ignored.
Unsupported Cache-Control directive post-check -- ignored.
Unsupported Cache-Control directive pre-check -- ignored.
Unsupported Cache-Control directive post-check -- ignored.
Unsupported Cache-Control directive pre-check -- ignored.
Unsupported Cache-Control directive post-check -- ignored.
Unsupported Cache-Control directive pre-check -- ignored.
Unsupported Cache-Control directive post-check -- ignored.
Unsupported Cache-Control directive pre-check -- ignored.
Unsupported Cache-Control directive post-check -- ignored.

... CUT (hundreds of the above) ...

Disabling disk cache: Permission denied
... cut ...
Couldn't create pid file /var/run/polipo/polipo.pid: File exists
Disabling disk cache: Permission denied
Couldn't create pid file /var/run/polipo/polipo.pid: File exists
Disabling disk cache: Permission denied
Couldn't create pid file /var/run/polipo/polipo.pid: File exists
Disabling disk cache: Permission denied
Established listening socket on port 8123.
Disabling disk cache: Permission denied
Established listening socket on port 8123.
Disabling disk cache: Permission denied
Established listening socket on port 8123.
Disabling disk cache: Permission denied
Established listening socket on port 8123.
Disabling disk cache: Permission denied
Established listening socket on port 8123.
Disabling disk cache: Permission denied
Established listening socket on port 8123.
Disabling disk cache: Permission denied
Established listening socket on port 8123.
Disabling disk cache: Permission denied
Established listening socket on port 8123.
Disabling disk cache: Permission denied
Established listening socket on port 8123.
Disabling disk cache: Permission denied
Established listening socket on port 8123.
Disabling disk cache: Permission denied
Couldn't create pid file /var/run/polipo/polipo.pid: File exists
Couldn't create pid file /var/run/polipo/polipo.pid: File exists
Couldn't create pid file /var/run/polipo/polipo.pid: File exists
Couldn't create pid file /var/run/polipo/polipo.pid: File exists
Couldn't create pid file /var/run/polipo/polipo.pid: File exists
Couldn't create pid file /var/run/polipo/polipo.pid: File exists
Couldn't create pid file /var/run/polipo/polipo.pid: File exists
Couldn't create pid file /var/run/polipo/polipo.pid: File exists
Established listening socket on port 8123.
Established listening socket on port 8123.
Established listening socket on port 8123.
Established listening socket on port 8123.

In fact I have hundreds of lines referring to Permission denied.

I made up polipo as a user in /home/polipo-cache but I don't believe this is the right way to go about things either (comments please)

Here is the relevant /etc/polipo/config

Quote:

### On-disk data
### ************

# Uncomment this if you want to disable the on-disk cache:

# diskCacheRoot = ""

# Uncomment this if you want to put the on-disk cache in a
# non-standard location:

diskCacheRoot = "/home/polipo-cache"

# Uncomment this if you want to disable the local web server:

# localDocumentRoot = ""

The reason for all the stuffing around with the cache directory in the first place is disk space. I've allocated 20 gig to /root (which includes /var/cache but I've got a 100 gig in my /home (heaps of space) on a SSD. Hence wanting to cache everything to help speed things up. If I leave polipo-cache in /var then I'll run out of space pretty quickly. I fully intend to let polipo cache 20, 30 or even 40 gig of web content before cleaning it out. Hence wanting to move the cache to a /home directory so I can browse it or see what the others in my household happen to be looking at. Also, we download ISO's by the dozen and I've found 4 of the same thing dotted all over the place. Polipo will help me mitigate some of that.

How can I move the relevant polipo-cache directory to a read write execute location in my home folder? (Either in /home/spinifex/ or /home/polipo-cache - or anywhere else that will enable polipo to cache properly? )

Thank you - I really appreciate this.

EDIT: FYI

Quote:

[root@localhost ~]# service polipo status
Redirecting to /bin/systemctl status polipo.service
polipo.service - A caching web proxy
Loaded: loaded (/etc/systemd/system/polipo.service; enabled)
Active: active (running) since Fri 2013-11-29 02:30:07 EST; 2h 45min ago
Docs: man

olipo(1)
http://localhost:8123/
Process: 2776 ExecStart=/usr/bin/polipo -c $CONFIG (code=exited, status=0/SUCCESS)
Main PID: 2778 (polipo)
CGroup: name=systemd:/system/polipo.service
└─2778 /usr/bin/polipo -c /etc/polipo/config

Nov 29 02:30:07 localhost.localdomain systemd[1]: Starting A caching web proxy...
Nov 29 02:30:07 localhost.localdomain systemd[1]: Started A caching web proxy.

So it appears to be running correctly bar the caching.

unSpawn · 12-02-2013, 01:35 PM

Quote:

Originally Posted by Trauma

Hi. Sorry for taking so long.

Me too :-]

Quote:

Originally Posted by Trauma

Code:

Couldn't create pid file /var/run/polipo/polipo.pid: File exists

Stop Polipo, run 'pgrep polipo' to see if an instance still runs and "kill -9" it if necessary then remove the PID file if it still exists.

Quote:

Originally Posted by Trauma

I made up polipo as a user in /home/polipo-cache but I don't believe this is the right way to go about things either

Polipo should be a system user with an UID greater than 100 and less than 500 (or 1000 on newer systems) with (in your case) home /home/polipo-cache and an inert shell like /sbin/nologin. Check with 'getent passwd polipo'. /home/polipo-cache should be owned by both user and group "polipo" and its context should be something like "polipo_cache_home_t" (not "polipo_cache_t": see 'man selinux_polipo' "File Contexts") and run 'ls -aldZ /home/polipo-cache' to check.

Trauma · 12-04-2013, 01:57 PM

Hi. Again, sorry for taking so long. I've got something running.

But, have been trying to figure out all this Selinux stuff and the folder permissions.

Polipo is caching and I see files when I access the directory http://localhost:8123/polipo/index. (Although subjective website browsing speed improvements seem to be minimal).

Re your instructions in your most recent post:

ls -aldZ /home/polipo-cache returns: (as I believe you said)

Quote:

[root@localhost ~]# ls -aldZ /home/polipo-cache
drwx------. polipo polipo unconfined_u

bject_r:user_home_dir_t:s0 /home/polipo-cache

getent passwd polipo returns:

Quote:

[root@localhost ~]# getent passwd polipo
polipo:x:987:984:Polipo Web Proxy:/var/cache/polipo:/sbin/nologin

I know this isn't exactly right. Selinux throws up a message:

Quote:

SELinux is preventing /usr/bin/polipo from getattr access on the directory /home/polipo-cache.

***** Plugin restorecon (99.5 confidence) suggests *************************

If you want to fix the label.
/home/polipo-cache default label should be user_home_dir_t.
Then you can run restorecon.
Do
# /sbin/restorecon -v /home/polipo-cache

***** Plugin catchall (1.49 confidence) suggests ***************************

If you believe that polipo should be allowed getattr access on the polipo-cache directory by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
# grep polipo /var/log/audit/audit.log | audit2allow -M mypol
# semodule -i mypol.pp

Additional Information:
Source Context system_u:system_r

olipo_t:s0
Target Context unconfined_u

bject_r:home_root_t:s0
Target Objects /home/polipo-cache [ dir ]
Source polipo
Source Path /usr/bin/polipo
Port <Unknown>
Host localhost.localdomain
Source RPM Packages polipo-1.0.4.1-10.fc19.x86_64
Target RPM Packages
Policy RPM selinux-policy-3.12.1-74.14.fc19.noarch
Selinux Enabled True
Policy Type targeted
Enforcing Mode Permissive
Host Name localhost.localdomain
Platform Linux localhost.localdomain 3.11.9-200.fc19.x86_64
#1 SMP Wed Nov 20 21:22:24 UTC 2013 x86_64 x86_64
Alert Count 2
First Seen 2013-12-05 02:59:53 EST
Last Seen 2013-12-05 02:59:59 EST
Local ID 7b9eae18-2f3a-4daa-8e10-a15053a3a931

Raw Audit Messages
type=AVC msg=audit(1386176399.41:492): avc: denied { getattr } for pid=2770 comm="polipo" path="/home/polipo-cache" dev="sdb3" ino=4849665 scontext=system_u:system_r

olipo_t:s0 tcontext=unconfined_u

bject_r:home_root_t:s0 tclass=dir

type=SYSCALL msg=audit(1386176399.41:492): arch=x86_64 syscall=stat success=yes exit=0 a0=8199c2 a1=7fffcf37bac0 a2=7fffcf37bac0 a3=24c9 items=0 ppid=1 pid=2770 auid=4294967295 uid=987 gid=984 euid=987 suid=987 fsuid=987 egid=984 sgid=984 fsgid=984 ses=4294967295 tty=(none) comm=polipo exe=/usr/bin/polipo subj=system_u:system_r

olipo_t:s0 key=(null)

Hash: polipo,polipo_t,home_root_t,dir,getattr

And to be honest, the selinux stuff is WAY over my head. I'm really not sure where to go with that.

I ran

Quote:

/sbin/restorecon -v /home/polipo-cache

as per the first few lines of the Selinux do-dad, and it appeared to complete without throwing up any warnings.

I then attempted to run

Quote:

grep polipo /var/log/audit/audit.log | audit2allow -M mypol

but it returned:

Quote:

[root@localhost ~]# grep polipo /var/log/audit/audit.log | audit2allow -M mypol
compilation failed:
sh: /usr/bin/checkmodule: No such file or directory
[root@localhost ~]#

and also ran

Quote:

semodule -i mypol.pp

but it returned:

Quote:

[root@localhost ~]# semodule -i mypol.pp
semodule: Failed on mypol.pp!
[root@localhost ~]#

Suggestions? I've sort of hit a wall with this. I just want to get Polipo working as it should and I believe the lack of subjective improvements I've noticed in internet browsing are related to Selinux the the permissions you talked about.

Where do I go from here?

Thank you

unSpawn · 12-06-2013, 01:38 AM

Try 'chcon -R -t polipo_cache_home_t /home/polipo-cache'.

Trauma · 12-07-2013, 11:13 AM

Hi.

Tried:

Quote:

Originally Posted by unSpawn

Try 'chcon -R -t polipo_cache_home_t /home/polipo-cache'.

And that completed without throwing up command prompt printout.

Then ran:

Quote:

[root@localhost ~]# ls -aldZ /home/polipo-cache
drwx------. polipo polipo unconfined_u

bject_r

olipo_cache_home_t:s0 /home/polipo-cache

Then ran:

[root@localhost ~]# semodule -i mypol.pp
semodule: Failed on mypol.pp!

Selinux is still throwing up warnings whenever I start a browser after reboot.

I'm not sure where this is going but when I look for mypol.te it is located in /root/

Quote:

[root@localhost /]# locate mypol
/root/mypol.te
[root@localhost /]# cd /root
[root@localhost ~]# ls
anaconda-ks.cfg el_dlurls.txt home:satya164:fedorautils.repo testdisk.log
easylist.txt fedora-chromium-stable.repo mypol.te
[root@localhost ~]# cat mypol.te

module mypol 1.0;

require {
type home_root_t;
type polipo_t;
type user_home_dir_t;
class dir { write remove_name search };
class file { read write getattr unlink open };
}

#============= polipo_t ==============
allow polipo_t home_root_t:dir { write remove_name search };
allow polipo_t home_root_t:file { read write getattr unlink open };
allow polipo_t user_home_dir_t:dir search;
[root@localhost ~]#

I'm not sure what it means but something doesn't feel right.