Website monitoring through cronjob

gloeckle · 04-24-2019, 02:50 AM

Hello guys,

i am trying to figure out, how to monitor a website through a cronjob with wget or curl.

What i want:
Let my LMint (VM @ home) automatically open or get something on a selfsigned website (company network) to simulate best the out of company network website visit.

what i have:
- URL (of course)
- selfsigned user Certificate (.p12) allready converted with openssl into *crt.pem & *key.pem

what i tried so far with the allmighty google:
1.

Code:

curl -E **.cert.pem --key **.key.pem https://companysite.asdf/index.php

2.

Code:

wget --ca-cert=/etc/ssl/certs/winhostname.pem --certificate=/etc/ssl/private/linuxhost.pem --private-key=/etc/ssl/private/linuxhost.key https://winhostname.home.net:8443/winhosturl.asmx

2. seemed to work for someone who searched and tried a lot: link
But do i need a server cert aswell? The "--ca-cert=/etc/ssl/certs/winhostname.pem" what is this third certificate?

In the end it does not need to be wget or curl. It just has to be timestamped results if reachable or not, which i can make a chart out of. pinging the webserver does not help since it is running but that horrible tool, reached through this site, is what i want to monitor.

any help would be much appreciated!

berndbausch · 04-24-2019, 03:35 AM

Quote:

Originally Posted by gloeckle

The "--ca-cert=/etc/ssl/certs/winhostname.pem" what is this third certificate?

It’s the certificate that identifies the Certificate Authority that issues the web site’s certificate. To obtain an RHCE certificate (not to be confused with a CA or RHCA certificate) you must be able to say this sentence three times in a row without stuttering.

My totally untrustful explanation: A certificate is like an ID card or passport. It is issued by some authority, and you also need a copy of this authority’s passport for things to work. Certificates of well-known CA’s like Comodo or Thawte are deployed when you install Mint, but wget or curl don’t know your company’s CA. Hence the need to explicitly provide the CA’s certificate on the command line.

gloeckle · 04-24-2019, 08:04 AM

Quote:

Originally Posted by berndbausch

... you must be able to say this sentence three times in a row without stuttering.

made my day - thanks for that

well we get the client cert sent. import it into our browser and only then we can open that website. Due to my understanding there is no authority?!

thinknix · 04-24-2019, 08:20 AM

If you are just interested in whether or not the site is responding at an application level, use curl -k or wget --no-check-certificate, Either will ignore the self-signed cert and connect anyway.

berndbausch · 04-24-2019, 09:30 AM

Quote:

Originally Posted by gloeckle

Due to my understanding there is no authority?!

Ni dieu ni roi ni maître. I’ll think of it in the morning.

gloeckle · 04-24-2019, 02:33 PM

Quote:

Originally Posted by thinknix

If you are just interested in whether or not the site is responding at an application level, use curl -k

could not resolve host

Quote:

Originally Posted by thinknix

or wget --no-check-certificate, Either will ignore the self-signed cert and connect anyway.

failed: name or service not known

can i open it through browser with imported cert? yes

=/

thinknix · 04-24-2019, 02:44 PM

Quote:

Originally Posted by gloeckle

could not resolve host

failed: name or service not known

can i open it through browser with imported cert? yes

=/

You still need whatever other arguments you want to supply, including the target host. I meant that you just add -k or --no-check-certificate to the existing command (whatever one you are using) in your cron.

gloeckle · 04-24-2019, 02:49 PM

Quote:

Originally Posted by thinknix

You still need whatever other arguments you want to supply, including the target host. I meant that you just add -k or --no-check-certificate to the existing command (whatever one you are using) in your cron.

i did:

Code:

wget --no-check-certificate https://subdomain.domain.tld/index.php

Code:

curl -k https://subdomain.domain.tld/index.php

do you mean anything different?

thinknix · 04-24-2019, 03:23 PM

Yes, correct that is what I meant. I'm not sure why you would get resolver errors then. Since DNS resolution works for you in a browser, what is left is mistyping the command or hostname.

Try something like this from a shell prompt just to see if it works.

Code:

curl -I -k https://subdomain.domain.tld/index.php

The -I just limits the response to showing the HTTP headers, so your screen won't fill with web content.

gloeckle · 04-24-2019, 04:22 PM

Code:

curl -I -k https://subdomain.domain.tld/asdf/fdsa/index.php
curl: (6) Could not resolve host: subdomain.domain.tld

then i thought i go for a ping... shortened the url ...no ping ... wait!

i take it back! I TAKE IT BACK! i had a typo!

oh dear... i am realy sorry but it is a stupid stupid looong url!

Code:

curl -I -k https://subdomain.domain.tld/asdf/fdsa/index.php
curl: (35) error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure

then i read those howtos again and voilà:

Code:

curl -k -E ***.crt.pem --key ***.key.pem https://subdomain.domain.tld/asdf/fdsa/index.php
<!DOCTYPE html PUBLIC .............

now i need to figure out how to script me some automated results in a cronjob

maybe i get back to you guys soon.

thanks for your effort and patience! especially thinknix! cheers!

gloeckle · 04-24-2019, 04:42 PM

if anyone comes across this and needs it too, this is my command for now:

Code:

curl -IkE ***.crt.pem --key ***.key.pem https://subdomain.domain.tld/asdf/fdsa/index.php >"$(date +"%F_%H%M%S").txt"

berndbausch · 04-24-2019, 09:07 PM

Quote:

Originally Posted by gloeckle

well we get the client cert sent. import it into our browser and only then we can open that website. Due to my understanding there is no authority?!

To close this: The browser keeps a list of CA certificates. When you import a web site's certificate, the browser will check the issuing authority against this list. If the result is negative, you get the usual warning that something is wrong with the web site's security.

With curl's --ca-cert option, you effectively add that CA cert to curl's list.

Or so I understand it.

In case the web site also requires a client certificate, I would guess that the web site would also check it against a CA list.

gloeckle · 04-29-2019, 11:41 AM

Hey guys,

i am now scratching my head because i can't get the cronjob to work.

added cronjob by

Code:

crontab -e

added job like this (tried with and without quotes)

Code:

*/6 * * * * curl -sSkE "/home/gloeckle/Desktop/PSOcheck/***.crt.pem" --key "/home/gloeckle/Desktop/PSOcheck/***.key.pem" https://subdomain.domain.tld/asdf/fdsa/index.php | grep Login | ts "%Y-%m-%d %H:%M:%S" >> "/home/gloeckle/Desktop/PSOcheck/log/https-secure-***.txt"

the command works just fine when i copy paste it into terminal with or without quotes. After fiddling around, i found out that it works without "ts "#%Y-%m-%d %H:%M:%S"" just fine.

i also tried to debug crontab by editing "/etc/rsyslog.d/50-default.conf"

Code:

cron.*                         /var/log/cron.log

and lookup "/var/log/cron.log"
i found that there is another file called "/var/log/cron.log.1" but both are empty.

Any advice?

I am also searching for an easy way to add an entry if "grep Login" gets no result. But a missing line would do aswell.

thinknix · 04-29-2019, 12:32 PM

I'm not sure what 'ts' does (generates a timestamp?), if it is a script you have written make sure you use the full path to it in the cron. In that environment, a non-standard path won't be in $PATH.

thinknix · 04-29-2019, 12:42 PM

Also, in terms of debugging cron entries - you can have cron send you an email with the output (if any) after a command runs, put this at the top of your crontab:

Code:

MAILTO=me@example.com

or you can capture output in a logfile by adding something like this to your crontab entry:

Code:

*/6 * * * * "command pipeline here" >> /tmp/cron.log 2>&1

The '2>&1' makes sure you capture both STDOUT and STDERR in the logfile.