[SOLVED] Selenium with python, works via CLI, but not with cron

iamtheliquor · 08-24-2018, 07:28 AM

Hello,

I have the following simple python/selenium script, that when run from the CLI works absolutely fine. Opens firefox to the BBC homepage, waits 5 seconds, and then closes the window;

Code:

#!/usr/bin/env python

from selenium import webdriver
import time

driver = webdriver.Firefox()
driver.get("http://bbc.co.uk/")
time.sleep(5)
driver.quit()

It is called using the following bash script, which also works with no problem;

Code:

#!/usr/bin/env bash

export DISPLAY=:0
/usr/bin/python /home/wfs-test/Documents/browser-test-v2.py

However, when scheduling this with cron, it does not work. I can see the script is being run by cron okay, it just does not open the browser and perform the tasks as it should do. I know cron does not know about the display, which is why I have the following entry in the bash script;

Code:

export DISPLAY=:0

This should tell cron where the display is, but it still does not work.

I have also tried using xvfb with PyVirtualDisplay with the following code;

Code:

#!/usr/bin/env python

from selenium import webdriver
from pyvirtualdisplay import Display
import time

def main():
    with Display(visible=True, size=(1680, 1050)):

        driver = webdriver.Firefox()
        driver.get("http://bbc.co.uk/")
        time.sleep(5)
        driver.quit()

if __name__ == "__main__":
    main()

Again, this works with no issue when called from the CLI or via a bash script, but does not work when scheduled with cron. I have tried scheduling both as the local user and root, with no difference.

Any ideas?

Ubuntu 18.04.1
Python 2.7.15rc1
pip (9.0.1)
selenium (3.14.0)
PyVirtualDisplay (0.2.1)
xvfb 2:1.19.6-1ubuntu4

pan64 · 08-24-2018, 08:55 AM

you ought to implement some error handling and logging to find out which command failed and why.

iamtheliquor · 09-05-2018, 02:50 AM

So I got a bit side tracked with some other work stuff but got back to this, and the solution turned out to be quite simple. Just in case anyone was interested, looks like cron was struggling to find things, so adding the following two lines to the top of the bash script has sorted it;

Code:

export DISPLAY=:0
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Should have known really, not the first time I've had this type of thing with cron. I think what threw me is I would normally just write a bash script and use the full path for commands to avoid this kind of thing, whereas this is calling a python script to do the work.