How to run Python Selenium with Flask on a VPS using PhantomJS

I love automation and web testing/bot making, the problem is I just can only run it on my local computer or make a GUI app. This week I tried to expand the use of it to the world-wide-web to get more audiences. Tired of doing “python __init__.py”, then why not do “http://www.yourwebsite.com/script/” ???

I’m not going into detail how to do each part, you are required to have a general idea and a common sense how to debug and fix the problem. It took me prolly 2 weeks to figure out how to run the server correctly and fix all the basic bugs. Let’s start.

1. You need to have a VPS, I’ll recommend DigitalOcean.com for the cheap and free 2 months VPS. If you decide to register then please register under my referral link so I can get an earning from you and you can get 2 free months.

2.Install either LAMP stack or ngix on your VPS. I’m using LAMP stack so if you decidde to use ngix you have to figure out the problem on your own. (Note: I tried ngix, there is 1 problem that because the script usually takes longer to finish thna normal, so you always be timed-out.) The framework I use it Flask.

How to install LAMP stack on your VPS: https://www.digitalocean.com/community/tutorials/how-to-deploy-a-flask-application-on-an-ubuntu-vps
How to install Ngix/Unicorn on your VPS: https://realpython.com/blog/python/kickstarting-flask-on-ubuntu-setup-and-deployment/

Just follow all the setup and you will be ready to run python on a web. From now on I will just give instruction based on LAMP stack (Apache) only.

3. Make sure you are logged as root, now do “source venv/bin/activate”, then “pip install selenium”.

4. Now follow this to install PhantomJS, make sure you have a copy of the executable bin in your current “yourweb” folder. https://gist.github.com/julionc/7476620

5. PhantomJS needs a ghostdriver.log file in order to run, you have to create a customize log file because visitors cannot trigger the script to create that file in the system. So now in “yourweb”, let create a blank “ghostdriver.log”

6. Supposed your web is in “var/www/yourweb/” then you have to set the group to www-data in order to let visitors execute the script. Just do “chown -R root:www-data /var/www/yourweb/”.

Now up to setting permission, this problem took me a whole week to figure out how to do that:

7. Now you have “phantomjs” and “ghostdriver.log” inside “yourwebsite”, let do chmod 750 for phantomjs and ghostdriver.log

8. Keep in mind, in order for visitors to run those files, all related folders have to be set to 750 also. Now do chmod 750 for “yourwebsite” AND “WWW”  <== I didn’t chmod the “www” folder and It drove me crazy for couple days.

9. Now open /etc/sudoers, add these lines:
apache ALL=NOPASSWD: /var/www/yourweb/ghostdriver.log
apache ALL=NOPASSWD: /var/www/yourweb/phantomjs

I’m not quite understand the cause here, but if you don’t do that you cannot run phantomjs and ghost driver correctly.

10. Now you are set and ready to go, make a simple Flaskapp and test it. Remember that your webdriver has to be customized to:

br = webdriver.PhantomJS(service_log_path=’somewhere/ghostdriver.log’, executable_path=”somewhere/phantomjs”)

Where somewhere is your absolute path to the folder that those files reside.

If you run into any problem, the log file will be saved in “/etc/log/apache2/error.log”, just refer to the file.

Hope you could run selenium with flask on the website, and make an awesome app 😀 PM me if you need any help.

If you’d rather pay for this

A note first, because this post is from 2015: PhantomJS is dead. Its maintainer stepped down in 2018 and development stopped, after Chrome shipped a headless mode that did the same job properly. Do not start anything new on it. The Flask-and-a-VPS shape below is still sound; swap the browser for headless Chrome driven by Playwright or Selenium and the rest of this holds up.

If you are standing this up today, the real question is whether you want to run browsers at all. A headless browser on your own VPS is cheap in dollars — a few a month — and expensive in everything else: Chrome eats RAM, the version drifts out from under your driver, crashed processes leak, and anything touching the public web drags you into IP reputation and blocking.

Hosted browser automation exists precisely because of that tail. You pay per session or per hour of browser time instead of per server, and someone else owns the version drift and the IP pool. It is more expensive per unit and much cheaper per incident.

My split: if the browser is incidental to your app and runs occasionally, keep it on your own box. If the browsers are the product, or a failed run costs you something real, rent them.