Selenium is a Python module used to simulate the use of a web browser with Selenium Webdriver. Basically it allows you to program moving around the web browser, clicking certain things and filling out authentication or forms. You can use Selenium with Firefox, PhantomJS, Chrome, Safari and Microsoft’s Edge, the ones I used the most were Firefox and PhantomJS. I did try Chrome but ended up sticking with Firefox. The advantage to using PhantomJS is that it is a headless browser, meaning it does the programmed simulation without opening any windows and works with servers because a display is not necessary. However, I have recognized that it has its disadvantages.

Working with Selenium for a while now, allowed understand it quiet well. When I first started with Selenium there was a lack of sources, so I thought I would share my knowledge on Selenium . Even though the documentation (http://selenium-python.readthedocs.io) was very helpful, there were many things that learned eventually though working with the module for a while.

Installing Selenium

Installing Selenium is simple, just type this command that uses pip:

pip install selenium

Drivers for selenium

However, you also need a Selenium Webdriver for the specific browser of your choice.

Firefox: https://github.com/mozilla/geckodriver/releases

PhantomJS: http://phantomjs.org

Chrome: https://sites.google.com/a/chromium.org/chromedriver/downloads

Safari: https://webkit.org/blog/6900/webdriver-support-in-safari-10/

Edge: https://developer.microsoft.com/en-us/microsoft-edge/tools/webdriver/

For Firefox, PhantomJS and Chrome you have to copy the executable file to a directory in PATH for Linux and Mac (**Note: for PhantomJS the executable is in the bin folder inside the unzipped folder). You should copy the executable into /usr/local/bin which is a directory in PATH.

The Setup

The first thing you have to do in the code is setup the Webdriver. Since, Webdriver is the bridge needed for Selenium to control the browser. The code below imports webdriver class and creates the instance of Firefox Webdriver:

from selenium import webdriver

driver = webdriver.Firefox()

Substitute Firefox with the name of the web browser of your choice (Chrome, PhantomJS, etc.)

For a simple start lets make a code that goes to https://www.google.com, and searches up “selenium”. So lets add to the previous code:

driver.get("https://www.google.com")

The code so far will open Firefox and go to the website https://www.google.com, make sure the string contains “https://”.

However, this code does not quit Firefox after it is done, which is quiet inconvenient. So lets add to the code driver.quit(), which will quit Firefox after it is done. Another option is driver.close(), but this closes the tab, not the browser. We can also add try/except/finally to quit the browser even when there is an error. I also added the sleep from time module, so the browser won’t quit instantly and we can see what happened. The code now looks like this:

from selenium import webdriver
from time import sleep

try:
    driver = webdriver.Firefox()
    driver.get("https://www.google.com")
except:
    print("Failed!")
    raise
finally:
  sleep(5)
    driver.quit()	# <- Quits the browser when done

Basic

Now things get a little be complicated. First you need to know what an element is, an element is basically any part of a website, and we can identify them by id, xpath, name, link_text, partial_link_text, tag_name, class_name and css_selector. These are HTML terms so if you understand them it will be easier for you to use Selenium (I haven’t learned HTML, so I had to google my way into working with these).  The most useful once for me have been id, css_selector and xpath.

Getting the Element ID:

1. Lets first start by using Firefox (Doesn’t have to be the same browser, you just need a browser with inspect element) and go to the website, being https://www.google.com.
2. Then you have to right click on the search box and click Inspect Element.

3. This will open the Inspector and information about the search box element will be shown.

4. The id for this element is shown (It is not always there), so we will use it.

Now lets go to the code and add the following:

from selenium.webdriver.common.keys import Keys

search_box = driver.find_element_by_id("lst-ib")    # finds search box element
search_box.send_keys("selenium")    # types selenium into the search box
search_box.send_keys(Keys.RETURN)   # presses ENTER/RETURN

This uses the find_element_by_id method to get the element using its id. Then it type “selenium” into the search box, with send_keys. The third line uses Keys to simulate the enter/return button. You can also get the id for the google search element and click it with ELEMENT.click(), both of them work the same.

This was only an intro to Selenium. You can do much more with Selenium like can filling out forms, logging in, taking screenshot etc.