jeudi 30 juin 2016

Web Scraping with Python - Looping for city name, clicking and get interested value


This is my first time with Python and web scraping. Have been looking around and still unable to get what I need to do.

Below are print screen of the elements that I've used via Chrome.

As you can see, it is from the dropdown 'Apartments'.

My 1st step in trying to do is get the list of cities from the drop down

My 2nd step is then, from the given city list, go to each of them (...url.../Brantford/ for example)

My 3rd step is then, given the available apartments, click each of the available apartments to get the price range for each bedroom type

Currently, I am JUST trying to 'loop' through the cities in the first step and it's not working.

Could you please help me out as well, if there's a good forum, article, tutorial etc that's good for beginner like me to read and learn. I'd really like to be good in this so that I may give me to society one day.

Thank you!

import requests
from bs4 import BeautifulSoup

url = 'http://www.homestead.ca/apartments-for-rent/'
response = requests.get(url)
html = response.content
soup = BeautifulSoup(html,'lxml')

dropdown_list = soup.find(".child-pages dropdown-menu a href")

print (dropdown_list.prettify())

Screenshot


Aucun commentaire:

Enregistrer un commentaire