lundi 13 juin 2016

Scrape webpage after form fill


Im trying to scrape the response to this website using a pre-filled zip: zip who (i.e. the zip code is already filled in.) I tried to do this using the scrapy shell as follows

scrapy shell http://zipwho.com/?zip=77098&mode=zip

but the response does not contain the form-filled page, but only the content from the main zipwho.com page and without the details specific to that zip code. I tried filling in the form information using requests and lxml, but clearly I am doing something wrong.

import requests
import lxml.html as lh
url = 'http://zipwho.com'

form_data = {
    'zip': '77098'
    }
response = requests.post(url, data=form_data)
tree = lh.document_fromstring(response.content)
tree.xpath('//td[@class="keysplit"]')

and the table element for the data (td where class = 'keysplit') still does not exist. If you have ideas to get this working (hopefully with something simple in like requests & lxml) that would be best.


Aucun commentaire:

Enregistrer un commentaire