lundi 13 juin 2016

Scraping data which loaded when you select something


I need to scrape car makes and model. Now I can scrape car makes select option list but can't scrape car models select option list because it is loaded when you select car make.

Maybe you have any ideas how I could get car models select options list which is loaded when you select car make.

Here is my code.

# -*- coding: utf-8 -*-
import requests
from bs4 import BeautifulSoup



url="http://autoplius.lt/redaguoti/naudoti-automobiliai/"
page = requests.get(url)

soup = BeautifulSoup(page.content, "html.parser")

car_makes_select = soup.find("select", {"id": "make_id"})

car_makes = car_makes_select.select("option")


for item in car_makes:
    itemMain = item

    itemMain = itemMain.get('value')

    payload = {
        'make_id': itemMain
    }

    form = requests.post(url, params=payload)

    soup11 = BeautifulSoup(form.text, "html.parser")

    model_select = soup11.find("select", {"id": "model_id"})

    print model_select

Aucun commentaire:

Enregistrer un commentaire