How to use CSS Selectors to get the data you need :

Introduction and Prerequisites

Hello! Thanks for taking the time to review this article and unlock the power of our CSS selector functionality. In this article we'll be walking through how you can get the price, product title, and product description from an ecommerce site. To get started using our API's CSS selector functionality (API documentation here), you'll need a Scraping Robot account.

To sign up for a new account and instantly get 5,000 free scrapes, please register on our site here. If you're an existing user, please visit your dashboard and login here.

I've included a video walkthrough of this article at the end of the article to show you how to do this live as well.

Building your API call

Once you've signed up for an account, you will be provided with an API key that you can use to scrape the data you need from your target site. You can find your API key in the "API Info" tab on the left menu in your dashboard. You will need this to insert it into the API call once we've built it out in our API documentation.

First, you'll visit our API documentation for making a GET call (POST calls are also available if needed) and you'll see the table of optional parameters. We're going to use this table to build out our API call.

The two parameters that I need to fill in order to use CSS functionality properly are "scrapeSelector" and "url". For the URL parameter I will be using https://www.entirelypets.com/greenies-supplements-hipnjoint-dogs-80chews.html.. See the next section for how to get the "scrapeSelector" variables.

Retrieve scrapeSelector parameters from your target site

Now, to get my first "scrapeSelector", I will use the right-click and "Inspect". And it will bring up the Google Chrome Elements window. You should right-click and hit "Inspect" directly on the element that you want to see (in this example it's the price). When I do that, I can see "sale-price" element already highlighted in the Elements window.

Once I've done that, I'll right-click on the element in the Elements window, hover over the "Copy" menu item, and then click "Copy selector".

Once I've done that, I'll paste this into my "scrapeSelector" parameter in the API documentation.

Finally, I need to paste my target URL into the "url" parameter and insert my API key to finish the formatting of my API call.

Using the API call

Now that we've built our API call through the documentation, we can easily port it over to the software of your choosing. You can select from our wide library of supported languages shown here. The "Request" box will update to the language of your choosing with the information that you need to copy and paste into your code. For this example, I'll be using Python to show you the finished product.

Once I've selected Python, I'm going to run the required installation steps. Then, I'll copy the "Request" box text and paste that into my IDE like this.

NOTE: Before you run your IDE project or the API call, you will need to move the token parameter ahead of the URL parameter or you will receive a "FAIL" message.

The API call generated by our documentation is:

https://api.scrapingrobot.com/?responseType=json&render=true&scrapeSelector=%23itemtable-prices%20%3E%20div%20%3E%20div.sale-price%20%3E%20span&waitUntil=load&noScripts=false&noImages=true&noFonts=true&noCss=true&url=https%3A%2F%2Fwww.entirelypets.com%2Fgreenies-supplements-hipnjoint-dogs-80chews.html&token=d5a79d6d-6b75-48f8-b417-e25f062cf54e

And you should make the change to this:

https://api.scrapingrobot.com/?responseType=json&token=d5a79d6d-6b75-48f8-b417-e25f062cf54e&render=true&scrapeSelector=%23itemtable-prices%20%3E%20div%20%3E%20div.sale-price%20%3E%20span&waitUntil=load&noScripts=false&noImages=true&noFonts=true&noCss=true&url=https%3A%2F%2Fwww.entirelypets.com%2Fgreenies-supplements-hipnjoint-dogs-80chews.html

You'll notice I moved the "token" parameter from the last spot in the URL, to the spot right behind "responseType".

JS rendering requirements

There are some sites that require JS rendering to get the data that you're looking for, my example site here being one of them. If you see this error message:

{"status":"FAIL","date":"Mon, 20 Dec 2021 12:26:54 GMT","error":"JS rendering required"}

Go back to the API documentation and set the "render" parameter to "true" and retry the request.

Output

After you've run your request, you'll be returned with a JSON object containing first the selector elements that you're looking for along with the full HTML of the page that we scraped for you for any additional parsing you may want to do. From there, you can either parse out this data yourself or watch the next video in our series about integrating with Google Sheets. That video will be linked in the Conclusion section once complete.

Using Multiple Selectors

If you want to use multiple selectors, you will need to add another "scrapeSelector" parameter to your API URL. A sample of this would look like:

https://api.scrapingrobot.com/?responseType=json&token=d5a79d6d-6b75-48f8-b417-e25f062cf54e&render=true&scrapeSelector=%23itemtable-prices%20%3E%20div%20%3E%20div.sale-price%20%3E%20span&waitUntil=load&noScripts=false&noImages=true&noFonts=true&noCss=true&url=https%3A%2F%2Fwww.entirelypets.com%2Fgreenies-supplements-hipnjoint-dogs-80chews.html

Conclusion and Video Walkthrough

That concludes the explanation of how to use our CSS selector functionality. I hope you were able to get what you needed by following along, but if you have any questions, please reach out to our support team at support@scrapingrobot.com and we're happy to help you out!

If you'd like to view a video of this functionality, you can see that here.