How to use our HTML API for general scraping using Python :

Introduction

Hello! This article will help outline how to use our HTML API for general scraping purposes. We will be using our site - www.scrapingrobot.com - as the sample here. There will be more articles added to our knowledgebase that expand on specific features, so if you don't find exactly what you're looking for here, please visit one of our other how-to articles for advice or reach out to our support team at support@scrapingrobot.com..

Prerequisites

Latest Python Requests Library - https://docs.python-requests.org/en/latest/

Sample Code

import requests


url = "https://api.scrapingrobot.com?token=YOUR_SR_API_TOKEN"

payload = "{\n    \"url\": \"https://www.scrapingrobot.com\",\n    \"module\": \"HtmlRequestScraper\"\n}"

headers = {
  'Content-Type': 'application/json'
}

response = requests.request("POST", url, headers=headers, data = payload)

print(payload)
print(response.text.encode('utf8'))

After inserting this code into your IDE and replacing your token into the "url" variable where it states "YOUR_SR_API_TOKEN", you'll be able to see the HTML returned form any website that you put into the payload. Also, please check out our API documentation for more features that you can use and samples in different languages!

Give it a try and let us know if you have any questions at all!