Skip to main content

6 Things you can Do with Python Requests Library

What is Python Requests Library about?


Python Requests Library allows you to send HTTP/1.1 request to Apache2 servers to serve data requested by clients. With request one can add form data, headers, multi part files etc. In this blog we have explained below how you can use Python Requests library in 6 different ways in your project to access web and raise HTTP request from server for the client.

1.Get HTML code of a website

We can use Python requests library to download and save the source files of a website. Do keep this in mind that we will be able to download only the client side files of that website, so no basically you can’t download the whole Facebook and replicate it on to your server.

This feature can come handy if you are trying to save all the links of a website or want to download all the images over a particular website. A basic example of HTML retrieval is as follows.

Python Code
# import requests library
import requests
# site whose HTML you want to retrieve.
site = "https://skwow.github.io/"
# send the request and recieve the response.
r = requests.get(site)
# response.text contains the website you recieved from its server
print(r.text)
# now you can store it in a file or process it the way you wnat.
<!DOCTYPE html>

<!--html>
<head>
    <title>Color Dodge</title>
    <meta http-equiv="refresh" content="1;url=https://play.google.com/store/apps/details?id=com.skwow.colorDodge&hl=en/" />
</head>
<body>
<img src="try.png" alt="image">
<br>
<button onclick="return myFunction()" type="button">Download</button>
</body>
</html>


<script>
    function myFunction() {
        window.location.href = "https://play.google.com/store/apps/details?id=com.skwow.colorDodge&hl=en";
    }
</script-->
 
Output
<html>
  <head>
    <title>WOW Games</title>

    <script src="js/jquery.js"></script>
    <script src="js/myOwn.js"></script>

    <link rel="stylesheet" href="css/style.css"/>
    <link rel="stylesheet" href="css/bootstrap.css"/>
    <link rel="stylesheet" href="css/animate.css"/>

      <meta charset="utf-8">
      <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
      <link rel="stylesheet" href="TemplateData/style.css">
      <link rel="shortcut icon" href="TemplateData/favicon.ico" />
      <script src="TemplateData/UnityProgress.js"></script>
  </head>




  <body>
    <nav class="animated" >
            <img src="Images/wow.png" id="navwow" class="img-responsive center-block animated" style="height: 40px; width: 60px;">
    </nav>
    <div class="sec1" >
      <img src="Images/ww.png" class="animated bounceInDown" id="ww" style="position: absolute; width: inherit; height: inherit; padding-bottom: 50px;">
      <img src="Images/games.png" class="animated bounceInRight" id="games" style="position: absolute; width: inherit; height: inherit; padding-bottom: 50px;">
    </div>
    <!--
    <div id="container" class="container">
      <ul id="scene" class="scene">
        <li class="layer" data-depth="1.40"><img src="Images/12.jpg"></li>
        <li class="layer" data-depth="1.20"><img src="Images/6.png"></li>
        <li class="layer" data-depth="2.00"><img src="Images/5.png"></li>
        <li class="layer" data-depth="0.80"><img src="Images/4.png"></li>
        <li class="layer" data-depth="0.60"><img src="Images/3.png"></li>
        <li class="layer" data-depth="0.40"><img src="Images/2.png"></li>
        <li class="layer" data-depth="0.20"><img src="Images/1.png"></li>
        <li class="layer" data-depth="0.10"><img src="Images/0.png"></li>
      </ul>
    </div>
    <script src="js/parallax.js"></script>
    <script>

      // Pretty simple huh?
      var scene = document.getElementById('scene');
      var parallax = new Parallax(scene);

    </script-->

    
    <div class="subclass">
        <div id="1" style="height: 700px;">
            My Content 1
        </div>

        <div id="2" style="display:none;">
            <iframe id="gameFrame" src="#" width=1150 height=700 name="IFRAME" style="padding-left: 150px;"></iframe>
        </div>

        <button id="btnClick">Click me!</button>
    </div>


    <h1 style="align-content: center; padding-left: 500px;"> Under Construction.</h1>
  </body>
</html>
=======
>>>>>>> origin/master
 

2. Download stuffs over HTTP or FTP


We can download stuffs from internet using python requests library. Do keep in mind that it can only download files over HTTP and FTP. It can take time depending on the size of the file so be patient if it seems to have stuck at request.get() statement. 
Python Code
# import requests library
import requests
# we are defining a function because we will be using it later as well.
def dnload(fileUrl,name): # takes link to the file to be download and name with which you want to save it
    # we send the request and recieve it in variable file
    file = requests.get(fileUrl)
    # we create and open a file with specified name and write our response bit-by-bit in it.
    with open(name, 'wb') as f:  
        f.write(file.content)
# now downloading file is as simple as passing Url and name of file to this function
fileUrl = 'http://speedtest.ftp.otenet.gr/files/test10Mb.db'
dnload(fileUrl,"requestfile10mb.db");


3. Accessing APIs


The most widely used application of python requests library is to access the APIs. We can access any available API using requests. Some of the examples are:
  1. Google Map API – Search of details of a place
  2. Weather APIs – we can access any weather API and ask it for current temperature or anything using requests.
  3. Bitcoin API – we can keep track of exchange rate of Bitcoin
and this endless list goes on…

we will see how to access Bitcoin API because it doesn’t require authentication and is very easy.
Python Code
# as always we start by importing requests
import requests
# send request and store the response in a variable
r = requests.get("https://api.coindesk.com/v1/bpi/currentprice.json")
# this is going to be a big json file you can print it and find a way to extract the info we desire.
print("current price of Bitcoin : "+ r.json()['bpi']["USD"]['rate'])
Output
current price of Bitcoin : 14,264.3038

And that’s the exchange rate at the time of writing this blog. 😉

4. Log into website (and extract data)


Using Python Requests library we can log into a website directly from our code and perform several tasks such as retrieve user info, Download data etc.

This can come in handy if you are collecting data for machine learning. It is also very useful if you want to automate things like if you want to be a blogger but don’t want to waste time writing blogs, instead you can steal blogs from other website and post it over yours after some tweaking.

And the pic is saved at the location of this script.
Python Code
# import requests library
import requests
# send request to the website with login credentials and store the response in a variable
r = requests.get('https://api.github.com/user', auth=('username', 'password'))
# retrieve the infor you want. here we are extracting profile pic of the user
avatarUrl = r.json()['avatar_url']
# then download the profile pic using the function we created in part 2.
dnload(avatarUrl,"me.jpg")


5. To diagnose problem with your website


We can easily see what’s wrong with a particular website by simply typing the URL in the web browser. But sometimes we want to automate things. What if we want to perform certain task automatically if something goes wrong with our website.

status_code can be used to get the http status of a website. Here is an extensive list of all the http status codes : https://en.wikipedia.org/wiki/List_of_HTTP_status_codes

we will now see a couple of examples of these http codes.
Python Code
# import requests library as always

import requests

# sites to check

sites=['https://www.google.com',' https://www.thisnofcmspkgnskfs.com','https://api.github.com/user']

for site in sites: # loop through sites and print its status code

print(site + " : "+ str(requests.get(site).status_code))
 
Output
https://www.google.com : 200
https://www.thisnofcmspkgnskfs.com : 404
https://api.github.com/user : 401
  • 200 means site is OK
  • 404 means not found
  • 401 means unauthorized request

6. Web scraping


Though there are awesome tools for web scraping like beautiful soup and scrapy, we can also scrap the web using Python requests. However, this will require a lot of effort as you will have to download the HTML and parse it yourself (if you choose not to use any other library). We will use lxml to parse the HTML response of the request. 
Python Code
# import requests library and 
import requests
import lxml.html as parseHTML
# lxml.html parses the response of the request we sent to the website
dom = parseHTML.fromstring(requests.get('http://www.google.co.in').content)
# we save all the links from the HTML except for the one which pointed to website itself
links = [x for x in dom.xpath('//a/@href') if '//' in x and 'www.google.com' not in x and 'www.google.co.in' not in x]
# and finally we print it
for link in links:
    print(link)
Instead of printing, we can run this code recursively on these new links to our heart desire.

Comments

  1. Really very informative and creative contents. This concept is a good way to enhance the knowledge.thanks for sharing. please keep it up


    python training in chennai

    hadoop training in chennai

    ReplyDelete

Post a Comment

Popular posts from this blog

How To Open PDF File in Android Application Example Included

Want to know How To Open PDF File in Android Application Example. To open a PDF File in Android Application, your app can take help from Free Android library available on Github.

Here are few apps made from PDF Library you can look at, click here to know more.

By using Android Library you are not only leveraging the hard work from another developer but also making it easier and quicker to develop an android app.
How To Open PDF File in Android Application Example  In this tutorial series, we will teach you how to open a pdf file in Android Application Example included with the help of Android PDFViewer library developed by Barteksc.

Get inspirtion from here: Top 7 PDF Download for Mobile Android Apps

Before I proceed further, here is the screenshot of the kind of PDF Viewer that’ll be created using Android Studio.



 How To Open PDF File in Android Application Example

The app will open a specified pdf file which will be copied in the assets folder of your Android studio project. I'v…

Open Weather Map API Example | Create Android Weather App

In this tutorial blog, using open weather map API example, we will create an Android weather app. The Android App will able to access Open weather map website server and access weather details of particular location.

The App will look similar to the image shown below. It will fetch real time weather of the location provided in MainActivity.Java file.


Software / Files used:Android Studio (Version 3.x) Files Edited / Created:activity_main.xmlMainActivity.JavaWeather.JavaAndroidManifest.xml Steps Involved in Open Weather Map API Example: Here are the Steps you need to follow on ‘Open Weather map API Example’ to create Android Weather App:

1. Open the Open Weather map API link provided here. Sign up for Current Weather Data API key here.

2. Once the key is received, save it to notepad or on stick notes.

3. Open Android Studio and create a new Project with name, ‘FIMC Weather App’, we will select an empty activity in the wizard and wait for Gradle build.



4. Open AndroidManifest.xml…

Android Live Wallpaper Tutorial (using Android Studio 3.x) | FaultInMyCode

The Android Live Wallpapers are the way to redesign your mobile background into something amazing animated GIF using different effects and motions. Many novice developers are making good money by Henceforth here is Android Live Wallpaper tutorial for you to build your own Live wallpaper and start earning today as a developer!



This tutorial is a part of our Blog series on ‘Roadway to Android Development Course’, its suitable for Beginners to programming as well as the android enthusiast to learn about Android development and build their own apps.

In this tutorial for beginners, you’ll learn how by using Android Studio 3.0 and by having basic knowledge of Java, we will teach you to build your own Live wallpaper for Android which you may later publish in the Google Play Store. We assume you have some experience with Android studio, if not please check how to build your own calculator app and how to create simple login app in Android Studio 3.x tutorial.
How to build Android …