March 9, 2021, 5:54 p.m.

How to use proxy in python requests | http & https proxies - Scraping

Sometimes We need to scrape a website .......bla bla bla. Does it make any sense?? I am not going to do any kind of bullshit here. You all know whatever you are doing. This short article is going to be a small part of your long journey.
Today I am going to explain to you how you can use proxies during web scraping using python. There are 2 reasons behind your presence here. Either you are blocked by the server/website which you are trying to scrape or you are already smart enough to use proxies to bypass the firewall of a website's server.
What is Proxy? A proxy is nothing but just an authority to represent someone else. Now you might have been using this word in your collage or class among your friends for your fake attendance. In the web world, the proxy is an intermediate server which request data from the main server from which you wants the actual response.
So the proxy is just like a waiter in a restaurant which acts as an intermediate between you and the food bucket.

Free Proxy Resources Almost every developer uses proxies for there personal/business purposes and using a proxy is not illegal. The proxy consists of 2 parts i.e an ip address and a port number. There are many free websites like SSL Proxies from where you can get a full list of both HTTP and HTTPS free proxies. You can also search google "Free github proxies" for an giant proxy list.

Using proxies in python requests requests is the largest downloaded library on pypi.org/. Using proxies with requests is pretty simple. All the proxies have to pass as an dictionary in requests.get() method as shown below .
The syntax and an example is shown below :

#Syntax
proxies={'http':'http://<ip>:<port>',
'https':'https://<ip>:<port>'}
requests.get(url,proxies=proxies)

Examples Here is an image in which i haven't used any proxy. httpbin.org/ip/ is the url which is used to get my ip address in response. So in the below example the ip address is the address of my public internet provider

So now in the below example i used proxies and you can notice that the ip address which i get in response from httpbin.org/ip/ is diffrent i.e equal to the proxy.

Tips Sometimes you may have to face Connection refuses or could not connect to the proxy server error. This means that the proxy server which you are using is not working at that time. So you may have to check a couple of proxies to get a fine working proxy server. So don't get frustrated if you face such kind of errors.

Some recent posts

How to deploy a django website on pythonanywhere | django Web app deploying...

Complete process of changing database from SQLite to MySQL - How to migrat...

" How to download a file in django | Making a large file downloadable from ...

How to use proxy in python requests | http & https proxies - Scraping...

Top Django Querysets, How to filter records | Fetching data from database e...

How to change base url or domain in sitemap - Django ...

How to Make a Website - Everything you should know about web development...

What is Javascript - Features and usages ...

Top 5 Interview Questions : Tips for HR round ...

How to get job in IT : Perfect resume guide for IT job ...

View all...

How to use proxy in python requests | http & https proxies - Scraping

Some recent posts

Popular Posts