7 Practical Steps To Rotate Proxies In Python

Reading time: 11 min read
Muninder Adavelli
Written by
Muninder Adavelli

Updated · Aug 24, 2023

Muninder Adavelli
Digital Growth Strategist | Joined October 2021 | Twitter LinkedIn
Muninder Adavelli

Muninder Adavelli is a core team member and Digital Growth Strategist at Techjury. With a strong bac... | See full bio

Florence Desiata
Edited by
Florence Desiata

Editor

Florence Desiata
Joined June 2023 | LinkedIn
Florence Desiata

Florence is a dedicated wordsmith on a mission to make technology-related topics easy-to-understand.... | See full bio

Techjury is supported by its audience. When you purchase through links on our site, we may earn an affiliate commission. Learn more.

A proxy is a helpful tool for more accessible, secure, and private browsing. It works by masking your IP address and using another to send requests to your target.

A static proxy works well for simple tasks. However, a site can stop you from web scraping or data parsing using a single IP. In this case, you'll need a rotating proxy for these tasks.

Rotating proxies, on the other hand, lets you use a pool of proxy IP addresses. By doing so, you can send multiple requests—each with a unique IP.

In this guide, you'll learn how to rotate proxies in Python. Discover how to do it by reading further.

🔑 Key Takeaways

  • For Python proxy rotation, you'll need a collection of proxy addresses. If necessary, free proxies can be found on the internet.
  • Re-evaluating unsuccessful proxies is crucial for finding alternatives that work.
  • Static proxies frequently employ a single, fixed IP from a data center.
  • Rotating proxies utilize IP addresses that shift randomly or at specified intervals.

How To Rotate Proxies In Python?

It can be a technical task to rotate IP addresses. Still, it's doable with a few steps. Here's how you can rotate proxies in Python:

Things You Need To Rotate Proxies In Python

There are a few things you would need to do in Python. These are:

  • Python 3: You must install Python 3 for the script to run. This version is usually pre-installed in most systems. Still, it's best to download and install the latest version of Python
  • Pip Install: Pip Install is a command in Python. It looks for the latest version and installs it. It also checks the package metadata for dependencies and installs them to ensure the package has everything it needs.
  • List of proxy addresses: Data center and residential proxies are used for IP rotation. Each request is assigned a residential IP address for residential proxies.  Meanwhile, requests are handled separately by data center proxies.

🗒️ Related Article:


A regular proxy works for basic tasks, but occasionally, advanced types are needed. Rotating proxies are especially useful for scraping large files, preventing IP bans effectively by sending multiple requests that appear as if from different users. Various tools enable proxy rotation. To learn more about the benefits, check out our article on rotating proxies in Selenium with Python.

What is the Best Proxy Server?

Best for: Best overall
Smartproxy

Smartproxy

Smartproxy is a top-rated proxy provider trusted by many. Its 40 million+ proxies from 195+ locations help bypass CAPTCHAs, geo-blocks, and IP bans. It offers a free trial and has a high rating of 4.7 on Trustpilot with 89% 5-star ratings, making it one of the best in the industry.

Visit Website

Steps To Rotate Proxies in Python

Here's a guide on how to rotate proxies in Python:

Step 1: Have A Set Of Proxy Addresses

A set of proxy addresses is needed to rotate proxies in Python. There are a lot of free proxies online if you do not have any.

To get a list, save the URLs as ".txt". For instance, save it as proxies_for_rotation_list.txt.

🗒️ Note:

Free proxies are not the safest and most reliable proxies. Most of them are short-lived. Using residential or data center proxies for scraping or other projects is better.

Once you have the list, read the file and make a list with all the proxies. Do this by removing blank spaces and splitting every line.

Here's an example:

A set of proxy addresses

Be careful when saving the file since there'll be no test for valid strings. 

Step 2: Start Testing The Proxies

Once you have the list, test to see if the proxies work. You can write a function using ident.me. This page returns a reply with the IP.

You'll also need to download the Python request library. It's an efficient HTTP tool for this test. It works by taking one proxy from the list and calling the URL provided.

The script is often repetitive, but it will be helpful soon. The results could be either:

  • If it works: you'll receive a printed status code and a reply text. This one is usually the proxy server’s IP.
  • If it does not work: you'll receive a printed error for reasons like a timeout. If this happens, the proxy server isn't usable or cannot work on the request. It's usually the case when working with free proxies.

Here's an example:

Sample script for proxies

Pro Tip:

When doing this test, use HTTP instead of HTTPS since most of the free proxies on the list won't support SSL.

Afterwards, save the script in a document. For instance, you can save it as proxies-rotation-for-python.py.

Then, you can launch it from your computer's command line:

Start Testing The Proxies

Once done, the result should appear like this:

Command line

Step 3: Do Extra Tests To Confirm

You must do more tests to verify if the proxies are working. 

If there's an exception, it will show that the command did not work. Besides the exceptions, you should also check other things—like status codes.

You can check by recognizing specific codes as valid. For the rest, mark them as errors. It is also best to try other codes if they don't work.

Another test you can do is confirm if the response includes a specific IP address. Here's an example:

Screenshot of sample test to confirm if responses include specific IP address

Step 4: Do The Same Process Over Each Proxy

After the testing, do the same process for every proxy on the list. You'll have to repeat the proxies using 'get' to do this step.

It's best to keep it simple through a sequence. You can use asyncio.gather and aiohttp to make the requests. Using async will make the code complicated. However, it's best for web scraping tasks.

It is set to receive only up to 10 packages for protection purposes. This way, you can reduce the number of unwanted commands.

Sample command

Step 5: Separate Working Proxies From Non-Working Ones

Once you have the working proxies, you must set them apart from the non-working ones. It's best to group these addresses into three classifications:

  • unchecked_proxies: when its state is unknown or still needs testing
  • working_proxies: when the proxy showed success during the last test
  • not_working_proxies: when the last test for the proxy failed

With these classifications, you can add and remove items with ease. It will also help you avoid having duplicate proxies.

Move the proxies from one list to another while avoiding any duplication. To do this, you can make tiny changes in how the proxy data is stored.

Include the proxy list from your file for the unchecked proxies. Here's how you can set it up:

Sample command

Afterward, use commands to transfer proxies from one group to another. Use only one for every group. It will include the working proxy and exclude it from the rest.

Use the command discard to delete the proxy, whether present or not.

For instance, if a request succeeds, you can name it 'working_set'. This command will separate the working proxy from the not working or unchecked list. Then, it will add the proxy to the active list.

Sample command

Next, you must change "get" to set these commands after every request. Here's an example:


Sample command

You can include some trails at the script's end to check if it works. After running all proxies, the unchecked list should be empty.

🗒️ Note: 

If you're using free proxies, you'll find only a few working proxies in the list. Like the one below.

Step 6: Start Rotating Proxies

Sample command

The worst case is that you won't find anything on the working list.

Once you place each proxy into groups, you can use the working ones. However, you must make a function to pick a proxy.

You can use the working proxies alone or with the unchecked ones.

With sets, the function "random" is not suitable. For this reason, you must change them using the function "tuple."

From here, you can create a simple Python proxy rotator. Here's an example that picks random proxies from a set of working proxies:

Sample command

You can then change the "get" command to select a proxy when you can't find any present ones.

With this step, the "proxy" value becomes unnecessary. Use it to test the first proxies like before.

Afterwards, you can leave the list and use "get" without adding it. You'll use a random proxy. It will go to the not_working list if it does not work.

If you're web scraping, you need to scrape actual content. To do this, you need to send back the reply. You can also apply for an exception.

Here's the last version:

Sample command

You can include the data you need to collect below this script. You can also call the testing URL for a trial.

Another thing to do is make a customized parser using Beautifulsoup. Below is the final code on rotating proxies in Python:

Sample command

Step 7: Re-Test The Failed Proxies

Re-testing the failed proxies is necessary as it lets you discover more working proxies for several reasons.

Python lets you use the function "Timers" for this step. This action runs at a specific time. It has various methods, but the result is the same. You can do this step with only three lines.

With the reset_proxy command, you can set up a timer and run it on every proxy that's not working.

Set it to around 20 seconds. It may sound too quick, but it's enough for this test. Then, you can remove a non-working proxy by returning it to uncheck afterward.

Here's an example:


Sample command

Static vs. Rotating Proxies

There are two main types of proxies: static and rotating. Static proxies use one IP address, usually from a data center, for as long as you want. This is good for consistency. Rotating proxies, however, provides more anonymity and variety by switching IP addresses.

Meanwhile, a rotating proxy lets you use an entire proxy list or pool of IP addresses. These addresses change at random or fixed intervals, depending on how you set them.  

Take a closer look at the similarities and differences between these two: 

Similarities Of Static And Rotating Proxies

Both static and rotating proxies serve the same purpose. 

They mask your actual IP address. These proxies enhance your online anonymity for a better browsing experience.

Rotating and static proxies can also let you use shared IPs, depending on the exclusivity you pick for the latter. 

Differences Between Static And Rotating Proxies

With a static proxy, you can use a single IP address for specific purposes. It is generally cheaper since you’re not paying for an entire proxy pool. 

However, static proxies are more susceptible to IP bans. This issue usually happens when you make too many requests to a particular server.

With rotating proxies, you can enjoy higher security and anonymity. The chances of being banned are less since you'll use a different IP for every minute or every request. 

The only drawback to rotating proxies is that there’s a chance that you can use IPs from unrelated geolocations. 

Pro Tip

Knowing about proxies is important because they help you stay private online, improve security, and access blocked websites. Aside from rotating proxies in Python, we have more articles that can help you learn more:

Benefits Of Rotating Proxies

While rotating proxies in Python seems like a lot of work, doing so comes with many benefits. Below are some advantages you'll enjoy when using rotating proxies:

  • Avoid trackers

Most websites track an IP address, which can lead to identity theft. 

With a rotating proxy, these trackers will stay closed since you use a different IP with every request. This lets you enjoy better security and anonymity while browsing.

  • Unique Requests

Since you'll be using a proxy pool of IP addresses, you can send unique requests every time.

  • Skip Anti-Bots

When making multiple requests to modern websites, you’re more likely to encounter anti-bots like CAPTCHA. However, a rotating proxy can help you avoid getting banned since it reflects different IPs for each request.

  • Multiple connections with various locations

You can split your connections with a rotating proxy. This allows you to connect with different geo-restricted websites at the same time.

  • Avoiding request limits

Rotating proxies help scrape large websites. Using a different IP for every request lets you remain under the radar and avoid restrictions.

  • Higher Conversion Rates

Rotating proxies can help you raise conversion rates since every request you make comes from a different IP address.

🗒️ Helpful Article:

A proxy server acts as a bridge between users and the internet. When considering Proxy Server Services, key factors are usability, features, speed, cost, and assistance. We offer a guide on top proxy server services for selecting the right paid or free proxy, ensuring secure internet access.

Bottom Line

A rotating proxy lets you make multiple requests to a target without getting banned since you're using a unique IP address per request. 

Using the steps above, rotating proxies in Python is now doable. It seems technical initially, but keep in mind that rotating your proxies will help you with any complex tasks

FAQs.


Are rotating proxies legal?

Rotating proxies is legal in most countries. It's commonly used to request HTML and other data from a website.

Should I use sticky or rotating proxies?

A static or sticky proxy is best for simple tasks like scraping social media or buying from a particular region. A rotating proxy is best for complex or large projects like web scraping or data parsing.

What does rotate mean in Python?

Rotate in Python is a method of rotation that goes by the code "rotate()". In this method, the deque class rotates the sequence in the deque object 'n' times.

SHARE:

Facebook LinkedIn Twitter
Leave your comment

Your email address will not be published.