How to send 100,000 HTTP requests quickly in Python?

Sometimes, we want to send 100,000 HTTP requests quickly in Python.

In this article, we’ll look at how to send 100,000 HTTP requests quickly in Python.

How to send 100,000 HTTP requests quickly in Python?

To send 100,000 HTTP requests quickly in Python, we can use the tornado library.

To install it we run

pip install tornado

Then we use it by writing

from tornado import ioloop, httpclient

i = 0

def handle_request(response):
    print(response.code)
    global i
    i -= 1
    if i == 0:
        ioloop.IOLoop.instance().stop()

http_client = httpclient.AsyncHTTPClient()
for url in open('urls.txt'):
    i += 1
    http_client.fetch(url.strip(), handle_request, method='HEAD')
ioloop.IOLoop.instance().start()

to create the handle_request function that gets the response and prints our the code.

Once i is 0, we have

ioloop.IOLoop.instance().stop()

to stop the async requests.

Then we create an HTTP client object with

http_client = httpclient.AsyncHTTPClient()

Then we loop through the URLs returned from urls.txt.

In the loop, we call fetch to make a GET request with each url and increment i.

Then we have

ioloop.IOLoop.instance().start()

to start making the requests asynchronously.

Conclusion

To send 100,000 HTTP requests quickly in Python, we can use the tornado library.