Quick Directory Scanner with Python Requests

- 2 mins


This is for educational purposes - do not do directory scanning on any host you do not own or have explicit permission to scan. You can/likely will face serious punishment.


python-requests is one of the libraries that makes me love Python: it’s simple, easy to understand, easy to use, and gives you tons of functionality with little code.

Requests allows you to make HTTP/S requests with Python code.

A quick demo from their site:

>>> r = requests.get('https://api.github.com/user', auth=('user', 'pass'))
>>> r.status_code
>>> r.headers['content-type']
'application/json; charset=utf8'
>>> r.encoding
>>> r.text
>>> r.json()
{u'private_gists': 419, u'total_private_repos': 77, ...}

Requests is amazing for building and testing web applications, especially ones that consume REST APIs because you can generate python dictionaries from the JSON response of the web server. You can modify headers (in case you want to perform some User-Agent spoofing maybe), stream downloads, chunk requests, tamper with cookies, and do tons of other cool stuff.

Quick directory scanner:

I’ll throw together scripts like this when I’m doing a CTF to test response codes of a target server without using something like DirBuster/GoBuster. With python-requests I can programmatically check for certain keywords in the web server’s response/source (easy key finding!) or classify pages by the returned status_code.

This script will open a dirbuster wordlist, make HTTP requests to a host, then collect a list of directories that return a certain status code (200 in this case)

Something like this can be thrown together in a few minutes but can save you a lot of time or run in the background while you perform manual probing of the target.

import requests
import time

victim = 'http://localhost:4000/'
dir_file = 'common.txt'

with open(dir_file) as f: 
    dirs = f.readlines()
    for directory in dirs:
        directory = directory.rstrip() # strip newline characters
        response = requests.get(victim + directory + '/')
        if response.ok:

Scanning my local test server (this site) returned the following results:

About, Blog, Index, Projects, about, assets, blog, index, projects, tags

Easy! Now we’ve got some directories to poke around in.

If I were making this more robust, I’d do things like:

Python-requests is an exceptional library and you should definitely be using it if you’re working with Python.

Tony De La nuez

Tony De La nuez

Software Engineer @ Atlassian. Security and Infrastructure.

comments powered by Disqus
rss facebook twitter github youtube mail spotify instagram linkedin google google-plus pinterest medium vimeo stackoverflow reddit quora