How to use Gobuster to find Interesting Directories & Files on Website.

Hey there,
Today we will learn how to use gobuster to enumerate the files and directories on a webserver.

Enumerating hidden folders and files is one of the initial steps in an assault on a web application.
This may frequently give vital information that makes it easier to conduct a precise assault, leaving less space for errors and lost effort.
There are several tools available to accomplish this, but not all of them are made equal.
Gobuster, a directory scanner developed in Go, is definitely worth exploring.

Conventional directory brute-force scanners, such as DirBuster and DIRB, perform properly but are frequently sluggish and prone to mistakes.
Gobuster is a Go implementation of these utilities that is available in a command-line manner.

The key benefit Gobuster offers over other directory scanners is its speed.
Go is well-known for its speed as a programming language.
It also provides great concurrency support, allowing Gobuster to take use of several threads for speedier processing.

The one limitation of Gobuster is the lack of recursive directory scanning.
Unfortunately, another scan will be required for directories that are more than one level deep.
This isn’t always a significant concern, and other scanners can step in and fill in the gaps for Gobuster in this situation.

Gobuster has an easy-to-use command-line interface.
It offers several helpful choices, but not so many that getting buried down in the intricacies becomes difficult.
Overall, it’s an excellent tool that is both effective and quick.
In this article, we’ll look at it using DVWA as the target and Kali Linux as the attacker system.
You may either follow along with them or use a comparable testing setup.

Step 1: Install Gobuster

# apt-get install gobusterReading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  gobuster
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/1,532 kB of archives.
After this operation, 4,963 kB of additional disk space will be used.
Selecting previously unselected package gobuster.
(Reading database ... 412624 files and directories currently installed.)
Preparing to unpack .../gobuster_2.0.1-1_amd64.deb ...
Unpacking gobuster (2.0.1-1) ...
Setting up gobuster (2.0.1-1) ...
Processing triggers for man-db (2.8.5-2) ...

Then, simply type gobuster in the terminal to run the tool.

~/gobuster# gobuster

2019/05/06 11:43:08 [!] 2 errors occurred:
    * WordList (-w): Must be specified (use `-w -` for stdin)
    * Url/Domain (-u): Must be specified

We can see it gives us a couple of errors. It needs at least two parameters (-u for the URL, and -w for the wordlist) to run properly. We can also display the help menu with the -h flag.

~/gobuster# gobuster -h

Usage of gobuster:
  -P string
        Password for Basic Auth (dir mode only)
  -U string
        Username for Basic Auth (dir mode only)
  -a string
        Set the User-Agent string (dir mode only)
  -c string
        Cookies to use for the requests (dir mode only)
  -cn
        Show CNAME records (dns mode only, cannot be used with '-i' option)
  -e    Expanded mode, print full URLs
  -f    Append a forward-slash to each directory request (dir mode only)
  -fw
        Force continued operation when wildcard found
  -i    Show IP addresses (dns mode only)
  -k    Skip SSL certificate verification
  -l    Include the length of the body in the output (dir mode only)
  -m string
        Directory/File mode (dir) or DNS mode (dns) (default "dir")
  -n    Don't print status codes
  -np
        Don't display progress
  -o string
        Output file to write results to (defaults to stdout)
  -p string
        Proxy to use for requests [http(s)://host:port] (dir mode only)
  -q    Don't print the banner and other noise
  -r    Follow redirects
  -s string
        Positive status codes (dir mode only) (default "200,204,301,302,307,403")
  -t int
        Number of concurrent threads (default 10)
  -to duration
        HTTP Timeout in seconds (dir mode only) (default 10s)
  -u string
        The target URL or Domain
  -v    Verbose output (errors)
  -w string
        Path to the wordlist
  -x string
        File extension(s) to search for (dir mode only)

Step 2: Install Some Extra Wordlists

Wordlists on Kali are located in the /usr/share/wordlists directory.

~/gobuster# ls /usr/share/wordlists/

dirb  dirbuster  dnsmap.txt  fasttrack.txt  fern-wifi  metasploit  nmap.lst  rockyou.txt.gz  sqlmap.txt  wfuzz

The dirb and dirbuster ones are fine, but there is another wordlist I like to use for directory brute-forcing. There is a whole repository of useful wordlists on GitHub called SecLists.

The “common.txt” wordlist contains a good number of common directory names.

We can download the raw file into our current directory using the wget utility.

~/gobuster# wget https://raw.githubusercontent.com/danielmiessler/SecLists/master/Discovery/Web-Content/common.txt

--2019-05-06 11:46:40--  https://raw.githubusercontent.com/danielmiessler/SecLists/master/Discovery/Web-Content/common.txt
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.148.133
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.148.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 35744 (35K) [text/plain]
Saving to: ‘common.txt’common.txt                                            100%[======================================================================================================================>]  34.91K  --.-KB/s    in 0.03s2019-05-06 11:46:40 (1.24 MB/s) - ‘common.txt’ saved [35744/35744]

Alternatively, if we wanted to install the whole SecLists repository, we can do so with the package manager.

~/gobuster# apt-get install seclists

Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  seclists
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/221 MB of archives.
After this operation, 795 MB of additional disk space will be used.
Selecting previously unselected package seclists.
(Reading database ... 412629 files and directories currently installed.)
Preparing to unpack .../seclists_2019.1-0kali1_all.deb ...
Unpacking seclists (2019.1-0kali1) ...
Setting up seclists (2019.1-0kali1) ...

That will install the entirety of SecLists in the /usr/share directory. It contains a ton of content, and if you’re looking for a go-to set of wordlists, SecLists will likely be the last thing you’ll ever need.

/usr/share/seclists# ls

Discovery  Fuzzing  IOCs  Miscellaneous  Passwords  Pattern-Matching  Payloads  README.md  Usernames  Web-Shells

Step 3: Scan Directories & Files

Now that everything is set up and installed, we’re ready to use Gobuster. Let’s run it against our target with the default parameters.

~/gobuster# gobuster -u http://10.10.0.50/dvwa/ -w common.txt

=====================================================
Gobuster v2.0.1              OJ Reeves (@TheColonial)
=====================================================
[+] Mode         : dir
[+] Url/Domain   : http://10.10.0.50/dvwa/
[+] Threads      : 10
[+] Wordlist     : common.txt
[+] Status codes : 200,204,301,302,307,403
[+] Timeout      : 10s
=====================================================
2019/05/06 11:50:25 Starting gobuster
=====================================================
/.htaccess (Status: 403)
/.htpasswd (Status: 403)
/.hta (Status: 403)
/README (Status: 200)
/config (Status: 301)
/docs (Status: 301)
/about (Status: 302)
/external (Status: 301)
/favicon.ico (Status: 200)
/cmd (Status: 200)
/index (Status: 302)
/index.php (Status: 302)
/php.ini (Status: 200)
/instructions (Status: 302)
/logout (Status: 302)
/robots (Status: 200)
/robots.txt (Status: 200)
/login (Status: 200)
/phpinfo (Status: 302)
/phpinfo.php (Status: 302)
/setup (Status: 200)
/security (Status: 302)
=====================================================
2019/05/06 11:50:38 Finished
=====================================================

We can see it gives us some information about the tool in the banner, and then it kicks off the directory discovery process. It returns the names of the directories and files as well as their status codes.

We can have it return the length of the body with the -l flag, which can be useful for enumeration.

~/gobuster# gobuster -u http://10.10.0.50/dvwa/ -w common.txt -l

=====================================================
Gobuster v2.0.1              OJ Reeves (@TheColonial)
=====================================================
[+] Mode         : dir
[+] Url/Domain   : http://10.10.0.50/dvwa/
[+] Threads      : 10
[+] Wordlist     : common.txt
[+] Status codes : 200,204,301,302,307,403
[+] Show length  : true
[+] Timeout      : 10s
=====================================================
2019/05/06 11:52:41 Starting gobuster
=====================================================
/.hta (Status: 403) [Size: 292]
/.htaccess (Status: 403) [Size: 297]
/.htpasswd (Status: 403) [Size: 297]
/README (Status: 200) [Size: 4934]
/config (Status: 301) [Size: 319]
/docs (Status: 301) [Size: 317]
/external (Status: 301) [Size: 321]
/favicon.ico (Status: 200) [Size: 1406]
2019/05/06 11:52:52 [!] Get http://10.10.0.50/dvwa/about: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
/php.ini (Status: 200) [Size: 148]
2019/05/06 11:52:54 [!] Get http://10.10.0.50/dvwa/cmd: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
/robots (Status: 200) [Size: 26]
/robots.txt (Status: 200) [Size: 26]
2019/05/06 11:52:59 [!] Get http://10.10.0.50/dvwa/index: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2019/05/06 11:52:59 [!] Get http://10.10.0.50/dvwa/index.php: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2019/05/06 11:52:59 [!] Get http://10.10.0.50/dvwa/instructions: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2019/05/06 11:53:00 [!] Get http://10.10.0.50/dvwa/login: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2019/05/06 11:53:00 [!] Get http://10.10.0.50/dvwa/logout: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
/phpinfo.php (Status: 302) [Size: 0]
/phpinfo (Status: 302) [Size: 0]
/security (Status: 302) [Size: 0]
/setup (Status: 200) [Size: 3549]
=====================================================
2019/05/06 11:53:01 Finished
=====================================================

Usually, if something is zero bytes, it isn’t even worth looking into. It can save loads of time, especially when dealing with a large website or a large number of directories. If we only want specific status codes to be displayed, we can do that using the -s flag followed by the code we want.

~/gobuster# gobuster -u http://10.10.0.50/dvwa/ -w common.txt -s 200

=====================================================
Gobuster v2.0.1              OJ Reeves (@TheColonial)
=====================================================
[+] Mode         : dir
[+] Url/Domain   : http://10.10.0.50/dvwa/
[+] Threads      : 10
[+] Wordlist     : common.txt
[+] Status codes : 200
[+] Timeout      : 10s
=====================================================
2019/05/06 11:54:16 Starting gobuster
=====================================================
/README (Status: 200)
/favicon.ico (Status: 200)
/cmd (Status: 200)
/php.ini (Status: 200)
/login (Status: 200)
/robots (Status: 200)
/robots.txt (Status: 200)
/setup (Status: 200)
=====================================================
2019/05/06 11:54:27 Finished
=====================================================

Use commas to specify multiple codes.

~/gobuster# gobuster -u http://10.10.0.50/dvwa/ -w common.txt -s 200,301

=====================================================
Gobuster v2.0.1              OJ Reeves (@TheColonial)
=====================================================
[+] Mode         : dir
[+] Url/Domain   : http://10.10.0.50/dvwa/
[+] Threads      : 10
[+] Wordlist     : common.txt
[+] Status codes : 200,301
[+] Timeout      : 10s
=====================================================
2019/05/06 11:54:58 Starting gobuster
=====================================================
/README (Status: 200)
/config (Status: 301)
/docs (Status: 301)
/external (Status: 301)
/favicon.ico (Status: 200)
/cmd (Status: 200)
/php.ini (Status: 200)
/login (Status: 200)
/robots (Status: 200)
/robots.txt (Status: 200)
/setup (Status: 200)
=====================================================
2019/05/06 11:55:10 Finished
=====================================================

Let’s say we just wanted a quick way to view the directories, without the extra noise of the banner and status codes. Use the -q flag to hide the banner, and the -n flag to hide the status codes.

~/gobuster# gobuster -u http://10.10.0.50/dvwa/ -w common.txt -q -n

/.hta
/.htpasswd
/.htaccess
/README
/config
/docs
/external
/favicon.ico
/about
/cmd
/php.ini
/index
/index.php
/instructions
/logout
/robots
/robots.txt
/login
/phpinfo.php
/phpinfo
/setup
/security

Another useful feature is the ability to save the results to a file. Use the -o flag to specify the output file.

~/gobuster# gobuster -u http://10.10.0.50/dvwa/ -w common.txt -o results

Now the results are saved and can be viewed at a later time.

~/gobuster# cat results

/.hta (Status: 403)
/.htaccess (Status: 403)
/.htpasswd (Status: 403)
/README (Status: 200)
/config (Status: 301)
/docs (Status: 301)
/external (Status: 301)
/favicon.ico (Status: 200)
/about (Status: 302)
/cmd (Status: 200)
/php.ini (Status: 200)
/instructions (Status: 302)
/index (Status: 302)
/logout (Status: 302)
/index.php (Status: 302)
/login (Status: 200)
/robots (Status: 200)
/robots.txt (Status: 200)
/phpinfo (Status: 302)
/phpinfo.php (Status: 302)
/security (Status: 302)
/setup (Status: 200)

And that’s it.
Today, have fun uncovering some secrets on hackthebox boxes or other bounty-targeted websites, and stay tuned for the next interesting article.

Feel free to Subscribe for more content 🔔, clap 👏🏻 and share the article With anyone you’d like.