Recursebuster is a tool that you can use run a recursive-HEAD-spider-assisted search with a single thread using a specific wordlist. Results will print to screen, but more importantly, will be written to a file ‘busted.txt’.

This tool will provide the following features:

• HEAD Based Checks – For servers the support it, HEAD based checks speed up content discovery considerably, since no body is required to be transferred.
• Recursion – When a directory is identified, it gets added to the queue to be brute-forced. By default, one directory is brute-forced at a time, however you can ‘cancel’ a directory interactively by hitting ‘ctrl+x’ if in UI mode.
• Spider Assistance – Some basic checks are done to look for links within the HTML response. This will whitelist and exclude certain content.
• Speed – the tool will make the scan faster.
• Proxy settings – Possible to use it with a proxy settings to avoid being blocked by web application security tools.

You can read more and download the tool over here: https://github.com/c-sto/recursebuster