From time to time everyone has a need to download bunch of files from some website. Sometimes there exists one index where links to every file can be found. But sometimes not. Analysis of a website and/or figuring out a way the link is created (especially if it is something like http://some-cdn.io/directory/file-generator?param=5acae975-7784-e511-9412-b8ca3a5db7a1&ws=b0c491db-ae8b-e011-969d-0030487d8897&uid=66dba4ec-bb13-e211-a76f-f04da23e67f6&switch=1) could take months and success is not guaranteed. If this happens, downloading files manually is the only way to do it. But manual download can be optimized too.
Burp is a sort of swiss army knife of penetration tester. Its main function is intercepting HTTP(s) traffic through built in proxy. It allows to decrypt traffic of any website or even Android app to third-party server. So this way you could configure web browser to use burp proxy and simply view every file you want to save. This would be ideal solution for backing up any image gallery including facebook galleries that can be viewed only by logged in users and uses weird links to facebook CDNs. But there is one problem. Burp does not allow to export many files at once (or at least its free license does not allow it). Or to be exact uses its own format to store both HTTP request, response and lot of metadata, we simply do not need, but we need to have directory filled with images, right?
To obey the problem, I have written simple bash script that extracts plain data from that exported data file. It gets XML file exported by Burp and unpacks plain responses, each to separate file. Usage is very simple.
- At first export files you want to save from Burp’s Target tab by selecting them, clicking on Save selected items and save file as whatever.xml.
- Then you just have to start the script with
and optionally appending desired file extension as second parameter.
Note that files will be named with iterator starting on 1 and going up and sorted the same way Burp had them exported.
As usual repo is available on github.