Wget with SSL/TLS support for Android

By | Published September 12, 2016
wget dependency tree

wget dependency tree

Lately I have tried to download some file from a website to my Android smartphone. Simple thing, yeah? Well, not really. Unfortunately mobile browser developers removed many features from their mobile distributions. One of them is a possibility of downloading random page to disk as is. Instead (this is the case at least with Mozilla’s product) they are forcing “Download as PDF” feature. I had a bit of luck, because the file I was trying to download was MP4 movie, which is downloadable, maybe not in an intuitive way, but it is. But before I have found that feature hidden in a player’s context menu, I tried another solution – wget. Since I am great fan of terminals, I have busybox installed on my phone. Those of you, who know what exactly is busybox should know that this is set of lightweight variants of most standard UNIX tools. So, if they are lightweight, they had to cut some part of tool functionality, right? And in case of my busybox’s wget, they cut HTTPS support. And today, it is more likely to encounter site which is only HTTPS than one that is only HTTP, at least when talking about popular sites. So I had to get my own distribution of wget, that will not be such constrained one.

Not to get you bored too much, here you can find binary distribution of what I achieved to compile. It was compiled for ARMv7 platform using NDKr12b and API level 24 (Nougat), so it will probably not work on most of current Android phones, but if you read later, it is probably working on your device or even is outdated. If you are interested in recompiling binaries yourself, you can find detailed how-to in the next part of this article.

Dependencies

Before compiling wget itself, you have to have whole bunch of its dependencies. But at first, you of course need Android compiler. It is distributed as part of NDK and I won’t describe its installation here. Sources of every program compiled here can be grabbed from its official sites (list at the end of this post). The only exception is libtasn1, which required few hacks to be done to make it compile with Android bionic libc. Its source, ported to Android can be get from my github repo.

Let’s start with programs that does not depend on anything. For all projects, the procedure is more or less the same and can be described with simplified bash script:

tar -zxvf program-1.00.tar.gz
mkdir build
mkdir install
cd build
CC=arm-linux-androideabi-gcc AR=arm-linux-androideabi-ar RANLIB=arm-linux-androideabi-ranlib CFLAGS=-pie \
    ../program-1.00/configure --host=arm-linux --prefix=/data/local/root
make
make install DESTDIR=$(dirname `pwd`)/install/
cd ../install
tar -zcvf program.tar.gz *

gmp, libidn and libffi

For these three programs, the procedure above should work without any modification.

nettle

Since nettle depends on gmp, it has to be configured with paths to gmp binaries and headers in its CFLAGS and LDFLAGS variables. They should look like this:

CFLAGS="-pie -I`pwd`/../../gmp/install/data/local/root/include"
LDFLAGS="-L`pwd`/../../gmp/install/data/local/root/lib"

when invoking configure script.

libtasn1

This was the hardest part for me, but should go smoothly now. Script below should do the work correctly:

git clone git@github.com:v3l0c1r4pt0r/android_external_libtasn1.git
mkdir build
mkdir install
cd build
CC=arm-linux-androideabi-gcc AR=arm-linux-androideabi-ar RANLIB=arm-linux-androideabi-ranlib CFLAGS=-pie \
    ../libtasn1/configure --host=arm-linux --prefix=/data/local/root --disable-doc
make
make install DESTDIR=$(dirname `pwd`)/install/
cd ../install
tar -zcvf libtasn1.tar.gz

p11-kit

This is the last dependency of gnutls which is the only, but very important dependency of wget. Just embedding libtasn1 and libffi should do the job well.

CFLAGS="-pie -I`pwd`/../../libtasn1/install/data/local/root/include"
LDFLAGS="-L`pwd`/../../libtasn1/install/data/local/root/lib -L`pwd`/../../libffi/install/data/local/root/lib"

Notice that libffi has no headers, so we add it just to CFLAGS here!

gnutls

This one was more complicated than the rest. As I mentioned above, it is very important to wget functionality. However wget’s dependency on it could probably be turned off, we would not have TLS support then. When compiling it I had some problems that seemed to be serious. There were a few errors while making it, so I had to call make twice and even though it failed. Despite that it seem to work after make install, which obviously failed too. In my case following script did the job:

mkdir build
mkdir install
cd build
CC=arm-linux-androideabi-gcc AR=arm-linux-androideabi-ar RANLIB=arm-linux-androideabi-ranlib \
    CFLAGS="-pie -I`pwd`/../../gmp/install/data/local/root/include -I`pwd`/../../nettle/install/data/local/root/include -I`pwd`/../../libtasn1/install/data/local/root/include -I`pwd`/../../libidn/install/data/local/root/include -I`pwd`/../../p11-kit/install/data/local/root/include" \
    LDFLAGS="-L`pwd`/../../gmp/install/data/local/root/lib -L`pwd`/../../nettle/install/data/local/root/lib -L`pwd`/../../libtasn1/install/data/local/root/lib -L`pwd`/../../libidn/install/data/local/root/lib -L`pwd`/../../p11-kit/install/data/local/root/lib" \
    ../gnutls-3.4.9/configure --host=arm-linux --prefix=/data/local/root --disable-cxx --disable-tools
make || make
make install DESTDIR=$(dirname `pwd`)/install/ || true
cd ../install
tar -zcvf file.tar.gz *

Compilation

Since we should now have all dependencies compiled, we can try compiling wget itself. The procedure here is the same as with dependencies. We just have to pass path to gnutls. And then standard configure, make, make install should work. However if your NDK installation is fairly new and you were not hacking it before, you most likely don’t have <sys/fcntl.h> header and make should complain about that. Luckily Android itself have this header present, but for reason unknown it is kept in include directory directly. To make wget, and any other program that uses it, compile you can just point “sys/” instance to <fcntl.h> with symlink or do something like that:

echo "#include <fcntl.h>" > $TOOLCHAIN/sysroot/usr/include/sys/fcntl.h

where $TOOLCHAIN/sysroot is path at which you have your headers placed. Depending on tutorial you were using for making it work it may have different structure.

Installation

All commands I presented above implies that you have your custom-compiled binaries in “/data/local/root”. I made it that way to have clear separation between default and busybox binaries. If you want to have them somewhere else, you should pass it to configure scripts of all programs you are compiling. After successful compilation of all tools, I have made single tarball containing all compilation output (this file’s link was placed above). Its content can be installed into Android by typing

tar -zxvf wget-with-deps.tar.gz -C/

using adb shell or terminal emulator.

Sources

Below you can find links to sources of all programs nedded to follow this tutorial.

Posted in Uncategorized | Leave a comment

Simple extraction of content from HTTP request

By | Published August 21, 2016

When writing a script for extraction of data exported to XML from burp, I was forced to make one very dirty hack on extracting body of HTTP request. It seems that there is no easy way of getting just the data, especially if the message transferred is binary file, i.e. image or executable. If it is simply text, it might probably done with some perl regexp wizardry. However it would be nice either way. The hack I mentioned was creating netcat server and piping whole request into it and then downloading it with help of wget. That way it works and takes only two straightforward lines of code, but it is still a hack. You can see it here.

To not be forced to do hacks like that in future I had a plan to make very simple utility program that will just delete HTTP header and print rest of input to stdout. Well, I decided to do this just after finishing the mentioned script, so about half year ago. Now I have found some time to do this and the result is very small utility written in C that does just that and by the way is able to parse the header itself and provides a way to be used easily by the user. The latter is done by spawning child process of default shell with header values set in its environment, so it may not be the nicest way to do this, but for it was the fastest to implement and should be fairly easy to work with. If you know better way that will result in similar behavior I will be graceful if you leave me a comment here.

Posted in Uncategorized | Tagged , , | Leave a comment

Method for intercepting (lot of) files from website using burp

By | Published February 26, 2016
Burp exporting requests to file

Burp exporting requests to file

From time to time everyone has a need to download bunch of files from some website. Sometimes there exists one index where links to every file can be found. But sometimes not. Analysis of a website and/or figuring out a way the link is created (especially if it is something like http://some-cdn.io/directory/file-generator?param=5acae975-7784-e511-9412-b8ca3a5db7a1&ws=b0c491db-ae8b-e011-969d-0030487d8897&uid=66dba4ec-bb13-e211-a76f-f04da23e67f6&switch=1) could take months and success is not guaranteed. If this happens, downloading files manually is the only way to do it. But manual download can be optimized too.

Burp way

Burp is a sort of swiss army knife of penetration tester. Its main function is intercepting HTTP(s) traffic through built in proxy. It allows to decrypt traffic of any website or even Android app to third-party server. So this way you could configure web browser to use burp proxy and simply view every file you want to save. This would be ideal solution for backing up any image gallery including facebook galleries that can be viewed only by logged in users and uses weird links to facebook CDNs. But there is one problem. Burp does not allow to export many files at once (or at least its free license does not allow it). Or to be exact uses its own format to store both HTTP request, response and lot of metadata, we simply do not need, but we need to have directory filled with images, right?

Solution

To obey the problem, I have written simple bash script that extracts plain data from that exported data file. It gets XML file exported by Burp and unpacks plain responses, each to separate file. Usage is very simple.

  1. At first export files you want to save from Burp’s Target tab by selecting them, clicking on Save selected items and save file as whatever.xml.
  2. Then you just have to start the script with 
    ./xtract-burp.sh whatever.xml

    and optionally appending desired file extension as second parameter.

Note that files will be named with iterator starting on 1 and going up and sorted the same way Burp had them exported.

As usual repo is available on github.

Posted in Tutorials | Tagged , , , | Leave a comment

HDCB – new way of analysing binary files under Linux

By | Published February 10, 2016

As any observer of my projects spotted, most of the biggest projects I do involves binary file analysis. Currently I am working on another one that requires such analysis.
Unfortunately such analysis is not an easy task and anything that will ease this or speed it up is appreciated. Of course there are some tools that will make it a bit easier. One of them is hexdump. Even IDA Pro can make it easier a bit. Despite them I always felt that something is missing here. When creating xSDM and delz utils, I was using hexdump output with LibreOffice document to mark different structure members with different colors. It worked, but selecting 100-byte buffer line by line was just wasting precious time.

SDC file analyzed by HDCB script

Solution

So why not automate whole process? What we really need here is just hexdump output and terminal emulator with color support. And that’s why I’ve made HDCB – HexDump Coloring Book. Basically it is just extension to bash scripting language. Goal was to create simple script that will hide as much of its internals from end-user and allow user to just start it will his shell using old good ./scriptname.ext and that’s it. HDCB is masked as if it was standalone scripting language. It uses shebang, known from bash or python scripts to let user shell know what interpreter to use (#!/usr/bin/env hdcb). Those who are python programmers should recognize usage of env binary.

In fact it is just simple extension to bash language, so users are still able to utilize any features available in bash. Main extensions are two commands: one (define) for defining variables and the other (use) for defining field or array of that defined type. Such scripts should be started with just one argument – file that is meant to be hexdumped and analyzed.

Internals

Bash scripts are just some kind of a cover of real program. Main core of the program is colour utility. It gets unlimited number of parameters grouped in groups of four. They are in order: offset of byte being colored, length of the field, background and foreground colors. As standard input, hexdump output (in fact only hexdump -C or hexdump -Cv are supported) is provided. Program colors the hexdump with rules provided as arguments. This architecture allows clever hacker to build that cover mentioned in virtually any programming language.

Downloads, documentation and more

As usual, program is available on my Github profile. Sources are provided on GPLv3 license so you are free to contribute to the project and you are strongly encouraged to do so or make proposals of a new functions. Program is meant to be expanded according to my future needs, but I will try to implement any good idea. Whole documentation, installation instructions and the like are also available on Github.

Posted in Uncategorized | Tagged , , | Leave a comment

Airlive WN-151ARM UART pinout & root access

By | Published November 24, 2015
airlive-pinout

Airlive WN-151ARM pinout

For curious ones. Here is pinout of serial connection. As you can see UART pins are at J4 header (should have pin 4 labeled and 1 be square).

J4 header
Num. Function
1 VCC
2 RX
3 TX
4 GND

Edit: Oh, and one more thing: goldpin header, you see in the picture is soldered by me, so do not be surprised if you have to hold wires all the time during the transmission.

Root access

There is also possibility to gain root access without removing the cover and possibly voiding the warranty. You have to connect to router’s AP and enter

http://192.168.1.254/system_command.htm

into your browser (panel authentication required). Now you can execute any command you want with root privileges! So let’s type

/usr/sbin/utelnetd -d &

into Console command field and press Execute button. If everything went well, you should now be able to connect to your router using telnet at its default TCP port 23. After that you should see BusyBox banner and command prompt.

It is worth noting that this hidden console cannot be accessed by unauthorized person, so only router administrator can use this (in theory, in practice there are surely a lot of routers using default credentials and security of httpd binary is unknown).

Posted in Just knowledge | Tagged , , , , , , , | Leave a comment

SDC file format description – Errata

By | Published November 6, 2015

Last year, I published a program for Microsoft Dreamspark’s SDC file decryption. Soon after that I wrote article about SDC file format and its analysis. Now it’s time to complete the description with newest data.

This article wouldn’t be written if not the contribution of GitHub’s user @halorhhr who spotted multi-file SDC container and let me know on project’s page. Thanks!

When writing that post year ago, I had no idea what multi-file container really looks like. Any suspicions could not then be confirmed, because it seemed that these files simply where not used in the wild. A days ago situation changed. I got a working sample of multi-file container so I was able to start its analysis.

Real container format

sdc-format

SDC files with different signatures

After quick analysis, I knew that I was wrong with my suspicions. Filename length and encrypted filename strings are not part of a file description. In fact they are placed after them and filename is concatenated string of all filenames (including trailing null-byte). So to sum up filename of n-th element starts at file[n].filename_offset and ends just like any other c-style string.

Whole header structure is like on the sample header on the right. Note that all headers beside 0xb3 one has been already decrypted for readability. In real header the only unencrypted field is header size at the very beginning of the file. 0xb3 sample has unencrypted header and header size is not present in a file. However file name is encrypted in some way, I haven’t figured out as of now. Encryption method is blowfish-compat (the difference between this and blowfish is ciphertext endiannes). Filenames are encrypted once again.

After header, all other data is XORed using key from EDV string and then deflated, so before reading them, you have to inflate and XOR again. Format of data in 0xb3 version is still unknown, however analysis of compressed and file size hints that it may be stored the same way. It is important to note that depending on file signature different configuration of deflater may be needed. It is now known that files older than 0xd1 header, which appears to be newest (because only this one supports files greater than 4 GiB) need to have deflater initialized with

inflateInit2_(&stream,-15,ZLIB_VERSION,(int)sizeof(z_stream));

or equivalent.

Unknowns

This errata does not contain all information needed to support all variations of SDC files. Beside unknowns I mentioned above, there is another variation that uses 0xc4 signature and which I had no sample of. The only trace of its existence is condition in SDM code. Because of that I cannot write support for that type of file. There is also possibility of multi-file containers having 0xb5 or 0xb3 signatures existence. That type of files seems to appear only lately, but it is probable that it existed in the past. Because of having no samples of them I cannot verify that xSDM properly handles them.

So if you have sample of any of variations mentioned here, just send them to me at my email address: v3l0c1r4pt0r at gmail dot com or if you suppose it may be illegal in your country, just send me SDX link or any other hint for me how can I find them.

Other way?

Few days ago, after I started writing this post Github user @adiantek let me know in issues that there is a method to obey SDM in Dreamspark download process. To download plain, unencrypted file you just have to replace ‘dfc=1‘ to ‘sdm=0‘ in a link Dreamspark provided in SDX file. If it true that it works in every file Dreamspark provides, my xSDM project would be obsolete now. However, because Microsoft’s intentions when creating this backdoor (it seems to be created just for debugging) are unknown, I will continue to support the project and fix any future bugs I will be aware of. But now it seems that this project will start to be just proof of concept for curious hackers and will start to slowly die.

Nevertheless, if you have something that might help me or anyone who may be interested in SDC format in future, just let me know somehow, so it will be available somewhere on the internet.

Posted in Uncategorized | Tagged , , , , | Leave a comment

imeitool – validate, generate and find out information about IMEI number easily

By | Published September 15, 2015

I just pushed imeitool to my Github. imeitool is small utility that can do few useful operations on IMEI numbers. It can check if number given is valid IMEI number, find information in its databases about given IMEI or TAC (Type Allocation Code, usually first 8 digits of IMEI) and generate fake IMEI based on conditions provided by user.

The reason this program was created should appear on blog soon. There is a possibility for anyone to contribute to imeitool’s db, so if you want to help, more info can be found on project’s README file.

Posted in Uncategorized | Tagged , , | Leave a comment

Changing your MAC address on new Android devices

By | Published September 11, 2015

Few months ago I flashed my Android device with Cyanogenmod-based LegacyXperia ROM. Before that I was using anonymization script, I created that was able to change MAC address, block some Play Store’s services sending unknown binary data and masking device hostname (used by DHCP). Obviously after upgrade, the script stopped working, so I tried to make it work again.

Apparently newer Android systems changed its way of turning wifi on and off (before, MAC changing was just invoking ip command), so now wlan0 interface is not present when it is off and after turning it on device will most likely authenticate to any known network, effectively leaking its HW address. Because of that I needed to find another way.

A bit of research and compiling iw tool later, I found out that when wifi is being turned off by GUI, wifi card kernel module is removed from kernel, so card’s interface as well as its physical device disappear from system. So the first step to make it work is to insert module back (wl12xx_sdio in my case, it seems to be very popular card on Android devices so probably you have the same). Inserting the module, however generates another problem: GUI cannot manipulate wifi now, so we have to do everything manually. The next steps are to invoke wpa_supplicant to authenticate to network and starting DHCP client daemon.

Doing this that way allows us to do one additional thing, increasing our privacy. Since we have to start supplicant ourselves, we can provide custom config to it. And if we have to write script anyway, we could have separate scripts for every network we know. Then we will be (almost) sure that we are connecting to the network we want.

The connecting script is available from my new repository with Android scripts for increasing privacy. I hope there will be more of them in future. As I have written in first paragraph, there are few other things to do. Or maybe there is something more I do not know about, yet…

PS: this script was tested only on Cyanogenmod 10.2 (based on Android 4.3) and may not work on new (5.*) systems.

Posted in Uncategorized | Tagged , , , , , , | Leave a comment

Decoding Aztec code from polish vehicle registration certificate

By | Published August 1, 2015

About a year ago I interested in mysterious 2D code placed in my car’s registration certificate. After quick research on Google it turned out to be even more mysterious because nobody knew how to decode it. There was even no official document like act or regulation that describes the code somehow. People knew that the code is Aztec code and that’s it. Some companies shared web and Android apps to decode this. And all of them was sending base64 to some server and receive decoded data.

Of course for me it wasn’t rewarding so I started my research on it. After initially scanning the code I’ve seen long string that I immediately recognized as base64. The real fun started after that, because stream I’ve got after that was so strange that at first I had no idea what to do. Upon closer examination it was clear that this data is not damaged but encoded in somewhat strange way. Few days later I was almost sure that this is not encoding but rather compression, because some unique parts of stream was easily readable by human. About a month of learning about compression, looking for even most exotic decompression tools and I was left with almost nothing. I had only weak guess on how decompression parameters could be encoded. I gave up…

Polish vehicle registration certificate (source: pwpw.pl)

Polish vehicle registration certificate (source: pwpw.pl)

About a year later I tried one more time. This time I was a bit more lucky. I found a program that decodes the code. Again. But this time was different. I shut down my network connection to make sure. And it worked! So now a bit of reverse engineering and it’s done. I will skip any details because I do not want to piss off the company which created this, even though I was right and I HAD right to do this.

As usual the source code is available on my Github profile. There is also a bit more information about whole scanning/decoding process. If you like to know more technical details about the algorithm or how to decode the data, everything can be found in README file in the repo.

Posted in Uncategorized | Tagged , , | 9 Comments

Kernel module for ST7565 based displays

By | Published January 27, 2015

Today I pushed kernel module driving ST7565 based displays to my github. I wanted to write kernel module some time ago and st7565 was first thing i thought of. Unfortunately I hadn’t enough motivation and when I got some I burnt my display 🙁 so it wasn’t done. This autumn I had other chance to make it real, because I started embedded systems course on university. And there it is: I’ve got working kernel module that can handle graphic LCD.

The module itself is fairly universal. Nevertheless I don’t know any existing gLCD driver that was included to the mainline kernel so have no idea how that thing should look. Because of that I implemented it as char device. It is storing bare pixel data so there aren’t any sophisticated functions to draw rectangle or something. Its advantages are that it isn’t limiting applications the module can be used to and make any userspace handling function very easy to write. It is also worth to note that the module is providing possibility to read data from buffer which is impossible in the module itself.

Beside that basic functionality it allows to switch off backlight and change brightness using sysfs attributes. The module is also portable thanks to possibility to change CS, A0, RST and backlight pin it is using so you are not tied to the ones I used and reconfiguration does not force you to recompile but just reload the module with different parameters. The exact scheme of the connection more or less is the same as on my previous approach to playing with that display.

PS: there is also new revision of my tool for font-making for that display. Now it is able to create any picture as long as its height is divisible by 8 (it is just simplification, made because of method of storing pixels). You can always implement algorithm that is OR-ing the picture with part of data that is already on display (of course using my module 🙂 ). It is still available on Gist.

Posted in Uncategorized | Tagged , , , , , , | Leave a comment
« Older