This blog has moved

I decided to transfer the blog to my own server. I’m going to leave all the content here for some time (there are websites with links to my posts already). For all the new content, however, please follow the link: http://blog.codepainters.com.

Android emulator and the SIM card serial number

After publishing the post about changing the IMEI number, I was asked about modifying the SIM card serial number. Yes, it is perfectly possible, but requires a bit more investigation.

In general, the mechanism used is the same (i.e. AT commands sent to the emulated GSM modem). The telephony subsystem sends the following command to the emulated modem to retrieve the SIM card serial number (use any GSM modem manual for reference):

AT+CRSM=176,12258,0,0,10

The first numerical parameter denotes a command to execute on the SIM card, 176 stands for READ BINARY. Second parameter is the field identifier – Integrated Circuit Card identification, that is the SIM card serial number (for the full list of fields, called Elementary Files, refer to 3GPP TS 51.011 specification). Emulated modem responds with the following hard-coded response:

+CRSM: 144,0,98101430121181157002

The string of digits following the second comma is what we’re looking for, however, with every pair of digits swapped. All the SIM card commands and responses can be found in external/qemu/telephony/sim_card.c.

Changing the emulated SIM card serial number is now as easy changing the IMEI. To change the SIM card serial to e.g. 12345678901234567890, one has to:

  • backup the emulator binary ;)
  • open the binary in the hex editor
  • search for 98101430121181157002
  • replace it with 21436587092143658709 (remember about swapping)
Posted in Android. Tags: , . 2 Comments »

Cherokee: first impression

I’ve been using Apache for many years now, thus recently I’ve made a decision to try some of the “new” web servers. One of my hosts runs lighttpd for quite some time without any problems, so this time I wanted to try something else. Cherokee has caught my attention, thanks to it’s web-based configuration utility – it is intuitive enough to go without reading the documentation really.
Indeed, configuring my site has taken about 15 minutes – HTTP to HTTPS redirection, WSGI application, PHP5 – all up and running. But… I’ve noticed that HTTP to HTTPS redirection fails occasionally for no obvious reason. It quickly turned out that the server’s worker process dies of segmentation fault. Source code investigation led to a quick patch and an issue reported. Not really a perfect first impression, huh? Time to try nginx, I guess… ;)

Django deployment, FastCGI and UNIX-domain sockets

For various (mostly security) reasons I prefer to deploy Django applications as separate processes, with web server and actual application running under different UIDs, communicating with each other via FastCGI or SCGI socket. This way application is safer from e.g. vulnerable PHP scripts running within the web server process – it’s pretty common for the attackers to use PHP vulnerabilities to read files accessible by the web server itself (for smaller sites it’s normal to use a single web server installation for many different applications).

For security reasons I also prefer to use UNIX-domain sockets over TCP sockets (only if web server and the application are deployed on the same machine, obviously). With TCP sockets in use, any process may connect to the FastCGI/SCGI socket and effectively bypass the web server running in front of the application. What if e.g. the web server itself is used to perform user authentication and authorization? With UNIX-domain sockets one can protect the application by means of regular UNIX permission mode.

Such deployment model is perfectly feasible with Django, as it uses flup internally for a WSGI to SCGI/FastCGI bridge, deployment procedure is described in the Django documentation. Following the manual, application process can be started e.g. with the following command line (note that I do not daemonize the application, as I prefer to use e.g. daemontools to supervise the application process):

./manage.py runfcgi protocol=scgi socket=~/django/testapp.sock daemonize=false

Unfortunately, the socket gets created with permission mode that do not allow any user other than the application process owner to write to the socket:

czajnik@lapsko:~/django$ ls -l testapp.sock 
srwxr-xr-x 1 czajnik czajnik 0 2010-08-29 20:49 testapp.sock

Obviously, this is not the way to go – web server will not be able to write to such a socket. One way to overcome the problem is to manually change the permissions mode and/or the uid/gid of the socket after the application is started, but this is far from perfect.

Digging in the source code I’ve realized that the flup module itself allows for specifying the umask value used while creating the socket, but this possibility is not used by Django. I’ve modified the code to pass 0 as socket creation umask, effectively making the socket accessible for every process:

diff -ruN Django-1.2.1.orig/django/core/servers/fastcgi.py Django-1.2.1/django/core/servers/fastcgi.py
--- Django-1.2.1.orig/django/core/servers/fastcgi.py	2010-08-29 21:00:37.000000000 +0200
+++ Django-1.2.1/django/core/servers/fastcgi.py	2010-08-29 21:06:32.000000000 +0200
@@ -177,6 +177,7 @@
         fp.write("%d\n" % os.getpid())
         fp.close()
 
+    wsgi_opts['umask'] = 0 
     WSGIServer(WSGIHandler(), **wsgi_opts).run()
 
 if __name__ == '__main__':

To control the accessibility I place the socket alone inside a designated directory and use directory permission mode to control who can access the socket. In my particular case the web server runs under www-data user/group. Directory where SCGI socket is created has the following permission mode:

czajnik@lapsko:~/django$ ls -ld socket
drwxr-x--- 2 czajnik www-data 4096 2010-08-29 21:10 socket
czajnik@lapsko:~/django$ ls -l socket/testapp.sock 
srwxrwxrwx 1 czajnik czajnik 0 2010-08-29 21:10 socket/testapp.sock

This works, but is not really a nice solution. Ideally, Django should allow for settings any socket permission mode from a command line. I’d even more prefer a model where application is started with UID 0, creates the socket owned by any configured UID/GID/permission mode, and then drops the root privileges.

Perhaps the best option is to drop flup altogether and switch to another WSGI host – uWSGI is an interesting option. It allows for chrooting, setting arbitrary socket permission mode, changing process UID and GID (if started as root), and much much more. However it doesn’t seem very popular yet, I’m yet to evaluate its quality.

As usual, any suggestions welcome!

Symbian: separate builds for different S60 editions

Symbian’s build environment was never really my favorite one. Recently I was working on a project using different APIs on S60 3rd, 3rd FP1, 3rd FP2 and 5th editions. Typical Symbian project contains a single bld.inf file and a corresponding .mmp file, so it is not immediately obvious how to compile conditionally, based on exact S60 release you’re targeting.

I’ve found a solution at Forum Nokia Wiki, but it requires adding a custom file to the SDK, I’m not really happy about it. However a closer look at the build process reveals that bld.inf is preprocessed using the following call:

cpp.EXE -undef -nostdinc -+  -I "C:\Symbian\9.2\S60_3rd_FP1\epoc32\include" -I . 
-I "C:\Users\czajnik\work\test\TestApp\group\"  -I "C:\Symbian\9.2\S60_3rd_FP1\epoc32\include\variant" 
-include "C:\Symbian\9.2\S60_3rd_FP1\epoc32\include\variant\Symbian_OS_v9.2.hrh"
"C:\Users\czajnik\work\test\TestApp\group\BLD.INF"

The most important observation here is the inclusion of Symbian_OS_v9.2.hrh. Every SDK release comes with such a file, containing a bunch of #define macros denoting features available on a particular platform. With a little help from grep, sort, sed and diff I’ve eventually got the following bld.inf file:

PRJ_PLATFORMS
WINSCW ARMV5 GCCE

PRJ_MMPFILES
gnumakefile Icons_scalable_dc.mk

#ifdef SYMBIAN_C32_SERCOMMS_V2
TestApp_S60_5th.mmp
#else 
#ifdef SYMBIAN_APPARC_APPINFO_CACHE
TestApp_S60_3rd_FP2.mmp
#else
#ifdef SYMBIAN_FLEXIBLE_ALARM
TestApp_S60_3rd_FP1.mmp
#else
TestApp_S60_3rd.mmp
#endif
#endif
#endif

Works like a charm :)

Posted in Symbian. Tags: , . 2 Comments »

Android’s HTTPS implementation slow under debugger

Update: I’ve just discovered that the problem described below only affects Android 1.5 (which is the target platform for my project). Thus another solution is to use AVD configured with Android 1.6 or higher for debugging, and use version 1.5 otherwise to ensure compatibility.

The project I’m working on now communicates with server via custom protocol on top of HTTP and HTTPS (using org.apache.http.client.HttpClient as a HTTP/HTTPS implementation provider). Recently I’ve observed interesting problem of HTTPS requests taking ages to complete if application is executing under debugger control. It makes debugging the code quite frustrating.

To investigate the problem let’s use the following code snippet:

		HttpResponse response = null;
		HttpClient httpClient = new DefaultHttpClient();
		HttpContext localContext = new BasicHttpContext();

		Log.d("--------", "1. Sending http request");
		try {
			response = httpClient.execute(new HttpGet("http://www.apache-ssl.org/"), localContext);
		} catch (Exception e) {
			Log.d("--------", "Request failed", e);
		}
		Log.d("--------", "2. Request finished, status = " + response.getStatusLine().getStatusCode());

		Log.d("--------", "3. Sending https request");
		try {
			response = httpClient.execute(new HttpGet("https://www.apache-ssl.org/"), localContext);
		} catch (Exception e) {
			Log.d("--------", "Request failed", e);
		}
		Log.d("--------", "4. Request finished, status = " + response.getStatusLine().getStatusCode());

This code snippet has been executed on Android emulator (I’ve no device at hand at the moment to try it), platform version 1.5, running on Windows Vista. To make results more dependable the code has been executed five times and average times have been calculated. Sure, it’s not a serious benchmark of any kind, but we only want to have a rough idea what’s going on, right?

A bit of log post-processing reveals the following average times:

  • HTTP, no debugger: 0.80s
  • HTTPS, no debugger: 3.32s
  • HTTP, with debugger: 3.15s
  • HTTPS, with debugger: 142.16s

HTTPS request takes ~45 times longer to complete compared to HTTP, with debugger attached. Without debugger the factor is ~4, which is acceptable.

For the time being I have no explanation for such a big difference. Looking at the stack trace of the thread paused while performing the HTTPS request shows a lot of classes from Bouncy Castle project, as well as classes from org.apache.harmony.xnet.provider.jsse package – JNI interface to OpenSSL library. Some profiling would be necessary…

In case of my application (which always connects to the same, HTTPS-only server) I’ve decided to use the good old stunnel running on my Linux box. It is enough to change the URL in the example above to http://linuxbox:8888 and start stunnel as follows:

czajnik@czajnik:~$ sudo stunnel -c -d 8888 -r www.apache-ssl.org:443

However, it only works for hosts that accept Host: HTTP header value different from actual host name. Ideally, we should be able to leave the original URL and only redirect the socket connection to a different IP and port. And in fact we are, using HttpClient’s proxy settings:

		HttpParams params = httpClient.getParams();
		params.setParameter(ConnRoutePNames.DEFAULT_PROXY, new HttpHost("linuxbox", 8888));

With this setting HttpClient connects to our stunnel process, but still uses original URL – exactly what we need. Of course, URL has to be changed from https:// to http://, as the connection between HttpClient and stunnel is obviously not encrypted. Also, resist the temptation to use localhost ;)

First Android platform build

After playing with Android SDK for a while, now it’s time to try to build the platform. Unfortunately, Android documentation is not the best one I’ve ever seen, and some research is needed. Do they try to promote their web search engine this way? ;)

Fist step is to clone the repository, as described in this guide (I’m going to use donut branch). It goes smoothly, but takes a lot of time (my Internet link goes up to 120kB/s only):

czajnik@czajnik:~/mydroid$ repo init -u git://android.git.kernel.org/platform/manifest.git -bdonut
czajnik@czajnik:~/mydroid$ time repo sync

Next step would be to select the JDK version (android seem to require JDK 1.5), and set basic environment by sourcing one of the shell scripts:

czajnik@czajnik:~$ export JAVA_HOME=/usr/share/jdk1.5.0_22/
czajnik@czajnik:~$ export ANDROID_JAVA_HOME=$JAVA_HOME
czajnik@czajnik:~$ export PATH=/usr/share/jdk1.5.0_22/bin/:$PATH
czajnik@czajnik:~$ cd mydroid
czajnik@czajnik:~/mydroid$ . build/envsetup.sh
including vendor/aosp/vendorsetup.sh
czajnik@czajnik:~/mydroid$

Now we need to select a particular build configuration to be used. On this page (which is a must-read anyway) they mention choosecombo, let’s use it to select generic device debug build (there is also a tool called lunch which allows to select one of the common build types, call it without any parameters to get the list of possibilities and select one interactively):

czajnik@czajnik:~/mydroid$ choosecombo
Build for the simulator or the device?
     1. Device
     2. Simulator

Which would you like? [1]


Build type choices are:
     1. release
     2. debug

Which would you like? [1] 2


Which product would you like? [generic]


Variant choices are:
     1. user
     2. userdebug
     3. eng
Which would you like? [eng]

============================================
PLATFORM_VERSION_CODENAME=REL
PLATFORM_VERSION=1.6
TARGET_PRODUCT=generic
TARGET_BUILD_VARIANT=eng
TARGET_SIMULATOR=false
TARGET_BUILD_TYPE=debug
TARGET_ARCH=arm
HOST_ARCH=x86
HOST_OS=linux
HOST_BUILD_TYPE=release
BUILD_ID=Donut
============================================

Finally we reached the point where we can start actual build by issuing make -j2. First attempt fails with the following error message:

development/emulator/qtools/trace_reader.cpp: In function ‘char* ExtractDexPathFromMmap(const char*)’:
development/emulator/qtools/trace_reader.cpp:1012: error: invalid conversion from ‘const char*’ to ‘char*’
development/emulator/qtools/trace_reader.cpp:1015: error: invalid conversion from ‘const char*’ to ‘char*’

It’s time for a first useful hint – if you want to see the exact commands executed, invoke make showcommands. Armed with this weapon it’s pretty easy to track the problem. It seems that trace_reader.cpp is expecting C-like string function prototypes in string.h, while string.h on my system exports a C++-compatible declaration. In this particular case:

// C way
char *rindex(const char *s, int c);
// C++ way (overload)
char *rindex (char *s, int c);
const char *rindex (const char *s, int c);

My Linux distribution (Kubuntu 9.10) has switched from glibc to eglibc, included /usr/include/string.h file uses the second type of declaration if used inside a C++ source file, causing the compilation failure. Look at this fragment of string.h:

/* Tell the caller that we provide correct C++ prototypes.  */
#if defined __cplusplus && __GNUC_PREREQ (4, 4)
# define __CORRECT_ISO_CPP_STRING_H_PROTO
#endif

Obviously, there are at least the following solutions:

  • fix Android code (it’s a C++ file, it should expect C++ prototypes)
  • use a system that uses regular glibc
  • use GCC compiler version lower than 4.4
  • comment out #define line above

Of course, I’ve chosen the last option. And that was enough, after almost 80 minutes I’ve got my first own Android image. Now it’s time to hack it a bit – stay tuned!

Posted in Android. Tags: . 2 Comments »
Follow

Get every new post delivered to your Inbox.