Until something like 1 or 2 months ago indeed it did not work in Android 4.0 due to some JNI issues. But I already fixed those. The VM image is quite old already, so "svn update" the Lazarus source tree that comes with it and try again.
Sometimes ant indeed reuses old pieces so deleting /bin helps sometimes. Lately I develop in Windows where I use a .bat script which calls directly javac, dex, etc, and I haven't had this issue. Most of the time I had issues lately I actually had forgotten to build something, or the Java building failed, or the Pacal build failed, or I was building the Pascal side for the desktop instead of for Android...
liblclapp.so comes from the directory android/lib/arm-eabi/ (or similar), but I think that maybe ant might reuse it from the previously built APK, so idea in which ocasion. Probably they made something to speed up building when you build multiple times without changing much and caused this issues.
About slowness, indeed it could be faster, but it is good enough for my usage. Improving the speed is essentially optimizing the routines in lcl/lazcanvas.pas for the pixel format of Android, but this is a rather boring job. It's in my ToDo, but not with a high priority. Of course that patches are welcome.
Right now I am working on networking support and sqlite support. Also just wrote this new thing: http://wiki.lazarus.freepascal.org/Custom_Drawn_Interface/Android#Using_the_Native_ListView_Dialog