Christoph Csallner
Reverse Engineering Mobile Application User Interfaces With REMAUI
August 03, 2016 at 3:30pm
CSE 305
Abstract
When developing the user interface code of a mobile application, in practice a big gap exists between the digital conceptual drawings of graphic artists and working user interface code. Currently, programmers bridge this gap manually, by reimplementing the conceptual drawings in code, which is cumbersome and expensive. To bridge this gap, we introduce the first technique to automatically Reverse Engineer Mobile Application User Interfaces (REMAUI). On a given input bitmap REMAUI identifies user interface elements such as images, texts, containers, and lists, via computer vision and optical character recognition (OCR) techniques. In our experiments on 488 screenshots of over 100 popular third-party Android and iOS applications, REMAUI-generated user interfaces were similar to the originals, both pixel-by-pixel and in terms of their runtime user interface hierarchies. REMAUI’s average overall runtime on a standard desktop computer was 9 seconds.
Bio
Christoph Csallner is an Associate Professor in the Computer Science and Engineering Department at the University of Texas at Arlington (UTA). Before joining UTA, he worked for Google and Microsoft Research and received a Ph.D. degree from the Georgia Institute of Technology (Georgia Tech).
Dr. Csallner is working on problems that are related to software engineering. Specifically, his research areas include program analysis, automated bug finding, software security, and mobile software development.
Dr. Csallner has received three Best Paper Awards, i.e., at the 22nd IEEE/ACM International Conference on Automated Software Engineering (ASE 2007), at the 21st IEEE International Symposium on Software Reliability Engineering (ISSRE 2010), and at the 4th Program Protection and Reverse Engineering Workshop (PPREW-4).
Dr. Csallner has also received three ACM SIGSOFT Distinguished Paper Awards, i.e., at the 15th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2006), at the 21th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2012), and at the 30th IEEE/ACM International Conference on Automated Software Engineering (ASE 2015).