08-runtime-and-performance-analytics.Rmd

# Runtime and Performance Analytics

In this chapter, we discuss the field of performance and runtime analytics. This chapter however
does not cover the entire field because it is too broad. Using Kitchenham’s method
[@kitchenham2004procedures], we have narrowed down the scope of this survey.

For inspiration, we explored five recent papers on runtime and performance analytics
published at top conferences. These five were selected because the papers discuss the software
side of performance and runtime analytics which is more consistent with the scope of this book.
However, focussing on only software, the field is still very broad. We have chosen to focus on the 
field of performance vs. energy consumption. This choice was made due to it being a very 
contemporary and thriving domain within runtime and performance analysis.

## Introduction
Energy consumption is an important factor in the day-to-day usage of software. Especially in the
field of software development for mobile devices, considering energy consumption determines the 
battery-life and limites the usage time of the device. Additionally, battery life is seen as very
important to most smartphone owners. Ninety two percent of potential smartphone
buyers consider battery life as a significant factor in their selection criteria[@OOC2017]. 
From the same research it is also found that 66% of smartphone owners would pay more for a device 
with longer battery life, and 63% is unsatisfied with theirs devices' battery life. 
Although the importance of energy efficient software is clear, programmers lack the knowledge on
the best practices to reduce software energy consumption, and education is not focused on this 
field either [@PHA2016]. In an analysis of questions posted on StackOverflow on the topic of 
energy efficiency performed by Pinto et. al. [@PCL2014] it was found that although programmers 
had questions related to energy efficiency, they rarely received appropriate advice. 
With over 2 billion daily smartphone and tablet users worldwide [@NPPPZL2017] it is clear that 
this lack of knowledge needs to be addressed. To help the programmers close this knowledge gap, 
hardware-based tools have been introduced. These tools can determine the energy profile of an 
application with high precision, but it means the costly hardware components need to be 
acquired[@NPPPZL2017]. Software-based solutions solve the problem of requiring expensive 
hardware and are easier to use, but also less precise. 

In this chapter we represent the current state of energy efficiency in software development for 
apps. For this we set up the following three research questions:

* **RQ1:** What is the state of the art of energy efficiency in software development for mobile 
apps?
* **RQ2:** What is the state of practice of energy efficiency in software development for mobile 
apps?
* **RQ3:** What future work needs to be done in the field of energy efficiency in software 
development for mobile apps?

In the following paragraph the methodology of the study we have followed will be presented. 
After that each research question will be answered. In the last paragraph of this chapter 
the conclusion on energy efficiency in software development for apps will be made.


## Methodology
In order to answer the research questions, we have retrieved over 30 papers from the selected 
field. These papers have been found by searching on Google Scholar with a set of filters. 
First off, only papers from the following journals and conferences are selected:

* ACM Transactions on Software Engineering Methodology (TOSEM),
* Empirical Software Engineering (EMSE),
* IEEE Transactions on Software Engineering (TSE),
* Information and Software Technology (IST),
* Journal of Systems and Software (JSS),
* ACM Computing Surveys (CSUR),
* Foundations of Software Engineering (SIGSOFT FSE),
* International Conference on Automated Software Engineering (ASE),
* Working Conference on Mining Software Repositories (MSR)
* Symposium on Operating Systems Design and Implementation (OSDI)

Furthermore, the preference is given to papers which are published not too long ago, preferably 
after 2012. But if the paper is cited a lot and if it contains interesting findings, older 
papers can also be used.

Given these two criteria the following search queries have been used:

* "Android AND Energy Efficiency AND Software"
* "Energy AND Android AND Runtime AND Analysis AND performance AND Software"

From the selected papers some backward and forward references have also been selected to get a 
complete picture. All papers are also checked for relevance to the topic.

To answer **RQ1**, a more in-depth investigation into available tools and guidelines for the 
energy efficiency is performed. For **RQ2**, the focus lies on what tools, guidelines and best 
practices are currently being used in mobile software development. **RQ3** is answered by the 
finding of the papers used for both **RQ1** and **RQ2**.

## RQ1: State of the Art

Many researchers found that there is an increasing demand for energy efficient software. In 
their research they often propose state of the art approaches which assist developers. These 
approaches can be separated into two main categories: tools and guidelines. This section will 
contain an overview of all state of the art tools and guidelines that satisfy our search 
criteria. This overview is used to answer **RQ1**.

### Tools
The table below shows a quick overview of the tools used to answer **RQ1**.

  |Reference   |  Tool                |  Description                            |
  |------------|-----------------------|------------------------------------------|
  |@CA2018      | Leafactor             | Analyze code smells                      |
  |@LWXM2017    | NavyDroid             | Locate energy inefficiencies             |
  |@KKK2016     | Static analysis tool  | Identify graphical energy bugs           |
  |@SKHA2018    | APOA                  | Compare energy consumption of apps       |
  |@BCBR2017    | Energy Patch          | Detect, validate and repair energy leaks |
  |@NPPPZL2017B | PETrA                 | Measure energy consumption of apps       |
  |@PSCS2018    | jStanley              | Detect and improve energy bugs in Java   |

_Leafactor_

Using static code analysis and automatic refactoring, Leafactor is able to apply
Android-specific optimizations of energy efficiency. The possible optimizations are
indicated by a leaf icon, and the fix priority is provided by the official Android lint
documentation. This priority reflects the severity of the energy performance, from 1 to 10
with 10 being the most severe energy consumption. Leafactor is able to detect five known
energy optimizations.

_NavyDroid_

NavyDroid is a tool created on top of the Java Pathfinder (JPF). Being constructed as a
strengthened DFA (deterministic finite automaton), it can accurately simulate the paused
state, the killed state and related state transitions of an activity. This way it can
detect complex patterns of wake lock misuses (for example multiple lock aquisitions). 

_Static analysis tool_

The proposed static analysis is a novel static optimization technique for eliminating
drawing commands to produce energy-efficient apps. The technique is exploiting the
insight that static analysis is able to predict future behavior of the app. With the
examples of _loop invariant texture analysis_, _packing_ and _identical frames detection_
it indicates total energy savings up to 44% of the total energy consumption of the device.

_APOA_

A recommendation system which can be implemented in any marketplace for helping users and
developers to compare apps in terms of performance. As an input APOA uses a set of metrics
and rating of apps in Comma Separated Value (CSV) format as well as metrics to optimize 
(context of usage). The result is a Pareto optimal front, from which the user selects the most
preferred solution.

_Energy Patch_

A framework that can systematically detect and fix energy bugs in mobile apps in a scalable
fashion. It uses a combination of static and dynamic analysis techniques to detect, validate
and repair energy bugs in Android apps. This enables EnergyPatch to quickly narrow down to
the potential program paths along which energy bugs may occur.

_PETrA_

A novel software-based tool for measuring the energy consumption of Android apps. PETrA
(Power Estimation Tool for Android) measures the energy consumption of Android apps by
relying on the tools and APIs provided with the publicly available Project Volta. This
means that all smartphones equipped with Android 5.0 or higher are compatible. The tool
provides similar performance to hardware-based solutions.

_jStanley_

Eclipse plugin that helps developers detect energy bugs. jStanley scans the open Java
collection for the usage of methods and functions that have more efficient alternatives.
Not only are energy leaks highlighted, but better alternatives are provided. The tool is
driven by a set of CSV files which contains energy consumption and run time of a wide
variety of methods. These values are harvested on a specific device of the researchers.
With the help of these values the possible performance boost is calculated.

Most of the state of the art tools can be classified as performing either measurements of energy 
consumption or code analysis. Although plenty of such tools have been proposed, with low citation 
numbers, none of them seem to have had a big influence in the research area. There is, however,
the possibility, with growing demand for energy efficiency, that the amount and quality of tools
will increase in the near future. 

### Guidelines
Another aspect of the state of the art are the currently proposed guidelines for energy awareness 
and improvements. To answer **RQ1**, seven papers have been selected that present guidelines. The 
research by Cruz et al. [@CA2017] shows that, interestingly enough, the best practices provided by 
Google fall short in addressing energy consumption. However they come up with guidelines that do 
give improvements. Their guideline shows that correct usage of Android methods such as 
_iewHolder_, _DrawAllocation_, _WakeLock_, _ObsoleteLayoutParam_, and _Recycle_ improved energy 
efficiency. From the research by McIntosh et al.[@MSA2018] it is noted that machine learning is
now also implemented by excited developers into their apps, having an impact on the mobile device's
battery life. They have combined empirical measurements of different machine learning algorithm 
implementations with complexity theory to provide the guideline that some implementations of 
algorithms, such as J48, MLP and SMO, generally perform better than others. However they also 
recommend that for optimal results developers must consider their own specific application since 
many factors, e.g. dataset size, can influence the performance.
Finally Li et al.[@LH2014] researched commonly recommended energy saving practices, to see if they
are valid. The topics investigated in this paper included optimizing the sending of HTTP packets, 
memory usage, array lengths, static invocations, and field access. This work gives guidance for 
mobile app developers. Especially bundling network packets up to a certain size and using certain 
coding practices for reading array length information, accessing class fields, and performing 
invocations all led to reduced energy consumption.


## RQ2: State of Practice
Now that we have an overview of the state of the art regarding energy consumption on Android driven 
mobile devices, we investigate what the current state of practice is. In this section, we try to 
find an answer to **RQ2** stated in the introduction by going through the papers we have retrieved 
using the methodology described in Section 2. First, we study how different software engineering 
approaches affect energy consumption and if developers have any knowledge or training regarding 
energy saving programming techniques. Furthermore, we try to see if there are any tools being used 
which help decreasing energy consumption. To make this more specific, we divide **RQ2** into 
several sub-research questions:

* **SQ1:** What are the current development approaches used in practice and how does that affect 
energy consumption?
* **SQ2:** What do programmers currently know and do about energy saving app development?

### Answering SQ1
Under development approaches, the language used to write the app is one of the things that is taken 
into consideration. There have been a number of studies into the effects of using a certain 
programming language in the applications. Java is still the most used language for Android 
applications. According to Oliveira et al. [@OOC2017], Java is actually not a good choice when 
limiting energy consumption is one of your priorities. One of the contributions of the paper is a 
study where hybrid implementations of applications are compared with the original, pure Java 
implementations in terms of energy consumption and performance. The paper suggests that hybrid 
implementations might be a good solution, but the problem is that the study was done on just four 
Android applications. The paper did use the Rosetta Code repository, but this does not represent 
commercial software development.

Another comparable recent research paper investigated the energy-delay product (EDP) which is 
defined as  “a weighted function of the energy consumption and run-time performance 
product” [@GKLS2018]:

$E * T^w$

where E is the total energy used to complete a task, T the time and w is a value that represents 
how important energy saving is with regard to time. The advantage of using this function is that 
performance is not neglected. The results in the paper show that the programming language should be 
chosen depending on the task. From these results, it is clear that Java is not performing well when 
comparing it to other languages as can be seen in the table below. However, it should be noted that
the paper does not test this on actual Android applications but rather the Rosetta Code repository.

![Programming Languages Average Weighted EDP Ranking, retrieved from [@GKLS2018]](figures/EDPranking.jpeg)

The above two papers are examples of research that suggest that the current state of practice, 
developing applications in Java only, is not in line with the current state of the art on energy 
efficiency. In spite of these research papers, developers are still working mostly with Java. One 
possible explanation is that Java is simply compatible with almost every system due to the Java 
Virtual Machine (JVM). We suggest that further research should investigate why developers are
sticking to Java. For example, looking at available libraries, coding difficulty and available
tools and knowledge might have a significant impact on this choice.

### Answering SQ2
In the previous sections we have seen examples of the state of the art, as well as development 
methods that could lead to improvements in energy consumption. However, this knowledge only holds 
value when the community can get developers willing to use these. There have been a number of 
studies that look into what is actually happening.

One example is the paper by Moura et al. [@MPEC2015]. In this paper a study is conducted by 
looking at a large number of energy-aware commits with GitHub as the primary data source. This 
analysis has yielded a list of approaches that are being used by developers in practice. These 
include: frequency and voltage scaling, using power efficient libraries and more. The study notes 
that the vast majority of the commits focus on the lower levels of the software stack. Furthermore, 
only 16.2% of the commits were related to using more efficient libraries or data structures. There 
are also a number of software qualities that have been shown to take precedence of the energy 
consumption. These include correctness, responsiveness and performance. Another point which the 
research addressed is whether software developers were certain of their energy saving commits. The 
paper suggests that there are definitely cases where developers are not certain about their energy 
saving change. This might be caused by the fact that there are few user friendly tools available 
for aiding developers in making energy-aware decisions. One example of such a tool that attempts to 
assist the developer in making energy friendly decisions is jStanley [@PSCS2018]. However, both 
the paper itself and our own replication of that study indicate that this work is not anywhere 
close to ideal. It only concerns static analysis of the code and is very limited in usability. 
This prevents such tools of being used in practice. Additionally, there is the problem of how to 
actually measure the energy being consumed. A number of papers explains how hard this task actually
is. Moura et al. [@MPEC2015] also alude to the fact that developers that do use third-party 
energy-management tools often lack trust in the accuracy of the tools that they are using. The
measuring of energy-consumption is not actually as straightforward as one may think. In a number
of papers it is explained how hardware-based solutions are accurate, but require expensive
components. This is a serious issue, especially in the field of mobile software development. 
On the other hand, software-based solutions are cheaper but less accurate. Now there is work being 
done on creating more accurate software solutions [@@NPPPZL2017]. Early work shows promising 
results in combining the best of both worlds in order to obtain improved results.

## RQ3: Future research
In this section we will take a look at the suggested future research directions that have been 
mentioned in the papers used in this survey. We attempt to extract some general trends from the 
future research suggestions that are included. A similarity between almost all of the papers 
discussed in this section is that they agree the research field is still not very mature. The 
conclusions of the majority of papers is that more research is required. However, in this survey 
we found that the consensus is that it is hard to make strong claims, mainly due to having doubts 
about both the measurement methods as well as the research methodologies. 

One of the fields of research we looked at in the previous sections was the relation between 
programming languages and energy consumption. Multiple papers suggest that the programming 
language used for implementation has an influence on the energy consumption and performance 
[@GKLS2018; @OOC2017]. These papers however do not look into the influence of specific 
features of the programming languages on energy consumption and performance. 
Or rather the specific characteristics of a programming language that cause the differences, 
as described in the paper by Georgiou et al. [@GKLS2018]

In the investigations of this report, we came across a number of tools which can be used to aid 
developers in their management of the energy consumption of applications. However, to our 
knowledge, there are no studies reporting on the actual use of such tools. For future research, 
we suggest investigating the amount of developers actually using such tools. Furthermore, it is 
important which features should be included in tools for optimal use. For example, we replicated 
the paper that introduces jStanley, a tool which can be used for energy and performance 
optimization [@PSCS2018]. We faced the issue that this tool cannot implement the suggested 
improvements in an easy and efficient way. For example, we took the AssertJ open source project 
from GitHub and there were more than 300 energy saving suggestions. A user has to click on every 
suggestion to actually implement the improvement which is tedious work.
Similarly, a lot of the papers reporting on tools that could be used in practice, are currently 
being tested with benchmarks. However, it is always the question whether these benchmarks are 
representative enough of the real world to hold actual merit.
Additionally, the outcomes of the papers introducing these tools are not comparable. Testing all 
these tools on the same project would give better insights in what tools are actually useful for 
the developers to use in terms of gaining as much energy efficiency as possible. 

Even though, to our knowledge, there have not been any studies into whether the tools described in 
the research are actually been used in practice, there have been some studies into energy-saving 
practices on GitHub. For instance, studies into the commit messages [@BLXWT2016; @MPEC2015] have 
shed some light into the state of practice. These studies looked into the commit messages 
containing keywords related to energy consumption and classified the corresponding code. The 
future research directions that are indicated by such papers is that it will be important to 
verify that the energy-saving commits actually have an impact on the overall performance of the 
software. That is both in terms of energy-consumption as well as metrics for the performance 
related to usability. Similarly, another direction is verifying that the energy-saving techniques 
are consistent across platforms.

Finally it is to be noted that nearly all research is based on open-source projects from GitHub 
repositories. Research on proprietary and closed-source software could possibly lead to different 
results and would therefore be interesting to conduct. 

## Conclusion
Having answered the research questions of this chapter in the sections above, we can conclude that 
energy awareness with regard to developing applications for mobile devices, and more specific 
Android, needs more attention. A lot of research is done in the field, resulting in guidelines and 
tools to help developers. But these guidelines are very generic and also conclude that energy 
awareness is project specific. The tools presented by papers hold claims to improvements, but for 
example the tested tool jStanley is not as easy to use as claimed in the paper. More research 
into the claims made in the papers describing these tools is needed.
In practice it is observed that the main language for creating Android apps (Java) is not the most 
energy efficient option. Furthermore, developers often do not give enough priority to energy saving 
options and when they do they are often unsure about the effects of their changes.