Heuristics Considering UX and Quality Criteria for Heuristics

Heuristic evaluation is a cheap tool with which one can take qualitative measures of a product’s usability


I. Introduction
T he term User Experience (UX) wasn't officially defined until 2010 in DIN EN ISO 9241-210. Before that usability, as defined in DIN EN ISO 9241-10 in 1995, was regarded as the contemporary paradigm. Through the definition of the UX, usability has become expanded (e.g., [1,2]).
There are various methods to ensure good usability, such as laboratory or field studies, which are often time-consuming and expensive. With heuristic evaluation [3], a type of expert-based evaluation has been established which requires no participants and is inexpensive as well as quick to carry out. Nevertheless, with this method, systematic results can be targeted. A set of rules, heuristics, form the basis of this method. Heuristics ensure that certain, desired characteristics of an object being investigated that lead to positive usability are examined.
In literature, there are some articles that deal with heuristic evaluation and recommend the use of heuristics [1][2][3][4][5][6]. In the analysis of the related work, it is striking that current heuristics are almost completely focussed on the evaluation of usability. Here, a research gap is shown, because heuristics have not yet been considered in depth for the evaluation of UX [2] and there are no studies on it that are worth naming. This article is aimed at analysing the current state of research on heuristics in order to evaluate UX. In addition, we would like to determine what quality criteria for the evaluation of UX heuristics look like. For this reason, we will answer the following research questions: • RQ1: What heuristics are used to measure the User Experience?
• RQ2: Which quality criteria exist for heuristics?
A literature research was conducted to answer the research questions. In a further step, the heuristics that were found are analysed and mapped with dimensions of the UX. In addition, quality criteria were extracted from the literature research, consolidated and made concrete in a recommendation, through additional criteria.
The main contribution of this research is a mapping of heuristics with dimension of the UX (RQ1) and, as far as we know, the first quality model for heuristic evaluation with a suggestion of quality criteria for heuristics (RQ2).
The paper is structured as follows: section II provides an overview of related work. Section III describes the methodological process and the selection criteria. In section IV, the results of both research questions are presented and analysed before they are summarised in section V.

II. Related Work
In 1990, Nielsen and Molich [3] already described heuristic evaluation as a convenient way to evaluate the usability of a system. Since then, many things have changed and it is often not just usability that is evaluated, but rather the entire UX [2]. Since then, the early heuristics from Nielson or Shneiderman have hardly developed further, yet are still frequently cited and used (e.g., [4]- [6]).
In doing so, the established heuristics are used for the validation of one's own results [7], [8]. These heuristics expand upon the area being evaluated. However, they do not take UX heuristics into consideration, because they do not cover the pragmatic and hedonic quality [1], which are components of the UX.
More frequently, the problem of generalisation is seen. General heuristics are easy to understand and implement [12], [16]; however, many specific topics could be overlooked [7].

III. Research Methodology
A literature review was conducted to find heuristics and quality criteria for heuristics used by the HCI community. The search string was defined as follows: ((Heuristic OR Heuristik) AND ("User Experience" OR UX)) OR "heuristic evaluation" OR ((Heuristic OR Heuristik) AND (quality OR "quality factor" OR "quality criteria")) In October 2016, searches were launched in the following search spaces: • scholar.google.com • Elsevier ScienceDirect • IEEE Xplore Digital Library (IEL)

• SpringerLink
The filtering process consisted of: (i) reading the title, (ii) reading the abstract, and (iii) reading the complete study. Studies were included if they met the following criteria: 1. The study reported how heuristic evaluation was used; 2. The study evaluated heuristics or heuristic evaluation; 3. The study proposed a method similar to heuristic evaluation; 4. The study proposed new heuristics; 5. The study was written in English.
The studies that did not meet the inclusion criteria or the full text of which was not available were excluded. After the selection of studies, backward snowballing [17] was used. For data analysis, we classified the paper according to objective, methodology (e.g., literature review, survey, heuristic evaluation) and relevant artefacts or results.
Our selection criteria for papers proposing new heuristics (RQ2) were: (i) heuristics needed to be presented in the paper, (ii) heuristics are formulated as rules, as mentioned in [3], (iii) heuristics differ distinctly (in terms of content) from existing heuristics and are general or easily adaptable to other contexts which are to be selected.
The articles found on RQ1 (What heuristics are used to measure the User Experience?) were examined to see which dimensions of the UX are evaluated with heuristics. In doing so, each heuristic was examined for its connections to the established UX dimensions of the UEQ [18]. These UX dimensions are: The UX dimensions dependability, perspicuity and efficiency are associated with the pragmatic quality and described in accordance with the model from Hassenzahl: the usability [1]. In comparison with this, stimulation and novelty are amongst the hedonistic quality characteristics and, in doing so, describe the UX. In contrast, attractiveness is a dimension of valency.
Furthermore, it was determined whether heuristics take a further dimension of the UX into consideration in a targeted way, which is not a part of the UX dimensions of the UEQ [18]. In this case, it was noted in an additional column in the table, under the header "Other (not related to UX dimensions").

IV. Results
The major studies that remain after application of the selection criteria for the RQ1 (What heuristics are used to measure the user experience?) are listed in Table 1. The works relevant to RQ2 (What qualification criteria exist for heuristics?) are depicted in Table 2.

A. RQ1: Heuristics and UX Dimensions
After the use of the selection criteria (see section III, on Research Methodology), six articles remain; they can be classified as follows: • Established usability heuristics from Nielsen [3] and Shneiderman [19] • Heuristics specially created for mobile end devices [7] • Heuristics for the examination of persuasive technologies [8] • Specific heuristics for the examination of the UX [10], [12] Amongst the articles selected, only two publications can be found [10], [12] that developed heuristics of the measurement of the UX.
In the following sub-sections, the six heuristics will now be presented and their connection to the UX dimensions of the UEQ examined.

Shneiderman
In 1987, in his book "Designing the userface" [19], Shneiderman created eight golden rules of interface design. According to Shneiderman, this guide was created in order to gain interaction design and therefore, also receive improved usability. Therefore, the intention was not to create heuristics for the heuristic evaluation in accordance with Nielsen and Molich, but these rules could still be used for it. Just like Nielsen's heuristics, these eight rules are considered established. The results in Table 3 show that all eight rules lie in the area of classic usability.

Nielsen und Molich
In 1990, Nielsen and Molich developed the method of heuristic evaluation [3] and empirically examined it in four experiments. The heuristics were expanded by Nielsen in 1994 [20] and are considered established. The expanded heuristics are reflected upon in this work. In this work, the expanded heuristics [20] are considered. The heuristics are also used as a benchmark of new heuristics [7], [8]. In Table 4, the contrast between the heuristics (lines) to the UX dimensions (columns) is shown. It should be noted that the heuristic "Aesthetic and minimal design" goes beyond classic usability [2] and can be attributed to the "attractiveness" dimension.

Väänänen-Vainio-Mattila and Wäljas
With her heuristics, Väänänen-Vainio-Mattila and Wäljas aims [10] at the evaluation of the UX of web services. They are based on text research and informal examinations of existing web services. Different from Nielsen [3], each heuristic was sub-divided into hedonistic and pragmatic aspects, which took place in accordance with Hassenzahl's UX model [1]. After a case study, the heuristics were expanded.
Of the selected heuristics, this is one of two that is explicitly created with the goal of measuring the UX. Table 5 shows that four heuristics go beyond classic usability.

Kientz et al.
Kientz et al. [8] present heuristics for the evaluation of persuasive health technology. In a literature review, a list of all heuristics is selected, with regard to persuasive health technologies. As a result, 13 experts were asked to list their respective 10 most important heuristics on this list. It could also be expanded through their own. Kientz et al. consolidated these 130 heuristics to a list of 10. These 10 heuristics were used in two case studies and compared with Nielsen's heuristics [20]. The control group could discover more issues while carrying out the evaluation, yet they were less severe. Table 6 shows that three heuristics go beyond classic usability.

Inostroza et al.
In 2016, with "SMASH", Inostroza et al. [16] developed a set of heuristics that have to do with usability within the context of smartphones. The 12 heuristics are based on prioritised and formallyrepresented characteristics of smartphones from literature and case studies and, in its approaches, have a connection to UX.
The heuristics were validated in a case study and every heuristic finally refined using the criteria of utility, clarity, ease of use and the need of additional elements (checklists). In Table 7, the 12 heuristics and their connection to the UX dimensions is depicted. It shows that two of the twelve heuristics go above and beyond the dimensions of classic usability.

Arhippainen
In a tutorial, Arhippainen [12] presented 10 user experience heuristics. The aim of the heuristics is to support designers shaping aspects of the UX. They are based on empirical studies from mobile services. In addition to those from Väänänen-Vainio-Mattila and Wäljas [10], they are the only ones that focus on the analysis of UX. The results from Table 8 also show that only five of the ten heuristics for classic usability are included.

Discussion
In the overview (Table 9), it appears that the established heuristics, with one exception, exclusively cover the usability dimensions of the UEQ. The UX dimensions mostly remain ignored.  Beyond all of the heuristics, it is shown that the dimension of dependability, with 35 considerations, is the heuristic that is observed the most. The UX dimension of the UEQ, which shows the fewest connections to the heuristics, is novelty (see Table 10). The number of individual heuristics varies between 8 [19] and 12 [16]. For better comparability, in Table 11, the considerations of the UX dimensions of the UEQ are weighted against the total number of connections. In the overall view of the heuristics, the number of dimensions is given that a heuristic takes into account, on average. Accordingly, the heuristics from Inostroza et al. [16] show the highest coverage of the UX dimensions of the UEQ. The heuristics from Nielsen and Molich [20] as well as Shneiderman [19], which were a basis for Inostroza et al. [16], follow.
The heuristics examined predominantly move in the directions of the UX dimensions of dependability and perspicuity, which can also be organised with the dimension of efficiency of usability. The dimensions of stimulation and novelty are to be attributed to the UX and are hardly taken into consideration.
Consequently, the heuristics from Inostroza et al., Nielsen and Molich as well as Shneiderman view usability to be the strongest, and Arhippainen and Väänänen-Vainio-Mattila and Wäljas, the UX. The connection to the pragmatic dimensions is more strongly pronounced than in the hedonistic. Nielsen and Molich as well as Shneiderman, do not view any UX dimensions of the UEQ that are attributed to the hedonistic quality.

B. RQ2: Quality Criteria for Heuristics
Based on the selected studies, as far as we know, the first quality model for heuristic evaluation (Fig. 1) was created. This depicts the central elements of heuristics, the test object, evaluators, users and the relationships between them.
Because in the selected papers, only individual characteristics but no list of quality criteria could be found for heuristics, quality criteria were deviated from the literature selected and the quality model for heuristic evaluation (Fig. 1). They should ensure the efficient use of heuristics: • Complying with the heuristics must help the user • Validated through comparison with established methods / heuristics [15] • Suitable for the context / object being examined [12], [14], [16], without negatively influencing the comprehensibility • Comprehensible and easy / simple to implement (also for nonspecialists) [16] Quality criterias Furthermore, the additional following quality control criteria are recommended, which expand these quality criteria: • Selective • Not more than 10-15 heuristics Additionally, quality criteria are recommended in order to ensure identical construction. In addition to this, heuristics should have the following elements: • Designation (a semantic summary, not coded) • Description

• Example
This recommendation for quality criteria should be ensured when considering the quality and the good applicability of heuristics in a heuristic evaluation.

V. Conclusion and Future Work
Within the scope of this paper, we have examined the status of research regarding heuristic evaluation. In doing so, we have focused on heuristics to measure UX. The heuristics identified were mapped with the dimensions of the UEQ for further analysis.
With heuristics, we examined as to whether not just the usability, but rather also the UX can be evaluated. To do so, every heuristic related to the UX dimensions was examined in accordance with the UEQ.
In this article we also introduced a quality model for heuristic evaluation and proposed a list of quality criteria for heuristics, which are based on research in literature.
With the heuristics identified, the pragmatic quality, which is attributed to usability, was examined most frequently. While research in the literature showed that many heuristics are particularly often based on the heuristics of Nielsen and Molich as well as those from Shneiderman which were created with a focus on usability.
The heuristics from Nielsen and Molich are still widely used for heuristic evaluation and as a foundation for other heuristics. They can be applied to most test objects. Other heuristics seem to be more focused on a particular domain. We recommend using the heuristics from Nielsen and Molich [3], as well as those from Inostroza et al. [16] when it is intended to use heuristic evaluation for UX as they have the highest weighted consideration of UX dimensions of the UEQ (Table 11).
The results of this study show that initial approaches exist to apply heuristics in a targeted way for the evaluation of the UX. However, this research field still requires further empirical research.
In the future, it would be important to examine how the results of a heuristic evaluation using a heuristic with a strong relationship to the dimensions of the UX appear in comparison to the established heuristics.