ORIGINAL ARTICLE
ALMEIDA, Hugo Leonardo Nascimento [1], CORREIA, Walter Franklin Marques [2], ALMEIDA FILHO, Adiel Teixeira de [3]
ALMEIDA, Hugo Leonardo Nascimento. CORREIA, Walter Franklin Marques. ALMEIDA FILHO, Adiel Teixeira de. Heuristic evaluations for accessibility decisions: a systematic literature review. Revista Científica Multidisciplinar Núcleo do Conhecimento. Year 08, Ed. 03, Vol. 03, pp. 107-153. March 2023. ISSN:2448-0959, Access link in: https://www.nucleodoconhecimento.com.br/computer-science/heuristic-evaluations, DOI: 10.32749/nucleodoconhecimento.com.br/computer-science/heuristic-evaluations
ABSTRACT
There are more and more people who are using technology on a daily basis to communicate and access information, including the elderly and people with disabilities. Assistive technologies, also called technical aids, are accessibility and inclusion resources that provide a greater degree of autonomy and assistance for people who have some degree of difficulty in using resources in their contact with real and virtual environments. The aim of this study is to investigate the perceptions about the heuristic assessment of accessibility in software by undertaking a systematic review of the literature that covers the last 5 years. This started off with a total of 8548 published papers that are indexed in the Scopus and Web of Science databases. Twelve relevant research questions were drawn up that were answered in accordance with the results found in the final set of articles. Two groups of keywords were created to assist in the search for papers and a filtering process was applied to the results of the searches. The research shows, by means of figures and tables, evidence about the growth of scientific papers in the area. The lead authors and journals that publish on evaluating the accessibility of software are listed, as are answers to the questions about which evaluative models, heuristics, metrics, limitations, assistive technologies and target audiences are most cited in published papers. This study also analyzes the co-citation of the references and couples bibliographic sources. The research showed evidence that there is no general consensus on the use of a single model to assess accessibility, but that the W3C WCAG guidelines are the sources most used to reference heuristics and metrics in studies of this type. It was also noticed that most studies do not focus on assistive technologies, but are widely used to support decisions on websites.
Keywords: Assessment and evaluation, Web accessibility, Heuristic evaluation.
1. INTRODUCTION
The use of heuristics can help to solve problems related to usability and accessibility in software projects (EUSÉBIO; SILVEIRO; TEIXEIRA, 2020; RANADA; LIDSTRÖM, 2019). Taking into account the growing search for and dissemination of heuristics (HORTON et al., 2017), which consist of methods or processes created to propose solutions to problems by using verification items, it is noticeable that several decision factors, which can be part of the solution, are not yet found in the literature (CHI; TSENG; JANG, 2012).
In scientific research, software engineering presents itself as one of the most referenced disciplines with regard to concepts and practices, as it brings together several techniques and activities for validation, verification and evaluation of results, including heuristics. The selection of experimental software engineering techniques that occur throughout the cycle of conducting research, when correctly used, can be the fundamental factor for the success of the research (WOHLIN et al., 2012).
Software engineering is seen as an interdisciplinary concept that brings together technological and managerial aspects, in order to systematically address all the processes of implementing, deploying and maintaining software projects, their quality being assured by constructing them while keeping within deadlines, costs and previously defined resources (MAFFEO, 1992).
The ways to conduct research related to software, when involving people, must be contextualized taking into account ethical concerns (BADAMPUDI, 2017). Based on the theoretical information obtained by the research, a choice/classification model will be proposed that is based on dominance relationships, in order to mitigate mistrust and inefficiency (SINGER; VINSON, 2002) regarding the conduct of empirical methods based on well-founded heuristics.
Currently, in the academy, different proposals for creating and extracting metrics (ways of measuring results) are being conceived (GARCÍA-SANTIAGO; OLVERA-LOBO, 2021; ARAÚJO; CARNEIRO; PALHA, 2020), although most of the proposals are not used in the same way in projects developed by industry (UMARJI; SEAMAN, 2008). The use of metrics is intended to assess the results obtained and add positively to the software process and the generation of value.
The metrics studied and proposed by industry have as an added value the search for costs and the timing of maintenance activities, in addition to the direct relationship with the number of errors found in business systems (UMARJI; SEAMAN, 2008), which demonstrates that issues related to accessibility are not the focus or are often not taken into account by the same proposals.
The concern with satisfying all the needs and requirements of its users and other potential individuals involved and the awareness that the acceptance of the system involves several factors, sees to it that usability and accessibility are thought about and considered together with cost, utility, reliability, social acceptance, etc (NIELSEN, 1994).
Usability and accessibility studies can take ownership of studies from other areas (GAMACHE et al., 2018; LAKSHMI; KUMAR; DAS, 2018; STITZ; BLUNDELL, 2018; VENTURI, 1995), in order to be able to monitor better the conduct of empirical studies, involving human beings, that are necessary for validating research in software engineering.
The user’s experience arouses a growing interest in the Human-Computer Interaction (HCI) community. Even though the HCI community appears to accept that functionality alone or usability principles are no longer sufficient, there is no coherent understanding of what the user’s experience actually is (HASSENZAHL, 2018). ISO 9241-210 proposes to define the user’s experience as all aspects of the user’s experience when interacting with a product, service, environment or facility (ABNT, 2011).
Building on previous studies, this article presents a systematic literature review (SLR) involving the use of heuristic assessments in accessibility decisions, with a view to answering a set of relevant questions that is still open in the literature in the area. Among the contributions of this study, it presents a set of data not observed in previous reviews, such as the analysis and grouping of the evaluation models, heuristics and metrics used, and carrying out a study on the lead authors. Initially, the 8548 papers that were considered in this systematic review were published in the Web of Science and Scopus databases. These were later analyzed and filtered and resulted in the 101 articles analyzed in this review. Therefore, the results of this article update the previous conclusions (CAMPOVERDE-MOLINA, LUJAN-MORA; GARCIA, 2020; NAGARAJU; CHAWLA, 2019; NATHAN et al., 2018; PAIVA, FREIRE; FORTES, 2021). Another contribution of this study is to verify a period before and after the European Accessibility Act, which was introduced after public sector bodies of the European Union were obliged to assure accessibility to websites. This came about as a result of the Web Accessibility Directive (CEN e CENELEC, 2019; EUROPEAN COMMISSION, 2015). In addition, a point to be emphasized as a differential in this article is the presentation of TreeMaps (SHNEIDERMAN, 1992) (analyzing expressions contained in the titles of the set of articles), WordCloud (analyzing the lead authors’ keywords), co-citation analysis and bibliographical coupling. The Bibliometrix R-tool (ARIA; CUCCURULLO, 2017) was used to capture these metrics.
This article is divided into five sections. Section 1 provides a brief introduction to the topic. Section 2 presents the SLR methodology. Section 3 presents the main results and lines of analysis of the systematic review. Section 4 summarizes the work undertaken, discusses some limitations, indicates research guidelines and makes suggestions for future lines of research while Section 5 draws some conclusions.
2. METHODOLOGY
This systematic review combines information from recent studies of the same nature, carried out and published in different areas of knowledge (ALMEIDA-FILHO; SILVA; FERREIRA, 2020; ARAÚJO; CARNEIRO; PALHA, 2020; GONZÁLEZ-PEREA; GALÁN; VILLARINY, 2019; LAENGLE et al., 2017; MARIZ; ALMEIDA; ALOISE, 2018; NASCIMENTO; ALENCAR, 2016; PEREIRA; COSTA, 2015; RUSCHEL; SANTOS; LOURES, 2017; TERASHIMA; CLARK, 2021; ZOPOUNIDIS et al., 2015).
In the methodology of this study, the questions that will be analyzed in the investigation are set; then the articles are listed and subsequently filtered, based on standard procedures; then, the main results obtained are presented; and finally, the results are analyzed.
2.1 DEFINITION OF THE ISSUES RELEVANT TO THE RESEARCH
The definition phase consisted of identifying specific points which the published articles need to fit in to in view of their presentation and discussion in the SLR. So, initially, the specific points that were addressed were listed and this led to compiling a series of research questions to guide the analysis of the results, to determine if there were gaps in the literature, and to define the scope of the review. These research questions, which are not classified according to their importance, are presented in Table 1.
Table 1 – Research Questions
| Research Questions | Description |
| Q1 | Has there been an increase in the number of heuristic assessment models for accessibility? |
| Q2 | Has there been an increase in the number of articles citing heuristic assessment for accessibility decisions? |
| Q3 | What are the main references found that address heuristic assessments for accessibility decisions? |
| Q4 | Who are the most relevant researchers in the field of heuristic assessment research for accessibility? |
| Q5 | Are heuristic evaluation models more present in journals focused on operational research? |
| Q6 | What are the most cited evaluation models? |
| Q7 | What types of assistive technologies do the models found most help? |
| Q8 | What groups of individuals are the focus of research? |
| Q9 | What are the existing limitations and gaps most cited in studies that address heuristic evaluation models? |
| Q10 | What are the heuristics most analyzed by the published models? |
| Q11 | What metrics are most used in assessments? |
| Q12 | Is there an association between the heuristic evaluation model used and the type of assistive technology used and/or the target audience studied? |
Source: author.
2.2 COLLECTION AND SELECTION OF THE ARTICLES
The set of published articles was compiled from the Web of Science database and the Scopus database. To conduct searches in the service provided by the databases, two sets of keywords were formulated, as shown in Table 2. A first set involved 9 keywords related to accessibility. The second set, with 12 keywords, is related to heuristic evaluation. Some of the chosen keywords were also considered in the review by (PAIVA; FREIRE; FORTES; 2021); however, the keywords presented in Table 2 are not limited to those used by them.
Table 2 – Key-words
| Accessibility: Key-words | Heuristics: Key-words |
| accessibility; assistive technologies; adaptive technologies; rehabilitative technologies; reachability; communicability; assistive devices; adaptive devices; rehabilitation devices | heuristic evaluation; heuristic model; heuristic method; heuristic techniques; inspection techniques; inspection methods; inspection evaluation; inspection model; heuristic inspection; evaluation model; evaluation methods; evaluation techniques |
Source: author.
Searches were made by combining the keywords of the two groups. Thus, each of the 9 keywords from the Accessibility group was combined with each of the 12 keywords from the Heuristics group, using the Boolean Operator “AND”. A total of 301 results were initially found in the Web of Science Core Collection database and 8247 results in the Scopus (Elsevier) database, totaling 8548 results found. This initial set of published papers was filtered according to predetermined rules in order to select only those that are consistent with the proposed systematic review as set out in Section 2.3.
2.3 FILTER PROCESS
First of all, a filter was used to retrieve only papers published from 2017 to 2021, the aim being to obtain the most recent and relevant features in the scientific literature, observing the search terms. After the filter, 158 papers were removed from the results of the Web of Science search, leaving 143 results. While in Scopus, 4480 works were removed, leaving 3767. In total, after Filter 1 there were 3910 papers left.
A second filter was used to remove papers that were not characterized as articles. The purpose of the filter was to focus only on the review of articles of great academic relevance, categorized as “Article” on the research platforms. Thus, books and other categories of published material were also removed. The papers that emerged from presentations at conferences, and that were later published in scientific journals, were consequently included in the analysis since they now had the status of articles. A total of 63 papers were removed by the second filter in Web of Science, and a total of 1281 papers were removed from the Scopus database by Filter 2. This left 80 papers from the Web of Science and 2486 from Scopus. In total, 2566 articles remained for analysis.
A third filter was used later. It was observed that some of the articles could be easily excluded since they do not belong to the research area of this paper. For example, the survey revealed many articles in the areas of Health, such as Medicine and Nursing. Such articles that were not categorized as “Computer Science”, according to the research areas of the databases used, were not considered in this systematic review. 55 articles were excluded from the Web of Science results, after the third filter, leaving 25, and 1795 articles were excluded from the Scopus results, leaving 691. Therefore, a set of 716 articles remained after Filter 3.
Finally, these articles were analyzed to exclude papers that would not add relevant information to this research. A verification of each article was carried out, in an attempt to select only the articles that presented, in short, assessments of the accessibility of technological applications. Thus, it was possible to eliminate duplicate studies found in the two databases used. In total, 101 articles were selected and formed the final set, which were analyzed and categorized according to the proposals of this paper. Figure 1 illustrates the process of applying filters to the initial set of articles.
Figure 1- Filtering process
2.4 COUPLING AND CO-CITATION ANALYSIS
Throughout this article, the results of and discussion about the systematic review will be presented. Many graphics are self-explanatory and answer some of the survey questions. Some graphics need additional explanations, and these are given in the text.
A bibliographic coupling analysis of the sources was carried out, so that the articles obtained from an initial set were grouped according to their sources. This analysis relates, how different sources are cited, by means of sets. Then, the articles that mention the authors of each group formed were listed. The strength of the bibliographic coupling grows as the number of similar articles is listed in the lists of the sources analyzed. This analysis allows us to understand which are the most relevant sources for a given number of articles (ECK; WALTMAN, 2010; PERIANES-RODRIGUEZ; WALTMAN; ECK, 2016; SMALL, 1974; WALTMAN; ECK; NOYONS, 2010).
A co-citation analysis of cited references was also performed, with the aim of measuring the strength of the connections between the cited references of a set of articles. Using a given set of articles, a list of cited references can be obtained from a database such as the Web of Science or Scopus. Considering a set of articles, the greater the number of articles in this set that cite two references at the same time, the greater the co-citation strength between these two cited references. This analysis allows us to understand how the most important references in a set of articles relate to the articles in the set based on where the references themselves are cited. (ECK; WALTMAN, 2010; PERIANES-RODRIGUEZ; WALTMAN; ECK, 2016; SMALL, 1974; WALTMAN; ECK; NOYONS, 2010).
In order to create a map linking the keywords of the most relevant authors of the articles analyzed, a network analysis of the authors’ keywords was performed. The strength of the connection between keywords grows as more articles use them (ECK; WALTMAN, 2010; PERIANES-RODRIGUEZ; WALTMAN; ECK, 2016; SMALL, 1974; WALTMAN; ECK; NOYONS, 2010).
3. RESULTS AND DISCUSSION OF THE SYSTEMATIC
First of all, in order to answer the first two research questions, the articles were separated by year of publication and by the frequency of their citation over the years. Figure 2 illustrates the results of categorizing the articles in relation to the year of publication by means of an area graph, which shows a trend in the growth of the number of articles published each year from 2018. For 2021, only articles published until July 2nd were accounted for. Therefore, for comparison purposes, it is evident that the total number of articles, published only in the first half of 2021, already exceeds half of the articles published in each of the other previous years. A more evident growth is seen in 2019, possibly because, in that year, the European Union introduced the European Accessibility Act, which is currently considered one of the main examples of digital accessibility legislation in the world. In addition to the fact that the member countries of the European Union needed to comply with the Web Accessibility Directive, approved by the European Parliament, which has required the websites of public sector bodies to comply with accessibility standards since September 23, 2018. On looking at Figure 3, note that the countries that produced the most articles in this sample were Spain, Malaysia, the USA and Brazil. However, more than 60% of the articles found come from countries belonging to the European Union, which explains the greater attention to accessibility due to the aforementioned legislation. Therefore, the answer to Q1 is: yes, the number of models used in the heuristic assessment for accessibility has been growing year by year since 2018.
Figure 2 – Annual Scientific Production
Figure 3 – Country Scientific Production
Regarding Q2, Figure 4 illustrates the evolution of the accumulated number of citations of selected articles per year, and a smooth growth curve can be visually evidenced. Thus, it is noticed that there has not been time for the most recent articles to be cited when compared to previous studies. There was an increase in the number of articles published, but the most recent articles, naturally, still have a lower number of citations, demonstrating that older papers have more relevance for recent discussions on the topic. In the first half of 2021, the selected articles received 6 citations, demonstrating that the authors are more interested in the state of the art in this area from the sources, but also demonstrating that the articles continue to be cited over the years, and showing that it has become natural for more recent articles to draw on earlier ones. Thus, the answer to Q2 is that the number of articles citing heuristic evaluation for accessibility decisions has grown.
Figure 4 – Number of citations per year
The verification of other citations was carried out, using information extracted from the databases of online journals, to evidence the listing of the articles. The Bibliometrix R-tool (ARIA; CUCCURULLO, 2017) was used to analyze the bibliographic coupling of sources and to perform a co-citation analysis via network visualization and cluster division. In bibliographic coupling research, the degree of kinship between papers is based on the number of shared references. In the co-citation analysis, the link between the cited documents is observed due to the way the reference items are cited together (SMALL, 1974). Figure 5 shows a co-citation analysis for 27 references, and three clusters are identified.
Figure 5 – Analysis of co-citation
Table 3 presents the three clusters, listing the articles that form each one. It was observed that general studies that address accessibility assessment were allocated to Cluster 1. On the other hand, Cluster 2 and Cluster 3 included papers that focused on assessing government websites.
Table 3 – Grouping of Clusters of co-citations
| Cluster 1 | Cluster 2 | Cluster 3 |
| vigo m. 2013 | abanumy a. 2005 | nielsen j. 1994-2 |
| nielsen j. 1994-1 | al-faries a. 2013 | al-khalifa h.s. 2017 |
| power c. 2012 | al-khalifa h.s. 2012 | youngblood n.e. 2012 |
Source: author.
Table 3 – Grouping of Clusters of co-citations
(conclusion)
| Cluster 1 | Cluster 2 | Cluster 3 |
| brajnik g. 2008-1 | baowaly m.k. 2012 | jati h. 2009 |
| acm: new york ny usa. | adepoju s.a. 2016 | king b.a. 2016
|
| aizpurua a. 2016 | akgul y. 2016 | |
| lazar j. 2007 | al-radaideh m. 2011 | |
| petrie h. 2007 | al mourad b. 2013 | |
| ahmi a. 2015 | grantham j. 2012 | |
| brajnik g. 2008-2 | latif m.h.a. 2010 | |
| brajnik g. 2011 | lujn-mora s. 2014 |
Source: author.
To analyze how the sources are linked based on their references, a bibliographic coupling analysis was performed and is illustrated in the visual representation of the network in Figure 6. In this case, three different clusters were formed. Universal Access in the Information Society, Library Hi Tech, IEEE Access and Advances in Human-Computer Interaction were identified as the most relevant sources for connections within the network.
Figure 6 – Analysis of bibliographic coupling
Taking into account the information extracted from the analyses, Q3 can be answered by indicating that the main studies that address heuristic assessments for decisions in accessibility are Al-Faries (2013), Al-Khalifa (2012), Al-Khalifa (2017), Vigo (2013) and Abanumy (2005) and the sources for these are Universal Access in the Information Society, Library Hi Tech, IEEE Access and Advances in Human-Computer Interaction.
To answer Q4, a study was conducted on the authors of the cataloged articles. The names of 307 authors were found in the published articles. 302 of these authors co-authored two or more of the 101 articles while only five of them are by a single author. The Bibliometrix R-tool was used to collect the impact of the authors and this impact is illustrated in Table 4 for the lead authors. Among the metrics presented are the number of published articles, total citations, the h index, g index and m index, considering the set of articles analyzed in this SLR. Each of the authors in this table is a co-author of at least three of the 101 articles identified for this study.
Table 4 – Impact of the authors
| Author
|
h_index | g_index | m_index | Total number of citations | Number of Papers | PY_start | |||
| ACOSTA-VARGAS P | 2 | 3 | 0.667 | 18 | 3 | 2019 | |||
| DOUSH IA | 2 | 3 | 0.4 | 22 | 3 | 2017 | |||
| ISMAILOVA R | 3 | 3 | 0.6 | 75 | 3 | 2017 | |||
| AZIZ N | 1 | 2 | 0.2 | 4 | 2 | 2017 | |||
| BRANCO F | 2 | 2 | 0.4 | 37 | 2 | 2017 | |||
| FREIRE AP | 2 | 2 | 2 | 5 | 2 | 2021 | |||
| GONALVES R | 2 | 2 | 0.4 | 37 | 2 | 2017 | |||
| HUSSAIN A | 1 | 2 | 0.25 | 4 | 2 | 2018 | |||
| INAL Y | 2 | 2 | 0.4 | 40 | 2 | 2017 | |||
| LUJAN-MORA S | 2 | 2 | 0.667 | 15 | 2 | 2019 | |||
| LUJN-MORA S | 2 | 2 | 0.5 | 17 | 2 | 2018 | |||
| MARTINS J | 2 | 2 | 0.4 | 37 | 2 | 2017 | |||
| MUTALIB AA | 1 | 2 | 0.2 | 4 | 2 | 2017 | |||
| PAIVA DMB | 2 | 2 | 0.5 | 5 | 2 | 2018 | |||
Source: author.
As shown in Table 4, of the lead authors, three are co-authors of more than 2 of the articles, viz., Acosta-Vargas P, Doush IA and Ismailova R. In fact, it is pointed out that Acosta-Vargas, although only first appearing in 2019 in this area, is already the most important author according to the indices shown. In response to Q4, considering the g index and the number of papers, these three authors stand out as the most relevant researchers in the heuristic evaluation research area for accessibility. Acosta-Vargas P also stands out for the m index, while Ismailova R stands out for the number of citations. Table 5 illustrates the relationship between these authors and the heuristic evaluation models analyzed.
Table 5 – Lead Authors x Models
| Author | Models |
| Acosta-Vargas P. | WCAG 2.1, WCAG 2.2, WCAG 2.1 + Brajnik, JClic, Ardora, Chanchí et al (2019), Case Study, Salvador-Ullauri L. et al (2020). |
| Doush IA. | AChecker, HTML Validator, CSS Validator, APrompt, Cynthia Says, EvalAccess 2.0, SortSite, TAW, WAVE, ISAB. |
| Ismailova R. | Survey, EvalAccess 2.0, WCAG 1.0, WebXACT/Bobby, Cynthia Says, The Functional Accessibility, WebInSight, TAW, AChecker. |
Source: author.
Q5 is directly related to scientific journals that publish articles that design and apply heuristic evaluation to help make better decisions in accessibility. In all, 55 different scientific journals were found. Only 16 of these journals published more than 2 articles that were included in the SLR. Table 6 shows each of these scientific journals according to the number of articles each of them published.
Table 6 – Published papers per Journal
| Sources
|
Articles |
| UNIVERSAL ACCESS IN THE INFORMATION SOCIETY | 19 |
| IEEE ACCESS | 7 |
| RISTI – REVISTA IBERICA DE SISTEMAS E TECNOLOGIAS DE INFORMACAO | 6 |
| APPLIED SCIENCES (SWITZERLAND) | 4 |
| INTERNATIONAL JOURNAL OF HUMAN-COMPUTER INTERACTION | 4 |
| ACM TRANSACTIONS ON ACCESSIBLE COMPUTING | 2 |
| ADVANCES IN HUMAN-COMPUTER INTERACTION | 2 |
| BEHAVIOUR AND INFORMATION TECHNOLOGY | 2 |
| COMPUTERS IN HUMAN BEHAVIOR | 2 |
| ELECTRONIC LIBRARY | 2 |
| INTERNATIONAL JOURNAL OF INTERACTIVE MOBILE TECHNOLOGIES | 2 |
| INTERNATIONAL JOURNAL OF TECHNOLOGY AND HUMAN INTERACTION | 2 |
| JOURNAL OF INFORMATION AND COMMUNICATION TECHNOLOGY | 2 |
| JOURNAL OF SPECIAL EDUCATION TECHNOLOGY | 2 |
| LIBRARY HI TECH | 2 |
| MULTIMEDIA TOOLS AND APPLICATIONS | 2 |
Source: author.
On analyzing Table 6, note that only 16 of the 55 scientific sources were cited, that is, 39 of the sources published only 1 scientific article related to the specified topic. These numbers show that authors can search for more specific journals that deal with a certain type of heuristic evaluation model, or specific sources in relation to environments where there may be a certain concern with accessibility in order to evaluate it.
A more specific analysis was conducted with the Bibliometrix R-tool (ARIA; CUCCURULLO, 2017) on the impact of scientific journals. Table 7 presents a set of metrics involving the most relevant journals based on the h-index in relation to citations of the studies from each journal. The number of published articles, total citations, h index, g index and m index are displayed, considering the 101 articles analyzed in this SLR. The calculated Impact Factor (IF) of each journal for the last two years is also displayed.
Table 7 – Impact of the Journals
| Element
|
h_index | g_index | m_index | TC | NP | PY_start | IF | ||||||||
| UNIVERSAL ACCESS IN THE INFORMATION SOCIETY | 7 | 13 | 1.4 | 183 | 13 | 2017 | 1.815 | ||||||||
| ACM TRANSACTIONS ON ACCESSIBLE COMPUTING | 2 | 2 | 0.666666667 | 23 | 2 | 2019 | 2.641 | ||||||||
| ADVANCES IN HUMAN-COMPUTER INTERACTION | 2 | 2 | 0.5 | 5 | 2 | 2018 | 1.355 | ||||||||
| COMPUTERS IN HUMAN BEHAVIOR | 2 | 2 | 0.4 | 33 | 2 | 2017 | 5.003 | ||||||||
| ELECTRONIC LIBRARY | 2 | 2 | 0.4 | 11 | 2 | 2017 | 0.792 | ||||||||
| IEEE ACCESS | 2 | 3 | 0.666666667 | 25 | 3 | 2019 | 3.745 | ||||||||
| Element
|
h_index | g_index | m_index | TC | NP | PY_start | IF | ||||||||
| INTERNATIONAL JOURNAL OF HUMAN-COMPUTER INTERACTION | 2 | 3 | 0.4 | 34 | 3 | 2017 | 1.713 | ||||||||
| INTERNATIONAL JOURNAL OF TECHNOLOGY AND HUMAN INTERACTION | 2 | 2 | 0.666666667 | 6 | 2 | 2019 | 0.711 | ||||||||
| JOURNAL OF INFORMATION AND COMMUNICATION TECHNOLOGY | 2 | 2 | 0.4 | 12 | 2 | 2017 | 1.833 | ||||||||
| RISTI – REVISTA IBERICA DE SISTEMAS E TECNOLOGIAS DE INFORMACAO | 2 | 2 | 0.666666667 | 8 | 3 | 2019 | 0.531 | ||||||||
Source: author.
It is interesting to observe in Table 7 that, in all the listed journals, the sample articles have, on average, more citations per article than the journal’s overall average. Therefore, they pull the IF of the respective journals up.
By examining the keywords indicated in the article catalog, information in the form of WordCloud could be extracted. Figure 7 illustrates the main terms present in the articles, thereby illustrating for which themes the journals were considered in this SLR and by what criteria they can be classified. We highlight the terms accessibility and usability as the most relevant among the 20 most relevant terms in the entire catalog. Terms that refer to disabilities and the elderly also entered into the composition of the figure. It is important to mention that a published article may have more than one keyword.
Figure 7 – WordCloud of the main terms
Figure 8 shows the 10 two-word expressions (Bigrams) that most appear in the titles of the articles analyzed in this SLR. Within this TreeMap, different information can be analyzed in relation to Figure 7, such as the terms ‘literature review’ and ‘systematic literature’, thus showing the format of some of these studies in the area. In Figure 8, the term accessibility is confirmed as the most relevant for journals that publish heuristic assessments for accessibility. Government websites are also shown as a relevant expression in this research area and are present in 12% of the articles. Thus, the answer to Q5 is “yes”: heuristic evaluation models are more present in journals focused on operational research.
Figure 8 – TreeMap of Bigrams
The answer to Q6 can be identified in Figure 9. The models most cited and used in the heuristic assessments for accessibility that are scientifically documented and indexed by Scopus and Web of Science are: WCAG standards, AChecker and WAVE tools. Note that, in addition to the models themselves, techniques, tools, standards, studies, reviews and metrics are used to assess accessibility in software. The WCAG has different versions that were cited by the studies (1.0, 2.0, 2.1, 2.2 and other variations (ACOSTA-VARGAS et al., 2020; ACOSTA-VARGAS; SALVADOR-ULLAURI; LUJAN-MORA, 2019; AKGÜL, 2021; ALAJARMEH, 2021; ALSAEEDI, 2020; BABU; XIE, 2017; BAI; STRAY; MORK, 2019; HASSOUNA; SAHARI; ISMAIL, 2017; ISMAILOVA; KIMSANOVA, 2017; MADEIRA et al., 2021; MARTINS; GONÇALVES; BRANCO, 2017; SALVADOR-ULLAURI et al. 2020; WENTZ et al., 2019). The tool most used was AChecker.
Figure 9 – Frequency of citation of the models
In Table 8 there are descriptions of the models that are mentioned more than once and their respective occurrences in the articles of the final sample. 91 different models were cited only once, while 25 different models were cited more than once.
Table 8 – Count of occurrences of the models
| Models/Techniques/Tools/Patterns | Frequency |
| WCAG | 17 |
| AChecker | 13 |
| WAVE | 10 |
| TAW | 6 |
| Interview | 6 |
| Survey | 6 |
| W3C HTML/CSS Validator | 5 |
| SortSite | 4 |
| Questionnaires | 4 |
| Experiment | 4 |
| EvalAccess | 4 |
| FAE | 3 |
| Case Study | 3 |
| WCAG 2.1 + Brajnik | 2 |
| Pingdom AB | 2 |
| Mobile-friendly Test | 2 |
| ARIA | 2 |
Source: author.
Table 8 – Count of occurrences of the models (conclusion)
| Models/Techniques/Tools/Patterns | Frequency |
| Total Validator | 2 |
| SiteImprove | 2 |
| Norman | 2 |
| Nielsen | 2 |
| Prototype | 2 |
| UX evaluations | 2 |
| Chanchí et all (2019) | 2 |
| Cynthia Says | 2 |
Source: author.
To answer Q7, the technologies involved in all studies of the articles in the final sample had to be collected. It was identified that the technologies evaluated were not necessarily directly or properly assistive, but they were used by a public that had people benefiting from good accessibility practices. Initially, a minority of the studies evaluated properly assistive technologies, as illustrated in Figure 10.
Figure 10 – Types of Technologies evaluated
Table 9 lists the brief descriptions of the assistive technologies evaluated by the articles. Only TalkBack technology was the focus of evaluation in more than one different study (ROBLES et al., 2019; YAN; RAMACHANDRAN, 2019).
Table 9 – Description of the assistive technologies
| Types of assistive technologies evaluated |
| Assistive Systems in general |
| Assistive Interactive System Design |
| Assistive Pointing Device Based on a Head-Mounted Camera |
| Chart interaction (Safe Magnification, Printing, Customization Real-time update, Voice interaction, Data export, Sonification) |
| Collaborative Robot Arm |
| Interactive Assistive Courseware |
| Keyboard Accessibility |
| Mobile Health Assistive Applications |
| Mobile-assisted and Gamification-based language |
| Multimedia Training Stimuli |
| TalkBack |
| Teleguidance-based remote navigation assistance |
Source: author.
Among the other technologies evaluated by the articles, we highlight the evaluation of websites in general, mobile apps, government websites, software in general and educational software, as illustrated in Figure 11.
Figure 11. Non-assistive Technologies evaluated
The answer to Q7 is that TalkBack is the type of assistive technology most supported by the models we investigated. However, in general, websites are evaluated the most due to accessibility heuristics.
To answer Q8, what had to be done was to review all the articles selected, and to identify and group the groups of individuals characterized as the target audience of each study. It was noticed that the group of individuals with visual impairments (including blind people, people with low vision and other individuals with visual problems) are the most frequent focus of research involving heuristic assessment for accessibility. Figure 12 illustrates the percentage of occurrence of each group of individuals described in the SLR articles.
Figure 12. Occurrence of the target public
Table 10 presents a description of the groups of individuals who were selected as the target audience of the research, the number of occurrences of these groups in the articles and, in some cases, some observations about the grouping. Some studies focused on more than one group of individuals, so the overall sum exceeds the number of articles analyzed in this systematic review.
Table 10 – Target public of the studies
| Target audience of studies | Occurrences | Observations |
| People with a visual impairment | 29 | These include blind, low vision, contrast problem and other visually- impaired users |
| People with disabilities | 21 | Unspecified |
| People in general | 17 | Unspecified |
| Target audience of studies | Occurrences | Observations |
| Educational website users | 10 | These include administrators, students (with unspecified disabilities or not), teachers |
| The elderly | 10 | |
| People with Autism Spectrum Disorder (ASD) | 6 | |
| People with a physical or sensory disability | 5 | |
| People with a hearing impairment | 4 | |
| Government website users | 4 | |
| People with intellectual disabilities | 3 | |
| People with learning disabilities | 3 | |
| Stakeholders involved with the product | 2 |
|
| People with Down Syndrome | 1 | |
| People with speech impairments | 1 | |
| People with Attention Deficit Hyperactivity Disorder (ADHD) | 1 | |
| People with photosensitivity | 1 |
Source: author.
In answer to Q9, the existing limitations and gaps most cited in the works that address heuristic evaluation models are: the technological resources used; the focus of the studies, which are not always focused only on accessibility; the limitation of research related to the objectives addressed here in this SLR; and the diversity of the population studied. The number of times each type of limitation appears in the sample for this review is shown in Table 11 below. It is important to point out that different articles cited more than one type of different limitation, thus causing the total number of occurrences to be greater than the number of articles in the sample.
Table 11 – Limitations of the studies
| Limitations | Frequency |
| Limitations on the technological resources used | 32 |
| Accessibility Assessment is not the focus of the search | 24 |
| Limited search | 24 |
| Diversity of the population studied | 18 |
| Barriers during interactions or data collect with stakeholders | 7 |
| The result is at the preliminary stage | 6 |
| Cost | 5 |
| Time | 5 |
| No emphasis on cultural, user’s needs and contextual aspects | 4 |
| Researchers’ bias and generalizability of their results | 4 |
| Old guidelines | 2 |
| Subjective guidelines | 2 |
| Knowledge of the target population | 1 |
Source: author.
Figure 13 shows a graph that lists the limitations mentioned as a percentage of their total number, and thus shows the most frequently mentioned limitations.
Figure 13 – Frequency of the limitations
To answer question Q10, the information had to be extracted from heuristics used in each of the studies reported by the articles in this SLR. Heuristics typically belong to a specific heuristic group or repository. As shown in Table 12, the groups, repositories, references and heuristic studies mentioned in more than one article were described.
Table 12 – Sources of the Heuristics
| Heuristics | Frequency |
| WCAG 2.0 | 28 |
| WCAG 2.1 | 11 |
| Nielsen | 10 |
| WCAG 1.0 | 9 |
| Heuristics | Frequency |
| US Section 508 standards | 8 |
| WCAG | 7 |
| W3C | 5 |
| Interview | 4 |
| WAI-ARIA | 4 |
| Own guidelines | 4 |
| Experiment | 3 |
| ISO 9241-11 | 3 |
| ISO/IEC 40500 | 2 |
| Silva, Holden, & Jordan, 2015 | 2 |
Source: author.
Considering all the citations from the final sample, a graph was constructed that shows the proportional relationship between the main sources of cited heuristics that are grouped, regardless of versioning, and the other less cited sources. This graph is shown in Figure 14.
Figure 14 – Sources of the Heurístics
From Figure 14, note that the most recurrent sources are the different versions of the W3C WCAG, which is authored by the same group that provides other internationally recognized sets of guidelines, and which are also cited in the final sample set of this SLR (UAAG; FERATI; VOGEL, 2020), ATAG (FERATI; VOGEL, 2020) WAI-ARIA (COSTA; DUARTE, 2017; FERATI; VOGEL, 2020; GARCÍA-SANTIAGO; OLVERA-LOBO, 2021) but are included in the Others category of the graph presented. There are also some studies that cited ISO/IEC 40500 (MARCO; ALONSO; QUEMADA, 2019; NAVARRETE; LUJÁN-MORA, 2018), which is the equivalent of WCAG 2.0. Therefore, the answer to Q10 is: the heuristics most analyzed by the published models are the heuristics described by WCAG (Perceptibility, Operability, Understanding and Robustness) (CALDWELL et al., 2008).
The metrics most used in the assessments are the WCAG compliance levels (A, AA, AAA) (CALDWELL et al., 2008). This is the answer to Q11, as shown in Table 13, which lists all the sources (cited more than once) from which the assessment metrics were extracted. From a total of 85 sources of different metrics cited in the articles, only 15 were cited more than once and only 1 source (W3C WCAG) stands out with a difference of 29 citations more than the other most flagged sources (Literature Review and Questionnaire).
Table 13 – Sources of the Metrics
| Source of the Metrics | Citations |
| W3C WCAG | 36 |
| Literature Review | 7 |
| Questionnaire | 7 |
| Survey | 6 |
| Interview | 6 |
| Experiment | 6 |
| Own study | 4 |
| Qualitative research | 3 |
| Nielsen | 3 |
| WAI | 2 |
| Section 508 | 2 |
| Pilot Test | 2 |
| NTC 5854 | 2 |
| Case Study | 2 |
| ARIA standards | 2 |
Source: author.
To answer the last research question of paper, what had to be investigated for all the assistive technologies studied was which evaluation model was used in the study. It was observed, however, that the number of assistive technologies themselves is low in relation to the total of technologies mentioned, and there was no repetition of the occurrence of assistive technologies in the sample articles, with the exception of TalkBack, thus making it impossible to make an association between the type of technology used and the model used to evaluate the heuristics.
The answer to Q12 is in part no. There is no evidence of an association between the model to assess accessibility and the assistive technology assessed. When observing the studies that involved the evaluation of TalkBack (MADEIRA et al., 2021; ROBLES et al., 2019; YAN; RAMACHANDRAN, 2019), it is noticed that the models mentioned are diverse and do not repeat themselves. In a quick check, the evaluation model most cited in the articles of this SLR (WCAG standards) does not present an association with any specific technology, as no repetition pattern was observed. However, a relationship could be established between the main non-generalized target audiences and the assessment models found in the literature as shown in Table 14. Note that the WCAG pattern is the most recurrent in studies aimed at the visually impaired, while questionnaires and interviews are the most used to evaluate products aimed at the elderly, while Wave and AChecker appear more in surveys focused on people with physical or sensory disabilities.
Table 14 – Models vs Target audience
| Models/ Target audience | The visually impaired | The elderly | People with Autism Spectrum Disorder (ASD) | People with a physical or sensory disability | The hearing impaired |
| WCAG | 5 | ||||
| AChecker | 3 | 2 | 1 | ||
| Interview | 3 | 2 | 1 | ||
| Brajnik | 2 | ||||
| Questionnaire | 2 | 2 | |||
| eXaminator | 1 | 1 | 1 | ||
| Wave | 1 | 2 | 1 | ||
| TAW | 1 | 1 | 1 | ||
| Models/ Target audience | The visually impaired | The elderly | People with Autism Spectrum Disorder (ASD) | People with a physical or sensory disability | The hearing impaired |
| Device-interaction model own | 1 | 1 | 1 | 1 | |
| Ivarsson and Gorschek (2011) | 1 | 1 | 1 | ||
| Experiment | 1 | 1 | |||
| Case Study | 1 | 1 | |||
| Survey | 1 | 1 |
Source: author.
4. DISCUSSION, FUTURE PATHS AND TRENDS
This research presented the analysis of the results obtained from the grouping of articles in an SLR. In total, 101 articles were grouped according to year of publication, references, authors, models, technologies, target audience, limitations, heuristics and metrics. Based on the presentation of the results, a set of research questions was drawn up and answered. Table 15 presents the conclusions from the analysis carried out on the set of 12 research questions initially presented in Table 1.
Table 15 – Conclusions of the Research.
| Research Questions | Description |
| Q1 | Yes, the number of models used in the heuristic assessment for accessibility has been growing year after year |
| Q2 | Yes. The number of articles citing heuristic assessment for accessibility decisions has grown, but new articles have not had much time to be cited more often |
| Q3 | Al-Faries A. (2013), Al-Khalifa H.S. (2012), Al-Khalifa H.S. (2017), Vigo M. (2013) and Abanumy A. (2005) |
| Q4 | Acosta-Vargas P., Doush I.A. and Ismailova R. |
| Q5 | Yes. Heuristic evaluation models are more present in journals focused on operational research |
| Q6 | The most cited and used models in heuristic assessments for accessibility scientifically documented and indexed by Scopus and Web of Science are: WCAG standards, AChecker and WAVE tools |
| Q7 | TalkBack is the type of assistive technology most supported by the investigated models, however, in general, websites are the most evaluated product due to accessibility heuristics |
| Q8 | The group of individuals with visual impairments (including blind people, people with low vision and other individuals with visual problems) are the most frequent focus of research involving heuristic assessment for accessibility |
| Q9 | The existing limitations and gaps most cited in the works that address heuristic evaluation models are: the technological resources used; the focus of studies, which are not always focused only on accessibility; the limitation of research related to the objectives addressed here in this SLR; and the diversity of the population studied |
| Q10 | The heuristics most contemplated by the published models are the heuristics described by WCAG (Perceptibility, Operability, Understanding and Robustness) |
| Q11 | The most used metrics in assessments are WCAG compliance levels (A, AA, AAA) |
| Q12 | There is no evidence of an association between the model to assess accessibility and the assistive technology assessed. However, the WCAG pattern is the most recurrent in studies aimed at the visually impaired, while questionnaires and interviews are the most used to evaluate products aimed at the elderly, while Wave and AChecker appear more in surveys focused on people with physical or sensory disabilities |
Source: author.
Given the results, the predominance of the use of heuristics and metrics from WCAG can be confirmed, as can the entire evaluative context of WCAG. Furthermore, the growth in the number of published articles in the area demonstrates the importance of heuristic evaluation in supporting accessibility in software products.
In summary, the main contributions of this study are to have conducted a review of the state of the art of heuristic assessment for accessibility that can serve as a basis for future research in the area of digital accessibility; to set and answer a set of research questions relevant to the literature; to extend previous SLRs as to the scope of the research by inserting approaches related to the models, heuristics and metrics that are most used in assessing the accessibility of technological products.
As to future lines of research, more specific investigations related to using different assistive technologies supported by heuristic evaluations are suggested. Investigations specifying a defined target audience, as well as setting and answering new research questions using the database created in this study. Future studies should continue to refine heuristic methods, and thus socialize the best accessibility practices.
Future research may also propose new methods for evaluating digital resources. Tests of these new methods with well-defined users are suggested to identify the barriers they may have in using the technological resources covered in the tests. Furthermore, a complement to this is suggested, namely, a survey of stakeholders involved with software development, to find out if they know the accessibility guidelines disseminated in the literature and if they have already applied some of the guidelines in software products.
Moreover, a study on the metrics used in the accessibility assessment will be necessary, in order to analyze the validity of existing metrics and propose improvements that allow the effective use of these techniques in software projects.
Although there are advantages in assessing accessibility in decision-making in software construction, that are well spread in the literature, there are still challenges to be explored in the field. Implementing heuristic evaluation for decisions on accessibility in organizations and the proper use of heuristics is widespread in the search for solutions in the most different areas of knowledge (VENTURI, 1995). Evaluative models can take time and involve financial costs and effort. Thus, the most commonly used models allow for the flexibility of some of their parameters in order to facilitate their applicability. Many hybrid models have emerged which combine automated tools, well-established theoretical principles and design techniques.
Regarding this SLR, assistive technologies correspond to only 13% of the technologies mentioned in the set of selected articles (see Figure 10) while 14% of the target audience of the studies is the general public, without characterization of the users (see Figure 12). This shows that although most studies are focusing on specific users, the technologies being evaluated were not designed to directly assist a specific group. From this research, it was noticed that there is no evidence of an association between the model to assess accessibility and the assistive technology assessed, but it was possible to establish which models are most used by specific audiences most cited in the studies.
To conclude, maintaining the analysis of permanent limitations in evaluative models is suggested as important for future perspectives. Comparison between different review studies should inform whether or not some of these limitations have been alleviated. From this study, the most frequent limitations refer to the technological resources used, the focus of the studies, research direction and the diversity of the population studied.
5. CONCLUSIONS
The evolution of decision-making techniques has transformed the way researchers analyze problems when developing a software/hardware solution. This article takes the form of a systematic literature review, and set out to provide a broad view of how heuristic assessment models have been used to support accessibility, whether evaluating assistive technologies or evaluating any other technologies in recent years. Overall, this SLR confirmed findings from previous studies and added new research questions to those SLR-format studies that have been used to date.
Taking into account the continued optimization of the set of articles selected for future studies, as it is a very specific topic, it is suggested that new filtering processes be considered, in order to avoid further variations in the use of evaluators for projects that value by accessibility being selected outside the context of this research. This SLR shows that some selected applied studies already demonstrate limitations regarding the objectives and specification of the assessment with a focus on accessibility. As for the authors and sources of the cataloged references, they were described by making use of ough indices of impact and frequency of appearances, ignoring the thematic depth established by their related articles and the immersion in the concepts in the area.
The number of published papers on heuristic assessment for decision-making in accessibility have increased in recent years, and this was particularly noticeable between 2018 and 2019. The Web Accessibility Directive, approved by the European Parliament obliged the member countries of the European Union to comply with accessibility standards of its public websites as early as September 2018. As a result, accessibility studies were naturally developed and evaluative models for accessibility became more widely used. Added to this directive, the European Accessibility Act was initiated in 2019, and has become a world reference on digital accessibility legislation.
The main objective of this SLR was to provide an overview of the state of the art of a research field that may imply pointing out gaps in the literature. In short, the techniques and methods used to ensure the main objective are intended to be solid and are subject to updating. The results in the present study showed the relevance of the research questions, and led to interesting discoveries about the literature of heuristic evaluations for decisions in accessibility.
ACKNOWLEDGMENTS
This work has been partially supported by CIn/Samsung, SiDi, CNPq (315245/2020-4, 305768/2016-6, 428832/2018-0) and Capes (001) for which the authors are grateful.
REFERENCES
ABNT – Associação Brasileira de Normas Técnicas Ergonomia da interação humano-sistema: parte 210 : projeto centrado no ser humano para sistemas interativos. 2011. Available: https://moodle.ufsc.br/pluginfile.php/4330158/mod_resource/content/4/ABNT_NBR_ISO_9241-210_2011.pdf. Acess in: 23 mar. 2023.
ACOSTA-VARGAS, Patricia et al. Evaluation of accessibility in teleconferencing systems for low vision users during covid-19. RISTI-Revista Iberica de Sistemas e Tecnologias de Informacao, p. 205-218, 2020.
ACOSTA-VARGAS, Patricia; SALVADOR-ULLAURI, Luis Antonio; LUJÁN-MORA, Sergio. A heuristic method to evaluate web accessibility for users with low vision. IEEE Access, v. 7, p. 125634‑125648, 2019. DOI: https://doi.org/10.1109/ACCESS.2019.2939068.
AKGÜL, Yakup. Accessibility, usability, quality performance, and readability evaluation of university websites of Turkey: a comparative study of state and private universities. Universal access in the information society, v. 20, n. 1, p. 157-170, 2021. DOI :https://doi.org/10.1007/s10209-020-00715-w.
ALAJARMEH, Nancy. Evaluating the accessibility of public health websites: an exploratory cross-country study. Universal Access in the Information Society, p. 1-19, 2021. DOI :https://doi.org/10.1007/s10209-020-00788-7.
ALMEIDA-FILHO, Adiel Teixeira de; SILVA, Diogo Ferreira de Lima; FERREIRA, Luciano. Financial modelling with multiple criteria decision making: a systematic literature review. Journal of the Operational Research Society, 1-19, 2020. DOI: https://doi.org/10.1080/01605682.2020.1772021.
ALSAEEDI, Abdullah. Comparing web accessibility evaluation tools and evaluating the accessibility of webpages: proposed frameworks. Information, v. 11, n. 1, p. 40, 2020. DOI :https://doi.org/10.3390/info11010040.
ARAÚJO, Adolpho Guido; CARNEIRO, Arnaldo Manoel Pereira; PALHA, Rachel Perez. Sustainable construction management: a systematic review of the literature with meta-analysis. Journal of Cleaner Production, v. 256, p. 120350, 2020. DOI: https://doi.org/10.1016/j.jclepro.2020.120350.
ARIA, Massimo; CUCCURULLO, Corrado. bibliometrix: an R-tool for comprehensive science mapping analysis. Journal of informetrics, v. 11, n. 4, p. 959-975, 2017. DOI: https://doi.org/10.1016/j.joi.2017.08.007.
BABU, Rakesh; XIE, Iris. Haze in the digital library: design issues hampering accessibility for blind users. The Electronic Library, v. 35, n. 5, p. 1052-1065, 2017. DOI: https://doi.org/10.1108/EL-10-2016-0209.
BADAMPUDI, Deepika. Reporting ethics considerations in software engineering publications. In: 2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), p. 205-210, 2017. DOI: https://doi.org/10.1109/ESEM.2017.32.
BAI, Aleksander; STRAY, Viktoria; MORK, Heidi. What methods software teams prefer when testing web accessibility. Advances in Human-Computer Interaction, v. 2019, p. 1-14, 2019. DOI: https://doi.org/10.1155/2019/3271475.
CALDWELL, Ben et al. Web Content Accessibility Guidelines 2.0 (WCAG). W3C, 2008. Available in: https://www.w3.org/WAI/WCAG20/versions/guidelines/wcag20-guidelines-20081211-a4.pdf. Acess in: 23 mar. 2023.
CAMPOVERDE-MOLINA, Milton; LUJAN-MORA, Sergio; GARCIA, Llorenc Valverde. Empirical studies on web accessibility of educational websites: a systematic literature review. IEEE Access, v. 8, p. 91676-91700, 2020. DOI: https://doi.org/10.1109/ACCESS.2020.2994288.
CEN & CENELEC. EN 17161:2019: design for all: accessibility following a design for all approach in products, goods and services: extending the range of users. Brussels: Cen-Celenic, 2019.
CHI, Chia-Fen; TSENG, Li-Kai; JANG, Yuh. Pruning a decision tree for selecting computer-related assistive devices for people with disabilities. IEEE Transactions on Neural Systems and Rehabilitation Engineering, v. 20, n. 4, p. 564-573, 2012. DOI: :https://doi.org/10.1109/TNSRE.2012.2193419.
COSTA, Daniel; DUARTE, Carlos. Visually impaired people and the emerging connected TV: a comparative study of TV and Web applications’ accessibility. Universal Access in the Information Society, v. 16, p. 197-214, 2017. DOI: https://doi.org/10.1007/s10209-016-0451-6.
ECK, Nees Van; WALTMAN, Ludo. Software survey: VOSviewer, a computer program for bibliometric mapping. Scientometrics, v. 84, n. 2, p. 523-538, 2010. DOI: https://doi.org/10.1007/s11192-009-0146-3.
EUSÉBIO, Celeste; SILVEIRO, André; TEIXEIRA, Leonor. Website accessibility of travel agents: an evaluation using web diagnostic tools. Journal of Accessibility and Design for All, v. 10, n. 2, p. 180‑208, 2020. DOI: https://doi.org/10.17411/jacces.v10i2.277.
EUROPEAN COMMISSION . Proposal for a directive of the European Paliament and of the Concil on the approximation of the laws, regulations and administrative provisions of the Member States as regards the accessibility requirements for products and services. EUR‑Lex, 2015. Available in: https://eur‑lex.europa.eu/legal‑content/EN/TXT/?uri=COM%3A2015%3A0615%3AFIN. Acess in: 23 mar. 2023.
FERATI, Mexhid; VOGEL, Bahtijar. Accessibility in web development courses: A case study. Informatics. v. 7, n. 1, p. 1‑15, 2020. DOI: https://doi.org/10.3390/informatics7010008.
GAMACHE, Stephanie et al. Measure of environmental accessibility (MEA): development and inter-rater reliability. Journal of accessibility and design for all, v. 8, n. 1, p. 1-32, 2018. DOI: https://doi.org/10.17411/jacces.v8i1.141.
GARCÍA-SANTIAGO, Lola; OLVERA-LOBO, Maria-Dolores. How accessibility guidelines are used in Spanish World Heritage websites: an exploratory study. Library Hi Tech, v. 39, n. 1, p. 144-165, 2021. DOI: https://doi.org/10.1108/LHT-05-2019-0113.
HASSENZAHL, Marc. The thing and I: understanding the relationship between user and product. Funology 2: from usability to enjoyment, p. 301-313, 2018. DOI: https://doi.org/10.1007/978-3-319-68213-6_19.
HASSOUNA, Mohammed Saleh; SAHARI, Noraidah; ISMAIL, Amirah. University website accessibility for totally blind users. Journal of Information and Communication Technology, v. 16, n. 1, p. 63-80, 2017. DOI: https://doi.org/10.32890/jict2017.16.1.8218.
HORTON, Emily L. et al. A review of principles in design and usability testing of tactile technology for individuals with visual impairments. Assistive technology, v. 29, n. 1, p. 28-36, 2017. DOI: https://doi.org/10.1080/10400435.2016.1176083.
ISMAILOVA, Rita; KIMSANOVA, Gulida. Universities of the Kyrgyz Republic on the Web: accessibility and usability. Universal Access in the Information Society, v. 16, n. 4, p. 1017-1025, 2017. DOI:https://doi.org/10.1007/s10209-016-0481-0.
LAENGLE, Sigifredo et al. Forty years of the European Journal of Operational Research: A bibliometric overview. European Journal of Operational Research, v. 262, n. 3, p. 803-816, 2017. DOI: https://doi.org/10.1016/j.ejor.2017.04.027.
LAKSHMI, Krishnan; KUMAR, PD Madan; DAS, Himangshu. Design considerations for a dental health care for patients with special needs. Journal of accessibility and design for all, v. 8, n. 1, p. 80‑101, 2018. DOI: https://doi.org/10.17411/jacces.v8i1.168.
MADEIRA, Stephane et al. Accessibility of mobile applications for tourism—is equal access a reality? Universal Access in the Information Society, v. 20, p. 555-571, 2021. DOI: https://doi.org/10.1007/s10209-020-00770-3.
MAFFEO, Bruno. Engenharia de software e especificação de sistemas. Rio de Janeiro: Campus, 1992.
MARCO, Lourdes; ALONSO, Álvaro; QUEMADA, Juan. An identity model for providing inclusive services and applications. Applied Sciences, v. 9, n. 18, p. 3813, 2019. DOI: https://doi.org/10.3390/app9183813.
MARIZ, Fernanda Bar; ALMEIDA, Mariana R.; ALOISE, Daniel. A review of dynamic data envelopment analysis: state of the art and applications. International Transactions in Operational Research, v. 25, n. 2, p. 469‑505, 2018. DOI: https://doi.org/10.1111/itor.12468.
MARTINS, José; GONÇALVES, Ramiro; BRANCO, Frederico. A full scope web accessibility evaluation procedure proposal based on Iberian eHealth accessibility compliance. Computers in Human Behavior, v. 73, p. 676-684, 2017. DOI: https://doi.org/10.1016/j.chb.2016.12.010.
NAGARAJU, M.; CHAWLA, Priyanka. A methodical search of web content accessibility for visual disabled people. International Journal of Engineering and Advanced Technology, v. 8, n. 6, p. 1713‑1719, 2019. DOI: https://doi.org/10.35940/ijeat.F8425.088619.
NASCIMENTO, Kayo Renato Da Silva; ALENCAR, Marcelo Hazin. Management of risks in natural disasters: a systematic review of the literature on NATECH events. Journal of Loss Prevention in the Process Industries, v. 44, p. 347-359, 2016. DOI: https://doi.org/10.1016/j.jlp.2016.10.003.
NATHAN, S. S. et al. Accessibility, disability and deaf: a review. International Journal of Engineering and Technology, v. 7, n. 4.19, p. 369-371, 2018.DOI: https://doi.org/10.14419/ijet.v7i3.12.16109.
NAVARRETE, Rosa; LUJÁN-MORA, Sergio. Bridging the accessibility gap in Open Educational Resources. Universal Access in the Information Society, v. 17, n. 4, p. 755-774, 2018. DOI:https://doi.org/10.1007/s10209-017-0529-9.
NIELSEN, Jakob. Usability inspection methods. New York: John Wiley & Sons, 1994.
PEREA, Lourdes González; GALÁN, Jesús Hernández; VILLARINY, Natalí González. The presence of universal accessibility in doctoral dissertations deposited in Spain between 1998-1999 and 2017-2018 academic years. Journal of accessibility and design for all, v. 9, n. 2, p. 118-140, 2019. DOI: https://doi.org/10.17411/jacces.v9i2.213.
PAIVA, Débora Maria Barroso; FREIRE, André Pimenta; FORTES, Renata Pontin de Matos. Accessibility and software engineering processes: a systematic literature review. Journal of Systems and Software, v. 171, p. 110819, 2021. DOI: https://doi.org/10.1016/j.jss.2020.110819.
PEREIRA, Valdecy; COSTA, Helder Gomes. A literature review on lot size with quantity discounts: 1995-2013. Journal of Modelling in Management, v. 10, n. 3, p. 341-359, 2015. DOI: https://doi.org/10.1108/JM2-07-2013-0029.
PERIANES-RODRIGUEZ, Antonio; WALTMAN, Ludo; ECK, Nees Jan Van. Constructing bibliometric networks: A comparison between full and fractional counting. Journal of Informetrics, v. 10, n. 4, p. 1178-1195, 2016. DOI: https://doi.org/10.1016/j.joi.2016.10.006.
RANADA, Åsa Larsson; LIDSTRÖM, Helene. Satisfaction with assistive technology device in relation to the service delivery process – a systematic review. Assistive Technology, v. 31, n. 2, p. 82‑97, 2019. DOI: https://doi.org/10.1080/10400435.2017.1367737.
ROBLES, Teresita de Jesús Álvarez et al. Adapting card sorting for blind people: Evaluation of the interaction design in TalkBack. Computer Standards & Interfaces, v. 66, p. 103356, 2019. DOI: https://doi.org/10.1016/j.csi.2019.103356.
RUSCHEL, Edson; SANTOS, Eduardo Alves Portela; LOURES, Eduardo de Freitas Rocha. Industrial maintenance decision-making: A systematic literature review. Journal of Manufacturing Systems, v. 45, p. 180‑194, 2017. DOI: https://doi.org/10.1016/j.jmsy.2017.09.003.
SALVADOR-ULLAURI, Luis et al. Combined method for evaluating accessibility in serious games. Applied Sciences, v. 10, n. 18, p. 6324, 2020. DOI: https://doi.org/10.3390/APP10186324.
SHNEIDERMAN, Ben. Tree visualization with tree-maps: 2-d space-filling approach. ACM Transactions on graphics, v. 11, n. 1, p. 92-99, 1992.
SINGER, Janice; VINSON, Norman G. Ethical issues in empirical studies of software engineering. IEEE Transactions on Software Engineering, v. 28, n. 12, p. 1171-1180, 2002. DOI: https://doi.org/10.1109/TSE.2002.1158289.
SMALL, Henry. Co-citation in the scientific literature : a new measure of the relationship between two documents. Journal of the American Society for information Science, v. 24, n. 4, p. 28-31, 1974.
STITZ, Tammy; BLUNDELL, Shelley. Evaluating the accessibility of online library guides at an academic library. Journal of Accessibility and Design for All, v. 8, n. 1, p. 33-79, 2018. DOI: https://doi.org/10.17411/jacces.v8i1.145.
TERASHIMA, Mikiko; CLARK, Kate. Measuring economic benefits of accessible spaces to achieve’meaningful’access in the built environment: a review of recent literature. Journal of Accessibility and Design for All, v. 11, n. 2, p. 195-231, 2021. DOI: https://doi.org/10.17411/jacces.v11i2.274.
UMARJI, Medha; SEAMAN, Carolyn. Why do programmers avoid metrics?. In: Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement, p. 129-138, 2008. DOI: https://doi.org/10.1145/1414004.1414027.
VENTURI, Gustavo. O universalismo ético: Kohlberg e Habermas. Lua Nova: Revista de Cultura e Política, p. 67‑84, 1995. DOI: https://doi.org/10.1590/s0102‑64451995000200005.
WALTMAN, Ludo; VAN ECK, Nees Jan; NOYONS, Ed CM. A unified approach to mapping and clustering of bibliometric networks. Journal of informetrics, v. 4, n. 4, p. 629-635, 2010. DOI: https://doi.org/10.1016/j.joi.2010.07.002.
WENTZ, Brian et al. Documenting the accessibility of 100 US bank and finance websites. Universal Access in the Information Society, v. 18, p. 871-880, 2019. DOI: https://doi.org/10.1007/s10209-018-0616-6.
WOHLIN, Claes et al. Experimentation in software engineering. Berlin: Springer; Berlin: Heidelberg, 2012.
YAN, Shunguo; RAMACHANDRAN, P. G. The current status of accessibility in mobile apps. ACM Transactions on Accessible Computing, v. 12, n. 1, p. 1-31, 2019. DOI: https://doi.org/10.1145/3300176
ZOPOUNIDIS, Constantin et al. Multiple criteria decision aiding for finance: An updated bibliographic survey. European Journal of Operational Research, v. 247, n. 2, p. 339-348, 2015. DOI: https://doi.org/10.1016/j.ejor.2015.05.032
[1] Teacher. ORCID: 0000-0003-4467-0113. CURRICULUM LATTES: http://lattes.cnpq.br/9433837114578364.
[2] Doctor. ORCID: 0000-0002-6491-9783. CURRICULUM LATTES: http://lattes.cnpq.br/3252289006108114.
[3] Advisor. Doctor. ORCID: 0000-0001-6069-3601. CURRICULUM LATTES: http://lattes.cnpq.br/9944976090960730.
Sent: 02 of March, 2023.
Approved: 15 of March, 2023.
