A System to Generate SignWriting for Video Tracks Enhancing Accessibility of Deaf People

been developed and described in [7]. This component adapts the application content for people with severe auditory disability by translating standard web applications to web applications based on SingWriting, a way of writing Sign Language. The present paper describes a solution that intends to complement this adapter, eliminating existing barriers in video subtitling. If a web page provides a standard subtitling file, the proposed solution translates the available plain text to vector graphics, representing SingWriting, which accompany the video sequence, allowing the perception of audio information of the video by Deaf people. Specifically, the adapter supports translation of subtitles written in English language into SignWriting for American Sign Language


I. Introduction
W EB content, specifically multimedia content, has increased much on the Internet during recent years [1]. New tools facilitating video editing and uploading to Internet have led naturally to the massive generation of videos, many of them of didactic, informative or illustrative nature [2]. Unfortunately, the poor accessibility level of much multimedia content increases the gap existing in access to information between not impaired and impaired people.
Therefore, there are organizations promoting recommendations in order to reach the adequate accessibility level, being the Web Content Accessibility Guidelines (WCAG) 2.0 [3], proposed by the W3C, one of the most important recommendations.
Moreover, in different countries, legislations exist that state that public administration websites, as well as those of other institutions that benefit from public financing, must comply with a certain level of accessibility. For example, in Spain, the websites of public administrations or other institutions receiving public financing or giving general services, must comply with the AA level of the WCAG. In United States, some states have codified laws to ensure web accessibility while others inform about standards and guidelines, applying to state entities [4].
However, in spite of these efforts, most multimedia content on the Internet is not accessible. Developing an accessible website is not only expensive but also difficult because there are some aspects that depend on interpretation. For example, while a textual description of an image could be adequate for a visual impaired person, it could not be appropriate for a person with cognitive problems [5].
In order to face these drawbacks, Crespo et al. [5] propose a new approach allowing the collaborative creation of different accessible adaptations of websites. A step-by-step assistant or wizard guides the semiautomatic adaptation that can be done by any user and that is saved on a repository, in which all adapted profiles of websites are shared.
(ASL), one of the visual-gestural languages most commonly used by the Deaf communities in the United States of America.
Next section of the paper explains reasons why Deaf People experience difficulties when consuming multimedia web content. Besides, some state-of-art solutions are described. Section III gives an overview of the accessibility global platform and the auditory disability accessibility adapter, both mentioned in previous paragraph. Besides, section III presents the solution that makes subtitled videos accessible for the Deaf, whose evaluation is presented in Section IV. Section V presents main conclusions and future works. Most multimedia content on the Internet is not accessible for Deaf people, who remain behind the digital barrier. Complex texts and multimedia content without alternative text are the most critical barriers for hearing impaired users according to the study of Pascual et al. [8]. Including captions or transcriptions to a video can make it more accessible [1] but reading oral language is still difficult for some deaf people, especially for those with prelocutive auditory disability, who find many difficulties using web sites. Prelocutive indicates that deafness appears before learning of language. Technical advances such as cochlear implants allow profoundly deaf to hear, as it simulates the function of the inner ear, providing sound signals to the brain. At present, many young deaf people benefitting from cochlear implants are achieving good results when learning oral languages. However not all people can be implanted and not all implanted people obtain an adequate proficiency level [9].
Oral languages are based on letters that represent sounds, which for prelocutive deaf, are completely unknown. Therefore, for these people, learning reading oral languages is merely memorizing sequences of letters [7]. This has consequences such as the fact that deaf children are also affected in their linguistic and, therefore, cognitive development because language is the tool that organizes our thoughts. This is used to describe the concepts we learn and mediates between the subjects and the objects of knowledge [10].
Therefore, for those with auditory disability who cannot benefit from the cochlear implant, a visual language will be their preferred language as it is received through vision, which is the natural communication channel for them. A sing language is a visual-gestural language expressed with hands, body and facial gestures, which is used, among others, by the Deaf. Sign languages allow the Deaf to communicate unhindered, focusing on the message and not on the medium. Moreover, sign languages are organized in the brain as spoken languages. Although many signs mimic the meaning of the concept expressed, the position and movement of the hands for many of them do not reveal the meaning. Therefore, a Deaf person using Spanish Sign Language will not understand someone using American Sign Language, as it happens with spoken language [11].
As well as there is not a unified way of signing in every country, Deaf people have not yet stablished a unified way of writing sign languages. SignWriting is a proposal candidate for that [12]. As above mentioned, reading spoken languages is a very difficult task for the Deaf because written words are made of letters representing sounds, thus learning to read is a task consisting in memorizing. SignWriting represents signs on the signer's perspective, that is, how the signers see their own hands when they sign. Also, it represents head, shoulder and trunk movements, as well as facial expressions, which can change totally the meaning of a sign [13]. Fig. 1 contains the word 'SignWriting' expressed in SingWriting. In this specific case, the facial expression does not take a role when expressing the sing so the graphic does not include the face expression. The upper part of the graphic shows the positions and configuration of hands and the arrows indicate the movement of the hands. Fig. 2 represents the word 'Hello', which includes the smiling facial expression.
As it can be seen, the elements composing SignWriting can be generated as vector graphics, easily treated by current web applications. Therefore, the proposed system described in this paper will use vector graphics to show SignWriting as subtitles of video content, enhancing video visualization experience for Deaf people.

B. Related Works
There are several systems that translate texts and video subtitles to Sign Language automatically. San Segundo et al. [14] have developed a machine translation system from Spanish to Spanish Sign Language. The system uses speech recognition, which is automatically translated and interpreted by an avatar animation module. Araujo et al. [15] propose a solution that translates video subtitles and generates a sequence in Brazilian Sign Language in real time, respecting linguistic constructions and grammatical rules. Signs are interpreted by an avatar and the authors are working on strategies that allow human collaboration to maintain, improve and extend automatic translations rules of the system.
López-Ludeña et al. [16] propose a bidirectional system that translates speech into sign language and generates speech from sing language. The system recognizes speech and uses a memory-based translation strategy, using a set of sentences in oral language and their translation into sign language as a learning stage that will enable the system to translate similar sentences.
Most of translators to Sign Language correspond to automatic translators focused on specific domains of activity, in which vocabulary and expected sentence constructions are quite limited, such as a hotel reception [16], a bus company [17], renewing the identity card or the driver's license [14], for example.
Another approach which can be applied to open domains and intends to be cost effective, is the one described in [18]. To solve the needs of Deaf people accessing to multimedia content, an architecture is proposed that includes intelligence in the streaming server so that it distinguishes clients' characteristics and properties of communication to provide the most suitable content. Specifically, for clients with hearing disabilities, a middleware is proposed to translate the subtitles tracks to SignWriting. The work described in this paper completes this work, taking elements from this architecture to implement a translator to SignWriting for subtitled videos, complementing the adapter that translates standard web applications to applications based on SignWritting [7], that is, the graphical representation of Sign Language.

III. Solution to Global Accessibility
There is not a universal accessibility, given that the needs of each disability are very different and therefore accessibility requirements are also very different. The work of Crespo et al. [6] [7] describes a solution for those hearing impaired but it intends to be a global solution to accessibility based on two main principles: • Separating content and presentation: Content has to be firstly presented in a language that can be processed by a machine and not by the user. The purpose is that the user is able to select the representation of the content according to his needs (this can be SignWriting or HamNoSys for example).
• Automatic generation of content in a specific format: According to the targeted user, a specific formatted representation of content is generated. For a hearing impaired user, the representation can be based on sign language symbols coded as SVG format, as in the specific solution described in this paper.
Therefore, the problem of constructing an accessible website is separated into different parts (see Fig. 3). First, there is a core component, which facilitates the collaborative adaptation of websites. This is called the WAI Accessibility Adapter, which automatically converts any web application, regardless its accessibility level, to the expected accessibility level. As Fig. 3 shows, this adapter is an intermediate component between the application server and the client. It is totally independent of the specific requirements of the users according to their disability. This component transforms code served by the application server so that it conforms with level AA of WAI guidelines. One of the main problems of implementing accessible websites is covered by this component, that is, facilitating the delivery of accessible web content through a gateway, saving the costs that would involve to make accessible each website separately by each company. However, this component does not allow covering all the problems specific to the different disabilities by itself. Fig. 3. Overall architecture of the global accessibility solution (Own elaboration based on [7]. The special features required by a specific disability are achieved by different targeted-disability adapters that would act as kind of user agents. Then, there can be a specific component for those with visual disability and another for those auditory impaired. Any specific accessibility requirement should be solved by the corresponding specific adapter.
As the present paper describes a solution to accessibility problems of those with auditory disability, from this point the paper will focus on the Deaf People Accessibility Adapter.

A. WAI Accessibility Adapter
The WAI Accessibility Adapter is an intermediate component between a web application and its clients. When a client makes a request to a web application, instead of directly sending the request to the web application, the request is sent to the WAI Adapter. Then, the adapter sends the request to the Web application and apply some transformations to the content returned so that this content is accessible. The adapter sends the accessible content to the client as response. This adapter has two main components [7]: • Proxy adaptor: it receives the requests of the clients, forwards them to a specific browser instantiated in the pool component, obtaining the HTML generated from that request by the web application. Then, it applies the transformations to deliver accessible contents to the client.
• Pool of browsers: it receives the proxy requests and instantiates browsers, managing their assignments to specific clients. Instantiated browsers send requests to the web application and receive the corresponding HTML code, which is delivered to the proxy adaptor.

B. Deaf People Accessibility Adapter
The Deaf People Impairment Adapter is a middleware that allows translating typical web applications into SignWriting web applications. This adapter is based on SWMLSVG (SignWriting Markup Language for Scalable Vector Graphics), a language defined to visualize signwriting graphics in any web browser supporting the visualization of SVG. [19]. It is an effort to allow the interoperability of web applications using SignWriting. SWMLSVG language extends SWML to render SignWriting symbol sequences as vector graphics SVG. It allows easy visualization of SignWriting elements in any web browser supporting visualization of SVG graphics.

SWML (SignWriting Markup Language) is based on XML
The Adapter uses SVG format because it is more adaptable than other formats. This development is intended to be a solution for the Deaf but these people can also suffer from partial lack of vision, for example. Therefore, enlarging the size of images without losing quality could be needed for some users. Other benefits of using vector drawings instead of rasters are smaller storage space for simple images or easy edition [7].
The overall architecture of this component in charge of the adaptation of web content to SignWriting is shown in Fig. 4. The HTML document received from the WAI Adapter is processed by the SWMLSVG Adapter. This transforms the input HTML document into a HTML document with SWMLSVG inside. Two databases are used while undertaken the transformation: • The standard database SSS-XXXX-SVG, which contains, in written form, specifically in SVG format, the different possible configurations a sign can have. SignWriting dictionaries are sorted by Sign-Symbol-Sequence (SSS). Therefore, SSS has the same function for sign languages as the alphabet has for oral languages. A file SSS-XXXX-SVG serves as database of SVG images, which compose the SignWriting alphabet.
• A database (SBML), which is a kind of cache of symbols previously configured and then stored for quick use [7].

C. SignWriting Subtitling System for Video Streaming
As described previously, many webpages contain video content currently and, therefore, the Deaf People Accessibility Adapter has been extended to support subtitle translations. This section describes this new component of this adapter, a SignWriting subtitling system based on HTML5 <video> tag and Synchronized Multimedia Integration Language (SMIL) Timesheets. When the adapter detects that a webpage contains a video with subtitles, it takes the subtitles expressed in an oral language and generates subtitles in SignWriting format, timely synchronized with the video stream.
This synchronization is achieved thanks to SMIL TimeSheets, which can orchestrate documents compound of HTML and SVG drawings, separating timing and synchronization of elements in the document from content and presentation [20].
SMIL is a W3C standard XML-based language defined to describe the temporal behaviour of a multimedia presentation allowing synchronizing multimedia content [21]. SMIL Timesheets, also proposed by W3C, reuse some SMIL timing primitives but separate content, styling and timing, with the objective of facilitating authoring and handling of multimedia contents. While CSS (Cascading Style Sheets) defines the spatial layout and format of the elements of a web page, SMIL Timesheets play a similar role in the temporal aspect. SMIL Timesheets define which elements are shown at a certain time and can be reused in different documents, as CSS [22].
HTML5 specification introduced the <video> tag, which facilitates video integration into webpages. The video element includes the <track> tag in which subtitles and caption files can be specified so that they are shown while media is playing. The adapter will take the subtitle file referenced in the track element and generate a web document with SVG graphics showing a SignWriting sequence synchronized with video. HTML5 does not support including SVG code in the HTML track element, therefore SMIL Timesheets are used to synchronize vector drawing on web page with video stream. As SMIL is not natively supported by web browsers [20], the use of JavaScript is required to govern the document timing from SMIL data. Specifically the open-source JavaScript library timesheet.js is used to integrate SMIL and render SVG drawings synchronized with video stream, following the approach proposed by Cazenave et al. [20]. For translation of oral language subtitles to SignWriting, the databases used by SWMLSVG adapter shown in Fig. 4 are used (see Fig. 5). Currently the first prototype of the SignWriting Subtitling System has been developed and tested. It provides an interface to upload a video and a subtitle file, supporting subtitles expressed in SubRip Text format and written in English language, translating them into SignWriting for American Sign Language (ASL). Fig. 6 shows how SVG graphics are visualized synchronized with video. This first prototype has been tested providing a first and prompt feedback for developers. The experiment undertaken for testing is described in next section. The support of WebVTT is under development and complete integration into Deaf People Accessibility Adapter is in progress.

A. Method
The application performance of first prototype of the SignWriting Video Subtitling System has been validated by using different types of tests: • Unit tests: some unit tests on the most important components of the system are done.
• Usability tests: to evaluate the system by testing on users.
• Accessibility tests: both using an automatic evaluation accessibility tool and manually evaluating with a usability and accessibility checklist.
• Integration tests: these are carried out each time a specific functionality is implemented.
This section specifically describes the methods used for accessibility and usability testing, as these are crucial taken into account the target user and objective of the application, which is improving accessibility to video content of Deaf People. Two questionnaires were used during usability testing with users. The first one included general questions to know the computer literacy and computer habits of participants in the experiment, that is, if they use computer regularly and the type of activities they do with computers (working, leisure, social networking…).
Therefore, this first questionnaire served to classify the 13 participants according to their computer literacy profile and habits: • User 1: person in the age range of 20 to 35, regular user of computers and social networks, but without experience in digital certificates. 6 participants belong to this profile.
• User 2: person in the age range of 35 to 55, regular user of computers and Internet for leisure activities, but without experience in social networks and digital certificates. This group is made up of 3 participants.
• User 3: person in the age range of 35 to 55 with little experience using computers, Internet and social networks. This group is made up of 4 participants.
During the next stage, the participants visualized three videos of short duration by using the developed interface. After that, they answered the second questionnaire, which contained 17 1-to-5 rating scale questions to measure the following different usability aspects: • Ease of use: is browsing simple? Is it easy to access to a specific content? … • Functionality: are there some missed options? Are tasks done fluently? … • User interface: is size of signs adequate? Are colors of interface adequate? Is interface nice and clean? Is the application wellstructured? Is transition between pages well structured? Is the needed information delivered at each moment? Does the interface help to focus on the task that is being done?
For accessibility evaluation, WAVE accessibility evaluation tool is used [23], which is an online web service that helps to determine the accessibility of web content. This includes many checks for compliance issues contained in the Section 508 [24] and WCAG 2.0 guidelines. As automatic tools cannot check all issues of these guides, moreover only persons can determine true accessibility, the guide of evaluation of usability authored by Hassan and Martín [25], which includes a section to evaluate accessibility, is also used for validation. This is an extensive form that considers different aspects such as identity and information, language and writing, informative labels or signs, or page layout. Tests are done in several browsers such as Internet Explorer, Firefox or Google Chrome. Table 1 shows the average score obtained for each evaluated usability aspect through the questionnaire of 17 questions previously mentioned. The questions were rated with a 1-to-5 rating scale, being 1 the more negative response and 5, the more positive. Therefore satisfaction of participants with every aspect is in general high, being every aspect rated above 4. Fig. 7 shows the average score obtained for each aspect by the different groups of users. Even with a different computer literacy background, responses of the participants belonging to groups User 1 and User 2 were very similar, with a high score for every aspect. As expected due to their lack of experience with Internet and computers, participants of profile User 3 rated lower the different aspects, as shown in Fig 7. Anyway, average scores for the three aspects were higher than 3.5 and no question received a score lower than 3 for this group. WAVE tool is also used to check the accessibility of the application and no errors are detected. However, as said before, the absence of errors reported by an automatic tool does not mean that the page is accessible. Moreover some warnings are informed by this tool.

B. Results
Therefore, accessibility is also verified with the guide of evaluation of usability previously mentioned [25]. All criteria of the checklist of this guide are complied except the following ones: • There is no a high contrast between the font color (SignWriting graphic) and background.
• Users have not total control over the interface.
The results of usability testing show that participants are satisfied with the SignWriting subtitling tool. Not every word had its translated symbol, and this may be the reason why some users did not give a high score to question "Is the needed information delivered at each moment?". A dictionary generator tool allows to modify and extend dictionaries fluently once symbols are available, so this result can be improved in the future. Besides, as accessibility tests show, the application does not suffer from accessibility problems.

V. Conclusion
This paper describes an extension of a Deaf People Accesibility Adaptor that translates oral language subtitles of videos embedded in web pages to SignWriting. This extension is under development, and a first prototype has already been tested to give a prompt feedback to developers. This first test has probed that users are satisfied with the tool and that this potentially will enhance multimedia content browsing experience of Deaf Users.
More tests, with a higher number of participants, will be done when the tool is completely integrated into the Deaf People Accessibility Adapter. The translator failed to translate some words during tests already done, as they were not available in the dictionary. In order to measure these types of errors, next tests will include new metrics such as Sign Error Rate (SER). SER is the percentage of wrong signs in the translation output compared to a reference, which is provided by a human expert [14]. Besides, new recent instruments [26] [27] to evaluate usability and accessibility will be considered to assure an efficient evaluation of user experience in next experiments.
As future work, some complementary existing solutions will be explored. For example, the proposed system may be complemented with the system proposed by Bouzid and Jemni [28], who describe an avatar-based system for automatically generating 3D signing animations from SignWriting annotations expressed with SWML (SignWriting Markup Language). Besides, some measures to optimize the performance of the system will be considered and evaluated, such as the fusion of the repositories SMBL and SSS-XXXX-SVG as proposed in [29].
Specifically the solution described in this paper is applied to improve the web accessibility for people with hearing impairment. However, thanks to the mentioned separation of content and representation, by using the WAI Accessibility Adaptor or core component, which can be extended by any specific adaptor, the approach can be extended to target other types of impairments, as intellectual disability. Therefore, the proposed translator could be adapted to support other types of graphics accompanying video, which would be adapted to each specific need.