Keyword Base Framework for DLNA Acceptance Testing ?· Keyword Base Framework for DLNA Acceptance Testing…

  • Published on

  • View

  • Download


<ul><li><p>Keyword Base Framework for DLNA Acceptance Testing</p><p>Bruno Filipe Guia de</p><p>Instituto Superior Tecnico, Lisboa, Portugal</p><p>November 2015</p><p>Abstract</p><p>A successful acceptance represents the minimum operational functionality required by the customerto accept a software delivery, along with validation by use case studies. So far, acceptance frameworksfocused on automating this testing type. We propose an acceptance testing framework based on akeyword model whereby each device is mapped into the network through a state machine and accep-tance testing scenarios mapping are translated into a unique code, replacing the manual testing for anautomatic approach. Results showed that the proposed framework can reduce the demand for manualtesting in the acceptance process, hence eliminating part of the limitations associated with human op-erations, so far required in testing on a regular basis. However, the framework itself does not producea fail or pass result. That result evaluation derives from the level of functioning achieved in the finaluser implementation test.Keywords: Testing framework , Acceptance testing, DLNA, keyword driven model</p><p>1. Introduction</p><p>Testing is an expensive activity, counting up to 55%of a project cost [5]. In fact, failure to detectand correct defects in early states of a project canincrease considerably its cost [3] and compromisethe company reputation. Therefore, optimizationof testing activities can lead to important savingson the overall project cost. Cost is reduced in twointerlinked ways: by reducing the testing activitiescost as a whole and by increasing success in earlyfinding of any testing bugs, hence avoiding costly so-lutions for later issues. The core of the testing activ-ities are test cases whereby a test case correspondsto a set of instructions and expected result. Com-bining several test cases originates initially a testingsuite and ultimately a test plan which defines a setof test cases to be implemented. A frequent testplan required to be performed either several timesper month or even per week, is a good candidate foroptimization. Manual testing of the test plan canbe tedious for the tester and time consuming for thecompany resulting in increased costs and increasinghuman error potential. Testing automation is a de-sirable solution as it overcomes manual testing lim-itations through operational optimization. Most ofthe work and research was conducted at ACCESSEurope GmbH1 offices in Oberhausen, Germany aspart of my Software Tester position in the QualityAssurance department. The primary focus for the</p><p>1</p><p>present work was to improve the acceptance testingprocedure by developing a new automation frame-work, easily expansible and able to perform simpleand complex workflow testing without overlappingwith existing tools. Currently, most of internallytesting performed by the quality assurance team isbase on a called End-To-End (E2E) approach. Inorder to apply a distributed testing based on enduser interactions, a manual setup of a test envi-ronment would be required. In a test environmentand prior to testing, it is required to setup the net-work configuration, deploy the System Under Test(SUT) and Device Under Test (DUT) while ensur-ing that all configurations are correctly configured.Testing operations involve one or two testers thatperform test cases as part of a test plan and reportthe results. Interactions between software sharingthe DLNA norm and digital media is one of themain products. Currently, an automatic solutionthat controls all the different devices potentially in-volved and their interactions occurring at a humanlevel and that simulates a wide range of end-userscenarios is unavailable. The present work aimedto develop a solution applicable not only to test-ing between network devices but also to additionalscenarios and products.</p><p>1.1. Objectives</p><p>The main objectives underpinning this thesis wasto develop an automatic testing framework solutionthat can save time and resources on the long run by</p><p>1</p></li><li><p>creating templates, reusable codes and an easy tofollow method for extending the solution. A suc-cessful testing framework would allow the release ofresources from manual testing to others activitiessuch as testing of other features not automated.</p><p>2. BackgroundDigital Living Network Alliance is a protocol foraccessing and sharing multimedia content betweendevices developed by an organization with the samename. DLNA categorizes the products in Deviceclasses, whereby each class has its own set of rulesand characteristics.The main device classes are thefollowing:</p><p> DMS : correspond to the servers where the con-tent including audio, video or images are storedand sent on request to another device class.</p><p> DMR : are devices where the content from theserver is displayed, like an TV.</p><p> DMC : is a controller that enables the user tosee a list of available servers and renders</p><p> DMP : corresponds to a DMC with an embed-ded DMR</p><p> DMPrs: Is a printer using DLNA protocol andable to print images.</p><p>2.0.1 Use Cases</p><p>To ensure interoperability between devices, whichcould have differing manufacture origins, a certifi-cation is issued by DLNA after verifying each prod-uct through a program partially described in sec-tion 3. The certification is attributed only whenthere is proof of meeting the necessary require-ments. DLNA provides a certification program, in-cluding tests against specific tools or testing withother already certified devices. Typical use cases ofDLNA protocol can be summarized in three types:</p><p>1. Content playback from the server to a renderer:</p><p>2. Upload or download content from the server:Clients can upload and download content fromDMS.</p><p>3. Printing content: Printers can be connected tothe DLNA eco system and print content sendby DMC or DMP.</p><p>2.1. Test Automation TechniquesAccording to various authors and studies [4], test-ing automation techniques can be divided in 5 largetypes: Capture-Replay Testing; Script Based Test-ing; Keyword Driven Testing ; Keyword DrivenTesting and Model Base Testing. All of those whichwill be describe in detailed in the next sub sections.</p><p>2.1.1 Script Based Testing</p><p>Script Based testing is supported on the idea of de-composing testing into small units and creating asmall testing script for each unit. Once the scriptsare implemented execution is done automatically.To create new tests, scripts can be combines amongthemselves and new tests can be created by chang-ing the script execution order. The script base test-ing abstraction and quality depends in large part onthe implementation used. Coverage is not easily ex-tracted and needs to be manually compared to therequirements.</p><p>2.1.2 Keyword Driven Testing</p><p>Keyword base script are based on the idea of hav-ing keywords and additional info, and the keywordis mapped to scripts and library executing test-ing. Adaptation between the script and SUT canbe achieved hence increasing the tool abstraction.Keyword-driven testing allows a high level of ab-straction and can be automatically executed how-ever coverage is difficult to measure.</p><p>2.1.3 Model Base Testing</p><p>Model base testing consists in the idea of repre-senting the behavior or a sub set of this behavior ofSUT into a written form. Further testing follows themodel to ensure the correct behavior of the device.The model testing is generated by a test modulegenerator that later is executed with the supportof test scripts.Some authors indicate that the mainmodel base testing advantage is the awareness ofthe system and coverage[1].</p><p>2.2. Continuous integration</p><p>Continuous integration is based on the idea of con-tinuously getting the latest version of the sourcecode and compilation to ensure proper working, ontop of which tests can be executed. This practiceensures that the source quality is tested in earlystates and broken source code can be detected andfixed early in the project. According to several web-sites, Jenkins is currently the most used continu-ous integration tool, also the one used by ACCESSGmbH making it the tool we are required to have agood operational understanding.</p><p>3. Implementation</p><p>Proposed Testing Framework</p><p>The main result and outcome of the present thesisis a testing Framework called GenFramework.Dueto the fact that this thesis is being conducted in abusiness environment, cost and time to market areboth two critical criteria. If the solution develop-ment costs are higher than others available in the</p><p>2</p></li><li><p>market, the present research would have limited in-terest and application. Concerning time-to-market,given that shareholders (testers, programmers andmanagers) can be consider the main end-users ofthis framework, the shorter the time to deliver thetool results the better, otherwise this tool wouldonly have internal use application. Negative feed-back or delay in presenting results may result in aproject cancellation. GenFramework is required toprovide easy integration and support of new hard-ware/operating system needs given that new oper-ating systems and mobile devices may arise. De-spite the fact that the SUT running hardware andsoftware operate with minimum change and prefer-ably in an invisible mode, the new element to beadded should be integrated in the most transpar-ent way possible. Once the setup and configura-tion is done, the development and maintenance oftest cases should be executable by a tester withoutprogramming skills. This approach releases the pro-grammers from this time consuming task. However,changes in the environment such as new platformsstill require a programmer support. After supportis added give, interaction between similar scriptsshould be applicable. In a distributed system suchas DLNA operations are not always sequential. Acommon example is a server access, when differentdevices can perform operations in the server simul-taneously. GenFramework should be able to controlmultiple devices simultaneously. When executing atest case all SUT present in the system need to bemonitored and controlled. A failure may occur inany of the SUT and GenFramework needs to be ableof detect individual SUT failures.</p><p>3.1. Solution overview</p><p>The idea behind the solution of GenFramework isusing a script files as the one in listing 1. Eachscript file represent one test script that should beexecuted. The language used is high level and isnot bound to any specific language. The technicalrequirements are the following:</p><p> Web sockets modules : pip support many mod-ules.</p><p> Communication with Appium</p><p> Communication with Selenium</p><p> Support multi platform : Python is supportedin Windows MacOS and Linux</p><p> Integration with Jenkins</p><p> Good community support (Subjective criteria)</p><p> Support of reflection and dynamic properties</p><p>Listing 1: Simple script example</p><p>1 \# Sta r t i ng a p p l i c a t i o n2 DMC myControl l i nux3 DMS myServer windows4 DMR myRenderer Android5 \# s t a r t i n t e r a c t i o n s6 myControl play myServer myRenderer v ideo</p><p>Using a keyword-driven framework approachtesters only need to write scripts similar to exam-ple in 1. Lines 2, 3 and 4 represents initiationof devices, a DMC, DMR and DMS are created.In line 6 testing start by the created DMC play-ing a video from DMS in the the DMR. To makethis possible the framework in background needsto perform a large amount of operations and moreinformation is needed. DMS needs to a have thecontent ; DMC needs to locate content and DMRneeds to be on a ready state to get input. To im-prove testing quality the framework should haveknowledge about current state of all SUT. This al-low to better differing if some behavior or action iscorrect. State Chart extensible Markup Language(SCXML) is a XML language created by W3C [2]to represent event base state machines. DLNA de-vice classes mainly only react to events, from otherdevices or users. SCML is presented as a as eventbase xml, this turns SCXML a good candidate tomodeling DLNA components. The goal is only touse SCXML to help modeling the DUT, compat-ibility to SCXML will not be completed but onlyinspired in SCXML. This approach can hopefullyhelp saving time by avoiding implemented instruc-tions that are not needed in this content and notfollowing SCXML convention completely. It is im-portant to mention that we do not aim to fully sup-port SCXML and/or generic implementations, butinstead only use the concepts and operations fromSCXML needed to better modeling DLNA systems.In the next sub sections some this tags and otherSCXML concepts will be explained in more detail.An event can have multiple transitions, howeveronly one can be selected. Every time a event match,condition will be evaluate. Only when the conditionmatch the event is executed. When none of theevents match the condition or name does not exist.Generic event called * can still be executed. Anyevent name can match with *. However condi-tion also needs to be verify. Transition are executedwith a up down approach stopped in the first exe-cutable one. Non accessible transitions should notoccur however they are possible if previous transi-tion are always invoked. SCXML allow compoundstates which means that internally a state can havemore states. Also allow parallel states which are</p><p>3</p></li><li><p>very similar to compound states, a group of statetags have before a parallel tag and all children aresimultaneous activated. All onEntry and onExitare executed. In the framework the implementa-tion for simplification the path of a parallel stateneed to be specify. In DLNA content, upload anddownload are two example of functionality inside aparallel state called streaming state.</p><p>DataModel is supported by SCXML supports inhis specification a data model embedded in theSCXML machines. However we decided not to useit. In alternative each SCXML will have an as-sociate Data Model using the is own specification.This option keeps the SCXML more clean and focuson devices functionality only, testing specific infor-mation are stored in different files and location.</p><p>3.1.1 New State Machine</p><p>When a generic line is read on the script a instanceneeds to be created. A new process with all relevantinformation is crated. this process should managethe binary to deploy ( on request ). This info (lo-cation of the delivery is passed to the process. Thebinary is not running, needs to Wait for onStart()instruction from GenFramework to start.</p><p>3.2. Solution Architecture and implementationFor a better understanding of the framework spe-cific parts of the framework will be more detailedexplained. To achieve a framework as much genericas possible some options had to be more complexthat expected. Allowing the framework to be moregeneric as possible.</p><p>3.2.1 TestRunner</p><p>TestRunner is the center of the framework, is fromthis module all framework is controlled. Once theframework is invoked TestRunner receives the no-tification with the information of the script undertest.</p><p>Every SUT will have is own separated process.This allow the framework to control an unlimitednumber of devices and control each one in a inde-pendent way....</p></li></ul>


View more >