Workshop Remote Testing Environment - OSFC · 2018. 9. 22. · Introduction Firmware developer routines Remote Testing Environment Infrastructure Hardware setup Automation Tests Fixing

Remote Testing EnvironmentWorkshop

OSFC 2018

Piotr Król, Michał Żygowski

1 / 41

IntroductionFirmware developer routinesRemote Testing EnvironmentInfrastructureHardware setupAutomation TestsFixing bugsFuture plans

RTE Control API (beta)New RTE revision for RPi

Q&A and RTE giveaway

Agenda

2 / 41OSFC 2018

CC BY 4.0 | Piotr Król, Michał Żygowski

Piotr Król Michał ŻygowskiFounder & Embedded Systems

ConsultantFirmware Engineer

[email protected]/in/michał-

żygowski-88954416b

@pietrushnic [email protected] linkedin.com/in/krolpiotrfacebook.com/piotr.krol.756859

Introduction

3 / 41OSFC 2018


https://www.linkedin.com/in/micha%C5%82-%C5%BCygowski-88954416b/

https://www.linkedin.com/in/krolpiotr

https://www.facebook.com/piotr.krol.756859

Narrowing down serious issue requires a lot of repetitive work:

modify & build firmware with potential fixflash affected firmware versionboot systemverify if the bug still exists

Problem with automation and remote development

To get rid of these problems we created Remote Testing Environment.

Firmware developer routines

4 / 41OSFC 2018


Remote Testing Environment

5 / 41OSFC 2018


Today, our Devices Under Test are PC Engines APU platforms. For workshoppurposes we will need:

Additional infrastructure that can be used is Debian machine with iPXE serverfor supplementary boot option.

Infrastructure

6 / 41OSFC 2018


connect RTE Hat with Orange Pi Zero board (~3min)

Hardware setup

7 / 41OSFC 2018


RS232 interface via a null modem cable (~2min)

Hardware setup

8 / 41OSFC 2018


SPI interface via IDC 2x4 pin cable (~3min)

Hardware setup

9 / 41OSFC 2018


APU PWR/RST pins with RTE OC buffers (~5min)

Hardware setup

10 / 41OSFC 2018


DC-DC jack cable (~2min)

Hardware setup

11 / 41OSFC 2018


take the connected setup to the power/network section (~10min):power up RTE and APU platformsconnect Ethernet cable to Orange Pi Zero

Hardware setup

12 / 41OSFC 2018


Generic test automation framework for acceptance test-drivendevelopment (ATDD)Utilizes the keyword-driven testing approachThe core is implemented using Python and runs also on Jython (JVM) andIronPython (.NET)Bug fixes need tests too - Test-Driven Bug Fixing (TDBF)

Automation tests - RobotFramework

13 / 41OSFC 2018


Each test suite should contain 4 sections:

*** Settings ***Library SSHLibraryResource robotframework/variables.robotSuite Setup SSH Connection and Log In ${rte_ip}Suite Teardown Log Out And Close Connections

*** Variables ***${username} root${password} meta-rte

*** Keywords ***Log Out And Close Connections [Documentation] Close all telnet and ssh open connections. Telnet.Close All Connections SSHLibrary.Close All Connections

*** Test Cases ***RTE: 1.1 SSH connection ${ssh_info}= SSHLibrary.Get Connection Should Be Equal As Strings ${ssh_info.host} ${rte_ip}

Automation tests - test suite structure

14 / 41OSFC 2018


First, prepare the environment (~10min):

Automation tests - setup environment

15 / 41OSFC 2018



start Debian RTE Workshop VBox and loginpassword: rte-workshop


16 / 41OSFC 2018




open terminal (Activities/Terminal)


17 / 41OSFC 2018





change directory: cd RTE-Workshop


18 / 41OSFC 2018





change directory: cd RTE-Workshop

activate virtualenv: source robot-venv/bin/activate


19 / 41OSFC 2018


RTE-Workshop directory tree

20 / 41OSFC 2018


Now we should run script which tests our hardware setup (~5min):

Connection validation - part 1

21 / 41OSFC 2018



robot -v rte_ip:RTE_IP validation.robot

where RTE_IP is your RTE IP address


22 / 41OSFC 2018



robot -v rte_ip:RTE_IP validation.robot

where RTE_IP is your RTE IP address


23 / 41OSFC 2018



24 / 41OSFC 2018



25 / 41OSFC 2018


Possible failures:ssh timeout - RTE lacks network connectionNoValidConnections - bad IP addressTelnet timeout - check RS232 cable or close established connection

Automation tests - logs

26 / 41OSFC 2018


Possible failures:ssh timeout - RTE lacks network connectionNoValidConnections - bad IP addressTelnet timeout - check RS232 cable or close established connection

After each RobotFramework test suite we can track generated log files:

Try it by yourself by typing: firefox log.html report.html output.xml

Automation tests - logs

27 / 41OSFC 2018


Flashing process can be started by flash-apu.robot script (~5min):

robot -v rte_ip:RTE_IP -v fw_file:PATH_TO_FILE flash-apu.robot

remember to change fw_file firmware path for appropriate APU2/3/4binary (if you are not sure, check the sign of life from previous tests)

Flashing firmware - automated process

28 / 41OSFC 2018


Flashing process can be started by flash-apu.robot script (~5min):

robot -v rte_ip:RTE_IP -v fw_file:PATH_TO_FILE flash-apu.robot

remember to change fw_file firmware path for appropriate APU2/3/4binary (if you are not sure, check the sign of life from previous tests)

Possible failures:

opening "/tmp/coreboot.rom" failed - SSH key not authenticatedNo such file or directory - bad firmware file path

Flashing firmware - automated process

29 / 41OSFC 2018


We have prepared 2 binaries to present the RTE and RobotFramework capabilityto track down unexpected bugs and flash APUs firmware:

Fixing bugs (~10min)

30 / 41OSFC 2018



I. First, let's try to update our DUT with custom firmware:


31 / 41OSFC 2018




run: robot -v rte_ip:RTE_IP -v fw_file:firmware/apu4_custom.rom flash-apu.robot


32 / 41OSFC 2018




run: robot -v rte_ip:RTE_IP -v fw_file:firmware/apu4_custom.rom flash-apu.robot


33 / 41OSFC 2018



II. Assumes that we made appropriate changes and build the new binary, flashAPU with the newest v4.8.0.3 firmware:


34 / 41OSFC 2018




run: robot -v rte_ip:RTE_IP -v fw_file:firmware/apu4_mainline.romflash-apu.robot


35 / 41OSFC 2018




run: robot -v rte_ip:RTE_IP -v fw_file:firmware/apu4_mainline.romflash-apu.robot


36 / 41OSFC 2018


To open rest API service, type: firefox RTE_IP:8000 (~3min)

Our plans - RTE Control API (beta)

37 / 41OSFC 2018


New RTE Hat compatible with 40-pin header RaspberryPi 2/3/Zero

Our plans - New revision for RPi

38 / 41OSFC 2018


New revision during testing phase

Our plans - New revision for RPi

39 / 41OSFC 2018


RTE website & shop

40 / 41OSFC 2018


Q&A and lottery

Cards received at the entrance will serve as numbers to select the winnerof the lottery. The prizes are three full RTE sets!

Where can I order the RTE for myself? The online store is starting soon, so please submit any orders to:

[email protected]

Please visit our RTE website: 3mdeb.com/rte

41 / 41OSFC 2018


http://192.168.3.110:8000/[email protected]

https://3mdeb.com/rte

Documents

Workshop Remote Testing Environment - OSFC · 2018. 9. 22. · Introduction Firmware developer routines Remote Testing Environment Infrastructure Hardware setup Automation Tests Fixing