If we look at current trends, the automation of homes and residential units is on the rise, healthcare is upgrading to smarter diagnoses, digitizing solutions, and solving complex medical conditions, automobiles are becoming smarter, more entertaining, and safer, telecommunications are setting up smart communications infrastructure, and railroads are improving their safety and security. To improve productivity and produce more intelligent products, large and complex manufacturing industries are adopting embedded systems. To support these trends, the development of embedded systems and software with short development cycles is becoming increasingly critical, along with catering to the need for more sophisticated, defect-free, reliable, and high-quality embedded systems for a competitive edge in the market.
In every industry and organization, embedded systems and devices solve specific problems in unique ways. Therefore, to determine the success of an embedded device or software solution, it is important to verify the reliability, robustness, and performance (speed of the testing process), which includes both functional and non-functional aspects, in a rapid, invasive, repetitive manner. In testing, functions and non-functions of a system are verified and validated, and the most critical defects are exposed to improve a system's quality and reliability. In this paper, we will examine different types of embedded software used across industries, as well as the challenges associated with embedded software testing and how it can be automated to improve test coverage and reduce time-to-market.
Embedded software is created to run on the specific device on which it is installed. This means that the restrictions on processing and memory are tied directly to the specifications of that device.
As shown in the above diagram, software embedded in a device consists of the following four building blocks:
Firmware is software written directly for hardware. It communicates with other devices directly or performs basic tasks and functions without requiring APIs, operating systems, or device drivers. Firmware is formed of the boot loader, board support packages, and hardware abstraction layer (HAL).
Operating systems, along with required device drivers, are defined as a software package that enables a user to run other applications on an embedded device in its most general sense. They are divided into two types:
• Embedded operating systems (EOS) such as embedded Linux, Yocto OS (operating system), and Ubuntu Core
• Real-time operating systems (RTOS) such as VxWorks, Azure RTOS, QNX, and so on.
In computing, the term middleware refers to a software layer that sits between a user application and an operating system. This layer is also referred to as embedded frameworks, and it is used to ease the rapid development of user applications on a wide variety of operating systems and platforms with more sophisticated interoperability capabilities. A few good examples of this would be application frameworks such as Android and iOS that address telecommunication needs, Qt that addresses automotive, aerospace, consumer, and industrial user interface needs, and ROS (robot operating system) that addresses robotic automation needs.
This refers to software that is installed on the operating system and uses middleware and firmware to perform the primary functions of an embedded system. Essentially, this application solves a specific problem within a specific domain. Every end application is unique, but operating systems and firmware may be the same from device to device. Embedded user applications can be classified into the following types:
a. Bare-mettle embedded application
This is the simplest type of embedded application to be self-contained without an operating system. All other types followed this type. To meet the needs of this embedded software, the software engineering team must also design and build the testing framework from scratch. As a result, we cannot use any off-the-shelf testing tools or frameworks available today.
b. Embedded Linux applications
This is an end-user application that does not have any real-time requirements and runs on a customized embedded Linux operating system. General purpose evaluation boards like Raspberry Pi, Beagle Bone, and Jetson are examples of embedded Linux devices that run on various embedded Linux distributions such as Yocto, Ubuntu Core, and Windriver for training and software development. There are also good off-the-shelf test tools that can be customized and extended to meet application-specific testing needs, e.g., LAVA.
c. Embedded RTOS software
When an end-user application has hard or soft real-time, deterministic, and repeatable behavior needs, they are developed and deployed on top of real-time operating systems like Azure RTOS, and QNX. Real-time operating systems applications include airline traffic control systems, command control systems, airline reservation systems, heart pacemakers, multimedia networks, and robots.
d. Embedded networking software
Software for embedded systems is primarily concerned with transferring data from one module to another or making two modules talk to each other. Embedded software is given a special category for networking since it requires a specific talent set to develop and manage custom network stacks. There are several types of network stacks, including Ethernet stacks, Wi-Fi stacks, and Bluetooth stacks.
e. Headless embedded UI software
Headless user interface (UI) software does not have any UI on the device but uses a Web server to enable remote access. The implementation of headless devices with remote access over the user's mobile device may become dominant in industries such as home automation. The future may see fewer refrigerators, ovens, and washing machines with large displays but more Wi-Fi-enabled devices. Using Web application test utilities, these applications can be tested with ease.
f. Embedded UI software
A GUI (graphical user interface) application is one that has a user interface embedded within it. They run on embedded microprocessors and controllers running embedded OS and RTOS that are hidden inside products such as cellular phones, communication equipment, automobile engines, laser printers, and medical devices. GUI applications are built on Qt, Azure GUIX, PEG, etc. GUI libraries are examples of such applications that are usually constrained by power, memory, and performance. Some of these libraries also provide testing capabilities that must be adapted for application testing.
g. Mobile application
Mobile applications, also known as apps, are software applications that run on mobile devices, such as smartphones or tablets. The complexity of embedded devices is hidden from developers so that they can develop rapid applications with smooth interoperability. This should be categorized as a separate embedded application type. Examples include applications developed for Android and iOS devices such as smart watches, TVs, mobile phones, tablets, etc. Testing tools are more mature for these types of applications, e.g., Appium, selenium, etc.
If you choose the right software architecture for embedded systems that proves measurable testability by controlling the inputs of each component (possibly manipulating its internal state) and seeing its outputs (and possibly its internal state), most of the above-mentioned challenges can be overcome by—
Know-how
Determinism and repeatability
Communication
Hardware dependency & Lack of access
Software vs hardware defects
Firmware update
Regression
Non-reproducible defects
1. Adopting testable software architecture
Use popular, stable, and mature OTS middleware or develop an in-house framework that encapsulates hardware, platform, and operating system intricacies. This will help to significantly reduce the learning curve of test engineers in case the OS, hardware, platform, or environment changes.
2. Provisioning injectable test hooks
The chosen middleware or in-house developed middleware must provide test hooks to ease integration with test fixtures and supply more control over input and output.
3. Abstracting device communication
Construct a portable device communication strategy that encapsulates all communication protocols and channels and is highly configurable and scalable.
4. Adopting modular and scalable test architecture
To achieve greater reusability and sustainability during the iterative development cycle, it is important to have modular and scalable test architectures.
5. Emulators and test containerization
Configurable and virtualized test execution environments will aid in first software testing where real hardware availability is limited. In an iterative development model, containerization will ease deployment and software upgrades.
6. Enabling remote and distributed testing
Using FOTA (firmware over the air) will allow devices to be upgraded remotely. And having the ability to control multiple devices remotely will improve agility and significantly reduce regression and execution time.
7. Enabling CI/CD integration
Continuous deployment and testing will help to achieve automated repeatability of test execution and development workflow management.
8. Using cognitive intelligence
Cognitive intelligence solutions will help testers to create self-healing tests and perform regressions on change triggers and change impact analysis.
9. Creating test scripts in Gherkin syntax
The use of Gherkin syntax for test script creation for embedded applications will abstract out platform dependency and enable validation/acceptance test creation by business analysts or stakeholders in plain English.
10. Test automation
For non-reproducible or intermittent defect verification, tests need to execute repeatedly, multiple times for longer durations. Using a modular framework for automated testing will simplify the process of automating and orchestrating flaky tests.
To address the challenges and solutions mentioned above, embedded device test execution processes must be automated. Testing can be made more efficient and less time-consuming by automating the process before manual testing. This results in organizations launching their products earlier and reducing software development costs significantly, which is crucial to their success.
Cyient's end-to-end test automation framework, named CyFAST, is designed to solve most embedded test automation challenges with an intuitive test management workflow. How multiple stakeholders will be able to automate test execution for multiple devices and applications is shown in the infographic below.
With CyFAST, embedded testing will benefit from the most advanced features of the robot framework, and CyFAST’s containerized distributed test execution capability will allow the parallel execution of multiple targets on multiple platforms. There is an embedded test library provided by CyFAST, along with portable and easy-to-configure test hooks that can be used to quickly develop and integrate test keyword libraries for every type of embedded software application. This library may be run with a remote server from the robot framework as well as in isolation.
Bhushan Mahajan, senior solutions architect at Cyient, has vast experience in medical device design and development. In the course of developing medical devices and frameworks for their verification, Bhushan has become well-versed in best practices and pitfalls to be avoided. Cyient’s CyFAST is aligned with identified megatrends and is a key support to technology driven disrupters specifically for automation.
