In this blog will discuss about how to do the Hardware and firmware testing, techniques used, then the scope of testing for both. To speed up your testing you can use tools mentioned end of this blog, all those tools are available on internet.
Knowing about the Hardware/Firmware and how to test all these will help you for upgrade testing of a product which involve firmware upgrade with OS upgrade and product upgrade.
Hardware testing is to check/ensure the functionality, stability of hardware component and ensure that it should not have process fault. It also include the heavy workload task for memory and CPU to check the performance and durability.
Now a days hardware design become much complex which demands the methods for testing to adhere and adapt to the challenges that arise, hence test development with new standards for hardware become advance.
There are many components involve in hardware testing like BIOS, CPU/Processor. Test the hardware to ensure its logical correctness and to ensure that follow appropriate standards. Using functional tests to determine whether met the test criteria. there are the few techniques commonly used for hardware testing.
- Software-based self-testing
- ATPG (Automatic test pattern generation)
- BIST (Built-in self-test)
- The Software-Based Self-Testing: Modern microprocessors impose significant challenges to the testing hardware, because of their high complexity and heterogeneity. The software-based self-testing alternate way to hardware based self-testing, which cover the testing of a microprocessor using its instruction set. The benefit of software-based self test is that it can be applied in the normal operation mode of the microprocessor, thus applying the required tests at-speed.
- The ATPG (Automatic test pattern generation): Starting with a chip netlist, inserting scan-chains (Scan chain is a technique used in design for testing), and generating vectors is the most direct and effortless approach and doesn’t required complete knowledge of the DUT (Device under test). The recent EDA (Electronic Design Automation) tools are capable of deducing how to partition a design into blocks, and to isolate them by scan-chains.
- Built In Self Test (BIST): BIST is best for testing complex system, due to less accessibility to internal nets as design complexity increased, has spawned various design techniques that increases testability. BIST implementations are based on a full scan architecture. This means that all the storage elements in the DUT concatenated to form several scan chains. This way test patterns can be serially shifted in and out of the storage elements. BIST requires no interaction with a large, expensive external test system. The testing is all built-in, and only tester is needed to start the test.
There are many tools available for hardware test and hardware diagnose like.
- Automatic Test Equipment (ATE)
- Sandra Lite – SiSoftware
To execute load tests, simulate and observe variety of conditions and using or exceeding the amounts of data that could be expected in an actual situation. Following tools allows to measure different aspects of a system.
- NFS Connectathon package
Firmware is a computer program that is embedded in a hardware device that provides control, monitoring and data manipulation of engineered products and systems. The firmware contained devices provides the low-level control program for the device. Examples of devices containing firmware are embedded systems, computers, computer peripherals, mobile phones, etc.
Importance of Firmware testing is the certification of firmware system meets its requirements with respect to functional correctness as well as performance, operational, and implementational properties. Then to reduce the risk and improve the performance.
The firmware functionality changed from conventional instruction set emulators to more extensive and powerful instruction sets, diagnostic programs, interpreters for high level languages, and operating system functions.
These are the three techniques of firmware testing.
- Tests the microprogram level that consider complete microprograms by analyzing their code or investigating the machine states after execution.
- Tests the microinstruction level that consider single microinstructions by analyzing the assignment of micro-operations to them or investigating the machine states after execution.
- Tests the micro-operation level that consider individual micro-operations by monitoring the execution
The firmware testing is huge and complex task to complete, to overcome this challenge there are some automated tools available.
- Firmware Test Suite (fwts): FWTS is a Linux tool that automates firmware checking. Tests that are designed to exercise and test different aspects of a machine’s firmware – including ACPI, UEFI, hardware configuration, power-saving and so on.
Firmware testing is complex and challenging as it needs to consider the concurrency between firmware and its interacting hardware/firmware components. This component of the system is increasing in scale and importance, and thus firmware validation is a critical part of system validation. This is generally addressed through co-simulating C/C++ based firmware code and HDL hardware models. Hence we can think of open source tool available in market for hardware and firmware testing.
Find the reference for more information and tools available for hardware and firmware testing :-
- The Unified Extensible Firmware Interface (EFI or UEFI for short): UEFI is a model for the interface between operating systems and firmware. It provides a standard environment for booting an operating system and running pre-boot applications.
- FirmWare Test Suite (fwts): fwts tool is used for PC’s firmware which check common BIOS and ACPI errors and provide some advice, how to fix issues or workaround firmware issues.
- CHIPSEC: Platform Security Assessment Framework: CHIPSEC framework will analyze the security of PC platforms including hardware, system firmware (BIOS/UEFI), and platform components. It cover a security test suite, tools for accessing various low level interfaces, and forensic capabilities. It support multiple platform like Linux, Windows, Mac OS and UEFI shell.
- Intel® UEFI Development Kit Debugger Tool: Debugger Tool provides the ability to debug UDK based firmware by co-working with the target side component of the debug solution.
- Greentea: Greentea is the automated testing tool for embedded OS development. It automates the process of flashing embedded boards, running the test and genrate test results into test reports. Developers use it for local development as well as for automation in a Continuous Integration environment.
- Nanotest: Nanotest work on the microcontroller to aid in both development and verification and to enable things like built in self-tests (BISTs) and power on self-tests (POSTs). The integrated self-tests provides a framework for basic diagnostics.
- gperftools (originally Google Performance Tools): The fastest malloc, works particularly with threads and STL. Also thread-friendly heap-checker, heap-profiler, and cpu-profiler.
- linux-uefi-validation: It provides a new level of support to developers identifying firmware implementation issues. Bringing together multiple separate upstream test suites into a cohesive and easy-to-use product with a unified reporting framework, LUV validates UEFI firmware at critical levels of the software stack.
- op-test-framework: This repository provides a collection of tools that enable automated testing of OpenPower systems. The op-test-framework suite is designed to test a machine largely out of band that is, it is designed for tests that do things like power cycle the machine, test booting different configurations. As part of the op-test-framework, we may run tests on the host itself.
- arm-trusted-firmware: ARM Trusted Firmware provides a reference implementation of secure world software for ARMv8-A, including a [Secure Monitor] TEE-SMC executing at Exception Level 3 (EL3). It implements various ARM interface standards, such as the Power State Coordination Interface (PSCI), Trusted Board Boot Requirements (TBBR, ARM DEN0006C-1) and SMC Calling Convention. As far as possible the code is designed for reuse or porting to other ARMv8-A model and hardware platforms.
- LPS node firmware: This project contains the source code for the Local positioning System node firmware.
- dns323-firmware-tools: These firmware images are what are used to “bundle” the kernel, initrd, and other data when uploading custom firmware images using the “stock” firmware update interface. If you are already running a custom firmware, it is likely that these tools will be of no use to you, and you will need to use whatever update mechanism is provided by your firmware.
- crazyflie-firmware: This project contains the source code for the Crazyflie 1.0/2.0 firmware.
- binwalk: Binwalk is a fast, easy to use tool for analyzing, reverse engineering, and extracting firmware images.
- phantom-firmware-tools: Tools for extracting, modding and re-packaging DJI Phantom 3 firmware. Tools for handling DJI Phantom quadcopter firmwares.
- Evaluating FRDM-KW41Z using NXP Test and Tool Utility: The Test Tool utility is a Windows® based graphical interface that communicates via serial interface to NXP development boards. This tool support to test NXP hardware with either the 802.15.4 MAC, Simple MAC (SMAC), SynkroRF, BeeStack, BeeStack Consumer ZigBee, Thread and Bluetooth Low Energy software
- HP StorageWorks Library and Tape Tools: HP StorageWorks Library and Tape Tools (L&TT) is a robust diagnostic tool for all of HP’s tape storage and magneto-optical storage products.
- IBM Tape Diagnostic Tool (ITDT): Run quick or extended diagnostics on tape drives to check if the tape device is defective. Invoke tape library self-test operations. Retrieve dumps from tape drives and libraries. Perform firmware update check and retrieval for tape drives and libraries. Perform a firmware update on tape drives or libraries. Test the performance of the environment by completely writing a cartridge and measuring performance.
- Secure Boot Manual Logo Test: This test verifies the proper functioning of boot time image authentication and the proper authentication and operation of UEFI secure boot variable updates.