Uncovering Testing Best Practices in a Communications Environment
October 11, 2022
Blog
Fundamental to providing reliable connectivity is the testing of products to ensure that they comply with applicable standards. Wi-SUN certified products undergo rigorous testing by independent authorized testing laboratories. It would be great to see the same meticulous approach applied across all sectors.
Testing is, of course, critical to the bottom line: it’s far more expensive to fix a product once it’s been released onto the market than to fix it during development and production. But there’s also a significant reputational impact to consider. Malfunctioning equipment isn’t just a headache for customers, who may think twice about placing orders in the future. It could have serious repercussions in safety critical sectors like healthcare and aerospace.
Highly stringent testing processes are absolutely required for these (and similar) sectors and use cases. But there must always be a commercial consideration: if you’re making a $1 product, it is not economically viable to perform $10 worth of testing on it.
Breaking It Down
Product type testing falls into three key areas:
- Conformance testing is required to ensure a product meets the requirements of a relevant specification or standard. In short, does the product do everything it’s supposed to?
- Interoperability testing is run to ensure that products which have passed conformance testing also work as intended with other relevant equipment. This provides an additional check to cover situations where conformance testing is not viable.
- End-to-end testing ties everything together to make sure all of a product’s hardware and software components function correctly in the final application. Typically, this covers testing across multiple specifications.
In a communications environment, end-to-end testing requires that a product is tested across each of the seven layers of the Open Systems Interconnection (OSI) networking reference model. That means the physical (radio) layer, the data link layer (including medium access control and link control), and the network, transport, session, presentation and application layers.
This helps us to understand why end-to-end testing is so important. While conformance or interoperability testing may only deal with a subset of the OSI model, end-to-end testing is holistic—requiring a meticulous layer-by-layer approach.
A Smart Grid Scenario
Let’s take a look at a typical scenario in the kind of communications environment in which Wi-SUN protocols are used. Smart grids and metering systems are helping energy providers to achieve significant operational efficiencies and deliver optimized customer experiences across the US and the world. But to maximize the benefits to businesses and consumers, they must be built with robust, conformant, and interoperable equipment, almost certainly standards based.
Take a smart meter. A rigorous testing process will check whether the meter can correctly measure energy consumption, with a variety of loads and supply conditions, that it can communicate reliably and securely, and successfully send relevant data to the utility back-end. In detail, the test coverage should include:
- Metrology: Connect the meter to an energy source and an energy load and also to a piece of test equipment. Run it for an allotted time and under a wide range of load and supply conditions and check that the smart meter has measured the correct energy usage.
- Communications: Check that a device correctly identifies the network to join, authenticates correctly, communicates reliably and securely over that network.
- Data exchange: Provide a testing environment to mimic the utility back-office environment which can then verify that the data sent from the meter is correctly formatted and matches the format expected by the application. For example, Wi-SUN Alliance works in collaboration with the DLMS-UA standards organization to provide an end-to-end testing environment for DLMS/COSEM over Wi-SUN FAN.
Each of these three stages uses a combination of conformance and interoperability testing. When integrated they provide an end-to-end test system capable of providing good coverage and reliability.
Managing Risk at Every Stage
In conclusion, multi-layered testing is essential to managing financial and reputational risk end-to-end. It must absolutely be run before equipment is put into production. But organizations may also benefit from running production line testing occasionally to prevent any costly mistakes from creeping through.
Phil Beecher is the president and CEO of Wi-SUN Alliance and a recognized global expert on wireless IoT. He can be reached at [email protected]
As smart cities, smart utilities, and large-scale corporate IoT networks continue to grow, so will the attention of cyber criminals looking for new ways to make money. That is why security-by-design is a prerequisite for all connected devices.
