The products we buy and use every day are far more complex than they were just a few years ago. IoT, technology convergence, voice control, and more trends are driving more and more capable devices, and with that comes more pressure on test teams. When was the last time you encountered a simple test coverage requirement? Does not contain new metrics, does not change over time? How complex is your test line setup compared to previous product versions?

Five years ago, your room might have been lit with simple incandescent bulbs or fluorescent tubes. Testing a bulb is easy: plug it in, measure the lumens, and the test is done. However, now you are most likely using LED lights. (Look up at the ceiling, is it an LED light!)

Figure 1: Even the light fixtures we use are rapidly increasing in complexity, and now they are wireless and voice-activated

Testing LED luminaires is more complicated. First, you have a driver board in front of you that needs to be tested for ICT and functionality. Then, once each accessory was added to the printed circuit board, lighting manufacturers quickly added “differentiating features” because adding circuitry to an existing green circuit board made good use of space and improved product margins . Next, you can imagine the conversation in the R&D team:

– “Why not add a wireless access point? That seems to work.”

– “Well, if we had wireless, we’d have to add Bluetooth!”

– “Then we can add speakers and then…”

– Such a list of requirements would extend indefinitely.

Long story short, teams of test engineers who once only needed to light a glass bulb are now facing issues with power electronics, wireless connectivity, acoustics, and more. I think this sounds familiar because a similar story is being told in everything from consumer electronics to industrial machines and life sciences applications. The complexity of this new DUT raises two fundamental questions for the test engineering team:

1. The increasing breadth and depth of test coverage requirements often requires new workstation architectures, instrumentation, and software. However, budgets and time-to-market were not scaled up, creating conflicts or forcing test teams to change, or both.

2. The field of knowledge required to implement the system is wider, and professional knowledge will not be achieved overnight. It is necessary to find an organic combination of personal development, team support and theoretical abstraction in order to effectively build a workstation.

Responsibility for addressing these issues rests with the leaders of the testing organization, who must plan, and the path forward is not always clear. Below we discuss some of the strategies available and the successes of leading test teams.

Test Coverage – The Story of COTS Hardware and Software

The additional complexity presents not only technical challenges, such as meeting ambitious specifications, it is a financial problem. Let’s look at an edge case where the tech/budget balance is met. The Saab Gripen E supersonic jet is radically optimized for both performance and cost efficiency. The result was that the total program cost of this aircraft was a fraction (over 10% by some estimates) of some other similar programs. Saab is focused on breaking down the direct relationship between the increase in functionality and the cost of testing – they call it the cost curve, and their methodology is something we can all learn from.

Figure 2: The Saab Gripen E aircraft uses an open COTS test method

Having COTS products means we can control development and maintenance costs, driving Saab’s initiative to break the cost curve.

They found that not only the COTS platform was adopted, but the COTS platform that was open to both hardware and software. This approach allows them to save 90% of the development cost of the use case and integrate 10% of the unique requirements. Saab saved weeks of documentation time per test system and reduced testing costs by 30 percent. Click here for a detailed Saab test methodology.

By adopting a standard that already tightly integrates instruments that meet most measurement needs, while operating in an ecosystem that meets niche needs, you can confidently accept any new test specification that will impact existing capabilities, footprints, and processes. have minimal impact. This philosophy is the basis for PXI based on an open modular architecture.

Figure 3: The PXI base layer maintains an open modular standard, ensuring user flexibility

Don’t fall into the common misconception that the benefits of COTS are limited to hardware. To avoid being constrained by proprietary, vendor-defined systems, production engineers often turn to custom solutions. While this can lead to huge coverage, run-time and cost advantages, it also means they have to start from scratch every time they write measurement code.

It doesn’t matter if these measurements are simple and can be coded quickly; but they usually involve complex algorithms that require a deep understanding of signals and sensors. This is where the modular COTS approach shines again. In audio testing alone, many experts offer excellent open software products such as CATS (CIM.AS) or Audio Expert (MegaSig). The CIM team summarizes their views on the benefits as follows:

With our CATS software, our goal is to be the electroacoustic testing expert on your team, while you can focus on being your product expert.

– Dennis Morini. Business Manager, CIM.AS

One of the barriers to our adoption of COTS is shifting the budget from operating expenses to prepaid capital. If you add up the price of adding a new test stand for each new measurement type, the steep cost curve quickly becomes unattractive. Longtime Philips Healthcare employee Neil Evans explains this well as he witnesses the exponential growth in the capabilities of the ultrasound products he is working on. Every year, he faces the challenge of continually demanding increased testing capacity without increasing his testing budget.

It is important to articulate the ability to test the business value that an organization can provide. In this case, exponentially increasing development and maintenance costs can be predicted based on increased product complexity. The vision to break the relationship between product complexity and test system cost informs executive decision-making.

-Neil Evans, Senior Manager, Philips

The cost of not changing tends to be more apparent when you consider the operational expenditures that go from organizational decisions to maintaining decision-making levels.

Not being able to “break the cost curve” and hiding it behind the decisions of many small test stations does not eliminate its existence. Best practice recommends escalating decisions to broader strategies where possible, which increases the stakeholder group and slows decision-making, but this practice has proven to have long-term benefits.

Figure 4: Unable to break the cost curve, the cost of testing continues to increase with the complexity of the product, and it quickly enters the uneconomical area for the business

Bottom line: If you are faced with increasing test coverage requirements, the best proven practice is to use an open COTS platform wherever possible. Find interoperability from multiple vendors that fits a broad ecosystem, and your team can focus on testing your DUT instead of troubleshooting compatibility issues or debugging analysis algorithms. Make budgeting decisions based on total cost of ownership to maximize returns.

Capability investment is critical for complex testing

Any testing strategy technique will only work if you have an equally good team. As the functionality of the DUT increases, so do the expectations of the test team. If more people cannot be hired, each team member must be prepared to do more. This is a well-documented product design challenge.

People are trying to bring more complex products to market with insufficient talent pools. We cannot rely on people who know only one area.We need engineers trained in systems thinking

-Professor Alberto Sangiovani-Vincentelli

Professor of Engineering at UC Berkeley

Co-founder of Cadence and Synopsys

The key to success, concludes Professor Vincentelli, an expert in the field, is to distill the specialized details of the field from engineers so they can work more comprehensively and contribute to the overall system. Applying this theory to production test, the conclusion is that engineers need value-added capabilities across the different measurement domains and ownership of the entire DUT test, which ensures that errors do not arise in the gaps between the various system silos.

By prioritizing system proficiency over task proficiency, each engineer’s efficiency can be maximized. The best way to facilitate this is to establish a scalable set of processes and tools that meet production line test coverage requirements. To effectively drive member adoption processes and easily deploy more complex test lines, workstations or entire test architectures, you need the understanding and support of every team member. Some engineers naturally see broader system proficiency as an opportunity for personal growth, while others resist due to short-term frustration.

To be successful, everyone must understand how the change will take place and by whom; share your vision for the future with them and support them in making it happen together. A strong technical leader should be able to identify and correct immediate technical challenges and actively mentor team members. This leadership resource can be hired permanently into the team or part-time through a consultant, which is important anyway. Senior Test Software Engineer Chris Cilino noted:

I have worked with a variety of companies, from semiconductors to consumer electronics. Most people get the first step right: they standardize on a single tool (like LabVIEW) and a set of development guidelines. Where they fail is the second phase: deploying a set of guidelines for each engineer to adhere to and support initial success until it becomes a habit.

-Chris Cilino, Founder and Owner, PetranWay

There’s never been a good time to shift your time from project management to investing in team capabilities. Any manager will say that the test engineers on his team are their most valuable asset. Good managers prioritize their proficiency, outsource their “busy work,” and support them as system-level and even organizational-level thinkers.