The 6 Best Practices for Developing Successful Test Engineering Solutions

This Evaluation Engineering article is reprinted here with permission.

What you’ll learn:

• What key cost drivers should be considered when developing a test engineering solution.
• How to sort through the build vs buy dilemma when planning a test strategy.
• Why it’s important to plan what to do once the test data arrives.

PRACTICE 1:

Analyze All of Your Cost Drivers

Many companies choose a solution because the initial capital outlay on the system appears acceptable; however other important key cost drivers should be considered. These include:

• Set-up costs: These are the costs associated with taking the hardware, integrating it into a mechanical enclosure, wiring the system, building the test fixtures, checking that the software drivers are compatible, and developing the software code to provide a suitable application. Reducing set-up costs and risks can be done by finding a suitable single supply resource partner who has domain and industry knowledge across multi-vendor solutions.
• Overheads: Recruiting and training staff to develop a successful solution can add considerably to the overall development costs. These costs can be reduced significantly if you have in-house, well-trained resources, or if you can hire an experienced test engineering outsource provider to enhance your internal teams.
• Lowering NRE costs: You can reduce NRE costs by project planning effectively and by using COTS (Commercial Off-The-Shelf) products wherever possible throughout system development.
• Customization costs of fixtures and tests: Most companies require individual test fixtures developed to interface the measurement hardware directly with the DUT that they are testing. Traditionally, this has been done through dedicated fixture companies. There are alternatives. multi-interconnects that provide high-density pin I/O can be permanently connected to the test instrumentation. A common interface test adapter (ITA) can then be used to serve many different products, meaning that fewer fixtures need to be developed. 

Test code development also adds significantly to the system cost. Test Executive software, such as National Instrument’s “TestStand,” provides high-level management of each product's test needs. Individual test steps are written as small measurement or functional blocks providing faster test software execution, as the complex management of test steps takes place within the Test Executive software. This solution also means that the test steps are easier to maintain in the long term.

• Time-to-market costs: Time to market is a key cost driver for most companies manufacturing products. Many companies buy separate test equipment for their R&D, validation, and production test departments. However, in many situations, the tests used in R&D are fundamentally the same as in production. Therefore, using the same test in both R&D and production makes an ideal economic option.
• Calibration costs: Any unplanned system downtime for calibration and maintenance increases the cost of the final solution. Much of the currently available PC-based instrumentation hardware has built-in calibration, which can be run regularly without the need for downtime or is calibrated for a minimum of two years. When choosing test instrumentation hardware, it is certainly something that needs to be considered.

PRACTICE 2:

Decide on the Strategy to Suit Your Business Needs

There are different product offerings in the test marketplace today, making it quite difficult to choose one suitable solution. Moreover, test managers are wary of choosing new technologies for risky high-value production test needs. It is worth reviewing the reasons for this concern. The following bell chart taken from the book “Crossing the Chasm” describes how the acceptance of new technology changes throughout the product development and manufacturing lifecycle. 

The Early Market

Technology enthusiasts and visionaries are those engineers who have been employed to find the latest available technology for designing market-leading products. The R&D engineer puts this new technology into a real saleable product. These engineers must have the latest test technology at their fingertips for testing their innovative designs. Re-use of test hardware is unusual within this group of engineers. They often use the latest innovations in test equipment as the risk to their business if the equipment fails during R&D or validation testing is extremely low.

Many innovative Instrumentation vendors successfully sell their products to this group of R&D and product Validation/ Verification engineers. However, the situation changes dramatically when the chasm is to be crossed.

The Mainstream Market

The decision-making process is completely different for those whose livelihood depends on successfully introducing and manufacturing their products. However, this group of senior production test engineers and test managers are very wary of using new technology. This is where hardware re-use comes in! Using the same proven technologies for testing each new product significantly lowers the risk of failure. 

Many manufacturing companies choose to outsource their end-of-line functional test systems to third-party test system integrators. These system integrators tend to develop their solutions using modern but well-proven technologies such as modular rack-mounted Instrumentation platforms, including the now industry-standard PXI modular instrumentation platform.

PRACTICE 3:

Reuse Your Test Software Development  

Today’s most effective test solutions are based on software. Software provides the flexibility and the speed required to satisfy the ever-increasing demands of testing for increasingly complex products and devices. 

The current trend in production test software is that a COTS test executive is used to manage the test process. The off-the-shelf solutions mentioned earlier in this article, such as TestStand by National Instruments, offer all the functionality of an in-house solution at a much lower price point. These low-level tests can often be reused as they are typically focused on doing one thing well. 

PRACTICE 4:

Build versus Buy

The Build-versus-Buy dilemma is one that all companies go through when planning their test strategy—develop an in-house solution or outsource your test solution to a third-party system integrator. 

If you don’t have the skill or resources to complete a solution satisfactorily, you can opt for outsourcing. The cost of outsourcing is typically higher, but it comes with a guaranteed solution from professionals.

Developing a solution in-house does come with higher risks. This risk, however, can be reduced significantly by partnering with experienced test service providing companies. They will help reduce some of the cost drivers discussed in Practice 1.

PRACTICE 5:

Ensure Effective Project Management

If you decide to develop your test solution in-house, its success depends on how well you manage the project. Effective project management should be planned in advance. Make sure to look at all the present as well as future needs of the solution.

On the other hand, when you outsource your solution to a third-party, you don’t have to worry about anything. The professionals will take care of everything, including project management, and deliver the guaranteed solution in a timely manner.

PRACTICE 6:

Gain Valuable Business Improvement Insights from Your Test Data

The final practice is one that is often overlooked in the planning stage. Many companies completely miss out on planning what to do once the test data arrives. You can make use of some excellent tools such as SPC and 6sigma. Also, choosing the right database structure to deal with the vast amounts of data can be challenging. SQL databases are probably the most common as they allow data to be collected and stored from many sources. 

The IT infrastructure of your test data management should be high up on the list of decisions made when planning your solution. Companies that get this right in the first place will ultimately be more competitive in their marketplace.  

Test data management solutions are available to seamlessly integrate into your test system development without any rewrite of your test sequences or code. These solutions pull data into the database as soon as it is available. The systems can be deployed on internal servers, virtual machines, or as secure cloud-based services. They provide high-value proposition solutions with low upfront costs.