Addressing Product Quality and Performance With Effective Test Methods

Download this article in PDF format

When it comes to ensuring that a product meets the standards established for performance, reliability, and safety, how well that compliance testing is performed directly impacts the product’s eventual success or failure in the field. Production test is an aspect of the true value-add and where the real mission to create a good product occurs. To talk about designing, developing, making, and fielding a product, we reached out to Daniel Knighten of Audio Precision, who does a lot of testing of consumer and commercial acoustic devices.

Daniel Knighten: It starts at the very beginning. There’s a very real issue for everybody designing a device today, regardless of what kind of devices they are. There are companies, whose fundamental selling proposition is, "Hey, do you want to build a smart speaker? Have we got a chipset for you? Buy our chipset. Follow our design guide. Here, we're going to give you schematics, and gerbers and such, and you can just do this."

It's the same model as Blue Apron and whoever else that's saying, "Hey, do you want to cook, but you don't know how to cook? We'll send you a box of ingredients, and instructions and, in theory, at the end of it you'll have a beautiful meal.” So, the reality is, the very first place where most of our customers start with using our gear, is they get those building blocks. They get evaluation modules with different IC's. They get sample speakers, and microphones and they test them to see if they actually have the characteristics that they need for their product.

At a paranoid level, they might even test them and see if they actually meet their own published specifications, but even more practically, one of the things you may have noticed if you cruise around data sheets, is there's no two companies that publish specifications the same way. At the very earliest parts of a design stage for a product, where a customer is qualifying the components they're going to go with, they're using supplied products and they're testing samples of those components and verifying that they'll work for them.

The next thing they do, is actually, in the world of audio, and I'll divide it between the electronics portion and the acoustic portion. The electronic portion is pretty well modeled. It's still not perfectly modeled, so there are second- and third-order effects. There are things like crosstalk and just stuff that never gets modeled perfectly, so there are a fair enough number of people who will design a circuit board, they've modeled it all up, and they just fab it up and it's good to go, but they're still going to test it (Figure 2).

But, as soon as you move into the acoustic realm, into speakers and microphones, no, there's no good models of even first order. By first order effects, what I mean is you have a model of a speaker driver and therefore you want to predict the frequency response. There are software packages that will do that, but they're going to be pretty far from whatever the actual frequency response is. So what you'll get in there and what you'll see customers doing is, they're going to totally mock up their design.

They will 3D-print what they think their physical form factor is, and there will be an iterative design process for the acoustic side of the product, both the speakers and the microphones, and that iterative process itself. If you're trying to sell a product for $25, you're not going to spend a lot of time on that process, but you're probably still going to iterate several times. For a high-end product, you might see thousands. It's kind of scary. You see these guys, there are graveyards of mechanical prototypes.

EE: At that point, Daniel, one of the things I'm fond of saying, as mentioned, I've been in the audio business myself, is you could give two sets of engineers a stack of wood, the exact same drivers, the exact same capacitors, and passives, and tell them to make two speakers and those speakers will look differently and sound differently, even though they're from the same components.

Knighten: Absolutely, so there our equipment gets used very intensively. Not only do you want to on-access frequency response, you want the off-access frequency response. You probably want to do it at multiple angles. Again, it all depends on what your ultimate goal is with this product, but for, let's call it a mass-market smart speaker from a Tier 1 company, this process is going to take months to years and it will involve... If you actually counted it, it would involve tens of millions of measurements, and they'll go through that process and, "Okay, we finally arrive at our first prototype."

Again, there are scales at every different level and products, but let's say you're a Tier 1 company, you're making a mass-market product. In a mass-market product, we often lose sight of the scale, but when you make a mass-market product, you're going to sell tens of millions of these things, maybe hundreds of millions of these things. It blows your mind the first time you go visit a factory in China that's making a true mass-market product. 

So, one of the things that can be stunning is, let's say you're a little boutique speaker manufacturer. Prototypes might be single digits. The R&D engineer who's designing it may, in fact, be the person who hand-assembles the prototype. But, if you're a mass-market company, your prototype runs may look like other companies' lifetime production runs.

One company that we work with, their prototype runs are 10,000 units. Lots of the ideas you have around, well, what does a prototype run look like, really don't apply. Here, a prototype looks like a full-scale-production run, but what remains true, especially about prototype runs, is you do a lot more measurements than you plan to do when you're in regular production.

EE: But, you still have to do QC when you're in regular production. You can't just stop testing once you've gotten a good product.

Knighten: There is, especially for audio products... What I like to point out is, speaker drivers are the last moving part in a consumer electronics product these days. Everything else is solid-state electronics, very well understood, very reliable. ICs, these days, it's very, very rare for somebody to get a defective IC, but a speaker is paper. We forget that, but it's made out of paper, maybe coated with something so it's not obvious that it's paper, but it's paper, and it's a spring, and it's a magnet and some type of compliant material.

From a manufacturing perspective, it is an incredible variable component. Speakers do not have the repeatability of any solid-state components and they're really, incredibly complicated. It's a mechanical device, it's an electronic device, it absorbs and exhausts humidity, and human beings listen to them and human beings are kind of terrible at making objective assessments, but especially when human beings experience something all the time, they become extremely good at picking up small variations.

This is one of the challenges that remains in the world, is that a completely untrained human being could pick up a phone and if the speaker is caught in the gap they will detect any issue... They may not be able to say what it is,-

EE: Can't put their finger on it.

Knighten: ... but they'll pick up that phone, they'll listen to their aunt talking and they'll say, "This phone is crap," and return it.

EE: Well, Dan, to add to what you were just saying, what about some of these newer piezoelectric glass-shaker technologies that some of these television companies are implementing. They still have to be tested as well, just because they are solid-state doesn't make them magically acoustically good.

Knighten: I'm curious to see how that technology goes. They're solid-state but they're still a moving part. Actually, they're still many moving parts.

EE: At that point, would you consider vibration... Obviously, a speaker vibrates, but that's pistonic motion. But, I mean in the case of a piezo electrically-stimulated display, you're vibrating 80 diagonal inches of engineered plastic, I was going to say glass, but you know what I'm saying? At what level do you consider a vibrational surface a moving part? Does it make a difference to your test equipment how that sound is generated?

Knighten: Ultimately, we're measuring air pressure and the question is, how is that air pressure being modulated? I would say that the MEMS-actuated vibrating surface transducers, and there's a few different technologies for doing that, those are a moving part and I'll give you a very simple reason why. One of the most common ways in which speakers go bad or since they go bad, are bad, is the adhesive isn't applied correctly, or the adhesive cures incorrectly or something goes wrong with the ingredients used in the adhesive. And, that's not less of a problem when you're vibrating the screen, that's more of a problem.

EE: Excellent point.

Knighten: And, the screen is now a giant diaphragm. Is the screen a uniform thickness? What is your repeatability in the manufacturing of that diaphragm? The problems aren't less, the problems are different. For that particular kind of thing, the problems are actually more challenging.

EE: Well, then they would need you more, actually. You know what I'm saying? These newer technologies are actually going to need more acoustics tests to ensure that these Blue Sky, advanced ways of emitting sound actually do something.

Knighten: Just very, very broadly speaking, at a couple miles off the ground, when you introduce a new technology, test requirements increase, and they increase until people have a grip on that technology and figure out how to make the technology repeatable. The goal of anybody who runs a manufacturing line is to figure out how to do the exact same thing over and over again. And, what happens with new technologies is, you haven't figured out how to do that yet.

EE: Why don't you put some of your product solutions in context with... I want to go back to what you were saying about highly reproducible, which goes back to manufacturing tests. We can talk more about the exotic aspects of it, but at the end of the day, I'm trying to manufacture. I want to be consistent in my tests. Why don't you give us an example of one of your solutions in context, in the production line, to help me get my product out the door properly?

Knighten: Right, so a very good analogy you made earlier was, test equipment has to be better, along whatever access you want to imagine in the device under tests. If you want to measure the noise floor of a power amplifier, the thing that has to measure that noise floor has to have a lower noise floor and if you want to measure the frequency response of something, you have to have a flatter frequency response that what you want to measure.

In manufacturing, if part of your goal is, I shouldn't say part of your goal, your basic goal of manufacturing is to do the same thing over and over again, then your test equipment has to be reliable and repeatable. And, I would say that is maybe not the number one thing AP is known for. People generally think of Audio Precision for the performance of our test equipment, but, certainly, part of the performance of our test equipment is that it will measure the same thing over and over again in a relatively harsh environment for decades.

EE: An industrial floor can be a harsh environment.

Knighten: An industrial floor can be a very harsh environment, because it turns out that when people build factories and think about manufacturing costs, they might decide that air conditioning is not something they actually need, so the temperature inside the factory could vary 20, 30 degrees C or Fahrenheit during the day. The humidity could suck. It could be a dusty, dirty environment. All of those things are very true. Particularly for our customers using our equipment in production tests, that's really the No. 1 thing, is that our equipment can be relied upon to be exceptionally repeatable and reliable in an adverse environment.

Typically, we're in that realm where, actually, our customers take it for granted. They just assume, "Yep, I can buy this piece of equipment, put it in this environment and never think about it again." That's something where, with our partners, it's true of their equipment. That's, actually, one of the things that distinguishes, I'll say, a measurement microphone from a non-measurement microphone, as a measurement microphone uses a stainless steel diaphragm instead of, say, a Mylar diaphragm, and it is quite robust and stable.

EE: It goes far beyond just putting an extra gasket on it and some plastic.

Knighten: Oh, yeah. It's generally, again, the stability and repeatability that will exhibit over time, temperature, humidity and dust, is orders of magnitude better than what you would see from any other kind of microphone.

The thing is, is that in terms of what we're doing these days, the key for us and our customers is, No. 1, we have a lot of different products targeted at some different segments, but they all run the same software, so somebody who starts out using, say, the APx555 in an RNT environment, they can recommend an APx517 or 515 for the manufacturing environment and they can assume that the measurements are going to be the same, the measurements are being made with the same algorithms, they can compare the measurements and they can assume that, after that product, after the analyzer has been on the assembly line for two years, that those measurements are still correct (Figure 3).

That, I would say, is the most fundamental thing that we're offering today, is that within this sub-segment of... You take a consumer electronics product and you're entirely right, it's amazing, all the different things that a smartphone has to do these days, but if you take the audio subsystem, and the speakers, and the microphone, and the codex, and amplifiers and all of that stuff, we're offering a solution to our customers from the design stage through the production, and we do a fair amount of stuff with return centers.

Our analyzer will actually sit at a warehouse that's collecting stuff returned by consumers and they will test it to see, "Well, is the consumer just returning it, and we can repackage it and resell it or is this actually defective?"

EE: That's a very interesting test aspect, successfully reclaiming useful return stock, that's another nice, cost-effective way to rationalize the equipment if you are on the fence about getting it.

Knighten: The thing is that speaking to stuff at the higher end, when you start talking about a smartphone that might have a value of something approaching $1000, all those companies have gotten, in the early days, that stuff would just get binned. But, I think I can say, authoritatively, that at this point, all the major smartphone companies have a process to reclaim smartphones and they will, in fact, do a repair. So when you return your trade-in smartphone, they will, actually, test it, make some assessment and quite possibly, depending on the component, repair it before reselling it, rather than just binning it.

EE: Why don't you give us a final thought for the audience, to make them think about acoustic tests in a way that they may not be?

Knighten: This is the quote I use, which is not original, by my part, this goes back a lot, but audio often gets not enough credit, in that most people... Think about the advertisement you see for smartphones. When was the last time that somebody touted the great audio of a smartphone?

The analogy I use is movies. Nobody would ever go back to silent movies. Audio is interesting, in that it doesn't get, I think, very much credit in the general populace, exactly. Every smartphone I see, they tout the screen. "Oh, it's got a fantastic screen. It's got so many pixels by so many pixels. It's this size." They never talk about, "Oh, and by the way, the built in speakers have this frequency response and this sound pressure level," right?

Nobody ever says that, but on the other hand the screens, the manufacturing of screens has gotten really good. You don't have issues like dead pixels anymore. Smartphones are not, generally, being returned by consumers because of a screen defect. Smartphones are being returned because a consumer picks the phone up, gets on a phone call and it doesn't sound right.

Audio test is important and often underappreciated. It's a very large cause of returns on consumer products. Consumer products and cars, that's another one we're dealing with, but the car companies actually do a lot of audio tests, because when a car gets-

EE: Well, audio and cars have evolved together.

Knighten: Audio and cars have evolved together, and the cost of a consumer returning their car is grotesque.