How much should a test engineer be valued?

Editor's note: This blog is a follow-up to a piece posted on EE Feb. 21

Recently I posted a blog entitled “How little is a test engineer valued?” in which I showed an example of how little test engineering services were valued by recruiters. I got an outpouring of responses. Since those responding almost unanimously agreed that the hourly rate offered test engineers was low, here I will try to offer some guidelines in calculating how much a test engineer is worth.

Test engineers—hardware, software, anywhere—are responsible for finding faults. This is often frowned upon socially as those who devote their lives to delivering bad news are often mistaken to be the cause of the news itself. While our gut reaction to bad news is often negative, if we consider that it is better to know early than to not know, we can see the value test engineers bring to an organization. Acting in an oversight capacity, test engineers contain failures and faulty products to the factory, saving untold amounts of penalty costs incurred when these mistakes are allowed to escape to the customer. As long as customers receive good products, they are not concerned whether it was produced perfectly or were made to work as they were supposed to work. If the product customers bought isn’t delivered as promised, the level of displeasure translates to money your company will need to spend on making the customer happy again. Alternatively, the customer could remain unhappy and while it may not be apparent, that carries a cost as well. Test engineering is intended to minimize the occurrence of delivering a faulty product. Also, accepting that it will be inevitable that faulty products will escape tests, the test engineers will design means to quickly diagnose failures and make repairs to the product, to the process, or to both.

Since most productions do not reach perfection (100% yield), any product quality approaching this level happens because of successful test engineering finding the faulty product and directing its repair. Unfortunately, this fact is usually not apparent to management and marketing and therefore, little value is attributed to the test engineers’ endeavors. In fact, many do not consider the test engineers’ contribution to the quality of the product, until such time that a large number of failures in the field occur.

In order to value the contribution of the test engineer, we should probably calculate the value that the test engineering department—with all its resources, including test equipment—can bring to a company. Today’s electronics circuit boards, for example, can consist of hundreds of components along with thousands of connections for each product. If, on the average, there is a potential of 10,000 faults per board and only one fault per board actually gets through, that manufacturing is exceptionally good at 99.99%. If, on the average, one fault per board exists, using poisson distribution, we can calculate that the fault-free yield (Yin) will be about 36.8% of the boards, and the rest (63.2%) of the boards will have at least one fault in them. Tests eliminate faults from escaping to the customer, saving the cost of dealing with an irritated client. We can represent this saving as:

S = (Yout – Yin) * P * Q – T [1]

Where S is the savings in dollars, Yout is the quality of the product after having been tested and repaired if necessary, while Yin is the quality of untested products (36.8% in our example). P is the penalty cost per faulty product delivered to the customer (up to you to predict, but never $0) and Q is the quantity of products you deliver per unit time (month). T is the cost of the test operation per that same unit time. T includes equipment depreciation, facilities, amortized software costs, and of course, personnel salaries—including the salary of test engineers.

Testing produces other benefits (B) that include product improvements, lowering production costs, lowering support costs, and a myriad of other long-term advantages. B may only be visible in other organizations. We will not go into details here about the source of B, but it plays an important role in determining the total value (V) of testing in Equation [2].

V = S + B [2]

V in Equation [2] needs to be positive, which means that S should also be positive. (B will likely not be negative, but it should not be the driving force in making testing valuable). If S is negative or it is insignificantly small, your test operation is not performing economically and your test strategy is probably ineffective. Test engineering can control the (Yout–Yin) and the T terms in Equation [1], so to “fix” your test strategy, you need to revisit both of these terms and optimize them.

To evaluate the value of test you should calculate the return on investment (ROI) in Equation [3]

ROI per year = V/T per year [3]

A 2% ROI is what we can expect in a risk-free investment, and at that level the company would not have any incentive to invest in test. However, ROI of 33% or more would double the investment in less than 3 years and would be a good investment. (Since test equipment becomes obsolete quickly, you would not want ROI less than 33% because you would not be able to realize a return on your investment before all your equipment would need to be replaced.)

Yin, P and Q are beyond the control of the test operation. Improving the value of test (delta V) can be achieved by creating delta Yout = (Yout-new - Yout-old) and delta T = (Tnew – Told) such that

Vnew > Vold [4]

Test engineers should maximize Equation [4]. When a test engineer can achieve this (s)he has earned the organization the amount of money in Equation [5]:

delta V = Vnew - Vold [5]

If a test engineer—whether an outside consultant, a contractor, or a captive full-time employee—can create a substantial increase in value, they are worth a high percentage of delta V. If, for example, the increase in value is $500,000 per year, why debate whether a consultant’s 1,000 hour at $250 per hour is too expensive? Yet we know that many organizations would reject the idea that a test engineering consultant should be paid $250,000 for a 6-month assignment. At the same time, the same manager would not blink an eye if you pointed out that the company owns $250,000 worth of capital test equipment collecting dust that no one uses any more (and increasing T because of depreciation and taking up real estate. It could also be true that they are buying more of the same equipment again, rather than use what they now own, because of accounting reasons since this is for a “different project.”)

I venture to guess that the simple equations I used above are not familiar to many seasoned test engineers or managers of any discipline. This is probably because no one seriously wanted to calculate the worth of a test engineer. Rather, the profession is often viewed as a necessary (some even think unnecessary) evil and minimizing the cost makes the most sense. This would be true if test engineering had no ROI or no role in improving bottom line profits for the company. Despite common misconception, test engineering is not simply a profession of facilitating a service. Some managers are too far removed from the test engineering profession to realize that the true role of a test engineer is to solve an economic problem.

Test engineering is not a rubber-stamping inspection service. Rather, it is a serious technical function intended to uncover hidden faults and prevent them from escaping to customers’ sites. Marketing should appreciate the value that such a task provides. “Quality Assurance” is meaningless without a test that can be used to assure that quality. Parts procurement should use test findings for future parts and vendor selection. Manufacturing should be eager to know what process improvements can be made by examining data collected with the tests. Reliability figures are only meaningful if we know that all products were good when the product was shipped and we also know what faults were identified when the product was returned. Finally, selection of test equipment resources can be a considerable cost factor. Test engineers influencing design for testability (DFT) and Built-in self-test (BIST) can offer many cost-saving options.

One professional who can influence so much in the organization should be hired based on her/his value to improving the test operation, indeed company profits. The hiring decision should not be predetermined to be captive vs. consultant vs. contractor. The criteria should not be based on the years of experience with a particular piece of test equipment or test language such as LabVIEW or TestStand or VEE or Python or C++ or ATLAS. Experienced test engineers can learn any of these in short order. More importantly, they should be judged on their experiences relating to circuit type (analog vs. digital vs. microwave), test types (manufacturing test vs. design verification test vs. environmental test vs. field return test and repair), cross disciplines (design for testability, system level built in test and diagnoses, prognoses, quality assurance, reliability, maintainability) and finally on their ability to affect the economics of test.

So how much should a test engineer be valued? Some percentage of the value (s)he can bring to the organization... which is a great deal more than what many organizations have given them credit for.