inner banner

Calculation of Standard Testability Metrics

Calculation of Standard Metrics for Defense/Aerospace

Since DSI is the original pioneer of the need for Testability as an integral component of the design development process for the US Department of Defense, and has mentored William Keiner, author of the first Testability Standard (MIL-STD 2165) prior to its acceptance in 1981, we have continued to lead industry in this purpose.

Testability Standards MIL-STD 2165 and IEEE 1522

In 1989, just a few years after the release of MIL-STD 2165 (1986), DoD and industry conferred to work on a more comprehensive capability, part of which initially revolved around “Testability”, by introducing “AI-ESTATE” (Intelligence Exchange and Service Tie to All Test Environments: IEEE1232). This new IEEE Testability Standard was going to eventually be identified as IEEE1522. The need for this new standard was a look at the future of what was possible to achieve through “Testability” (later identified and distinguished as “Diagnosability” in IEEE1522). The document will describe the branching of the term, “Testability” into the forming of two (2) tangential paths described as Testability and Diagnosability. The difference being that Diagnosability would also include and require calculations for “Fault Isolation” in conforming with a minimum set of standard design assessment metrics.

One of the core objectives of Testability Standard IEEE 1522 was to enable the open interchange of diagnostic data between design and support tools and technologies. The objective assumed that if the Fault Isolation characteristics and metrics were universal, then a program could swap in diagnostic or test tools that could equally interpret the format used to exchange the diagnostic sequencing data. This format would give rise to new interchange prospects by that would eventually use ATML. DSI had preceded IEEE’s push to support ATML, with its own open XML diagnostic interchange format, “DiagML”. But while DSI’s ISDD Tool Suite uses the broader diagnostic capability of DiagML, ISDD also supports the most popular and advanced diagnostic uses of ATML.

 

DSI had preceded IEEE’s push to support ATML, with its own open XML diagnostic interchange format, “DiagML”. But while DSI’s ISDD Tool Suite uses the broader diagnostic capability of DiagML, ISDD also supports the most popular and advanced diagnostic uses of ATML.
>> Integrate Test and Diagnostic Tools via ATML

Testability Analysis that Lives beyond Design Development

While complex board level designs have been benefiting from using eXpress Diagnostic Engineering tool over the years, the primary value area has been on the application of System Testability as a method to influence the complex or critical design for sustainment over the lifecycle of the product. As such, DSI has had to transcend the purpose of Designing for Testability beyond the simple generation of Fault Detection (FD) and Fault Isolation (FI) metrics, and into a full systems’ integration process that seeds the design and support disciplines with unparalleled and continued value.

Why stop at generating the (FD/FI) numbers when you can export the diagnostic data through a one-button push to a Diagnostically-savvy XML schema, or “DiagML”? Or, publish the operational run-time diagnostic “Test Sequencing” into DSI Workbench to open up a vast new data interchange capability with ATML or IEEE S1000-D? The value of reuse of any investment made during design development is certainly a tremendous value, but to reuse this investment over and over again, for many purposes or yet evolving purposes, is another tremendous leap in value.

Testability – Variance in Goals

With ISDD, the Testability analysis does not solely target ineffective “one-&-done” goals. Unfortunately, the “one-&-done” get the “checkbox checked” goal has been ubiquitous in too many areas where customers and contractors alike dismiss the value and opportunity that diligent Testability Analysis will absolutely produce.

 

ISDD is DSI’s process that avails the use of many “back-end” tools for use in a myriad of seamlessly integrated production and sustainment activities. DSI’s high-end eXpress tool enables the capturing and analyzing of the design and provides unmatched facilities to perform and service every type of diagnostic application imaginable – and beyond. But that’s for you to investigate throughout this website on your own.

“Testability” Confusion

Many programs have “Testability Requirements”, which are, today, a bit confusing. In attempting to sort out some of the confusion, when we discuss eXpress, in the context of this website, we are primarily referring to very large, critical or complex fielded assets. This will help keep us on track when other folks are describing a different interpretation of Testability, which essentially limits Testability to be an analysis of a chip on a circuit board, the circuit board itself, or a set of digital circuit boards, which by themselves, are not deployed or fielded products.

Much of the confusion originates in discussions where circuit board manufacturers are discussing specific techniques to ensure that their circuit board is testable by the design engineer or, in some more advanced applications, in some production labs that use very expensive circuit design tools. Such techniques are more commonly referred to as Design For Test, or DFT.

More of the confusion arises when this section of circuit board designers refer to a “system”, which, in that world, is far too constrained to simply the chip, board or collection of boards or any other associated digital hardware piece(s).  When DSI refers to “system”, it shall mean everything the DFT world considers, but also everything else that truly is included in a fielded product that may support any or all of its operational characteristics. So, when caught in any discussion or paper presented by an expert in the DFT community, be sure to remember to ask them what they mean by “Testability” and “System”, so you’ll better acclimate others involved in the discussion. This may help to reduce some of the “Testability Confusion”.
>> Design for Test (DFT) vs. Design for Testability (DFT)

DFT Data Reuse

The best use of the output from the DFT community, however, is to reuse most everything that’s able to be exported from any of their design tools. With ISDD, we are interested in grabbing nomenclature, netlist of components with dependencies (internal and external), part numbers, costs, functional flow, test points and test coverage, schematics, board design layout with “X and Y” coordinates, images or any other documentation. We will have use for all of it and more. Reusing all of this data will need to be worked into the design process. Of course, eXpress will be a tremendous “cross-check” tool that inherently, rechecks and cross-checks all naming identifiers, functional flow, test coverage and ultimately any design level Fault Detection or Fault Isolation metrics if produced by the DFT tool or by the designer performing their testability analysis. This inherent rechecking and cross-checking process will enable the data be better prepped for inclusion and integration in the other levels of design hierarchy, which can now be referred to as establishing “Data Fitness”.

Testability Analysis in eXpress

Once the design is captured in DSI’s eXpress functional modeling tool, any other designs can also be captured and represented at the same design hierarchy as the initial design modeled, or at any other functional or “diagnostic level”. The hierarchies can be based on the hardware design or simply to represent any diagnostic and maintenance objective, or can be used to do both. Typically, a “system” in eXpress will consist of any number of “lower-level” designs that may support the function of the next “higher level” design. Each design modeled in eXpress only has to concern itself with that particular design as eXpress will automatically generate all of the implied functional dependencies between any objects modeled in that design. This is the simple paradigm that is used to characterize, or “model” the assemblies, subsystems and systems, or even System of Systems (SoS).

So the Testability Assessments in eXpress must first establish the boundaries of the assessment. But be mindful that any model captured in the eXpress modeling paradigm, may need to be assessed in conjunction with any other same level, lower level or higher level model as a condition to really generate actual Fault Detection or Fault Isolation values for the design as integrated with the other models. This is a very important concept and consideration because FD/FI assessments that are typical in the DFT community, for example, are not typically able to consider any other companion designs external to the boundaries of their technology.

Testability across Integrated Design Domains and Hierarchy

The eXpress models provide a robust canvas to include multiple design domains (digital or analog electronic, hydraulic, software, pneumatic, mechanical, optical. etc. or any mixture thereof, and environmental inputs or outputs), and unlimited levels of hierarchy. This is critical because a fielded “system” may have many unknown design-induced constraints that impede or “interfere” with, not just “Fault Detection” at the system level, but also “Fault Isolation” due to such restrictions. The eXpress analysis will, objectively and, sometimes abruptly point out these non-considered diagnostic weaknesses. As a result, the eXpress assessment will educate the analyst on many factors that have unknowingly contributed to over-presumptuous values previously represented in non-eXpress generated Testability assessments.

Capturing the Functional Design

Initially, the functions for each object are represented in the eXpress model. As each object is added and connected by “nets” to any other object, eXpress automatically creates that “functional dependency” or “relationship”.  As other objects are connected via these nets, the functional dependencies are fully propagated. When any net misrepresents any logical use, eXpress, upon user action, will check the entire model for logical errors, and provide in depth guidance and assistance for the detailed understanding and the performing of corrected the modeling errors.

It is important to initially capture the functional design as a baseline as early as possible during design development. In this regard, early testability feedback to design development may serve to be timely. There are many more advanced safety and supportability dividends that can be discovered in throughout the design development as discussed throughout the context of this website. But for the purpose of this more fundamental discussion, the performing of early testability analysis simply from the functions captured in the eXpress model, provide an invaluable opportunity to provide a new perspective of the diagnostic integrity and its impact upon the “integrated system”.

Let’s say, for example, failure detection or isolation is unable to support the maintenance philosophy at higher levels of the fielded design. If this shortcoming is discovered early during design development (i.e. before part selection is finalized), then the ability to address this impending and costly sustainment weakness can be worked at minimal development cost, if any. This is a keen place to start using feedback from the eXpress model, if the program is considering risk reduction or “smart” lean development and support objectives.

Capturing and Propagating Failures in the Design

During the process of selecting parts in the development phase, early input from the Reliability engineering can also be captured in the eXpress modeling environment.  This is very advantageous because now we can begin to learn which components fail more frequently, are more reliable, and which failures are more critical than others. Combining this ability to “overlay” failure data and their propagation onto the functional design, we have now established a “hybrid diagnostic model” that will produce or integrate with a broad range of functional and failure-based testing assessments, tools and technologies. This melding of failures and functions within a diagnostic model is unique to eXpress and facilitates other advanced operation simulation assessments and run-time capabilities that are also unique to ISDD.

Because there is so many reasons to reuse this data that is captured in the eXpress model, we will be expecting eXpress to establish a database that will generate “turn-key” assessment products as outputs based upon the data detail captured at any time. As the design evolves and matures, the assessments will be able to absorb any new data, reorganize it, and then restructure the data so a full set of new assessment products can be generated with a push of a button. Of course, the assessed data can be “exported” for immediate use in run-time diagnostic environments, so we need to validate the diagnostics to ensure that any tests performed will cover and report the intended objectives.

Evaluating Tests to be used for Deployed Diagnostic Objectives

Once the test coverage in represented in eXpress, we can also accurately determine if the candidate “Points of Tests” are capable of actually obtaining diagnostic status from their “test placement” within the architecture of the design.  This is all performed as a core purpose of the “Desktop Fault Insertion” (DFI) facility in eXpress. No other tools can approach this sort of diagnostic validation thoroughness during design development.

The DFI capability enables full design evaluation, which can include any other design(s) contained in the fielded system. It’s best to first use it as another way to validate the test coverage in each eXpress model contained in the integrated system model.

At this point, we’re ready to move on to examining more advanced testability metrics and capabilities that are available and currently being used within eXpress for some of the more sophisticated applications.
Some of the more popular capabilities are addressed in more detail in “Calculation of Advanced Metrics for Defense/Aerospace”.
>> Diagnostic Validation through Fault Insertion

Related Links

Subscribe To Our Newsletter