Share this:

The Problem of Software Quality Metrics

A fine example of a problem posed by software risk was over a decade ago when the then CIO of the United States Air Force divulged that the US military forces were dependent on hundreds of thousands of copies of a specific piece of software. This piece of software compromised around 65,000,000 lines of code and because it was a trade secret, the Pentagon had not even been allowed to see it. This information was interesting yet terrifying, particularly because the US knew that some of this code had been written by developers in what was considered to be a potentially belligerent nation. However the code, of course, turned out to be Microsoft Windows and the CIO of the US Air Force wasn’t worried about Microsoft or even the potential threat of adversarial software developers. No, his problem, like so many others, arose from his software supply chain.
 

Supply Chain Risk and Service Chain Risk

Whenever a major manufacturer purchases parts from suppliers, there are a number of acceptance tests that must be carried out prior to integrating those parts into their assembly line. Manufacturers are responsible for ensuring that their sample parts meet the requirements to make sure their supply chain risk is under control. This is just one the aspects that contribute to supply chain risk. Other aspects that need to be considered include the ability of suppliers to develop, make and deliver the parts in the first place.
 
The process of conducting acceptance tests is equally critical to non- material supply chains or service chains. For example banks are responsible for deploying vigorous testing to ensure software works correctly.
 

Risk Mitigation

Supply or Service chain risk is just one of the many risks that any company or individual must face—this is where the role of risk mitigation comes into play. Risk mitigation is not a new concept and although there is risk mitigation information available on supply chain risk there is very little information on the risks of software integration. Where supply chains consist of the design of parts for manufacturing purposes the service chain deals with software which is typically trade secret and features a design that is largely incomprehensible to the buyer.
 

Software Integration Problems

Over the years there have many detailed studies of the problem of software risk. The U.S. Department of Homeland Security’s Build Security In program covers a great deal of the troubles caused by software integration. This study even includes some recent approaches to software supply chain risk mitigation.
 
According to the U.S. Department of Homeland Security’s Build Security In program “The programmatic and product complexity inherent in software supply chains increases the risk that defects, vulnerabilities, and malicious code will be inserted into a delivered software product. As a result, effective risk management is essential for establishing and maintaining software supply-chain assurance over time.”
 

Assessing and certifying levels of software risk

However in order to assess software supply chain risk, one must have access to the source code, which is trade secret and most unlikely to be given to the majority of buyers. This begs the question, how can assessment be carried out effectively? Well, there are a number of tools on the market but they are largely primitive at present. These tools are unable to support standard metrics for measuring software quality from several perspectives such as security, privacy and likely defects caused by known defective coding practices.
 

Risk Management for Software Supply Chain Assurance

However the Consortium for IT Software Quality (CISQ) through the joint efforts of Carnegie Mellon University’s Software Engineering Institute, the leader in process quality standards and the Object Management Group, the leader in technology standards are developing new standards that specifically focus on the problem of software quality metrics.
 
At last, by directly measuring software artifacts’ quality the IT industry is unifying on standards for assessing and certifying levels of software risk. The first new standard metric to be released is a consistent, dependable, automatable measure of code size that is designed to deliver metrics of quality related to complexity and size—a consistent Function Point metric.
 
The fact that these metrics are consistent, dependable and automatable has led to a decrease in cost and an increase in the amount of users. Ultimately this means that software users will have a consistent, standardized and readily-accessible method of measuring, mitigating and preventing risk.
 
CISQ will be hosting a webinar on Standard Metrics to Manage Software Risk on Thursday, December 6th at 11:00 AM EST. The webinar will be hosted by Dr. Bill Curtis, the Director of CISQ, and Jasmine Alexander, SVP and CIO of Penton Media. In addition, I will be participating in a CAST Software webinar discussing how to Reduce Software Risk through Improved Quality Measures on Thursday, November 29th at 11:00 AM EST. I hope you will join us for both of these enlightening events.

Leave a Reply

Your email address will not be published. Required fields are marked *

*

Comment validation by @