Complexity Economics Engineering Uncategorized

How to ‘Fix’ Standard Deviations



Standard deviations are a popular and often useful measure of dispersion. To be sure, a standard deviation is merely the most likely deviation from the mean. It also doesn’t take into account the shape of the probability distribution function (this is done better using, for example, entropy, which is a more versatile measure of dispersion).

Standard deviations, however, may be ‘fixed’ to take into account an interesting aspect of data, namely complexity. Let’s see an example. Say you have a system of 28 variables, some of which are independent (i.e. uncorrelated) some are not.


One computes the standard deviation of each variable and may them use it to measure volatility, dispersion or some measures of risk or even performance. However, this simple exercise neglects one fundamental issue – complexity, i.e. the fact that variables are interdependent with other variables, see the complexity map below.


Variables 5 and 7, for example, are correlated with numerous others, while 3, 6 and 25 are uncorrelated. This is reflected in the Complexity Profile (or Complexity Spectrum) which ranks the complexity footprint of each variable in the system. This is illustrated below.


Variable 7 has a footprint of just over 17% while 5 is responsible for nearly 15% of the complexity of the system.

The question now is this: why not use the information in the Complexity Profile to ‘adjust’ standard deviations by adding a correction originating from complexity? Clearly, a variable that is heavily correlated to others could be more ‘dangerous’ than an uncorrelated one.

One simple way to accomplish this is the following:


Adjusted STD = (1 + Complexity contribution) x STD


Basically, variables that increase complexity see their standard deviations corrected (increased) by a complexity-based factor. The (ranked) result is illustrated below.


The bar chart below shows the complexity-induced corrections of standard deviations.


For example, the standard deviation of the biggest complexity contributor – variable 7 – which is 3.81, is incremented by 17.1% (its complexity footprint) to yield a value of 4.46.

The above correction increases standard deviations, illustrating eloquently the concept of complexity-induced risk. Think of the impact of complexity in very large data sets when building a model. Remember, the most important things in a model are those it doesn’t contain.

Doing classical stats may produce overly optimistic results if complexity is neglected. In reality, every system has some degree of complexity, which is invisible to conventional analytics . In reality, there is often more risk than one may think. The above method shows how one can incorporate, albeit superficially, complexity-induced risk into any calculation or procedure without disrupting it.

Established originally in 2005 in the USA, Ontonix is a technology company headquartered in Como, Italy. The unusual technology and solutions developed by Ontonix focus on countering what most threatens safety, advanced products, critical infrastructures, or IT network security - the rapid growth of complexity. In 2007 the company received recognition by being selected as Gartner's Cool Vendor. What makes Ontonix different from all those companies and research centers who claim to manage complexity is that we have a complexity metric. This means that we MEASURE complexity. We detect anomalies in complex defense systems without using Machine Learning for one very good reason: our clients don’t have the luxury of multiple examples of failures necessary to teach software to recognize them. We identify anomalies without having seen them before. Sometimes, you must get it right the first and only time!

0 comments on “How to ‘Fix’ Standard Deviations

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: