The Force Awakens

The latest star wars film is one of the biggest box office hits to date. Millions of people saw the renewed confrontation between the dark and the light side of the Force. A renewed battle between the Sith and the Jedi. In this latest installment of the Star Wars franchise we truly get to walk into the past, with the reappearance of past characters, cute droids and a clueless Jedi running around while powerful Sith Lords demonstrate their vileness with over-the-top planet scale destruction machinery. 
Still, one of the most interesting differences is the speed in which the plot reveals the true heritage of the main antagonist, revealing that the players of light and dark side are (again) closely related.

Star Wars and Philosophy.

There has been written a lot about the connection between the “Force” from the Star Wars franchise and real world philosophies and religions. Star Wars contains a mix between typically classical Manichaeism, that goes for the more biblical opposing forces of good and evil, and the more philosophical Tao and  yin/yang, that strive more of a balance between forces. Because while the traditional good vs evil is prominent in the whole film, culminating in spectacular lightsaber (I’ll admit I have one myself) battles, there is indeed a more subtle notion a balance of forces visible underneath. There is clearly just one Force which is used in 2 different ways. It is clear that the dark and light side are entwined and cannot be separated or seen apart.

It should be obvious that this conflict is in fact the age old conflict inside every human between his different sides (not just plain good/evil), and that the true solution to the conflict (but alas also the best way to terminate the whole Star Wars franchise 😉 is internal resolve and balance between those 2 sides, as dictated by eastern philosophies like Tao. Instead the films show us the result of  a direct conflict which in the end just leaves us with more opposition later on (as the latest film aptly shows).

Yin and Yang

In Chinese philosophy, yin and yang (alsoyin-yang or yin yang,   yīnyáng “dark—bright”) describes how opposite or contrary forces are actually complementary,interconnected, and interdependent in the natural world, and how they give rise to each other as they interrelate to one another. Many tangible dualities (such as light and dark, fire and water, expanding and contracting) are thought of as physical manifestations of the duality symbolized by yin and yang.

Source: Wikipedia

Yin/Yang and Datamanagement

We can apply this image on Yin and Yang to data management. Indeed, in data management we have seen several conflicts that have stifled both maturity and acceptance of the data management field. These conflicts clearly need balance instead of conflict.  For example, there is the age old battle between the “data organization” (yin/dark) and “data valorization”(yang/light) (with a good throw in of a lot of technical issues for good measure). The need for good data organization has always been difficult in the light of the eagerness of monetization, and a lot has been said around the death of data organization. This attitude is visible for both providers of services and solutions as well as customers and users. But neither can win this argument since without any data organization it is hard to see or use data, let alone create value from it.
We also see other lesser conflicts between e.g. technology and data, and even between different levels of representations (e.g. conceptual, logical, implementation) that all want to control the data by themselves. So there is enough opposition if you go looking for it, but it might be a good idea not to try to win a war, and instead strive for some sort of balance instead.

Use the (un)Balance

The current unbalance in data management has serious consequences. It means wasting time and resources, and creates a lack of focus on what is important. It also makes us susceptible to arguments that try to win us over to one side, creating a never ending search for better technology to end the conflict that isn’t one. This generates heaps of opportunities for (large) (technology) organizations that are servicing this field. There is a great perversion here, that rewards those with no real interest in ending this conflicts (in balance), since it’s continuation allows them to keep selling new installments of their service and software, fuelling the conflict instead of ending it (See? It is the the Star Wars franchise in disguise;) So there is really nothing new here. But for true data management maturity, this conflict needs to transformed into a collaboration. A higher maturity is becoming a necessity in the light ascent into the information age. Bad practices need to go because they are eating up precious resources and practitioners alike. We need to start focusing on how manage our data in a flexible, reliable, sustainable and quality fashion, instead of (getting) run over to by the latest fads. It has kept our field hostage for over 40 years, only serving as money machine for the largest IT corporations forcing us into endless technological sequels, just to get a little bit of extra valorization. Even the “Big Data” movement, starting as a club of “data freedom fighters” are being been assimilated into this process.

The true “Dark side” of data management is the side trying to corrupt and disrupt the striving for a healthy and sustainable “data balance” for it’s own gain.

Data Valorization growing pains

In the past data valorization was not seen as a true asset to an organization. Most organizations considered data organization as operational expense. But in the current information age the cat is really out of the bag (and possibly into the fridge as well), and the monetization of data is undeniable at the forefront of business these days. It is also becoming clear that a lot of data technologies were not up to the task of delivering true value, and hence the rise of the grassroots ‘big data/nosql’ movement. It was clear that the IT service companies from the past where not up to the task. Alas, a lot of incorrect conclusions where drawn from the apparent imbalance between structure and value, not resulting in an improved  balance in our industry, but in a (technologically) more fractured landscape. Indeed, the rise of analytics and ‘data science'(sic) shows us that data valorization has grown up, just to find that ‘data-organization’ has still not grown up. This resulted in trying to ignore ‘data-organization’, thereby shifting the balance in data management. This is the (Big Data) technocracy trying to find a new technologically driven balance and might be leveraged for your advantage, but certainly does not hold any fundamental solutions to data management.

Data Management in balance

In data management I see Yin as the force that organizes the data. It is the force that strives for example for data modeling, architecture and data governance. I see the Yang as the valorization of data. It is the force that strives for insights and value creation through for example analysis and exploration. We should realize that these activities heavily depend on each other. Without data valorization there is no need for data organization, and without organized data, there is nothing to create value out of. So it is important we balance these two aspects. We see a lot of activities affecting the balance and strength either way. We note that for example Master Data Management initiatives strengthen the data-yin, while hiring data scientists strengthens the data-yang. We also need to understand how activities affect the balance. For example Business Intelligence programs, while setup as a vehicle for valorization usually put most of their effort in data organization due to an imbalance created by the program itself. It is the same reason that most data scientist put the majority of their time into data structuring and not actual valorization. This clearly shows a lack of balance and hence bad data management. 
An interesting phenomenon is the “Rise of the Data Chiefs” notably the CDO(Chief Data Officer) and de CAO(Chief Analytics Officer). Clearly they are champions of the data-Yin/organization and data-Yang/valorization. Let’s hope they understand the balance they should be maintaining, else we are in for some boardroom (lightsaber) data battles.

(Logical) Data Modeling: Separating Yin from Yang

An important aspect of Yin and Yang is it’s demarcation line. this line separates and connects both aspects. In data management this line hides and abstracts the data organization from the data valorization. This is exactly the role of a logical data model, whose role it is to form logical (not technical!) abstraction and access layer to the data.

a Better definition of “Data Management”

Data Management is the art and science of creating a balanced andcoordinated separation between  data organization and data valorization

This definition assumes some very important aspects of data management:

  • A clear separation between data organization and valorization is a precondition for good data management.
  • At the same time these two aspects are dependent on eachother
  • The main effort to get a good separation goes into good data architecture and good data modeling.
  • Data management facilitates valorization, but it’s execution is a business activity.
  • A large effort with data organization is in containing and shielding the valorization from the influence of data/information entropy!  (This also requires good data modeling and architecture)
  • Data organization also needs to be applied to people, processes and organizations, not just in the actual data itself.
  • Since balances can and do shift, data management is also about finding new equilibriums between data organization and valorization.

 Yin/Yang and Data management Frameworks.

We now have a very important criteria when we are looking for good data management frameworks, theories or approaches:  How good are they in creating and maintaining a balance between data organization and valorization? Alas, not many show promise in this respect. A good example of a framework that does honor our yin/yang principle is the Data Quadrant Model.

 

“The next time you face a data management challenge, try to look for the imbalance first instead of looking for an issue to fix. Try to address this with true data management, trying to separate data organization from data valorization using good modeling and architecture principles. Use a data management framework or approach that honors this balance instead of trying to corrupt it”

 

May the balance be with you!

 

ABOUT THE AUTHOR:

Martijn Evers is Chief information architect at and co-founder of I-Refact, delivering top data engineers and architects to high profile organizations helping them to managing their data effectively. He’s an expert facilitator and researcher on a large range of data modeling topics and is regarded as one of the best data architects around.