Creative New

Via it.toolbox , dhtow01

There has been a dramatic recent shift in sentiment in relation to the most appropriate model for developing software systems. The shift has marked a change from the tradition of preparing a detailed requirements specification as the first phase in the development cycle, to a less rigid adaptive evolutionary approach. 
The ongoing goal of software engineering is to ensure that a system meets its primary aims in terms of the quality criteria of functionality, performance, reliability and efficiency. Achieving rigorous standards of reliability and efficiency has never been the major problem for developers; rather it has been the potential for basing the system’s requirements specification on obsolete or inadeqaute premises, which on completion delivers sub-optimal outcomes. 

This is similar to the mathematical problem of using excellent deductive logic to draw conclusions from a set of axioms, but reaching a wrong conclusion because the axioms themselves are incorrect or incomplete. 

Time and again this Archilles heel of software development emerges- particularly when a project is large, complex and operates within a dynamic environment. Systemic failure is more often the norm and the litany of collapsed projects keeps growing; particularly in the government and multinational business domains of procurement, supply, logistics, human resources, health, education, customer services etc. 

A vast literature has accumulated on this endemic problem- how best to capture the requirements of a system. It is the elephant in the room at almost every CIO seminar and conference. 

A number of techniques have been applied over the past fifty years, each hopeful of delivering the magic silver bullet including- functional, data, entity relationship, process and object-oriented analysis, applied at various levels of sophistication. Each manages to capture a particular facet or dimension of user aspirations- but never the whole set. 

Libraries of tools and methods also cover all phases of the traditional software development cycle- requirements analysis, design, coding, testing, implementation, as well as project and quality management- but still the seminal problem remains. 

Organisations attempt to deal with the problem in a number of ways. 
First by buying off-the-shelf, pre-packaged software, hopefully flexible enough to be easily tailored and adapted to an organisation’s requirements. But this solution only works if a reasonable functional match exists in the first place and if the level of built-in flexibility is sufficient to avoid costly re-working over time, beyond ad hoc version updates. 

The second way is by linking together multiple functional components like a Lego set. But this also only works if the components are available and can be updated regularly and if they fit together without the need for complex middleware. 

In past decades, these approaches often worked adequately for standard systems such as accounting, inventory, sales, mainntenance, CAD, HR, job scheduling, project control, office systems etc. But even these became obsolete or unmanageable over time as protocols changed, customer expectations increased, technological change accelerated and the enterprise’s products and services evolved. 

In the 21st century we live in a vastly different world of web services and SOA’s, cloud and mobile computing and enterprises which must continually adapt to a bewildering mix of competitive and economic pressures, almost on a daily basis. 

On the other hand we have proof that immensely complex systems can be built and remain viable and continue to deliver real value over time- communication systems such as the Internet and World Wide Web, operating systems such as Linux, social networks such as Facebook and Myspace, scripting languages such as Javascript and Ruby, search engines including Google and Safari, as well as numerous designer business applications from eBay, Amazon and Google. 

These are cooperative, innovative works in progress, which have been tested through many iterations by scenarios and prototypes, before emerging in beta form- all developed in close consultation between developers and their user communities. And they continue to be adapted as community needs evolve on a daily basis. 

These are examples of the new emerging class of evolutionary adaptable systems. 

The major driver for the emergence of this radical evolutionary paradigm is the accelerating rate of social, technological and economic change, particularly over the past twenty years. In almost all cases this acceleration will mean that long lead times for systems development are now untenable and almost certain to lead to obsolescence or outright failure- certainly before an adequate ROI is achieved. 

It is becoming rapidly recognised that any realistic requirements engineering methodology must incorporate an evolutionary approach, combined with an efficient mechanism such as Agile Programming for converting evolving functional and process requirements to a useable system. This enables the enterprise to adapt to the continuing dynamics of social, business and technological change, by continuously spawning new functions or incremental amendments, without disrupting its core processes. 

The same change imperative applies to small as well as large systems. The risks inherent in smaller systems in the past just haven’t been as obvious or critical. In fact any significant system build that hopes to meet its user aspirations of long-term support and value contribution, must adopt an evolutionary approach. 

Risks in evolutionary development also exist as for traditional systems – the risk that managers misread the environmental signals, such as in the case of GM’s disastrous planning decisions and continue supporting ineffective analytical and reporting systems; or that the updates and changes become so pervasive that the system becomes unwieldy and opaque, as in Microsoft’s early Vista system. 

But the risk impacts from not following the evolutionary canon are far greater. Systems supporting wrong management decisions can be quickly turned around by agile methods, if recognised in time. Building inappropriate functions can waste resources and cause annoying disruption, but don’t cause catastrophic project collapse and massive system re-design time delays and budget overruns. 

Future trends 

In the future, the trend towards applying evolutionary techniques to software development will become embedded in IT best practice, particularly as this will be coupled with the parallel trend towards autonomic management of enterprises; interacting with the human and physical world on a real-time basis. 

The record of systems development to date is appalling, but not through lack of the enormous level of innovation, effort and professional skills applied. It is because we have found it difficult to come to terms with a constantly evolving world impacting our built environment. We have ignored the fundamental principle that systems must continually adapt to changing environments if they are to survive. 

This is as good a silver bullet as the IT industry is likely to get. 
Evolution has been the universal driver of all systems- biological, social and now economic and computing, since the universe began and we ignore its wisdom at our peril.

The Future of Life

Via dhtow01

The Information Imperative

Evolution in all its manifestations is therefore synonymous with the processes of ongoing adaptation and optimisation. However, the critical conclusion missing from most analyses is that the overall change is not random, but directed. Allied with change is the acquisition of information and knowledge, which as the fossil record over four billion years has proven, results in more complex DNA and nervous systems, a larger brain, more flexible behaviour and increased intelligence; resulting in more sophisticated system decision-making.

The understanding of information as a basis of evolution is rapidly unfolding and is at the core of the new decision-based network theory of evolution defined in this book. For example, computer science owes a huge debt to biological systems. The field originated largely as an attempt to understand and replicate the functions of the brain. Understanding the encoding of life’s blueprint in DNA and more recently the processes of gene regulation, have shed light on better evolutionary computation techniques.

It is now realised that information processing is common to all biological systems ⁽⁹⁾, in the form of the chemical switches and amplifiers used for gene transcription and encoding, cell metabolism, signalling and cycle timing, neural network functions and in the mechanism for encoding the evolutionary history of a species in its genome.

Information relating to transformational events in life’s environmental past – climatic, geographical, social etc. can be measured in the cumulative level of genetic diversity and mutational variation developed over generations. This is applied to dating past paleoclimate events using the regular mutational rate in the genes of mitochondria.

As mentioned, even bacteria absorb and process a great range of information from the environment, allowing the continuous selection of the most appropriate genetic configuration for survival. Each step of this process leaves an information marker, whether in the form of a genetic mutation, fossil record, neural connection or cosmic explosion, which can be used to retrace the evolutionary path of the system through time to its source.

This information vector allows the system to achieve higher levels of organisation and complexity, providing a bulwark against the physical process of depletion of energy and increasing entropy. Processed information therefore acts as a form of negative entropy protecting the species against loss of identity and potential. As the system’s knowledge increases, positive feedback enables it to expand its capacity to acquire new sources of energy and greater flexibility and complexity of organisation, thereby increasing the probability of its survival. Contemporary examples include man as a highly successful innovator, corporations as monopolists, national imperialism and development of advanced technologies to explore space and harness new energy sources in response to climate change.

The earliest hominids four million years ago had a brain size of approximately 450 cubic centimetres. This had increased to 750 cc in Homo erectus 2.5 million years later and doubled to 1400 cc. by the time modern man evolved 130,000 years ago. This rapid evolution of cerebral development has been ascribed to a variety of factors such as increasing use of tools, more advanced social relationships as well as development of language and numbers. It is probable that the drying out of the African forests some four million years ago triggered a more upright stance in man’s early ancestors in order to forage for scarce food resources, as well as the need to employ cooperative strategies and tools for hunting and food gathering. Obviously each advance triggered a further development, catalysing the need for greater intellectual capacity to cope with increasing environmental diversity. More recently it has been discovered that two vital mutations have occurred in the brain of modern humans; one around the time of the explosion of cave art across Europe 35,000 years ago and another 6,000 years ago at the beginning of the period of building more permanent dwellings. 

Darwinian selection does not fully explain the unfolding of life, because the time to evolve a species as complex as humans based on random localised mutations would be unthinkably slow. Only by incorporating the insights provided by modern information theory within the evolutionary paradigm can we realistically explain the ascent of modern humans within such a comparatively short timeframe.