by Pat Weaver
Complexity theory helps understand the social behaviours of teams and the networks of people involved in and around a project. The ideas apply equally to small in-house projects as to large complicated programs. In this regard, ‘complexity’ is not a synonym for ‘complicated’ or ‘large’.
This paper will briefly examine the underlying ideas and philosophies that have created ‘modern project management’ together with some emerging ideas such as projects being ‘temporary knowledge organisations’ (TKOs) and the importance of ‘social capital’. It will also consider the key elements of ‘Complexity Theory’ from its origins in Chaos Theory to the ideas of ‘Complex Responsive Processes of Relating’ (CRPR) and seek to link the ideas within two other strands of research; ‘Social Networks’ and ‘Temporary Knowledge Organisations (TKOs)’, to Complexity Theory.
From this theoretical framework the true nature of a ‘project’ will be described from the perspective of the ‘knowledge workers’ or ‘actors’ engaged in the creation, execution, delivery and closure of the project. And two critical aspects of project management practice will be re-evaluated from a ‘complexity’ perspective:
- The purpose and use of project management artefacts such as Schedules and EV Charts as communication tools to influence the ‘actors’ that make up the organisation surrounding the project (its ‘stakeholders’).
- Understanding risk and uncertainty from the viewpoint of ‘complexity theory’.
The paper will conclude by developing a range of practical suggestions for improving the effectiveness of project management practice based on an understanding of ‘complexity theory’ applied to the project environment.
Major projects & programs
Whilst the ideas of ‘complexity theory’ are applicable to all projects, size does have an impact. From a complexity theory perspective, every project is complex, the project team are working together to deliver their project and in the process have to deal with issues and tensions within the project and issues and tensions (if not outright conflict) with stakeholders external to the project. The actions and influences of these external stakeholders trigger the need for the project team to adapt to its environment and engage proactively with the external stakeholders for the project as a whole to survive and deliver a successful outcome. And importantly the behaviour of the team cannot be predicted from the behaviour of any one person.
As the size of a project increases some of the issues surrounding ‘complexity’ will tend to increase, particularly in relation to the project’s stakeholders. The two major differences to be expected are:
As Spike Milligan said: “Money can't buy friends but it can get you a better class of enemy” – the size of the project is likely to make external stakeholder management more important. Increasing the scale of a project is unlikely to significantly change the supportiveness of supporters but may well galvanise the interest and reactions of opponents.
- Stakeholder management issues within the project are likely to emerge as major issues – a small team or 3 or 4 people will generally resolve issues if well lead. A project team of 300 or 400 people will require significant formal processes as well as effective leadership for the project to be successful. The number of potential communication channels (or relationships) within the project team increase exponentially based on the formula:
(N x (N-1))/2
where N = the number of team members.
The two factors outlined above are common to both programs and large projects; although the objective and purpose of a program (to deliver benefits) is significantly different to a project (the efficient creation of a deliverable).
However, for the purposes of this paper the different objectives of programs and projects are not important and ‘size’ is only one of the influences on the degree of complexity in, and importance of, effective stakeholder management to creating a successful outcome; therefore we will simply consider ‘generic projects’.
The creation of ‘Modern Project Management’ - a scientific view
The greatest challenge facing project management in the 21st Century is managing the shift from the ‘command and control’ paradigm based in the theories of ‘scientific management’ developed by Taylor and others in the early 20th Century to a recognition of the inherent uncertainty and complexity involved in managing every project, and in particular, projects focused on the outputs of knowledge workers.
The key underpinnings of project management theory derive from the principles of ‘scientific management’. These principles are very effective in optimising and controlling simple manual tasks such as loading iron into rail cars and laying bricks. Managers can see and measure the work, quality is an ‘obvious’ factor and production rates can be established. Similarly, scheduling and cost estimating are relatively straightforward; you cannot build a brick wall until after the foundations are laid and all of the cost elements are measurable.
However, even for this type of simple project some project management ideas are overly optimistic. Project ‘control systems’ (schedules, cost plans, etc) don’t control anything and to a large extent, neither do project managers. People control their actions and the environment dictates many ‘uncontrollable’ variables. Apart from providing ‘guidance’ to the project team, project plans only provide a tool for estimating the likely levels of uncertainty in each element of the plan and then measuring the actual degree of variance as it occurs.
3000 years ago people believed it was possible to accurately foretell or control the future by using devices such as chicken entrails. Modern man should recognise that writing numbers into a spreadsheet during cost planning cannot ‘control’ the price suppliers will actually charge the project at some time in the future. Similarly entering durations into a scheduling tool cannot control the actual time needed to perform a task; arguably, much of the dogma surrounding scheduling in particular is founded on very uncertain premises.
Pollack’s review of project management literature (Pollack 2007) shows that project management has developed based on positivist and realist philosophies, as an essentially purposeful, functionalist activity with an emphasis on reductionist techniques and control.
Some of the key assumptions include:
The idea of ‘reductionism’; the characteristics (and behaviours) of a complicated entity can be understood by studying the characteristics of it parts. The underlying assumption being complex things can always be reduced or explained by understanding the simpler more fundamental elements from which they are assembled. Consequently, it is generally assumed the project’s goals and objectives can be decomposed and fully understood using techniques such as the Work Breakdown Structure (WBS).
The idea of ‘the clockwork universe’ - The outcome of an action is predictable and repeatable and outcomes (outputs) scale in proportion to inputs (i.e. more effort results in a larger or quicker output). The most obvious manifestation of this idea is the ‘critical path schedule’ one key assumption in traditional CPM is that task durations change predictably based on the level of resources applied to the task.
The future is essentially controllable. Management control over the workers actions, and consequently the project outcome can be achieved through developing accurate schedules and cost plans with adequate levels of detail and proper risk assessments incorporated in the plans; and then diligently managing in accordance with those plans. The natural extension of these ideas is that if adequate control cannot be achieved at the current level of decomposition, adding more detail will bring ‘better control’ and that human destiny is controllable.
- That it is possible to manage or eliminate ‘all risk’. If adequate effort is applied to risk management all risks can be transferred, mitigated or identified for acceptance, and appropriate contingencies and risk response plans can be calculated for all of the accepted risks. The caveat on risk management is that it is very expensive to undertake a ‘complete risk assessment’ and it is easier to accept there will be some ‘unknown unknowns’ that can be covered by a general ‘management reserve’.
This traditional project management paradigm (philosophies and assumptions) has some validity when the goal the project has been created to achieve remains stable and the work required is largely straightforward and obvious (e.g. the construction of a well-designed, traditional, building). This ‘knowability’ and ‘understandability’ of both the project work and its intended deliverable is implicit in the relative success of traditional project management in its base industries such as construction and engineering. The value of the paradigm become less certain when the ultimate project goals are ‘to be discovered’ or change during the course of the work (e.g. delivering a cultural change program) or when the nature of the work changes from essentially manual tasks to knowledge work.
Certainly project management writings of the last few years are starting to suggest that ‘people skills’ and leadership are seen as important attributes of a successful Project Manager and effective stakeholder management is definitely seen as a major item in delivering project success.
Note: The author strongly believes all of the processes described above are invaluable tools to help manage projects successfully, particularly effective risk management and scheduling. The objective of this paper is to re-frame their purpose within the complex processes of managing a project team.
The knowledge work conundrum
Consider the software engineer tasked with developing an algorithm to solve a data transcription problem. The primary ‘work’ is thinking through the problem and creating the idea that will allow its solution. This happens in the engineers mind. Counting ‘outputs’ is useless, the number of lines of code written do not measure the effectiveness of the solution; the most efficient and elegant solution may have far fewer lines of code than some inefficient ‘clunky’ solution.
The effectiveness (quality) of the solution cannot be fully tested until several other components being developed by other people are created and integrated and as these other elements are developed their final structure may require changes in ‘our engineer’s’ algorithm. In these circumstances, good managers can minimise identifiable risks through effective risk management and lead, motivate and provide direction assisted by their project plans but the only person that can actually ‘control’ the work is the knowledge worker.
The knowledge worker also needs to be continually adjusting his work to remain coordinated with the work of other knowledge workers in the team – the goals of his work are changing as the work of the project unfolds. In these circumstances, the manager is just one of many people relating and communicating in the complex network of interactions needed to successfully deliver the project8.
The branch of management science studying these complex multi-dimensional problems phenomena is known as ‘complexity theory’.
Temporary knowledge organisations (TKOs) and social networks
Two relatively recent views of projects, founded primarily in the work of European and Scandinavian academics are the ideas of projects as TKOs and the project team as a social network within a larger social network comprised of the projects stakeholders and surrounding community. These ideas have much in common.
Viewing a project as a temporary knowledge organisation (TKO) moves the focus of project management from the observation of the output of the project (its deliverable) to the processes needed to transform inputs received by the project team into the project deliverable(s). This is achieved by the gathering, melding, processing, creating and using of knowledge. TKOs share characteristics such as uniqueness, finiteness, uncertainty, and transience with the traditional project organisation.
The difference between them is the recognition that ‘linearity and predictability are not the realities of project management,’ and that resolution of ‘multi-causal problems within a complex and chaotic environment’ requires the team members as knowledge workers to generate new knowledge.
This represents a shift from viewing projects as ‘tools’ applied to solving problems, where people are outside the project; to the creation of a sense-making community of practice by the people involved in the project. This requirement for project team members to also be knowledge workers leads to additional expectations of the leadership qualities of the project manager.
A social network is a social structure made of nodes (which are generally individuals or organizations) that are joined by some form of relationship. The shape of a social network helps determine a network's usefulness to its individual members. The project team is a social network and it exists within a larger network primarily consisting of the project’s stakeholders. The project network can be considered as being both independent of the larger organisational network and an integral part of it.
Fig. 1 – Influence Networks (Bourne 2007)
Each network contains a level of ‘social capital’ - the sum of the actual and potential resources embedded within, available through, and/or derived from the network of relationships that connect its members. In the context of this paper, the two key aspects of social capital are the ‘know how’ required to create and deliver the project outcome and the ‘willingness’ to exert effort to achieve the project outcome.
Importantly, the level and availability of social capital within a social network is not fixed, it can be increased by developing:
- a more effective network by creating stronger relationships (links),
- a better alignment of the actor’s objectives through developing clear, agreed goals,
- effective collaboration and leadership (i.e. by developing a ‘high performance team’).
Conversely social capital can be dissipated by ineffective leadership, lack of agreement, contradictory visions, etc. (i.e. by allowing a dysfunctional team to develop).
More importantly, the social capital available through the network is not constrained by the team ‘as is’, the level of social capital can be increased by improving the absolute level of some or all of the current actors’ resources (e.g. by training), by expanding the network to connect to new actors with the required knowledge (e.g. by employing a consultant or connecting to an expert) and/or by increasing the efficiency of the network by increasing the level of interconnectivity in the network by improving conductivity of existing relationships (the ability to transfer information) and/or increasing the number of relationships between actors (network density).
Key elements in enhancing interconnectivity are raising the levels of trust and respect between actors (Brookes 2006); which are consistent with the ideas of a ‘high performance team’ mentioned above.
Combining TKOs and social networks
Combining these ideas, it is reasonable to assert that it is the actual transfer of knowledge through the ‘social network’ that allows the project team, functioning as a TKO, to develop the new knowledge needed to create the project’s deliverables. It is also important to note the actual transfer and creation of knowledge and the implementation of the ‘new knowledge’ to create the project deliverable is absolutely controlled by the willingness of the actors within the network to engage positively in the work. Therefore, effectiveness of these processes are constrained:
in part by the extent of knowledge actually available to the network,
in part by the efficiency of the network in transmitting the information to and between the actors who need to make use of it, and
in part by the willingness of the actors to actively engage in the processing and implementation of the knowledge in an aligned and effective manner.
The observation of a ‘high performance team’ is evidence of the knowledge processing and social networking systems working effectively.
These ideas will be picked up in the discussion on ‘complexity theory’.
The complexity view (Cooke-Davies, et al. 2007)
Complexity theory has become a broad platform for the investigation of complex interdisciplinary situations. It has developed from and includes the earlier fields of study known as ‘chaos theory’ and ‘Theory Building (Dubin 1978)'. Complexity theory can be defined as the study of how order and patterns arise from apparently chaotic systems and conversely how complex behaviour and structures emerge from simple underlying rules.
Some of the ideas appear directly relevant to understanding project management from a relationship perspective.
The first idea is from the early days of ‘chaos theory’. The ‘Tipping Point’ describes the way natural systems can absorb influences with minimal (or predictable) change until the tipping point is reached and then there is a sudden catastrophic change. This idea is particularly relevant when thinking about ‘culture change’ in a network. The social network can absorb a lot of pressure to change and targeted individuals may change whilst under direct ‘pressure’ but the ‘cultural norms’ prevail and there is little real change until the ‘tipping point’ is reached, then there is a sudden shift to a new set of ‘cultural norms’ and people retaining the ‘old ideas’ are seen as being out of touch. It is impossible to predict the ‘tipping point’ until it has been reached at least once.
The idea of ‘nonlinearity’ builds on from this. Nonlinearity suggests that you can do the same thing several times over and get completely different results. Small differences may lead to big changes whilst big variations may have minimal effect. The ‘butterfly effect’ describes the situation where minute changes in the starting condition can have major and unpredictable consequences in non-linear systems. Importantly, all human relationships are non-linear.
These ideas seriously question the validity of ‘detailed programming’ attempting to predict and control the future path of a project.
The complete unpredictability of nonlinearity is counteracted by the idea of ‘strange attractors’. Strange attractors are most easily thought of as recurring patterns that have quasi-predictable features. The behaviour of dynamical systems in nature (e.g. the weather) has a degree of predictability. However, dynamical systems can follow a number of qualitatively different attractors depending on their initial starting condition and the effect of external influences. The idea of a ‘normal degree of predictability’ underpins modern civilisation and most project processes including estimating, scheduling and risk analysis; however, the actual outcomes are highly dependent on the starting condition and the ‘strange attractors’ encountered along the way.
This brings us to the concept of ‘complex dynamical systems’. These systems are continuously both receiving and transmitting ‘energy’ to their environment, e.g. a Hurricane; at the detail level they are in ‘chaos’ but overall are a quasi-predictable ‘system’. After a period of time transferring energy, these systems reach a point of irreversible change (bifurcations) where the outcome is inherently unpredictable.
Self-organising systems are complex dynamical systems that appear capable of self-organisation and exercising choice in a manner that makes them inherently unpredictable.
Feedback loops contained within the system ensure that ‘rich patterns’ are produced and the system itself behaves in its own unique way. Importantly, how the system will behave cannot be determined by studying its parts. These ideas apply to shoals of fish, ant colonies and human social groups. As these self-organising systems go about their daily business they are continually exchanging energy and matter with their environment. This allows them to remain in a state that is far from equilibrium and allows spontaneous behaviours and new patterns to emerge in response to stimuli; ‘living on the edge of chaos’.
The feedback loops in self organising systems can amplify or attenuate the effect of the stimuli. Work underway in the UK at Strathclyde University is identifying the way many project ‘feedback systems’ can quickly turn into ‘vicious cycles’ through ‘normal’ management responses to cost and schedule slippages that amplify the effect and cause greater problems. The implication of this research is that traditional project ‘controls’ may be inappropriate once a project ‘starts to go off the rails’.
Complex adaptive systems are self-organising systems that have the capacity to learn from their experience. This ‘system description’ appears to relate very closely to a project team, living on the ‘edge of chaos’; responding and adapting to its surroundings (ie the project’s stakeholders) and learning (or creating new knowledge) as it advances.
These ideas offer a new set of insights on the management of projects; the key strand of research into complex adaptive systems that this paper will focus on is the concept of ‘Complex Responsive Processes of Relating’ (CRPR), which puts emphasis on the interaction among people and the essentially responsive and participative nature of the human processes of organising and relating.
‘Organisation is an emergent property of many individual human beings interacting together through their complex responsive processes of relating’. People use ‘language’ in conversations to simultaneously transfer information and ideas, negotiate social status and develop power relationships.
The ‘actors’ intentions, choices and actions / reactions are influenced by and influence their conversations as they operate within the dynamic of their daily interactions with other people. The process of ‘organising’ is the human experience emerging from the interactions between actors who are all continual forming intentions, choosing and acting in relation to each other as they go about their daily work together implementing the project. The future seen from this perspective is therefore under perpetual construction by the movement of the human action itself.
Consequently, the actors, interactions and emerging organisation are located in a specific context (the organisation’s social network, culture and ‘project team’) and are oriented towards an ‘unknown future’ (the project outcome) that the group is in the process of continually creating (or working to achieve). In this context, the intended (or planned) future needs to be differentiated from the actual future that unfolds over time.
Tying complexity theory, TKOs and social networks together
Traditional views of projects and project management have tended to treat the ‘idea of a project’ as a real object.
This is an easy enough assumption to make when the product of the project team’s effort is a building, aircraft or other tangible object itself. It is less useful when the product is intangible (e.g. a business culture change).
The ideas in ‘complexity theory’ reverse the traditional views of project management developed over the last 50 years and move from a Cartesian/ Newtonian/ Enlightenment paradigm, from which the practice of project management has emerged, to a more ‘complex’ view! The underpinning ideas in these theories separate and re-define the three key elements involved:
The ‘objects’ are the people engaged in planning, managing and executing the project work.
The ‘idea’ they collectively create and share through their social networks is the concept of ‘this piece of work being managed as a project’.
And through their coordinated efforts and the use of ‘social capital’, the objectives of the project are achieved (the project’s artefacts or deliverable).
Within this framework, the people and their relationships can be described as follows:
- Each individual, or stakeholder, is an ‘actor’; members of the project team are ‘project actors’.
- One actor interacts with another actor to form a relationship. Each relationship can conduct ideas, knowledge and influence, through the use of an appropriate ‘language’ and has a power dimension.
- The combination of many relationships forms the ‘social network’ around the project and within the project. The project network can be considered as being both independent of the larger organisational network and an integral part of it.
The project network can be considered to hold ‘social capital’, the knowledge, desire and capability to achieve the projects outcome.
The social capital (resources) of the project actors (their knowledge and willingness to expend effort) combine through their relationships to make the achievement of the project outcomes possible.
The larger organisational network (and the project network within it) defines the culture within which the project team operates.
The consequence of accepting these theories is to shift the focus of ‘project management’ from the object of the project to the actors involved in the project (i.e. its team members and stakeholders), and to recognise that it is people who create the project, work on the project and close the project.
Consequently the purpose of most if not all project ‘control documents’ such as schedules and cost plans shift from being an attempt to ‘control the future’ - this is impossible; to a process for communicating with and influencing stakeholders to encourage and guide their involvement in the project and create a jointly held objective for the team to work towards achieving. The rich symbolic languages of schedules, Earned Value reports, etc. are powerful modes of communication provided both the sender and the receiver within a relationship are trained to properly understand the language being used.
The outcome of the project seen from these perspectives is, as noted above, under perpetual construction by the project team itself. The individual decisions made by people in the network ‘create’ the future:
- different information, will lead to
- different decisions, which will cause
- different outcomes, leading to
- a different ‘future’!
And one of the key influences on the multitude of decisions being taken by team members every day should be the project plans and schedules, updated, adjusted and agreed by the project team.
There are two factors requiring caution:
Firstly: effective project networks are analogous to ‘high performance teams’; however, creating and leading high performance teams requires specific skills (Thamheim 2007). The key elements are:
- a focus on creating a supportive work environment with relatively low levels of conflict and cross functional support from all parts of the organisation;
- a specific effort to encourage team formation and the development of the ‘high performance team’;
- the development and use of efficient and effective work processes and
- the existence of supportive organisational processes.
The project team cannot easily achieve this state on its own; many of the factors are the province of senior management and are influenced by the culture of the organisation.
Secondly: the use of project management artefacts and other incentives to motivate and align project team members also needs to be implemented with care. If the wrong measures are selected behaviours can be driven in counter-productive directions, e.g. by focusing exclusively on the easily measured element of ‘time’ at the expense of less obvious elements such as stakeholder satisfaction (Roberts M, 1998). All members of the project team need training to understand their roles in this ‘complex project environment’.
Quoting Prof. Andrew Leicester (Bond University) “The term ‘simple project’ is an oxymoron – every project is complex, some are big and complicated as well!”
The ideas contained in ‘complexity theory’ define a number of challenges for traditional project management whilst at the same time offering a potential solution to the unacceptably high rate of failures in knowledge based projects such as software development. Some of the challenges are:
- Accepting that ‘project control systems’ cannot control anything.
- Recognising the future is inherently unpredictable.
- Also recognising it is essential to assume a degree of predictability and importantly, assuming the degree of uncertainty can be estimated15.
- Adapting project management systems to support a collaborative and negotiated style of leadership that fosters the development of ‘high performance teams’.
- Training project managers and senior managers to lead and motivate their teams rather than attempting to apply outdated 20th Century ‘command and control’ ideas to knowledge workers.
- Embracing risk management:
- Immature organisations ignore risk,
- Mature organisations ‘manage’ risk,
- Wise organisations will recognise that after all of their necessary and prudent risk management and risk mitigation activities, they still face an uncertain future!
The best defence against uncertainty is a well led, motivated and skilled project team ‘living on the edge of chaos’ and adapting quickly and effectively to the unforeseen changes in their surrounding circumstances.
Complexity theory, as linked to TKOs and social network theory in this paper, suggests that the creation of a successful project outcome will always be an uncertain journey, but the path to success or failure can and will be influenced by the actions and attitudes of the actors within and around the project team. The key element is how effectively the project team uses its social network to gather the resources (knowledge and support) needed to create success.
These ideas are closely linked to Stakeholder theory, which suggests that a project is only successful if its key stakeholders perceive the project to be successful. An obvious part of this requirement is the delivery to the stakeholders of a ‘useful deliverable’, in an appropriate way, that includes elements of the ‘right output’ at the ‘right time’ for the ‘right cost’; but none of these factors are absolutes. Project success is created by effectively managing the stakeholder’s expectations to align with what is reasonable and feasible for the project to achieve. Managing the stakeholder’s expectations takes place through the relationships that exist within project’s larger ‘social network’ and is ‘complex’. The emphasis for successful project managers is shifting to a focus on influencing, motivating and leading the people who can make projects successful.
From a project management perspective, complexity is a direct consequence of having people involved in the execution of the project’s work and other people perceiving they will be impacted by either the execution of the work or the project’s deliverable or both. To borrow from ‘Game Theory’: true complexity lies in understanding the intentions of others. In the project context this means leading the project team to deal effectively with the project’s stakeholders real (usually hidden) attitudes, expectations and willingness to take action to support or oppose the project.
The final conclusion to be drawn from this paper is that successful project management is a far more complex process than simply dealing with the iron triangle of time, cost and output first described by Dr. Martin Barnes in 1969. Successful project managers in the 21st Century will develop and lead high performance teams that create project success through the effective use of the ‘social capital’ within their networks, to proactively manage the expectations of their stakeholders and then deliver the projects outputs in alignment with those ‘managed expectations’.
Bourne, L. (January 2007). Avoiding the Successful Failure. PMI Global Congress Asia-Pacific, Hong Kong SAR, China.
Brookes N.J. Morton S.C. Dainty A.R.J. Burns N.D. (2006). Social processes, patterns and practices and project knowledge management: A theoretical framework and empirical investigation.International Journal of Project Management 24, 474-482.
Cooke-Davies, T., Cicmil, S., Crawford, L., Richardson, K. (2007). We’re Not In Kansas Anymore, Toto: Mapping the Strange Landscape of Complexity Theory, and its Relationship to Project Management. Project Management Journal, Vol 38, No. 2, 50-61
Dubin R. (1978). Theory Building. Free Press, New York.
Pollak J. (2007). The changing paradigms of project management. International Journal of Project Management 25, 266-274.
Roberts M. (1998). Modern Management Systems for Higher Margins. Proceedings, Micro Planning International User’s Conference.
Thamheim J. (2007). Managing Project Teams in an Age of Complexity. Proceedings, PMI Global Congress 2007 – Atlanta.