A network perspective on assessing system architectures: Foundations and challenges

Organizations are increasingly faced with the challenge of architecting complex systems that must operate within a System of Systems context. While network science has offered usefully clear insights into product and system architectures, we seek to extend these approaches to evaluate enterprise system architectures. Here, we explore the application of graph‐theoretic methods to the analysis of two real‐world enterprise architectures (a military communications system and a search and rescue system) and to assess the relative importance of different architecture components. For both architectures, different topological measures of component significance identify differing network vertices as important. From this, we identify several significant challenges a system architect needs to be cognisant of when employing graph‐theoretic approaches to evaluate architectures; finding suitable abstractions of heterogeneous architectural elements and distinguishing between network‐structural properties and system‐functional properties. These challenges are summarized as five guiding principles for utilizing network science concepts for enterprise architecture evaluation.


INTRODUCTION
Organizations are increasingly faced with the challenge of architecting and realising complex systems that must operate in the context of a System of Systems (SoS), or architecting and realising an entire SoS itself. 1 The Engineering Systems community has championed the importance of the architecture of an engineered system, in terms of its influence on functional behavior and desirable properties, such as robustness, flexibility, and resilience. 2,3 Traditionally, architecting approaches are reductionist in nature; seeking to decompose systems into smaller, more manageable chunks that can be reassembled into a whole with no adverse effects. [4][5][6][7][8][9] When architecting a complex system or complex SoS, such approaches are less likely to yield useful insights into the System of Interest (SoI; the system whose life cycle is under consideration 10 ) due to a combination of significant system scale and heterogeneity, a high degree of interconnectedness or contribute to a research agenda for the engineering of complex systems laid down nearly a decade ago; "How can we better model, visualize and understand networks of interdependencies, to achieve insights on the likely consequences of variations and perturbations (e.g., impact of changes to schedule or changes in performance of one component on other parts of the enterprise)?" 18 The paper is structured as follows. For the remainder of this section, in order to avoid issues created by the ambiguity of key terms (eg, systems architecting, complex system, and SoS), they are defined for the purposes of this paper. Furthermore, in order to contextualize the systems architecting challenge, typical questions that an organization would ask when architecting systems are discussed. Two real-world use cases are then introduced; a generic Search and Rescue architecture (SAR) and a tactical military communications enterprise architecture (MComms). The theory supporting the assessments of architectures using a network perspective is presented. The following section presents a literature review of current graph-theoretic approaches to assessing system architectures. Architecture topology is then assessed for each of the two use cases. The discussion turns to the challenges of adopting a network perspective to assess a complex SoS architecture, highlighting the areas where care must be taken.
Finally, further work is detailed, five guiding principles are suggested, and conclusions are drawn.

Definitions
Some of the terminology used thus far can be prone to ambiguity. The purpose of this section is to make clear what each term means for the purposes of this paper.

Complex system
The term complex system is poorly defined partly as a consequence of the variety of uses to which it is put. Various publications examine what complexity means to a systems engineer. [19][20][21][22][23] Complex systems can be described in terms of dynamic complexity, that is, how a system evolves or changes over time and the difficulty associated with predicting behavior, 24,25 or socio-political complexity, which is the complexity arising from human cognitive limits or from multiple stakeholders exerting different effects on a system. 9,22 Alternatively, complex systems can be described in terms of their structural complexity, which arises from a combination of system scale, connectivity, and heterogeneity. 21,22 The starting point for this manuscript is that a systems engineer endeavoring to architect or design a complex system will initially be concerned with more objective characterizations of structural complexity and its relationship to system behavior, function, or performance. Thus, a complex system, for the purposes of this manuscript, is defined as a system formed of many interdependent components which, as a consequence of the nonlinear character of these interdependencies, has the capacity to exhibit emergence, implying that overall system behavior cannot simply be inferred from the behavior of system components. 2,11,24,26-28

System of systems
The term SoS is also loaded with ambiguity. A recent review of pertinent publications on SoS Engineering 29 identified the increasingly rich descriptions and classifications used, and noted that authoritative guidance and robust methodologies are still lacking for SoS Engineering. Early thinking by Maier 30 defined an SoS as "an assemblage of components with operational and managerial independence." 30 Maier also postulated that the geographical distribution of a system's parts could be a contributory characteristic, but argued that geographical distribution alone is not sufficient for a system to qualify as an SoS.
International standards do not provide much clarity. The draft annex to ISO/IEC/IEEE 15288, System Life Cycle Processes 2015, 10 defines an SoS as a "set of systems for a task that none of the systems can accomplish on its own," echoing Maier's description but adding the notion of some emerging functionality that can only arise through the bringing together of the constituent systems. This point is further reinforced by Boardman and Sauser who set out five key characteristics of an SoS: autonomy, belonging, connectivity, diversity, and emergence. 31,32 The defining separation between a system and an SoS is equivalent to the separation between a component and a system of components-the notion of emergent properties at the aggregate level that cannot be attributed to their parts. Boardman and Sauser say it best: "The distinction lies in the meaning and significance of 'gathering together'an SoS is much more because its parts, acting as autonomous systems, forming their own connections and rejoicing in their diversity, lead to enhanced emergence, something that fulfills capability demands that set an SoS apart." 31 The draft ISO 21839, Systems of Systems Considerations in Engineering of Systems 2017, 33 similarly places emphasis on emergence as a defining characteristic of an SoS; the emergence of some new capability that none of the constituent systems have.
Alternative definitions assert that an SoS is deliberately brought together for the purpose of a goal , 34 but this is problematic because evidence for this may be limited (eg, the City of London is arguably an SoS without a designer, purpose, or goal). The a priori existence of a

Context
With working definitions presented, our focus can turn to contextualizing the problem of assessing complex SoS architectures. The case studies analyzed below are used to exemplify the exploration of complex SoS architecture evaluation. To contextualize the problem, consider an organization faced with an Invite To Tender for the upgrade of a system that operates as part of an SoS. The upgraded system, the SoI, may be complex and the SoS of which it is part may also be complex.
Several challenges are inherent to such a scenario; whether to bid for delivery of such a system; how to characterize design challenges for the SoI (eg, boundary definitions); how to make good design decisions that will determine the subsequent functionality of the SoI; and, potentially, how to evaluate and select one of several candidate architectures. If an organization utilizes systems thinking to assist in the characterization of such challenges, they are likely to maintain a broad and diverse perspective during the early phase design process. [50][51][52] This work aims to explore the extent to which a graphical representation of an architecture can be useful in such a situation. The next section introduces the real-world use cases considered in the remainder of the paper. resented there) or from expert opinion, for example, using some estimate of system importance. 58 However, in the absence of data on the relative significance of the relationships between architectural entities represented by network edges, they were treated here as unweighted. Service Functionality. 56,57 The relationships between these entities were modeled as unweighted directed network edges.

THEORY
This section introduces and defines the graph-theoretic topological properties used to explore the SoS architectures. This section aims to make these concepts and terms less opaque for systems engineers who may not be familiar with graph theory or network science concepts.

Vertex-level topological properties
The characteristic path length of a graph can be used to characterize efficient resource flow in the graph where a shorter average path length indicates that resources, information, energy, and so on, can flow through the network relatively easily.
The Clustering Coefficient of a graph, C, quantifies the probability that vertices i and j are connected, given that i and j are both directly connected to a shared neighbor vertex k. The Clustering Coefficient can be calculated globally (for the whole graph) as the proportion of connected triples in the graph that form two sides of a connected triangle: There are several approaches to identifying which entities in a graph are the most "important," with different approaches defining the term "important" in different ways. This paper concentrates on a few commonly used approaches. For a more detailed review of topological measures of significance, the interested reader is directed to Newman, 59 Guzman et al., 60 and Scott. 61 A simple approach to characterizing the importance of networked entities is to simply consider the entity's degree. Vertex degree could be considered an indication of vertex importance in the sense that Closeness Centrality is thus a measure of importance in terms of which entities are the most central in the architecture, and consequently closest to the other vertices. For a system architect, Closeness Centrality could be used to identify entities in the architecture that, if removed, would have a large impact on the average Closeness Centrality of the architecture, as a proxy for cohesiveness of the architecture.
As other authors have noted 62 A similar approach to Closeness Centrality is to examine importance in terms of the change in the sum of distances between vertex pairs, if the vertex in question were removed. 63 The Closeness Vitality of vertex i, CV(i), is calculated as the difference between the Wiener Index (the sum of the lengths of the shortest paths between all pairs of vertices, in effect the total shortest distance for graph G) of the graph with vertex i removed, I W (G∖{i}), and the Wiener Index of the graph without the vertex removed, I W (G). Important entities can thus be identified as those whose removal would increase the geodesic distance between vertices: An alternative approach to considering importance as geometric closeness is to consider importance to be associated with enabling communication between many entities. The Betweenness Centrality, BC(i), of a vertex i is the number of shortest paths in the graph that pass through vertex i and is given below, where N is the set of vertices in graph G, (s, t) is the number of shortest paths between vertices s and t, and (s, t|i) is the number of those shortest paths that pass through some vertex i: A further approach to characterizing the importance of a vertex, i, is to consider the well connectedness of its neighbors, which involves considering the well connectedness of the neighbors of these neighbors, and so on. How to calculate this recursive definition of importance? Consider a population of random "walkers" traversing the network. At each step, each walker picks a random edge leaving their current vertex and follows that edge to a new vertex. Over time, the initially randomly distributed walkers will come to be distributed across the network in a way that favors well-connected nodes, that is, those that are adjacent to many well-connected nodes. The Eigenvector Centrality, x i , of a vertex i captures this intuition, and is defined below: where A is the adjacency matrix of the graph G (the adjacency matrix is a square matrix representing the graph G, where a i,j = 1 if vertex j is connected to vertex i and a i,j = 0 otherwise) and ≠ 0 is a constant.
For an architect, the Eigenvector Centrality may be a more suitable measure of importance of an entity in an architecture than simply looking at degree since it highlights influential vertices in terms of their location within the network. The directed acyclic nature of the networks used to represent the two use cases presents challenges in the use of Eigenvector Centrality. Namely, vertices that have no in-coming edges have a null Eigenvector Centrality score, as do vertices whose only in-coming edges are from null scoring vertices. Katz Centrality is a similar measure but that deals with this issue by allocating each vertex an initial positive value of centrality. The Katz Centrality for vertex i is where A is again the adjacency matrix of the graph G with eigenvalues , with parameter controlling the initial centrality.
There are several other measures related to Eigenvector Centrality, such as Google's PageRank measure, which identifies important web pages and Hyperlink-Induced Topic Search (HITS) hub and authority score, but these are not discussed in detail here. Instead, an interested reader is directed to Newman. 64

Graph-level topological properties
Further topological measures of significance can be explored at the graph level (corresponding to the level of the SoS as a whole). The first is the density, D, of the directed graph, where |E| is the number of edges in the graph and N = |V| is the number of vertices in the graph.
Density is a simple measure of how densely connected the directed graph is, where a value of D = 1 corresponds to a graph for which every possible connection between vertices is present. Density con- where (c i , c j ) is 1 if c i = c j and 0 otherwise and the number of edges in the network is E. By randomly rewiring the network, but preserving the degrees of vertices, the probability of an edge existing from i to j is is the out-degree of j, and E is again the total number of edges in the network. Q is then given by We have presented several vertex-level and graph-level properties that can be used to quantify the importance of an entity or set of entities in a complex SoS architecture. However, each focuses on different interpretations of how importance manifests itself. Clearly, some of these properties will be more suited to some architectures than others, but the point here is that care must be taken to understand what the property means in the context of the system or SoS when it is modeled as a graph. A brief summary of the concepts discussed in this section is shown in Table 1.

APPLYING NETWORK SCIENCE TO SYSTEMS ENGINEERING
The notion of applying network science to enterprise architectures is Similarly, graphical models have also been suggested as an approach to assess SoS architectures in a quantitative manner 75 , where architectures could be evaluated in the early phase of their design for their ability to meet mission goals. The work has resulted in a recommended methodology that uses mission goals to represent resources, operations, stakeholders, and policies relevant to an SoS and assigns these mission goals to mission threads (suggested to be represented as sequence diagrams), which can be used to evaluate the ability of different architectures to meet the mission goals. The methodology recommends developing executable architectures that can be tested for their ability to meet the mission goals before a commitment is made to the engineering design of the SoS architecture. The authors stop short of detailing how the graphical models can be constructed for architectures; however, and the focus is instead on a framework to be fleshed out later with further use cases.
Such work often seeks to build on the success of network approaches to representing large-scale design products, engineering products, and engineering projects. 76

Network concept Description
Characteristic Path Length The length of the shortest path between two nodes, or average of all such lengths for a graph. Characterizes efficient resource flow in a graph where a shorter average path length indicates that resources can flow through the network relatively easily.
Clustering Coefficient Quantifies the probability that two vertices are connected, given that they are both directly connected to a shared neighbor.

Degree Centrality
The degree centrality for a vertex is the fraction of a network's nodes that it is connected to.
Closeness Centrality Closeness Centrality of a vertex represents its average distance to every other node in the network, measured in terms of shortest paths. Measure of importance in terms of which entities are the most central in the architecture, and consequently closest to the other vertices.
Harmonic Closeness Centrality Harmonic Closeness Centrality removes the impact of undefined path lengths between vertices that can hinder the use of Closeness Centrality by taking the sum of the reciprocal of the shortest path distances.
Closeness Vitality Impact of removing a node on a network's characteristic path length.

Betweenness Centrality
The Betweenness Centrality of a vertex is the proportion of shortest paths in the graph that pass through the given vertex. Important entities are those that enable connectivity between many entities.
Eigenvector Centrality Eigenvector Centrality suggests influential vertices in terms of their location within the network, where influential vertices are those that are adjacent to many well-connected nodes.
Katz Centrality Similar to Eigenvector Centrality but by assigning each vertex an initial value of centrality, the issue of vertices that have no in-coming edges have a null Eigenvector Centrality score, as do vertices whose only in-coming edges are from null scoring vertices, is removed.

Density
Proportion of pairs of network nodes that are connected. Could assist in evaluating competing architectures as a higher density solution may have a greater integration challenge, or a greater dependency management challenge, but conversely could enjoy greater resiliency.

Strongly Connected Components
If every vertex in a set of vertices is reachable from every other vertex in the set, the set is a strongly connected component. Strongly connected components may reveal a core and periphery structure in a directed graph.
Community Structure A community can be described as a set of vertices that are more strongly, or more frequently, connected to each other than to other vertices in the network. There are several algorithms for community detection.
Alternative frameworks exist within systems engineering to model of such an approach requires that the problem at hand is concerned with throughput and that the complex system or SoS architecture can be adequately modeled as a network that transports some resource from a source to a sink. Although the example used for a complex SoS architecture is a simplified manufacturing network, the justification that such a system is an SoS is not provided, nor is it determined how well a typical SoS can be modeled as a network with resources passed between sources and sinks. The work does highlight that there is potentially a wealth of theorems within graph theory that could assist the architecting of a complex SoS. This paper starts from a different modeling perspective to the work above. Instead of using a DSM or MDM, or creating a new applicationspecific abstraction based on system entities, a graphical model is constructed directly from AVs created in accordance with an Enterprise Architecture Framework. Rather than analyzing synthetic architectures, this work uses two real-world architecture products. Our work explores the extent to which the metrics and methods described in the sections above can be usefully exploited in each case and highlights the challenges inherent with taking such an approach. We summarize our findings by suggesting five guiding principles for the effective mobilization of networks concepts for complex SoS architectures. To examine the level of agreement between the various concepts, every correlation between the topological measures of significance was calculated for both architectures (

Graph-level topological properties results
At a more aggregate level of consideration, the edge density for each of the two architectures is calculated and is shown in     Figure 2) and another is the collection of the high-degree system discussed earlier (the secure deployable broadband voice, data, and video communications system) grouped together with several functions, capabilities, artifacts, and a service (shown in the teal community in Figure 2). This teal community supports the earlier visual interpretation of this system as a potential particular area of focus for this architecture. Interestingly, this system is grouped with a range of entities more operational in nature than technical communications. For the SAR use case, consider the orange community detected (Figure 3 This appears to make sense, and may be something that is neglected or overlooked in a "divide and conquer" approach to system development.
However, as we discuss later, this assessment requires some caution.

DISCUSSION
There is potential for the results of graph-theoretic analysis to be Although visualizing and examining the structure of these complex architectures, in terms of their connectivity, may uncover patterns that help architects to understand how their SoS operates, or how they may be able to leverage interventions into their SoS, it is again an area where care must be taken. Even a simple visual interpretation of networks can be misleading, with different layout algorithms telling different visual stories that can be interpreted in different ways by dif-ferent stakeholders, a challenge further magnified by the plethora of algorithms available for graph visualization. 89 The analysis used here deals with a static view of the system based on an enterprise architecture. In taking a static representation of the SoI, it is difficult to examine if any emergent properties will be present in the SoI, and future work should investigate how a network perspective could inform or predict the presence of this property by moving toward dynamic analysis.
Network approaches are also available to offer alternative partitions of a system architecture using community detection algorithms.
However, the modularity of such a partition, to evaluate the presence of this structure, does not evaluate how "correct" or "useful" the analysis, such as the capability, services, and operational viewpoints. 42 A further study could therefore specifically explore the extent to which a network perspective on these AVs can provide meaningful insights into the entire architecture, or at least how the analysis can be usefully bound to those AVs.
Network perspectives suggest examining whether a "core" and "periphery" structure exists and further work should seek to identify exactly what constitutes a "core" of an architecture in a real SoS. A network perspective may also suggest asking questions that address

CANDIDATE GUIDING PRINCIPLES FOR THE APPLICATION OF NETWORK CONCEPTS TO COMPLEX SOS ARCHITECTURE EVALUATION
1. Partition architectures to manage heterogeneity. The interpretation of network concepts and measures is considerably less conceptually challenging if the aspects of the architecture that are modeled are more homogenous. Graph-theoretic models can struggle to represent adequately the heterogeneity of entities and relationships present in a complex SoS. While it may be tempting to improve the fidelity of such models, perhaps by including further node properties, edge weights, or more sophiscicated network types, there is a compounding challenge in establishing confidence in the data underpinning these features, given the early lifecycle stages architecting activity tends to focus on. The more sophisticated the network representation of an architecture, the more challenging meaningful interpretation of network properties becomes, especially for more complicated aspects, such as assortativity and examining dynamic processes on the network. Instead, a more suitable approach is to consider partitioning an architecture into more homogenous AVs where network concepts may be more readily mobilized, noting that care should be taken to consider the diversity in the character of system entities (different types of network nodes) and the relationships between them (different types of network edges). However, this divide and conquer approach leaves a challenge of reintegrating the analyses of these separated components of an SoS. While it is beyond the scope of this work to recommend particular partitioning approaches, one useful contribution of a network perspective on architecture evaluation is to bring this challenge to the fore. However, even in such a case, comparison with other centrality measures will contextualize and thereby strengthen this analysis, and architects should be cognisant of the implicit modeling assumptions that underpin these metrics, so as to avoid potentially being misled.
3. Combine quantitative and qualitative assessment. Further to principle two, any system architect hoping to utilize a "network perspective" should temper the results of numerical analysis with a qualitative assessment. Simply because a set of enterprise architecture entities have strong connectivity between them, even taking into account inter and intraset connectivity does not guarantee that it makes logical sense to treat them as one distinct architectural "module" or "unit." The same is true of exploring if a "core" and "periphery" structure dominates, or if one architecture is more "complex" than another using one particular measure of complexity.
Thus, architects may well make use of network science analyses to examine the structure of their architecture, but they should temper these results with their own qualitative evaluation. 4. Maintain awareness of modeling depth. Any graph-theoretic approach to architecture evaluation should explicitly reflect on the modeling assumptions that underpin their findings. While this task is conceptually challenging, it is a necessary one to avoid misunderstanding or placing overconfidence in some numerical analysis.
Simply put, the network models are not the complex SoS architectures. Further, the architectures are not the real-life SoS in question. At each level, the artifacts and models are representations, abstractions, of the higher level. Care must be taken to not loose sight of this and test if the assumptions that underpin the network led enquiry are still valid for the architecture and for the SoS itself.

5.
Reflect on the questions a network perspective enables. Finally, the methodology used here cannot be seen as an alternative to more traditional approaches to architecture evaluation, but instead should be understood as offering a complementary perspective on architecture evaluation. It is instead recommended as a complimentary perspective, in line with a broader systems thinking approach, that enables architects to ask questions of their architecture that may otherwise go unanswered. Such questions include; "what makes an entity important in this architecture, what role does reciprocity, assortativity, or community structure play in this architecture, what makes this architecture more robust or resilient than another?," or "where is there more merit in considering this diverse and rich architecture at a greater level of abstraction or a more fine-grained level of detail?," or even "what is it about this architecture that led us to towards the toolbox of network science?" As graph-theoretic approaches continue to mature, it will remain important to critically assess the nature of the questions that they are capable of answering.

CONCLUSIONS
In this paper, we have taken a networks perspective on SoS architecture analysis in order to explore the potential for this approach to inform architecture design and selection, and also to highlight a series of challenges that need to be addressed if the approach is to be useful for system architects. A range of simple network metrics were applied to the two use case networks, real-world enterprise architectures, in an attempt to identify key entities and assess gross structural organization. Given the diversity of, for example, measures of vertex significance and community detection, it is challenging to either select particular measures (which may supply partial or idiosyncratic results) or, alternatively, apply a wide range of measures (which will typically disagree in ways that require careful analysis), without first considering what such measures correspond to in the real-world architecture.
Graph-theoretic approaches provide an alternative approach to exploring and understanding a complex SoS, by representing it as a graph and examining this graph's structure, but critically this structure is the modeled graph-theoretic structure. It is not the architecture itself, only an idealized representation of it. In creating a graphical model, there may be aspects of the SoS that are not modeled and thus making assertions concerning the structure of the graphical model may not be equivalent to making assertions concerning the entire SoS.
Analyzing SoS networks in terms of components or communities, for instance, may neglect potentially important factors, such as geographical, functional, or organizational separation between components if these are not explicitly represented in the network structure.
Taking a network perspective on SoS architectures could help inform which entities in an architecture are most important to one another, or assist architecture evaluation by helping determine whether one candidate architecture is more robust, efficient, or manageable than another. In seeking these insights, however, we argue that the tools from network science cannot straightforwardly be applied without developing a more sophisticated understanding of how they map onto the diversity, richness, and context sensitivity characteristic of complex SoS architectures. While developing a set of conceptual tools for the analysis of complex SoS architectures remains an open research challenge, by developing guiding principles for the effective mobilization of network concepts to architecture evaluation, system architects can better take the advantage of these tools. Further, in bringing to light the challenges system architects are faced in taking a network perspective on their architectures, they are better able to avoid being misled by some numerical analysis that lacks contextual awareness.
Instead of using network analysis as an off-the-shelf tool for empirical analysis and decision support in the design of SoS architectures, we advocate a more contextual approach in which network analysis is employed as one of many perspectives that can be taken on an architecture, one that may reveal insights that would otherwise be overlooked, but also one that requires cross-validation against more qualitative or systems theoretic perspectives that may do a better job of capturing the rich and heterogeneous properties of SoS architectures.