Experiences and challenges from developing cyber‐physical systems in industry‐academia collaboration

Cyber‐physical systems (CPSs) are increasing in developmental complexity. Several emerging technologies, such as Model‐based engineering, DevOps, and Artificial intelligence, are expected to alleviate the associated complexity by introducing more advanced capabilities. The AIDOaRt research project investigates how the aforementioned technologies can assist in developing complex CPSs in various industrial use cases. In this paper, we discuss the experiences of industry and academia collaborating to improve the development of complex CPSs through the experiences in the research project. In particular, the paper presents the results of two working groups that examined the challenges of developing complex CPSs from an industrial and academic perspective when considering the previously mentioned technologies. We present five identified challenge areas from developing complex CPSs and discuss them from the perspective of industry and academia: data, modeling, requirements engineering, continuous software and system engineering, as well as intelligence and automation. Furthermore, we highlight practical experience in collaboration from the project via two explicit use cases and connect them to the challenge areas. Finally, we discuss some lessons learned through the collaborations, which might foster future collaborative efforts.

In fact, MDE is a relevant software engineering paradigm to raise the abstraction level and improve the ability to handle complexity.The use of models, as first-class abstractions of systems and environments, is a fundamental element for technologies in current and future CPS engineering platforms. 1Also, with the advent of DevOps, CPS engineering would benefit from continuous development approaches proposing a smooth continuum from design to runtime (and vice versa).In parallel, IT leaders envision the productivity boost of tomorrow to be brought by the application of Artificial Intelligence (AI) principles and techniques.Thanks to AI and ML for IT operations (AIOps/MLOps), 5,6 the standard DevOps pipeline can be rethought by allowing continuous monitoring, alerting and remediation more securely and reliably.A major challenge is providing a generic, reusable AI-enhanced approach intended to support comprehensive and continuous CPS engineering.* In such a context, the ongoing European project AIDOaRt † aims at providing a model-based framework incorporating methods and tools for continuous software/system engineering and validation leveraging the advantages of AI/ML principles and techniques.The project brings together leading technology providers and research institutions, as well as cutting-edge methods and tools validated in highly relevant European industry case studies.AIDOaRt is a three-year-long H2020-ECSEL research project involving over 30 organizations from several different European countries which aims to apply and evaluate various technologies for developing complex CPS in industrial contexts 7,8 In more detail, the project looks at beneficial applications of MDE, DevOps, and AI/ML.The project divides the partners into use case providers who provide industrial use cases in several domains such as space, maritime, railway, smart grid, smart warehouse, and telecommunications industries.Use case providers drive the project by supplying real-world requirements and case studies, as well as validating and supporting AIDOaRt research and technical deliverables.Solution providers on the other hand consist of various academic and industrial partners who contribute with existing or new solutions to be applied in these contexts.
The continuing gap between academic research and industry is well known, for example, Garousi et al. 9 conducted a systematic literature review discussing the challenges and best practices of university-industry collaborations in the field of software engineering.The review highlights the potential tensions that can arise when academic and industrial partners have different objectives, goals, methodologies, and terminology.To overcome these challenges, the review suggests implementing best practices such as focusing research on real-world problems, being agile, and holding regular workshops and seminars between partners. 9his paper focuses on understanding the challenges that emerge when developing CPS in industry-academia collaboration.This is an important area of research that can help to identify strategies to overcome challenges and improve the outcomes of industry-academia collaborations.In the context of AIDOaRt, a task force has been created to explore this topic.This provided a focused effort to identify specific challenges and opportunities related to CPS development in industry-academia collaborations.The findings from this task force can contribute to a greater understanding of the broader challenges and opportunities associated with industry-academia collaborations in CPS development.To ensure that this research is effective, it is important to use rigorous research methods and to gather input from a diverse range of stakeholders, including researchers, industry experts, and end-users of CPS.It may also be useful to review the existing literature on industry-academia collaborations in CPS development to identify common themes and areas for further exploration.Overall, this research goal has the potential to make significant contributions to the field of CPS development and industry-academia collaborations.By identifying challenges and opportunities, this research can help to inform best practices and strategies for maximizing the benefits of these collaborations.
The main contributions of this paper are (i) a set of challenges from industry-academia collaboration on CPS (data, modeling, requirements engineering, continuous software and system engineering, as well as intelligence and automation), (ii) experience report from two distinct industrial use cases within AIDOaRt, and (iii) lessons learned based on experiences from industry-academia collaborations.
The rest of this paper is organized as follows: in Section 2 we introduce background and related work, in Section 3 we illustrate the method we used for identifying challenges, whereas in Section 4 we describe the extracted challenges.
In Section 5 we report our experience and lesson learned in industry-academic collaboration, while in Section 6 we discuss the considered topics.Finally, Section 7 concludes the work.

BACKGROUND & RELATED WORK
CPSs can be described as the integration of computation with physical processes, in which embedded computers and networks monitor and control physical processes, usually within feedback loops. 10,11In the modern day CPSs are present in many different domains and can be characterized differently, for example wired or wireless. 126][17][18] Examples of the general foreseen challenges relate to, for example, design constraints due to multi-disciplinary work and increase of software-driven components. 19However, there are different important issues related to the CPS deployment, such as maintainability (i.e., the ability to improve, repair and maintain the deployed CPS in real-time in a simple and fast effective way). 20Besides, the deployment of large-scale and heterogeneous interconnected CPS and its integration with the physical world increases the systems' complexity, demanding the management of non-functional aspects of design architectures and dealing with specific requirements and constraints. 36As a consequence, the CPSs require to be continuously updated and improved to meet these quality attributes. 21onstantino et al. 22 investigate a means of evaluating collaborative compatibility through examining co-changed files.They evaluate two strategies and through interviews conclude that they can predict suitable collaborators based on their previous history in development.Fritzsch et al. 23 performed a rapid literature review in conjunction with expert interviews to identify challenges, practices that may address the challenges, and opportunities for micro-services and DevOps in the CPS domain.While some of the identified challenges are similar, like data (discussed in Section 4.1 more in detail), there are also major differences in their scope.For example, their review discusses largely on the management and team oriented aspects, which while important, are not a significant focus of this work.
This paper focuses on the development of CPSs.The information obtained from CPSs, either large historical data or real-time traces, is of very high value for evaluating the validity and properties of the system design.However, specific methods and tools are needed to extract useful knowledge and analyze and reuse it in a productive way.To this end, AIDOaRt aims to integrate AI, MDE, and DevOps innovations to ensure that systems are designed correctly and to increase our confidence in their behavior.The project aims to create a framework that includes methods and tools for continuous software/system development and validation by taking advantage of several techniques.It is expected that the use of mature techniques within the mentioned domains can alleviate parts of the concerns related to complexity in developing CPS, and in turn improve several metrics in regards to system development in industrial use cases (e.g., significant improvement in productivity, quality, and predictability of CPSs).
In this work we report on the active collaborations in the project and extract findings from these industrial collaborations.We present an analysis of identified challenges in the project with a deeper dive with two complementary use cases.Since several key parts of the project activities cannot be directly disclosed, at times, we present an abstract view of the findings so that it can be shared with a wider audience.

General challenges in industry-academia collaboration
Industry and academia may look at the role of research from different perspectives.It seems that the distance between industry and academia has grown 24 over the last decades.Universities have focused on more narrow problems, making innovations difficult to use for industry.Companies instead focused on development.According to a case study from Sweden, 25 there could be many driving forces for companies to join industry-academia collaborations: knowledge, competence, improved products and processes, and legitimacy.Smaller companies were more motivated to improve products and services, whereas larger companies in more general has goals such as developing competence.
7][28] This can be a challenge because they typically have different perspectives on problem formulation, methodology, and result; as well as conflicting differences in their approach to knowledge and in their driving forces.A good evaluation criteria for collaboration is if the new knowledge drives further research or interest in continued collaboration. 26,28Weyuker and Ostrand 29 recommend academics to involve at least one industry participant that sees the value in collaborative studies.However, this might still not be enough for research to be significantly more relevant than academia-only research.Practitioners perceived only a minor increase in the relevance of research when papers had industrial co-authors, according to two studies from the domain of software engineering. 30,31They also found no correlation between citation count and perceived industrial relevance.
Sannö et al. 28 proposed a model for increasing the impact in these types of collaborations.A more detailed approach was proposed by Marijan and Gotlieb. 32Their model includes seven phases starting with problem scoping and knowledge conception to focus on industrial problems and formulate research questions.In this paper, we build on previous work to evaluate practices and gain practical insight from industry-academia collaborations in AIDOaRt.

METHOD
The contribution of this paper is based on a study conducted within the AIDOaRt project, by two parallel working groups (hereafter called WG1 and WG2) composed of several partners from academia and industry (i.e., seven universities and four companies from five countries).The general method is concerned with identifying challenges from two different perspectives.Specifically, on the one hand, we performed a bottom-up approach starting from the relationships with the CPSs stated by the use case providers that led to the identification of related challenges (conducted by WG1).On the other hand, we performed a top-down approach by identifying the challenges and aspects from the literature (conducted by WG2).Finally, the results were put together and analyzed to understand the correspondence.The adopted process is described in Figure 1.In particular, to gather the challenges we performed the following steps, as follows: 1.The WG1, composed of three universities and three companies, analyzed the reports of the development work performed by the use case providers within the AIDOaRt project (mainly, the group referred to the deliverable D5.2 AIDOaRt Integrated framework-initial version, ‡ as shown in Figure 1).To describe the point of view of AIDOaRt industries, WG1 carried out a thematic analysis 33 of such deliverable data.Specifically, WG1 extracted information regarding the relationships between use cases and CPS into a column in an online spreadsheet.The partners described these relationships in natural language by referencing the collaborations between use cases and solution providers established within the first and second AIDOaRt Internal Hackathon. 34Additionally, various data was gathered from working groups, meetings and project documents collected among the use cases about their activities in order to gather input describing the results of development activities and collaborations in a structured manner.Then, the extracted data were summarized, and qualitative analysis of these inputs has been performed using thematic analysis. 33Finally, through a series of iterations, WG1 identified several themes, where each theme was relevant to at least two partners.2. In parallel, WG2, composed of five universities and one company, considered existing secondary and tertiary studies and scientific approaches in the field of CPS development and extracted a set of characteristics/themes and challenges.Each selected paper was analyzed by two members of the group.Once the results were extracted these were iteratively refined by the group members.3. Finally, themes and challenges extracted by WG1 were compared and amalgamated with the themes and challenges identified in the literature by WG2.
Going into more detail, the Use Case CPS Relation Spreadsheet in Figure 1, extracted as part of the AIDOaRt deliverable D5.2 ‡ grew to nearly 1800 words before being exported as a summary spreadsheet file.Next, WG1 performed a series of iterations that reduced the set of themes and sub-themes extracted from the Use Case CPS Summary Spreadsheet.To support the thematic analysis and the iterative process of refining themes, WG1 implemented a Python script to identify themes with few links to partners and suggest themes that could be merged.After several iterations, the original 95 sub-themes were reduced to 27 themes.For example, sub-themes such as "heterogeneous systems of devices" and "large ‡ The concrete application of partner solutions is detailed in the openly available deliverable D5.2, available at: https://www.aidoart.eu/results/deliverables.

F I G U R E 1
A flow chart for the overall method followed in this paper.In the AIDOaRt project, partners from industry and academia collaborated on the public deliverable D5.2 which covers CPS themes from collaborations.These themes were analyzed by CPS working groups for the internal deliverable D5.6 on the use case development.This paper extracts parts of these confidential findings, enhances them with examples from Volvo CE and Westermo, and places these results in context with related work.

F I G U R E 2
Results of WG1 thematic analysis: Mapping between the preliminary CPS challenges/themes and the number of AIDOaRt use cases addressing each of these challenges/themes.SW-HW systems" were combined into the more general theme of "system-of-systems".By iterating and generating ideas for challenges, WG1 created a candidate list of 11 challenges from the 27 themes, as shown in Figure 2.This list was reviewed along with reading the results from the other WG, WG2.By revising this list and combining a top-down and bottom-up perspective, WG1 and WG2 arrived at a final list of five challenge areas, as described in Section 4. The result from the working groups was presented in the deliverable D5.6-Use Case Development Report-1 which is not public.As some of the data related to AIDOaRt is confidential, we developed a generalized view of the findings from WG1 and WG2, which is presented in this paper.In doing this we aim to share some of the findings of the project with a wider audience while keeping the sensitive information hidden.

CHALLENGE AREAS IN DEVELOPING CPS
In this section, we refer to the challenge areas for CPS development that were identified by means of the previously described method; below, they are summarized from an academic-industry perspective.Figure 3 provides an overview of how the different themes identified in the previous section from WG1 have been merged with input from WG2, creating a five challenge areas for developing CPS.
In the AIDOaRt project, there are five application dimensions; Requirements Engineering, Modeling, Coding, Testing, and Monitoring.These application dimensions correspond to the context of both WG1, but also the analysis from WG2.By extracting the challenges from industrial partner use cases, in addition to the state of the art, a merge was done as shown in the map in Figure 3.In essence, the five challenge areas consist of topics that are commonly found in both the industrial use cases but also in the state of the art for the application dimensions (in the context of CPSs).As such, some topics inevitably fall outside the intersection of WG1 and WG2, for example Reliability or Safety which are two important concerns in the CPS domain.At the same time, new topics emerge, such as Data detailed later in this Section (Sub-section 4.1).The rest of the section will discuss each identified common challenge area more in detail, and later in Section 5 we delve deeper with two use case examples.

Data
Managing and processing the huge amounts of data generated by CPSs is a major challenge for CPS engineering.With the increasing complexity of CPSs, the amount of data being generated is also increasing exponentially.This data needs to be properly managed, stored, and analyzed to extract meaningful insights from it. 35,36ata management in CPS engineering involves handling different versions of subsystems and components, which may have different data formats, storage requirements, and update schedules.This requires a systematic approach to data management, which involves defining data models, data integration strategies, and data governance policies.
In addition to data management, CPSs also require efficient data processing techniques that can handle the high volume, velocity, and variety of data generated by these systems.From the industrial practitioners' perspective, this requires the use of advanced data processing technologies such as stream processing, real-time analytics, and machine learning.
Finally, the data generated by CPSs needs to be easily accessible and interpretable by humans, especially in situations where critical decisions need to be made based on the data.This requires the development of user-friendly interfaces, data visualization techniques, and data explanation tools that can help users understand the data and its implications.Section 5.2 explores how data can act as a facilitator through one of AIDOaRt's industrial use cases.The use case regards how Continuous Integration and Continuous Deployment (CI/CD) principles can assist in CPSs design, particularly communication equipment for robust industrial communication applications.Automated testing speeds up development, and by collaborating on real industrial data, the solutions can be better tailored to the industry's needs.

Modeling and model-driven engineering
Model-driven engineering (MDE) can play a crucial role in addressing the challenges of modeling CPSs.MDE emphasizes the use of models as the primary artifact of the development process, enabling stakeholders to communicate, analyze, and refine system requirements and design decisions.
To achieve model fidelity, it is important to verify and validate models to ensure they are accurate and suitable for their intended purpose.This can involve a range of techniques, including simulation, testing, and formal verification, to assess the model's correctness and identify potential errors or inconsistencies. 11odeling and simulation also provide a means to explore uncertainty and the solution space.By creating different scenarios and testing them through simulation, stakeholders can gain a better understanding of the system's behavior under various conditions and make more informed decisions.
For industrial practitioners, finding the right level of modeling can be challenging, as over-modeling can lead to unnecessary complexity and increased development and simulation costs.It is essential to strike a balance between modeling each component to an appropriate level of detail to ensure meaningful analysis and decision-making can be performed, without becoming overwhelmed by the complexity of the CPS itself.

Requirements engineering
In the context of the AIDOaRt project, companies face challenges of variability that can occur in design time, in the product itself, or stem from the environment in which the CPS operates.These requirements can affect the performance of a CPS and may lead to strict requirements on reliability and safety.In such cases, it is relevant to monitor the performance of a CPS to ensure it operates with the expected quality of service.It is also desirable to have a test setup that matches a customer setup to make sure the CPS performs as intended in the actual operating environment.Automated ways to analyze the design of a CPS are necessary, as there are often many ways to design a CPS, leading to a potentially large number of possible configurations.This can make testing the relevant combinations challenging, as some combinations may reveal shortcomings while most do not.Additionally, CPSs may be slower due to resource constraints, which can lead to slower testing.
Model-based requirements engineering (MBRE) has been proposed as a relevant approach to overcome the challenges of traditional Requirement Engineering approaches in the context of large and complex CPSs.By relying on the intensive use of models to represent the requirements and the various system elements and their interrelations, MBRE can support the main needs of CPS design and development. 37

Continuous software and system engineering
CPSs are more challenging to evolve than conventional software, and require a tailored continuous engineering (or DevOps) process that takes into account the specific demands of testing and verification for CPSs.While DevOps practices have been effective in improving software development for conventional systems, applying them to CPSs requires overcoming unique barriers and challenges, such as the need for suitable testing and simulation techniques, as well as the complexity introduced by the combination of diverse hardware devices and software.Therefore, it is important to consider new facets when setting up a CPS development process, particularly for continuous integration and delivery pipelines. 21odels have been used extensively in the past for driving the development of complex systems at design time and as a reasoning layer for deployment, monitoring, and runtime adaptations at runtime.However, these approaches have remained mostly independent.With the emergence of DevOps principles, it is now possible to support a smooth continuum of models from design to runtime and vice versa.This can help to ensure that CPSs are designed and implemented in a more efficient and effective manner and that they can be adapted quickly and easily to changing requirements and conditions.
From the industrial perspective, testing CPSs can indeed be a complex and challenging process, given the interplay between their physical and cyber components.Automating testing on unit, integration, and system levels can help improve the efficiency and effectiveness of the testing process.This involves the use of advanced physical or virtual test systems or test benches, which can help reduce the time and effort required for testing.Several of the partners in AIDOaRt aimed at improving testing processes, for example, by automating testing on unit, integration, or system level, or by reducing test space.The testing process may be intertwined with a complex requirements process.During design time, requirements must be traced from multiple sources and on different layers of design and granularity. 38This can be a challenging process, but it is necessary to ensure that the CPS meets the intended specifications.A related challenge is that of ensuring the quality of the DevOps tools and testware used for quality assurance. 39

Intelligence and automation
The integration of intelligence into CPSs brings many exciting possibilities, such as increased efficiency, improved decision-making, and the ability to operate autonomously.However, there are also risks associated with the development and deployment of intelligent CPSs (e.g., related to safety, security and reliability).Intelligence is particularly expected to be an enabler for control, monitoring, testing, management, optimization, prediction, and security.Automation is also considered to be a key enabler for the considered CPS.
Intelligence for industrial partners is often connected with automation.In particular, it is reflected that AI/ML could assist in tackling complex processes in design or testing.Automation could be part of the considered CPS itself or a component of the workflow for any stage of development through operations.The use of automated testing can significantly reduce the time and effort required for testing, as well as help to ensure that the CPS meets the intended specifications.This can involve the automatic generation and execution of test cases, as well as the automatic reduction of large test space.Moreover, AI-automated tools can assist in generating accurate and complete requirements, and in the modeling stage, automation can help in creating formal models and verifying their correctness.
In Section 5.1 we explore collaborative efforts on this topic through one of the industrial use cases in AIDOaRt.The use case regards the design and specification of large construction machines, and the automatic transformation from legacy artifacts (e.g., Visio) to model-based representations (e.g., SysML).The model-based system descriptions can be leveraged with AI systems for additional specification capabilities, such as a recommender system.
In the next sections, some of these challenges are addressed in practice, with reference to two industrial use cases, with the aim to highlight some success stories of cross-collaboration between academia and industry within AIDOaRt.We present a summary of what our experiences have shown as particular challenges from academia and industry, with some suggestions for bridging the gap.

COLLABORATION EXPERIENCE
The AIDOaRt project brings together partners belonging to the problem domain with partners belonging to the solution domain.This means pairing teams who have expertise or experience in the area of the problem that needs to be solved and teams in the area of potential solutions to that problem.Most often the problem domain is represented by companies presenting use cases from industrial contexts (use case providers), while the solution domain corresponds to industrial or academic partners aiming to solve the challenges presented in the problem context through various methodologies and tools (solution providers and their explicit solutions and relationship to use cases are detailed in the deliverable D5.2.‡ While use case providers indicate the challenges and needs of use cases, solution providers indicate the technical solutions they make available.These inputs are collected and compared through various inter-project activities in order to enable collaborations among partners; for this purpose, one of the most successful tools are such as internal hackathons. F I G U R E 4 AIDOaRt seen through the collaborative model of Sannö et al. 28 This type of collaboration closely resembles the model described by Sannö et al., 28 from which we take inspiration to illustrate the collaborative model put into practice within AIDOaRt (see Figure 4).In particular, the project aims to bridge the gap between industry and academia by expressing common problem formulations to be jointly addressed via concrete collaborations.In turn, the joint work is expected to provide value for the academic and practitioner partners, mainly in form of research contributions (e.g., academic publications) and company improvements.Additionally, it is expected that these types of projects provide value for society, for example in the form of increased knowledge and improved technologies.
To highlight collaboration more in practical terms we discuss the experiences of Volvo Construction Equipment (Volvo CE) § and Westermo Network Technologies AB (Westermo).¶ Then, we provide our lessons learned toward common observations in industry-academia collaboration.

Volvo CE: Intelligence and automation
Volvo CE develops construction machines, such as dumpers, haulers, and excavators via various product lines.The development of these machines is a complex process and robust systems engineering is employed to tackle the many challenges related to product lines with high variability.In the AIDOaRt project, Volvo CE aims to improve the current capabilities of systems engineering with the technologies of MDE, DevOps, and AI.

Goal of collaboration and challenges
Particularly the defined use case(s) are aimed towards technical challenges in the domain of systems architecture definition and analysis at the early stages of development.A general overview of the Volvo CE use case is presented in Figure 5, where there is a need to automatically generate SysML models describing construction machines, in the figure a dumper, from non-model artifacts such as Excel and Visio with the possibility of two-way transformations.Utilizing a model-based representation the project explores how value can be added as compared to the traditional non-model methods, mainly through augmenting model-based approaches with AI/ML and DevOps.The main goal of the activities from the Volvo CE perspective can be formulated as the reduction of manual activities and increased development speed.By referring to Figure 4, the industry challenge relates to the reduction of manual activities in the Volvo CE context by employing intelligent and automated solutions to replace actions performed daily and often by engineers.These tasks might be relatively simple, for example translating table data to model elements, but they are fundamental for development.Even if the use of intelligent solutions is desirable it could be difficult to generate input data "from scratch" that is suitable for an academic tool.In this regard, the aim is to reduce the time for less critical tasks to enable engineers to focus on more essential engineering activities.However, the challenge relating to academic partners in AIDOaRt for § https://www.volvoce.com/.
automation and intelligence is to solve complex problems through novel solutions.Viewing the context of Volvo CE from an academic perspective, the novelty intuitively relates to processes related to, for example, variability, safety, and simulation through the use of the considered technologies.The collaboration requires finding the middle ground between these two perspectives, which, at a glance, see value in automating the simple contra complex tasks.While the overall goal is shared, that is, the reduction of manual activities, the expected value from these activities differs somewhat for the partners.The academic partners might be reluctant to spend effort implementing features already discussed in the literature, as it might be hard to attribute any research contribution.At the same time, the industrial partner might find it difficult to introduce emerging novel solutions in existing processes that are already quite complex and expect any improvements or large-scale adoption.

Collaboration
A concrete use case in the Volvo CE collaboration is the architectural definition of a dumper like in Figure 5.In particular, the definition of modeling patterns, enabling automation and intelligent solutions for the development of system architectures at early stages of development.At a glance, the problem can be considered an optimization problem where the academic view aims to find the optimal solution given certain constraints and conditions.However, in practical terms, this can be very difficult, and often from a practitioner's standpoint, it is more important to find a solution that is "good enough".In particular the notion of quantifying "good enough" proved a major inhibitor for the work, especially as the considered process occurs at a stage in development with a high degree of uncertainty.In the end, what proved to be the path to success was the decision to not replace the current practices but rather add to the current processes.In particular, the work emphasized interoperability with the current way of working and should the end user not forcefully introduce change, and the developed solutions were all built with this condition in mind.In this way, automation was included in the transition from legacy artifacts to models, and intelligence was added optionally.Specifically three aspects were essential to a fruitful collaboration in the Volvo CE use case.(i) Involvement from Volvo CE as a driver of the collaboration, which has enabled a useful clarity of direction and scope early.(ii) The aim of solutions has not been to replace but rather an add-on to existing processes.(iii) Alignment of expectations through shared work requiring active participation of both parties via common deliverables.
More in detail aspect (i) has included several evaluations in the industrial context with different types of stakeholders, from end users to managers, which has given broad feedback on the direction of the work.In particular, the feedback has been driven through sets of semi-structured questionnaires with scored questions.In particular, we highlight the following example questions as having been valuable feedback for the overall collaboration: 1. Is the tool-chain usage suitable in complexity? 2. Is the process easy to follow? 3. Can the approach enhance the specification of the considered system? 4. Is it easy to understand the created models (as compared to legacy artifacts)? 5. Are the models seen as useful for engineering workflow?6. Can the models accurately capture the information from Legacy artifacts?7. Are the chosen notations suitable for the considered systems?
The questions particularly asked for feedback regarding usability, relation to existing processes, expected model artifacts, and if there is perceived value in the proposed approach.These topics are a good means of driving the overall direction of collaboration towards a goal deemed valuable from a practitioner standpoint, promoting active participation via feedback.We also aimed to get a broad range of respondents to capture a holistic view of the developed solutions.Through this approach, Volvo CE has maintained a constant presence in collaboration with multiple stakeholders.Aspect (ii) in turn has consisted of the clear definition of the various input and output artifacts in the current workflow.For example the use of office tools such as Excel for defining various parts of the architecture along with Visio as a useful communication tool.Instead of replacing these tools which might be considered lacking when put in comparison to Modeling tools utilizing standards such as SysML, emphasis was put to support these existing tools instead.Aspect (iii) practically meant that the deadlines internally at the company were aligned with academic publications.Typically this could mean that the evaluation of the proposed solutions would have a double purpose of being targeted for internal evaluation as well as input for academic publications.

Results of collaboration
Through the collaboration, the main results have been the creation of a customized tool-chain for the use case.The tool-chain and associated documentation is openly accessible via a public GitHub repository.# In terms of the value defined in Figure 4 the practitioner value mainly comes from the concrete evaluation of the considered technologies in a realistic context.By integrating aspects of the considered technologies in the current way of working and performing structured workshops and evaluations, a more applicable view of the state of the art is gained for the Volvo CE context.The academic value in the collaboration instead comes from the insight into the state of practice at companies dealing with complex CPS development.The gained insights can be valuable input and motivation for scientific papers, and has enabled the evaluation of for example recommender systems, 40 and modeling languages in an industrial context. 41Furthermore, the evaluation has enabled the continued evolution of the considered tools.In fact, through direct feedback from industrial practitioners changes have been made to most of the involved tools.Examples include how to improve user-friendliness in the data representation (e.g., in the tool and not the command line), an extension of modeling tools to cover industry standards (e.g., SysML), and the inclusion of legacy tools and artifacts as tooling input (e.g., Excel and Visio).

Westermo and data
Westermo develops switches, routers, and other communication equipment for robust industrial communication applications such as on-board rail, track-side rail, power distribution etc.The software in these CPS devices is developed in an agile feature-driven process with high demands on quality, and over the years the company has invested heavily in system-level automated testing.The software development process consists of CI where each code change undergoes static code analysis and compilation, and CD in nightly automated system-level on test systems built up of network topologies with physical devices.

F I G U R E 6
In the Westermo use case in AIDOaRt, data such as test logs from the software development process is used as input to one or more AI tools in order to support monitoring and alarms.The goal is to speed up the process and increase the quality of products.

Goal of collaboration and data challenges
In the AIDOaRt project Westermo desires to continue work on continuous integration to increase the flow in the development process (e.g., by automating steps in the process), as well as increase the product quality, see Figure 6.In practice, this means that there is a desire for AI-based tools that monitor various data sources in order to raise alarms if anomalies such as suspected quality shortcomings are identified.
For the development of AI-based tools, data is essential, and real industrial data is sought after by many academics.However, there are many challenges for companies in general when sharing data.(i) Resource constraints: It might be impossible to collect some types of data due to resource constraints such as disk space, CPU shortage or timing issues on the CPS.(ii) Implementation effort: It could be impractical or time-consuming to implement data collection from a source, even if data collection would be possible.(iii) Data retention and pruning: It could also be the case that data collection is in place, but long-term retention of data is not seen as very valuable when compared to other costs such as disk space.(iv) Information security concerns: Even if data collection and long-term storage are in place, a company might be unwilling to share data with academics for legal reasons, or for fear of data leaks.

Collaborations and overcoming the challenges
With respect to (i), the challenge of data collection, resource constraints, the assumption that data collection is impossible should be challenged from time to time.Perhaps newer products, with increased resources allow for data collection.For Westermo, some new products have a 1.4 GHz processor, and some have 1 GB of RAM, whereas old products are significantly weaker.As a concrete example with respect to the (ii) on implementation effort, Westermo implemented a test results database more than a decade ago.Implementation of this database of course required resources such as IT infrastructure, disk space, and so forth, as well as skill and time for implementation, and maintenance.These are concrete costs, and with an unclear idea of benefits or return on investment, it may be unlikely that companies invest in such data collection.For Westermo, this data collection has enabled the possibility to better track SW quality and test results over time.It has also enabled the development of tools for test selection, 42 investigation of flaky tests 43 and presentations of test results. 44hallenge (iii), data retention and pruning, becomes more severe with an increasing amount of data sources.At the company, some data is put in long-term storage, other data is instead pruned after some time.In one collaboration in AIDOaRt, we recreated historic test effort allocation by interpreting stored test verdicts.This was needed because the raw data on the allocation was not retained.The work around worked rather well but resulted in data with a lower level of detail than what could have been expected if the raw data had been stored.
Westermo dealt with (iv), the challenge of information security, in two ways.First, there has been a close collaboration with some partners, with extended data sharing and regular meetings, leading to AI-based tools that might fit Westermo's needs quite closely.One example is the LogGrouper tool 45 that uses large amounts of test execution logs and the test results database to cluster failing tests where one can expect one root cause to have triggered several failing tests.For this collaboration, Westermo has created personal non-disclosure agreements with individuals from the partner organization, in addition to the AIDOaRt-level contracts.Second, another type of collaboration in AIDOaRt has been broader and used anonymized data such as a test results dataset, an anonymized subset of the test results database released on GitHub.|| For both types of collaborations, Westermo organized information security risk workshops, where risks were first identified and then mitigated, for example, by means of data minimization and data anonymization.

Results of collaboration
In the model proposed by Sannö et al., see Figure 4, the two central steps are a common formulation of a problem and common collaboration work.By sharing industry data, the process of joint understanding can be accelerated.As mentioned, this has led to both published and ongoing work with several solution providers in AIDOaRt.
From an academic perspective, scientific contributions are expected in the form of published peer-reviewed paper(s) and is commonly used as a metric for success (e.g., publications in established journals).However, for a company like Westermo, publications may be secondary to the value of prototypes, tools, knowledge or competence (previously observed by Assbring et al. 25 ).Therefore, in addition to sharing data and communicating over data, the AIDOaRt practice of conducting Hackathons was valuable for the industry-academia collaboration in the project as it also supported joint and often co-located work.Well-formulated Hackathon challenges and pre-prepared data sets could rapidly show the feasibility and value of research ideas.From the perspective of Figure 4, a Hackathon could be seen as an accelerator that makes the feedback loop spin faster.

Lessons learned
In this paper, we have reported on lessons learned from industry-academia collaborations in AIDOaRt.As previously reported, for example, by References 28,32, three lessons that seem to echo in the academic literature on industry-academia collaboration are to jointly formulate the problem, jointly work on solutions and knowledge, and an idea that collaborations could continue with the same parties after or beyond the project.In this section, we discuss our lessons learned towards these commonly identified success factors.

Jointly formulate the problem
In the paper, we have stressed that the perspectives of industry and academia differ.It is seen as a best practice to find common ground, and preferably a unified problem formulation that can be followed by both parties.However, while it is recognized as a best practice, it can be difficult to achieve practically due to the underlying differences.At a glance, it should be quite straightforward to match collaborators.However, it can often be the case that the problem formulations, often defined at a high-level, omit information that is necessary to fully understand the context.There might exist nuances of a use case that are not apparent until evaluated in the company case which in turn creates effects which are not understood until put in the wider context.Similarly, a tool developed in academic settings might not have been exhaustively tested or examined and might demonstrate unforeseen behavior once put in a, typically, harsher industrial context.
Additionally, what might be seen as a clear view of the problem at the start could turn out to be less clear than originally thought due to the differences in terminology and expectations.A scientific publication might not have value for a company, and company improvement might not be useful for a scientific publication.Similarly, from the academic view, a negative result might be very interesting while it often is not for the company, at the very least in terms of publications.It might be difficult to measure the effectiveness of the || https://github.com/westermo/test-results-dataset.collaboration in a common metric.Especially this is true for situations where there is a need to "convince" industry stakeholders of the value of the collaboration, and there might be a reluctance for involvement in case of unclear incentives.Similarly, the academic partners have the need to motivate their work through the lens of academic contributions, which might be blocked by work with little or no scientific contribution.Such work might in itself dissuade researchers, but it could also not be a guarantee that there eventually will be a clear scientific contribution, in turn further diminishing the expected value.
In our experience, data can act as a vector to speed up the formulation of problems.Support for this idea came during one of the AIDOaRt Hackathons when discussing the Westermo test results dataset.As use case providers Westermo asked "are there test cases that pass and fail together?"whereas a solution provider asked "what do you mean: If there is a high Pearson correlation coefficient or if there is a high Granger causality?Let's look at both!"This anecdote illustrates that data did speed up collaborations in this case.(More details are presented in a video from the Hackathon 46 ).Similarly, we experience that structured feedback from the industry is a good driver for continued collaboration.In the Volvo CE use case, a main inhibitor was the barrier to communication of the problems at hand.A useful tool for increasing clarity of the common problem was to show and not tell by directly involving practitioners in the considered technologies in the form of prototypes or demonstrators.In this way the discussed topics would be centered around the practical use cases at hand, and more meaningful problem formulations could be constructed to increase potential academic and industrial value in the collaboration.

Joint work on solutions
The AIDOaRt research project emphasizes the use of different mediators to bridge the gap in terms of academic and industrial problem formulations and promote active collaboration from the involved parties.A reoccurring example is through hackathons which are scheduled regularly in the project. 34In more detail, a hackathon consists of various challenges defined by the various use case partners following a semi-structured template.Each challenge should in turn be manageable during a session scheduled for a day of intensive collaboration.These hackathons act as a common place to work towards the use cases and project objectives and can strengthen existing collaborations or be a call for new collaborative efforts.Practically the types of hackathons that have been performed in the project vary from case to case, and while each concrete challenge is structured via the same format, the concrete work will vary between each particular instance.The underlying motivation and goal of these activities are to foster collaborations in the project, and highlight internally and externally some success stories. 46hile hackathons are a meaningful tool to create and strengthen collaborations, the effort spent in these activities is not by itself enough to create meaningful outcomes in projects of this scale.Indeed, it is more common for the vast majority of the work to be performed before and/or after the hackathon itself.In this way, it can be seen as a meeting and workshop for the involved partners to steer the planned or ongoing work, with direct feedback from the partners involved.A benefit of utilizing a format such as hackathon is that it becomes part of the project activities, and can enable discussions between partners who might not be collaborating directly in addition to creating and maintaining strong collaborative efforts towards commonly defined problems.
In our experience, another good practice to promote joint work is to have regular planned meetings focused on the industry use case progression.For example, in the Volvo CE collaboration, weekly meetings are held with all involved partners to discuss the progression of the use case.In this way, the goals of the use case are continuously discussed and used to motivate the other activities.Regular contact between partners could be seen as a pre-condition for activating collaboration, the aim and execution of these meetings are also important.Without having the use case in focus there is a risk of the collaboration drifting from the original intent and becoming locked on goals that are less related to the project's goals, and for example, more towards a specific solution or idea.So while the original problem formulation might have been a good match, there is additionally a need to regularly keep the collaboration on a path that is expected to provide value for both the academic and industrial partners.

Continuation of collaboration past project
Collaborative projects are a good means of identifying and matching potential partners from industry and academia.However, in practical terms, there are also some limitations that could hinder an effective collaborative environment.
One aspect is that the nature of collaborative projects, particularly research projects, might have dissemination plans that do not match the intended collaboration activities.Similarly, the allocated time of the project and expected activities limits the scope of collaboration.The academic expectation might be that company input is timely, concrete, and sufficient to apply their ideas and tooling directly.Company expectations might be that academic partners will adapt their ideas and tools created for another purpose for the specific use case at hand.Aligning these expectations, even when successful, take a considerable amount of time.Additionally, collaborations between industry and academia are rarely static and the scope might move during the time of collaboration.Indeed, adapting to the unforeseen needs that arise efficiently is often necessary to reach a meaningful outcome that satisfies all the involved parties.However, adapting to a moving scope poses several challenges in itself, from a company perspective it might involve lengthy processes to share data and sign contracts such as Non-disclosure agreements.On an academic level, a moving scoping might instead miss the original research intent or perhaps change the target contribution.
From the perspective of data, we would like to point out that an industrial dataset released to the general public, and not only shared within a research project, is an artifact that could be valuable for many future researchers.In and of itself, the dataset could trigger research that, from an industry perspective is realistic and therefore more valuable for the company that released it, and for society in general.In addition, the partner that released the dataset can use it to get a rapid start in follow-up projects-collaborations may start as soon as the research project start (or even before) and there would be no lead time or processing time for creating a new dataset.
Considering the nature of these collaborative environments it is therefore seen as a good practice to discuss and promote collaboration past and beyond the project.However, as mentioned in previous sections, it is also important to remember why there is a collaboration and to have a plan for extended collaboration.Indeed, the expected value of these collaborations needs to be considered, especially if considered outside an existing project scope which has already in place the plan, scope and resources.In this regard again we note that a clear focus on a use case, and the problem at hand is critical to enable a collaboration which also includes a path for future collaboration.By identifying clear problem formulations that through collaboration can be jointly described potential paths for extended collaboration can be more readily formulated, motivated, and planned for.

DISCUSSION
The presented work discusses what our experiences and suggestions for future endeavors in collaboration between industry and academia when developing complex CPS.As mentioned in Section 3 the challenges that are defined in the paper originate from the combined view from the industrial practitioners and the existing scientific literature.Originally the number of potential themes in the challenges from the observed use cases were many, and subsequently they were merged until a stable view was defined by the working groups.Similarly the scientific literature was searched and summarized into re-occurring challenges, resulting in the challenges presented in the paper.The effort of identifying these challenges and merging the different views has been a iterative process.We note that the particular challenges are all related to the AIDOaRt key domains of MDE, AI, and DevOPS, and in particular the intersection of these technologies.
Three lessons learned are to jointly formulate the problem, joint work on solutions, and a continuation of the collaboration past the project, summarized in Table 1.In our experience, joint problem formulation is a strong foundation for successful and meaningful collaboration.Particularly, aligning partners on expected value of collaboration is a good practice, as often the traditional value streams is different for industry and academia.Through the AIDOaRt project we also highlight how hackathons can be a very valuable effort to practically bridge this gap.In this case, the activities themselves should be consolidated to compact events, that can act as a kick-start for collaboration or "check-off" for some developing solution.Similarly, the use of industry data helps promote realistic solutions, and can re-enforce a sense of trust among partners.In a similar fashion, jointly working on the solutions is often a pre-requisite to keep the collaboration interests aligned over time, due to changes in scope, requirements, or target dissemination.Particularly, we believe that making structured feedback between partners is a good facilitator for concurrent work, and can in the best case be used for dissemination purposes as well.Finally, having a long term collaboration as the ultimate goal will enable a stronger foundation for trust and adaptability, as typical restraints of These can bridge the gap between industry and academia, and rapidly show the feasibility of research questions for example, "are there test cases that pass and fail together?".Hackathons can also promote the "show not tell" principle by rapid-prototyping through intensive shared work with involved partners.
1.3: Share data Sharing data can speed up industry-academia collaboration.However, industry partners may find this difficult due to resource constraints, implementation effort, data retention and pruning, as well as for information security concerns.Sometimes simply having a small set of "real" data in the beginning of a collaboration can foster trust and pave the way for gradual introduction of more data.

2: Joint work on solution
In addition to hackathons, structured feedback and sharing data, the project could hold regular meetings with all involved partners for discussing the progression of the use case.These gives all partners part of progress and get motivated, and limits the risk of drift of focus from the original problem formulation while promoting shared responsibility of he work.

2.1: Structured feedback
Feedback from industry partners to academic partners could be collected in a structured way, for example, triggered by show and not only tell.Having structured feedback in terms of surveys and extensive workshops are good practices which can directly highlight areas for improvement in the collaboration.Particularly, such feedback could extract valuable insights from a wider range of users and stakeholders, which might not be directly involved in the collaboration.

3: Continue past project
Collaborations that work towards long term goals past project boundaries can foster trust in addition to more freedom in adaptation.Additionally, companies might be more willing to invest resources in collaborative efforts that will stand on their own in contrast to typical research projects.
research projects are expected to eventually be alleviated (for example a shift in focus from one type of research area to another).The two use cases presented in the paper are relatively different from one another.They differ in technologies, target, and number of collaborators.We believe this gives a more holistic view on the topics discussed in the paper as the differences can complement one another, and therefore provide a broader view.The Volvo CE use case can be considered relatively wide, and weekly collaborative efforts is conducted with between five to seven partners consecutively.As such it can be considered as a relatively large investment from the side of the industrial partner, and one of the enabler in this regard is the active supervision of researchers from the company.Sometimes in the literature this is referred to having a "champion" in a company, and has been identified as a best practice in collaboration. 9In this way, the company gains value in the form of having a researcher producing knowledge that is applied directly.
This paper aims to, in part, disseminate data published in confidential deliverables in a research project.In this paper, we have highlighted collaboration and emphasized the different types of value that can be obtained through industry-academia collaboration.However, apart from the impact on the practitioners and researchers, this type of collaboration is expected to also impact a more general audience, for example, as seen in the model by Sannö et al. 28 Therefore, we believe that disseminating information that is typically hidden, perhaps due to it partly being sensitive company information, via more open publications is a way to increase the general value of a research project.However we do not want to dissuade company participation by giving the feeling that we will publish all of their data.Therefore we have tried via this publication to extract what we believe to be the essence of the information concerning a few selected topics to make the information more readily available.In particular, by involving two use cases directly and providing direct insights, we provide a view of collaboration in practice.Similarly, using thematic analysis and connection with scientific literature for the perceived challenges is a means of anonymizing the concrete data.While this process cannot fully capture the obtained knowledge, we believe it captures a large part and makes it available for a wider audience while maintaining the required confidentiality.
All research has limitations.This paper reports from experiences from the use cases in one research project focusing on themes and challenges identified by the working groups.One could argue that a limitation of the study is the lack of security-related issues.While we agree that this topic plays a significant role both from an industry and academic perspective for CPS, these topics were not at the core of the data extracted from the use cases of AIDOaRt project.An interested reader could turn to Humayed et al. 47 for an extensive literature review on security and CPSs and Lun et al. 48for a state of the art of research in CPS security considering an automatic control perspective.The topic of knowledge 25 (mentioned in section 2.1), could also have been raised as a challenge in a research project such as this one.Anecdotally, one could expect this to be related to talent acquisition, which has caused delays for some partners in the project.A third limitation could be the lack of hardware-related challenges, with the exception of the challenge of collecting data from resource constrained hardware devices in section 5.2.1.Another potential limitation of this study, is the topic of how generalizable our findings are outside of the AIDOaRt context.These limitations are of course real.However, despite these limitations, we argue that findings from industry-academia collaborations are relevant.One way to mitigate these limitations would be for more studies from other ongoing and future research projects.

CONCLUSION
In this paper we discuss the view from industry-academia collaboration through the experience of the AIDOaRt research project.The challenges extracted as part of the AIDOaRt project originate from a broad range of use cases and contexts.As such we believe that the consequent findings and discussion could be useful for a wider audience with the scope of developing complex CPSs.Since there has been a plethora of previous reporting on the challenges of developing CPSs, this paper has focused on a somewhat more narrow scope on the industry-academia collaboration.Indeed the AIDOaRt project aims to foster the collaboration between industry and academia so that the beneficial outcomes can be reached by applying research in real use cases.To this regard the paper can assist future endeavors in similar contexts.In particular, we report that alignment of expectations, and that sharing industrial data can be an enabler for collaborations.This paper has presented five challenge areas for developing complex CPS extracted from the AIDOaRt research project industrial use cases in conjunction with state of the art extracted from the literature: data, modeling, requirements engineering, continuous software and system engineering, as well as intelligence and automation.Furthermore, we have presented the lessons learned of industry-academia collaboration in a large European research project.To make the discussion more explicit two use cases have been presented and observed regarding this particular topic, namely Volvo Construction Equipment and Westermo Network Technologies AB.By discussing the two use cases we highlight how effective collaboration can be fostered and what are good practical practices observed in this regard, such as sharing industrial data and performing joint practical work.Based on our experiences we discuss the lessons learned and some recommendations for future collaboration that might help future collaborative ventures.
Future work could detail further advancements from the project in the last stages of performing the project.The use cases presented, along with the ones not detailed, are progressing continuously through the project and are expected to deliver more concrete outputs as part of the final deliverables.Another line of research could explore best practices (e.g., anonymization approaches) to simplify the transition of companies from sharing data internally in projects to sharing dataset publicly.

F I G U R E 3
Overview of the merge of WG1 and WG2 into five distinct challenge areas.
Lessons learned.Due to the different stakeholders in academia and industry it is common to work towards closely related but in the end separate goals.Effort should be spent on identifying dissemination strategies that can fulfill the needs of all involved parties, preferably without disjoint parallel activities.
TA B L E 1