Current trends in the car industry mean that developing software projects has become an increasingly complex task. Software teams and project managers require specialized error search and process optimization frameworks in complex and heterogeneous software systems.

In this Q&A with Torsten Mosis and Sebastian König of Elektrobit Automotive, the pair discuss mechanisms for detecting costly bugs in complex automotive software supply chains before they disrupt development projects.

Embedded Computing Design: What factors are contributing to the increasing complexity of automotive software development projects, and who is ultimately responsible for ensuring software quality amidst the complex automotive supply chain?

König: Today, a wide range of software technologies can already be found in the Head Units. They are typically based on standard systems such as HTML5, Java or Qt and the applications and services on the level below them exploit the advantages of established frameworks such as Android, QNX Car Platform or GENIVI Linux.

In the interim, functions that can be experienced by the user are supplemented by smartphone apps and the integration of mirroring methods such as Android Auto or Apple CarPlay. Added to these are the relocation and processing of large data quantities from the vehicle to the cloud. Market developments show that in the future, several different operating systems that are distributed across different hardware entities or run in a virtualized environment on a hypervisor will often run at the same time. Here, a complex feature such as a navigation system generally penetrates all software and hardware layers.

While the basic integration typically is done by the OEM itself, several suppliers usually contribute to the overall function to varying degrees. For developers and programmers, major coordination efforts are therefore required in order to test and guarantee the quality and robustness of the software system overall. Alongside the tried and tested standard systems and technologies, numerous proprietary extensions also need to be taken into account.

Embedded Computing Design: Bugs found earlier in the software development lifecycle are far less costly than those identified in later stages of production. What mechanisms are available to developers that simplify the detection of errors before they negatively impact production?

Mosis: Certain software development tools enable engineers to identify and localize functional and non-functional errors in early development phases. A typical example from everyday project work involving the development of a navigation system is that the language instructions for a turn-off maneuver are frequently announced too early or too late. The potential error sources for this vary widely and can often only be reproduced in certain situations. Often, the cause of the error does not lie in a single software module itself, but in the interaction between the modules, since their interfaces are frequently used incorrectly, e.g. in the wrong sequence, with the wrong values or at the wrong point in time. Rectifying this type of error is costly, since often, a large number of developers from different suppliers are involved in searching for the error and reproducing it.

EB solys, for example, is a tool for highly complex car development projects that supports error searches and process optimization not only in individual components, but at a higher level in the system structure overall. The focus here is on gathering, aggregating and correlating the data and operating states of a software system being investigated. Often, errors and anomalies can only be detected when the data from different sources are set in relation to each other. In order to gather the data, a target agent will be installed on the system to be monitored, which Elektrobit will provide as open source software. Here, the monitoring on the target system is passive. The analysis observes the data processing in progress and the inter-process communication, and is based not only on the access to instrumented source codes.

The target agent supports a plug-in architecture in order to enable access to specific inter-process communication or developer traces on the target system (Figure 1). By contrast, the data is aggregated and correlated on the host system, which runs on a Windows PC. In a similar way to the target agent, the architecture of the host system is also designed to allow itself to be adapted easily, for example, to specific data formats and to different inter-process communication procedures. Individually adapted importers can provide data to the core system from any log files required in order to convert specific data content such as binary traces into structure text formats.

Figure 1. The inter-process communication architecture of EB solys.

Many detail functions have their origins in the development practice at Elektrobit, and thus address specific needs of programmers, integrators and system developers. For example, visualized data always remains interlinked in different ways - if the user sets a marker in a graphic image, for example, this mark is automatically adopted and displayed in all linked charts and tables as well. As required, the analyses and depictions can be conducted at higher abstraction levels than for functions or processes, or at lower levels such as interfaces, services or objects. (Figure 2)

Figure 2. Different abstraction levels enable analyzing specific functions, processes and services within EB solys.

Currently, this agent is available for Linux, QNX, Android and Windows Embedded target systems, with others to follow.

Embedded Computing Design: What about automated monitoring and validation? This would seem critical to system analysis and error finding given the growing amount of software in vehicles.

König: Typically, in later development phases, the system analysis and error search is followed by continuous monitoring and validation of critical KPIs and processes. For this reason, the “EB solys Auto” version supports operation in batch mode, enabling its functions to be integrated in automated test environments. These also include methods for recording and visualizing KPIs such as the use of system resources, the performance of individual partial systems and hotspots and the communication between components and processes.

In this way, developers can assess the health and stability of the system, for example, or test whether certain specifications and development standards have been met, monitor performance metrics and detect trends in order to define measures at an early stage. As a result, the development processes are improved, in particular among dispersed development teams. Project managers can give their teams feedback early on and identify targeted measures in order to improve quality and stability.

An in-built script language based on the Xtend programming language designed for Java enables function extensions without the need for adaptations to the host system or to the source code of the target agent. The scripts and the gathered data can be accessed and new operating elements generated via a programming interface. Furthermore, the script API also provides methods for recording and storing KPIs in the Auto version. In order to display KPIs over a longer period of time, EB solys uses the interfaces of the technology pairing of InfluxDB (as a metrics database) and Grafana (as a dashboard).

As a development tool aimed primarily at the auto industry, EB solys also supports the correlation of log data with geodata. This allows it to, for example, show system events in combination with a map display and, if required, recorded camera images.

Even though it is clearly rooted in the automotive market, the open architecture generally also allows it to be adapted to other markets or development environments.

Torsten Mosis studied information technology at Mannheim University of Applied Science (FH) and successfully completed his diploma thesis at Siemens VDO in Regensburg. There, he was a software developer in the vehicle navigation systems division. In 2007, he joined Harman International and supervised the reorganization of the navigation software as a software architect. He has been working in the Software Integration Systems domain at Elektrobit in Munich since 2014, and is responsible as Product Manager for the development of EB solys.

Sebastian König is a consultant and Product Owner for EB solys. He completed his studies in media and information technology at Hof University of Applied Sciences in 2008. Following this, he began to work as a software developer and project manager in the Navigation Systems domain at Elektrobit. Between 2012 and 2015, Mr König pursued other projects including the independent development of his own business idea. In 2015, he returned to Elektrobit and has been supporting the EB solys product development team in Munich.