In engineering sciences, the handling of digital research data plays an increasingly important role in all fields of application (Sandfeld et al. 2018). This is especially the case, due to the growing amount of data obtained from experiments and simulations (Hey & Trefethen 2003). The extraction of knowledge from these data is referred to as a data-driven, fourth paradigm of science, filed under the keyword data science (Hey 2009). This is particularly true in materials science, as the research and understanding of new materials are becoming more and more complex (Hill et al. 2016). Without suitable analysis methods, the ever-growing amount of data will no longer be manageable. In order to be able to perform appropriate data analyses smoothly, the structured storage of research data and associated metadata is an important aspect. Specifically, a uniform research data management is needed, which is made possible by appropriate infrastructures such as research data repositories. In addition to uniform data storage, such systems can help to overcome inter-institutional hurdles in data exchange, compare theoretical and experimental data and provide reproducible workflows for data analysis. Furthermore, linking the data with persistent identifiers enables other researchers to directly reference them in their work.

In particular, repositories for the storage and internal or public exchange of research data are becoming more and more widespread. Especially the publication of such data, either on its own or as a supplement to a text publication, is increasingly encouraged or sometimes even required (Naughton & Kernohan 2016). In order to find a suitable repository, services such as re3data (Pampel et al. 2013) or FairSharing (The FAIRsharing Community et al. 2019) are available. These services also make it possible to find subject-specific repositories for materials science data. Two well-known examples are the Materials Project (Jain et al. 2013) and the NOMAD Repository (Drax & Scheffler 2018). Indexed repositories are usually hosted centrally or institutionally, and are mostly used for the publication of data. However, some of the underlying systems can also be installed by the user, such as for internal use within individual research groups. Additionally, this allows full control over stored data as well as internal data exchanges, if this function is not already part of the repository. In this respect, open-source systems are particularly important, as this means independence from vendors and opens up the possibility of modifying the existing functionality or adding additional features, sometimes via built-in plug-in systems. Examples of such systems are Ckan (CKAN Association 2014), Dataverse (King 2007), DSpace (Smith et al. 2003) or Invenio (CERN 2016), where the latter is the basis of Zenodo (CERN & OpenAIRE 2013). The listed repositories are all generic and represent only a selection of the existing open-source systems (Amorim et al. 2017).

A second type of system in addition to the repositories, which is also increasingly used in experimentally oriented research areas, are the electronic lab notebooks (ELN) (Rubacha, Rattan & Hosselet 2011). Nowadays, the functionality of ELNs goes far beyond the simple replacement of paper-based lab notebooks, and can also include aspects such as data analysis, as seen, for example, in Galaxy (Afgan et al. 2018) or Jupyter Notebooks (Kluyver et al. 2016). Both systems focus primarily on providing accessible and reproducible computational research. Specifically, the boundary between unstructured and structured data is more and more blurred, the latter being traditionally only found in laboratory information management systems (LIMS) (Bird, Willoughby & Frey 2013; Elliott 2009; Taylor 2006). Most existing ELNs are domain-specific and limited to research disciplines such as biology or chemistry (Taylor 2006). According to current knowledge, a system specifically tailored to materials science does not exist. For ELNs, there are also open-source systems such as eLabFTW (CARPi, Minges & Piel 2017), SciNote (SciNote LLC 2015) or Chemotion (Tremouilhac et al. 2017). Compared to the repositories, however, the selection of ELNs is smaller. Furthermore, only the first two mentioned systems are generic.

Thus, generic research data systems and software are available for both ELNs and repositories, which, in principle, could also be used in materials science. The listed open-source solutions are of particular relevance, as they can be adapted to different needs and are generally suitable for use in a custom installation within single research groups. However, both aspects can be a considerable hurdle, especially for smaller groups. Due to a lack of resources, a structured research data management and the possibility of making data available for subsequent use is therefore particularly difficult for such groups (P. Bryan Heidorn 2008). What is finally missing is a system that can be deployed and used both centrally and decentrally, as well as internally and publicly, without major obstacles. The system should support researchers throughout the entire research process, starting with the generation and extraction of raw data, up to the structured storage, exchange and analysis of the data, resulting in the final publication of the corresponding results. In this way, the features of the ELN and the repository are combined, creating a virtual research environment (Carusi & Reimer 2010) that accelerates the generation of innovations by facilitating the collaboration between researchers. In an interdisciplinary field like materials sciences, there is a special need to model the very heterogeneous workflows of the researchers (Hill et al. 2016).

For this purpose, the research data infrastructure Kadi4Mat (Karlsruhe Data Infrastructure for Materials Sciences) is being developed at the Institute for Applied Materials (IAM-CMS) of the Karlsruhe Institute of Technology (KIT). The current logo of the project is shown in Figure 1. The aim of the software is to combine the possibility of structured data storage with documented and reproducible workflows for data analysis and visualization tasks, incorporating new concepts with established technologies and existing solutions. In the development of the software, the FAIR principles (Wilkinson et al. 2016) for scientific data management are taken into account. Instances of the data infrastructure have already been deployed and show how structured data storage and data exchange are made possible (Brandt 2020). Furthermore, the source code of the project is publicly available under a permissive Apache 2.0 license (Brandt et al. 2020).

Kadi4Mat is logically divided into the two components ELN and repository, which have access to various tools and technical infrastructures. The components can be used by web- and desktop-based applications, via uniform interfaces. Both a graphical and a programmatic interface are provided, using machine-readable formats and various exchange protocols. In Figure 2, a conceptual overview of the infrastructure of Kadi4Mat is presented.

Kadi4Mat is built as a web-based application that employs a classic client-server architecture. A graphical front end is provided to be used with a normal web browser as a client, while the server is responsible for the handling of the back end and the integration of external systems. A high-level overview of the implementation is shown in Figure 4. The front end is based on the classic web technologies JavaScript, HTML and CSS. In particular, the client-side JavaScript web framework Vue.js (Vue Core Development Team 2014) is used. The framework is especially suitable for the creation of single-page web applications (SPA), but can also be used for individual sections of more classic applications, to incrementally add complex and dynamic user interface components to certain pages. Vue.js is mainly used for the latter, the benefit being a clear separation between the data and the presentation layer, as well as the easier re-use of user interface components. This aspect is combined with server-side rendering. Due to the technologies and standards employed, the use of the front end is currently limited to recent versions of modern web browsers such as Firefox, Chrome or Edge.

In the back end, the framework Flask (The Pallets Projects 2015) is used for the web component. The framework is implemented in Python and is compatible with the common web server gateway interface (WSGI), which specifies an interface between web servers and Python applications. As a so-called microframework, the functionality of Flask itself is limited to the basic features. This means that most of the functionality, which is unrelated to the web component, has to be added by custom code or suitable libraries. At the same time, more freedom is offered in the concrete choice of technologies. This is in direct contrast to web frameworks such as Django (DSF 2005), which already provides a lot of functionality from scratch. The web component itself is responsible for handling client requests for specific endpoints and assigning them to the appropriate Python functions. Currently, either HTML or JSON is returned, depending on the endpoint. The latter is used as part of an HTTP API, to enable an internal and external programmatic data exchange. This API is based on the representational state transfer (REST) paradigm (Fielding & Taylor 2000). Support for other exchange formats could also be relevant in the future, particularly for implementing certain exchange formats for interoperability, such as OAI-PMH (OAI 2015). Especially for handling larger amounts of data, other exchange protocols besides HTTP are considered.

A large part of the application consists of the core functionality, which is divided into different modules, as shown in Figure 4. This structure is mainly of an organizational nature. A microservice architecture is currently not implemented. Modules that access external components are particularly noteworthy, which is an aspect that will also be increasingly important in the future. External components can either run on the same hardware as Kadi4Mat itself or on separate systems available via a network interface. For the storage of metadata, the persistence module makes use of the relational database management system PostgreSQL (PostgreSQL Global Development Group 1996), while the regular file system stores the actual data. Additionally, the software Elasticsearch (Elastic NV 2010) is used to index all the metadata that needs to be efficiently searchable. The aforementioned process manager (Zschumme 2021b), which is currently implemented as a command line application, manages the execution of workflows by delegating each execution task to an available process engine (Zschumme 2021a). While the current implementation of the process engine primarily uses the local file system of the machine on which it is running, users can add steps to synchronize data with the repository to their workflow at will. To increase performance with multiple parallel requests for workflow execution, the requests can be distributed to process engines running on additional servers. By wrapping the process manager with a simple HTTP API, for example, its interface can easily be used over a network. A message broker is used to decouple longer running or periodically executed background tasks from the rest of the application, by delegating them to one or more background worker processes. Apart from using locally managed user accounts or an LDAP system for authentication, Shibboleth (Cantor & Seavo 2005) can be used as well. From a technical point of view, Shibboleth is not a single system, but the interaction of several components, which together enable a distributed authentication procedure. Depending on the type of authentication, user attributes or group affiliations can also be used for authorization purposes in the future.

Another component shown in Figure 4 are the plugins. These can be used to customize or extend the basic functionality of certain procedures or actions, without having to modify or know the corresponding implementation in detail. Unlike the tools in a workflow, plugins make use of predefined hooks to add their custom functionality. While such a plugin has to be installed centrally by the system administrator for all users of a Kadi4Mat instance, the possibilities are also evaluated to be able to make use of individual plugins on the user level.

The current functionalities of Kadi4Mat can either be utilised via the graphical user interface, with a browser, or via the HTTP API, with a suitable client. On top of the API, a Python library is developed, which makes it especially easy to interact with the different functionalities (Schoof & Brandt 2020). Besides using the library in Python code, it offers a command line interface, enabling the integration with other programming or scripting languages.

In the following, the most important features of Kadi4Mat are explained, based on its graphical user interface. The focus of the features implemented so far is on the repository component, the topics of structured data management and data exchange in particular, as well as on the workflows, which are a central part of the ELN’s functionality. After logging in to Kadi4Mat, it is possible to create different types of resources. The most important type of resource are the so-called records, which can link arbitrary data with descriptive metadata and serve as basic components that can be used in workflows and future data publications. In principle, a record can be used for all kinds of data, including data from simulations or experiments, and it can be linked to other records of related data sets, e.g. to the descriptions of the software and hardware devices used. The metadata of a record includes both basic metadata, such as title or description, and domain-specific metadata, which can be specified generically, in the form of key/value pairs. The latter can be defined using a special editor, as shown in Figure 5. With the help of such metadata, a description of subject- and application-specific records becomes possible. This is particularly relevant in an interdisciplinary research field such as materials science, where using a fixed schema would be impracticable, due to the heterogeneity of the data formats and the corresponding metadata. The value of each metadata entry can be of different types, such as simple character strings or numeric types like integers and floating point numbers. Numeric values can also be provided with an arbitrary unit. Furthermore, nested types can be used to represent metadata structures of almost any complexity, for example in the form of lists. The input of such structures can be simplified by templates, which are specified in advance and can be combined as desired. While templates currently offer the same possibilities as the actual metadata, it is planned to add further validation functionalities, such as the specification of a selection of valid values for certain metadata keys. Wherever possible, automatically recorded metadata is also available in each record, such as the creator of the record or the creation date. The actual data of the record can be uploaded by the users and are currently stored on a file system, accessible by Kadi4Mat. It is possible to upload any number of files for each record. This can be helpful when dealing with a series of several hundred images of a simulated microstructure, for example, which all share the same metadata.

The created record can be viewed on an overview page that displays all metadata and linked files. Some common file formats include a preview that is directly integrated into the web browser, such as image files, PDFs, archives or textual data. Furthermore, the access rights of the record are displayed on its overview page. Currently, two levels of visibility can be set when creating a record: public and private visibility. While public records can be viewed by every logged-in user, i.e. read rights are granted implicitly to each user, private records can initially only be viewed by their creator. Only the creator of a record can perform further actions, such as editing the metadata or deleting the entire record. Figure 6 shows the overview page of a record, including its metadata and the menu to perform the previously mentioned actions. In order to grant different access rights to other users, even within private records, different roles can be defined for any user in a separate view. Currently, the roles are static, which means that they can be selected from a predefined list and are each linked to the corresponding fine-grained permissions. Because of these permissions, the possibility of custom roles or certain actions being linked to different user attributes becomes possible. In addition to roles for individual users, roles can also be defined for groups. These are simple groupings of several users which, similar to records, can be created and managed by the users themselves. The same roles that can be defined for individual users can be assigned to groups as well. Each member of the group is granted the corresponding access rights automatically. Finally, the overview page of a record also shows the resources linked to it. This refers in particular to the so-called collections. Collections represent simple, logical groupings of multiple records and can thus contribute to a better organization of resources. In terms of an ontology, collections can be regarded as classes, while records inside a collection represent concrete instances of such a class. Like records and groups, collections can be created and managed by users. Records can also be linked to other records. Each record link represents a separate resource, which in turn can contain certain metadata. The ability to specify generic metadata and such resource links already enables a basic ontology-like structure. This structure can be further improved in the future, e.g. by using different types of links, with varying semantics, and by allowing collections to be nested.

To be able to find different resources efficiently, especially records, a search function is included in Kadi4Mat. This allows searching in the basic metadata of resources and in the generic metadata of records via keywords or full text search. The values of nested generic metadata entries are flattened before they are indexed in Elasticsearch. This way, a common search mapping can be defined for all kinds of generic metadata. The search results can be sorted and filtered in various ways, for example, by using different user-defined tags or data formats, in the case of records. Figure 7 shows an example search of records, with the corresponding results.

While the execution of workflows, via the web interfaces, and the ability to add user-defined tools are still under development, it is possible to define a workflow using a graphical node editor, running in the web browser. Figure 8 shows a simple example workflow created with this editor. A selection of predefined nodes can be combined and parameterized, while the resulting workflow can be downloaded. A custom JSON-based format is currently used to store the representation of a workflow. This format contains all the information for the node editor to correctly display the workflow and to derive a functional workflow representation for execution. The downloaded workflow file can be executed directly on a local workstation by using the command line interface of the process manager. All tools to actually run such a workflow need to be installed beforehand. A selection of tools is provided for various tasks (Zschumme et al. 2020), including connecting to a Kadi4Mat instance by using a suitable wrapper on top of the aforementioned API library. Several common use cases have already been implemented, including the task of extracting metadata from Excel spreadsheets, often used to replace an actual ELN system, and importing it into Kadi4Mat. An overview of such a workflow is shown in Figure 9. Developments are also underway for data science applications, especially in the field of machine learning. The combination of the ELN and the repository fits particularly well with the requirements of such applications, which typically require lots of high-quality input data to function well.

The development and current functionality of the research data infrastructure Kadi4Mat is presented. The objective of this infrastructure is to combine the features of an ELN and a repository in such a way that researchers can be supported throughout the whole research process. The ongoing development aims at covering the heterogeneous use cases of materials science disciplines. For this purpose, flexible metadata schemas, workflows and tools are especially important, as is the use of custom installations and instances. The basic functionality of the repository component is largely given by the features already implemented and can be used with a graphical as well as a programmatic interface. This includes, above all, uploading, managing and exchanging data as well as the associated metadata. The latter can be defined with a flexible metadata editor to accommodate the needs of different users and workgroups. A search functionality enables the efficient retrieval of the data. The essential infrastructure for workflows is implemented as a central part of the ELN component. Simple workflows can be defined with an initial version of the web-based node editor and executed locally using provided tools and the process manager’s command line interface. Both main components are improved continuously. Various other features that have not yet been mentioned as part of the concept are planned or are already in the conception stage. These include the optional connection of several Kadi4Mat instances, a more direct, low-level access to data and the integration of an app store, for the central administration of tools and plugins.

The development of Kadi4Mat largely follows a bottom-up approach. Instead of developing concepts in advance, to cover as many use cases as possible, a basic technical infrastructure is established first. On this basis, further steps are evaluated in exchange with interested users and by implementing best practice examples. Due to the heterogeneous nature of materials science, most features are kept very generic. The concrete structuring of the data storage, the metadata and the workflows is largely left to the users. As a positive side effect, an extension of the research data infrastructure to other disciplines is possible in the future.