The FAIR data principles provide technical guidelines to enable the Findability, Accessibility, Interoperability, and Reusability (FAIR) of research data. The main focus is on the machine-actionability of these aspects, i.e. the technical capability of performing these operations in an automatized way, with minimal human intervention.

Metadata. Metadata and metadata models play a major role in this process, and should contain:

• Global and persistent identifiers to datasets
• A number of attributes providing descriptive information about the context, quality, condition, and characteristics of the data
• Metadata attributes should be linked to controlled, shared vocabularies (or ontologies).

Identifiers and ontologies. To enable machine-actionability, the metadata needs to be indexed in a searchable resource and made retrievable via the identifiers using a standardized communication protocol. Moreover, a high level of standardization is required to achieve semantic interoperability allowing, e.g., for integration of different datasets. Linking metadata fields to ontologies provides context to the dataset as a self-describing information bundle where the links to ontologies provide the foundation to machine interpretation, inference, and logic.

Track data and FAIR principles. The degree to which deposited track data comply to the FAIR principles vary greatly, from near-perfect FAIRification practices in the context of certain consortia to the almost complete lack of metadata linked to track files in a range of smaller projects. Some common issues are listed in Figure 1.2. One of the major weaknesses is the lack of suitable uniform metadata schemas that can work across track collections. The lack of uniform metadata strongly limits the possibility of reusing or repurposing track data and hinders automation of these processes, especially when it comes to the "long arm" of deposited track data files. Furthermore, the lack of provenance information might introduce artefacts in the analyses. This lack of proper annotations and of a well-defined and universally adopted metadata standard is related to the lack of a central repository for track data, as described in the section Track collections.

The ambitions of the FAIRtracks project are two-fold:

• Provide a set of pragmatic metadata schemas for genomic tracks that comply with the FAIR principles and are adopted and further developed by the community as a minimal metadata exchange standard, providing a unified view into both:

  • novel track data depositions
  • legacy track collections/data portals.

• Provide a set of services to be integrated with downstream tools and libraries so that analytical end user can more easily discover and reuse from the massive amounts of track data that has been and is being created:

  • for various species
  • from different types of sample material
  • by applying diverse types of experiment assays and in silico processing workflows.

See Figure 1.3 for an overview of the secondary data life cycle that falls within the scope of the FAIRtracks project.

FAIR principle F1 stipulates the need to assign identifiers to data and metadata that are both 1) globally unique and 2) persistent. The
GO FAIR website page on the topic further asserts:

Principle F1 is arguably the most important because it will be hard to achieve other aspects of FAIR without globally unique and persistent identifiers.

Globally unique and persistent identifiers allows the linking of research data with different aspects of the research environment (Figure 3.1).

The need for track file identifiers: Track data files are seldom assigned identifiers directly; often it is only the raw sequence files used to generate the track files that are assigned identifiers, typically the accession numbers to data repositories such as the Sequence Read Archive (SRA) or the European Genome-Phenome Archive (EGA). The ENCODE project represents a notable exception: each track is associated with an identifier resolvable through Identifiers.org (see Figure 3.2) and a dedicated web page. Furthermore, a universally accessible service to assign and register identifiers to single track files and collections is currently missing.

Track files contain condensed data from bioinformatics workflows and are thus dependent on specific parameter settings and cannot be perfectly recreated from the raw without also perfectly reproducing the full analysis workflow, which is often a difficult task. We therefore strongly recommend the implementation of a track registry that preserves the full context of tracks by assigning global identifiers not only to the track data files but also to the associated metadata.

Built for track file identifiers: The FAIRtracks draft standard is developed from the ground up to support globally unique and persistent identifiers for track files and could be suitable for use as a basis for a potential global registry of track metadata (see Figure 3.3). For now, global track identifiers are allowed, but not enforced by the FAIRtracks standard. Instead, we require the inclusion of local track identifiers within the dataset as well as Uniform Resource Locators (URLs) to track files, which, unfortunately, come without any guarantees of persistence or uniqueness. The FAIRtracks standard still provides globally unique and persistent identifiers to track files in an indirect manner, using document DOIs.

DOI as document identifier: In case a direct identifier is not attached to a track file, the identifier of a parent record (e.g. study or experiment) can be used instead. On top of this, FAIRtracks requires a global identifier for the metadata file itself using a document identifier (DOI). In principle, a track file can thus be uniquely pinpointed by a joint identifier containing the DOI of the FAIRtracks document and the locally unique track identifier. As our policy requires support for DOI versioning and DOI reservation prior to publication, we currently recommend Zenodo for publishing FAIRtracks documents. We would extend our list of recommended repositories and archives to any domain-specific services meeting our requirements.

"Mix-tape" track collection identifiers: Apart from the main use case of annotating primary track collections deposited in some repository, FAIRtracks is designed to also allow a more novel use case: to annotate secondary "mixtape" track collections of track files originating from different primary sources. The main example of this use case is to annotate the exact track data files used to generate the findings of a scientific publication, whether these track files represent novel data, are directly reused from other repositories, are regenerated from the raw data or in other ways derived from the original track files. To allow the provenance of such "mixtape" reuse of tracks, assigning globally unique and persistent identifiers to track collections would be advantageous. Full support for secondary track collections is scheduled for version 2 of the FAIRtracks standard (coming soon). Currently, this concept is most fully developed in the form of GSuite files in the context of the GSuite HyperBrowser.

References to external records: FAIRtracks supports and recommends the inclusion of global identifiers to external records containing detailed metadata. We require these global identifiers represented in Compact Uniform Resource Identifies (CURIE) form resolvable through Identifiers.org (see Figures 3.2 and 3.3). A mapping service from existing URIs to the corresponding CURIEs is desirable, as it would enhance the conversion of existing metadata to the FAIRtracks standard.

We developed the FAIRtracks metadata standard and the associated ecosystem of services with the aim of fulfilling two main objectives:

• To facilitate deposition of novel track data and unification of metadata from legacy track collections in compliance with the FAIR data principles.

• Improving downstream reuse of track data through services for data discovery and retrieval applied to unified and FAIR metadata.

Metadata for data providers vs end users: In the initial development phase we quickly discovered that these two objectives are somewhat at odds with each other. Downstream users will typically prefer strict, homogeneous, and human-readable metadata. Data providers, on the other hand, will typically prefer streamlined deposition, variation in metadata content, and machine-operable metadata.

Human-readable labels for ontology terms? This gap can be illustrated in the case of ontology terms. If data providers were to only provide the machine-operable identifiers for the terms, human users will not be able to directly understand them. Furthermore, it would be cumbersome for downstream developers and analytical end users to have to implement ontology lookup functionality in order to make use of the metadata. On the other hand, forcing data providers to provide both identifiers and human-readable labels for all ontology terms would discourage data deposition.

Bridging the gap: Based on the above considerations and similar examples, we discovered a need for intermediate solutions that provide ontology lookup functionality and other FAIR-oriented features to help bridge this gap:

Minimal and augmented versions of FAIRtracks: Data providers that adopt the FAIRtracks metadata standard only need to fill the minimal set of fields that are marked as "required", which together constitutes the minimal version of the FAIRtracks standard (see Figure 5.1). The FAIRtracks augmentation service is implemented as a REST API that takes a minimal FAIRtracks-compliant JSON document as input, adds a set of fields with human-readable values to the document, and provides this augmented FAIRtracks-compliant JSON document as output. The fields added include human-readable ontology labels, ontology versions, human-readable summary fields, and other pragmatic fields useful for interactions with the end user (see Figure 5.2). This service simplifies the job on the data providers side and, at the same time, improves the quality of the data discovery and retrieval operations on the users' side in both automated and manual usage scenarios.

FAIR by design, FAIRification and Data brokering: There is a distinction between research data that has been made FAIR by design at the outset and research data that require a process of retroactive FAIRification (see Figure 7.1). In the FAIR-by-design case, metadata can be entered more or less directly according to the FAIR metadata schema, after which a validator can be applied to locate eventual errors. Retroactive FAIRification will, on the other hand, typically first require a process of transformation, cleaning, and mapping of the existing metadata from its previous shape into a new shape that follows a more FAIR-compliant schema. A similar process of metadata transformation and mapping happens in the process of data brokering, where metadata entered according to one metadata schema is automatically transformed to follow another schema and submitted to a data deposition service. In both cases, a validator will only be able to check the end result of a transformation process, it will not help much with the actual work of getting there.

"Parse, don't validate": A complementary approach to applying a validator at the end is to uphold the schema restrictions through a set of data parsers in the transformation process itself. In contrast to validators, parsers will typically allow a level of variation and noise in the input data, but still make sure that the output data follow the relevant restrictions. This alternative approach to data wrangling can be summed up in the slogan "Parse, don't validate". See Technical note #2 (to the side) for a more in-depth look at this concept.

Validators as the authority: Parsing-based approaches are powerful, but still, in our view , complementary to traditional validators. As one often needs to set up several transformation processes, there is still the need of a single validator to be the authoritative judge in case of disagreements. Also, as mentioned above, parsers are not applicable to all scenarios.

FAIRtracks is a secondary metadata standard: Genomic track files are most often created as secondary by-products derived from the raw experiment data (see e.g. Finding Tracks). Hence, the choice of primary metadata schemas are typically dictated by circumstances relating to the raw data, e.g. to follow schemas required by research data management (RDM) or data deposition services (see Figure 1.3 in the section FAIR data and FAIRtracks above for an illustration of the secondary data life cycle in scope of the FAIRtracks project). For projects that are part of larger consortia undertakings, required metadata schemas are typically defined centrally (see Track collections). In both cases, the FAIRtracks standard is a secondary metadata exchange standard to that aims to facilitate interoperability and reuse of the data through a central indexing service and downstream software integrations (see Finding Tracks).

We are developing Omnipy, a systematic and scalable approach to research data transformation and "wrangling" in Python

The importance of a good solution for metadata transformation: Being a secondary metadata standard, it is imperative for community adoption of FAIRtracks that we provide good solutions for transforming metadata from its primary form into the FAIRtracks standard. To this end, we are developing Omnipy, a systematic and scalable approach to research data transformation and "wrangling" in Python.

Generic functionality: Omnipy is designed primarily to simplify development and deployment of (meta)data transformation processes in the context of FAIRification and data brokering efforts. However, the functionality is very generic and can also be used to support research data (and metadata) transformations in a range of fields and contexts beyond life science, including day-to-day research scenarios (see Figure 7.2).

Data wrangling in day-to-day research: Researchers in life science and other data-centric fields often need to extract, manipulate and integrate data and/or metadata from different sources, such as repositories, databases or flat files. Much research time is spent on trivial and not-so-trivial details of such "data wrangling":

• reformat data structures
• clean up errors
• remove duplicate data
• map and integrate dataset fields
• etc.

General software for data wrangling and analysis, such as Pandas, R or Frictionless, are useful, but researchers still regularly end up with hard-to-reuse scripts, often with manual steps.

Step-wise data model transformations: With the Omnipy Python package, researchers can import (meta)data in almost any shape or form: nested JSON; tabular (relational) data; binary streams; or other data structures. Through a step-by-step process, data is continuously parsed and reshaped according to a series of data model transformations.

Omnipy tasks (single steps) and flows (workflows) are defined as transformations from specific input data models to specific output data models.

"Parse, don't validate": Omnipy follows the type-driven design principles introduced in Technical note #2: "Parse, don't validate" (to the side). It makes use of cutting-edge Python type hints and the popular pydantic package to "pour" data into precisely defined data models that can range from very general (e.g. "any kind of JSON data", "any kind of tabular data", etc.) to very specific (e.g. "follow the FAIRtracks JSON Schema for track files with the extra restriction of only allowing BigBED files").

Data types as contracts: Omnipy tasks (single steps) or flows (workflows) are defined as transformations from specific input data models to specific output data models. pydantic-based parsing guarantees that the input and output data always follows the data models (i.e. data types). Thus, the data models defines "contracts" that simplifies reuse of tasks and flows in a mix-and-match fashion.

Catalog of common processing steps: Omnipy is built from the ground up to be modular. As also exemplified in Figure 7.3, we aim to provide a catalog of commonly useful functionality ranging from:

• data import from REST API endpoints, common flat file formats, database dumps, etc.
• flattening of complex, nested JSON structures
• standardization of relational tabular data (i.e. removing redundancy)
• mapping of tabular data between schemas
• lookup and mapping of ontology terms
• semi-automatic data cleaning (through e.g. Open Refine)
• support for common data manipulation software and libraries, such as Pandas, R, Frictionless, etc.

In particular, we will provide a FAIRtracks module that contains data models and processing steps to transform metadata to follow the FAIRtracks standard.

Refine and apply templates: A Omnipy module typically consists of a set of generic task and flow templates with related data models, (de)serializers, and utility functions. The user can then pick task and flow templates from this extensible, modular catalog, further refine them in the context of a custom, use case-specific flow, and apply them to the desired compute engine to carry out the transformations needed to wrangle data into the required shape.

Rerun only when needed: When piecing together a custom flow in Omnipy, the user has persistent access to the state of the data at every step of the process. Persistent intermediate data allows for caching of tasks based on the input data and parameters. Hence, if the input data and parameters of a task does not change between runs, the task is not rerun. This is particularly useful for importing from REST API endpoints, as a flow can be continuously rerun without taxing the remote server; data import will only be carried out in the initial iteration or when the REST API signals that the data has changed.

Scale up with external compute resources: In the case of large datasets, the researcher can set up a flow based on a representative sample of the full dataset, in a size that is suited for running locally on, say, a laptop. Once the flow has produced the correct output on the sample data, the operation can be seamlessly scaled up to the full dataset and sent off in software containers to run on external compute resources, using e.g. Kubernetes (see Figure 7.4). Such offloaded flows can be easily monitored using a web GUI.

Industry-standard ETL backbone: Offloading of flows to external compute resources is provided by the integration of Omnipy with a workflow engine based on the Prefect Python package. Prefect is an industry-leading platform for dataflow automation and orchestration that brings a series of powerful features to Omnipy:

• Predefined integrations with a range of compute infrastructure solutions (see Figure 7.5)
• Predefined integration with various services to support extraction, transformation, and loading (ETL) of data and metadata
• Code as workflow ("If Python can write it, Prefect can run it")
• Dynamic workflows: no predefined Direct Acyclic Graphs (DAGs) needed!
• Command line and web GUI-based visibility and control of jobs
• Trigger jobs from external events such as GitHub commits, file uploads, etc.
• Define continuously running workflows that still respond to external events
• Run tasks concurrently through support for asynchronous tasks

Pluggable workflow engines: It is also possible to integrate Omnipy with other workflow backends by implementing new workflow engine plugins. This is relatively easy to do, as the core architecture of Omnipy allows the user to easily switch the workflow engine at runtime. Omnipy supports both traditional DAG-based and the more avant-garde code-based definition of flows. Two workflow engines are currently supported: local and prefect.