Quality annotation

According to Article 2 (2 aa) of the EHDS Regulation, a « 'data quality and utility label' means a graphic diagram, including a scale, describing the data quality and conditions of use of a dataset;».

 
 
In this context, the EU Data Portal has provided a general approach for the HealthDCAT-AP draft specification. The EU Data Portal defines a quality framework for checking metadata against various indicators, termed "Metadata Quality Assurance". This quality assurance is based on five dimensions: Findability, Accessibility, Interoperability, Reusability, and Contextuality. These dimensions are practical implementations of the FAIR data principles, and derived from the computation of selected DCAT-AP properties.

For instance, the Reusability dimension is evaluated by analysing information provided for "Licence information," "Licence vocabulary," "Access restrictions," "Access restrictions vocabulary," "Contact point," and "Publisher". An average score, termed "Rating evolution," is generated, and a visualisation output is available for each dimension:
Figure 10: Example of the Reusability graphic diagram of the data.europa.eu portal
The metadata quality evaluation in the EU Data Portal aids data providers and portals in improving their metadata. It also serves as a crucial filter for data users, enhancing their search experience.

Article 59 of the EHDS Regulation mandates that Health Data Access Bodies (HDABs) produce a biennial activity report. This report must include the '(l) number of data quality labels issued by data holders' for their data portal, 'disaggregated by quality category'. The expected quality categories for the reporting are listed in the Article 78 "Data quality and utility label". For paragraphs (a), (c), (d), (e), and (f), DCAT-AP and the new properties of HealthDCAT-AP provide the necessary metrics for HDABs to generate data quality labels from metadata (Metadata Quality Dimensions according to the EU Data Portal). This alignment is because part of the requirements for the HealthDCAT design are directly derived from the FAIR data principles and the EHDS Regulation.
EHDS Regulation Article 78) Data quality and utility label
The data quality and utility label shall cover the following elements, where applicable:

  • (a) for data documentation: metadata, support documentation, the data dictionary, the format and standards used, the source of the data and, where applicable, the data model;
  • (b) for assessment of technical quality: completeness, uniqueness, accuracy, validity, timeliness and consistency of the data;
  • (c) for data quality management processes: the level of maturity of the data quality management processes, including review and audit processes, and bias examination;
  • (d) for assessment of coverage: the period, population coverage and, where applicable, representativity of the population sampled, and the average timeframe in which a natural person appears in a dataset;
  • (e) for information on access and provision: the time between the collection of the electronic health data and their addition to the dataset and the time needed to provide electronic health data following the issuing of a data permit or a health data request approval;
  • (f) for information on data modifications: merging and adding data to an existing dataset, including links with other datasets.
EHDS Regulation Article 60) Duties of health data holders
4. Where a data quality and utility label accompanies the dataset pursuant to Article 78, the health data holder shall provide sufficient documentation to the health data access body for that body to verify the accuracy of the label.
The missing category, where no metrics are available from metadata, is paragraph (b) 'for assessment of technical quality: completeness, uniqueness, accuracy, validity, timeliness and consistency of the data;'. Moreover, user feedback, which provides valuable insights for assessing the quality and utility ranking of the dataset, is not covered by Article 78.

To include information on any applicable quality and utility labels for a dataset, HealthDCAT-AP utilises the dqv:hasQualityAnnotation property. This property, sourced from the Data Quality Vocabulary (DQV), is an RDF vocabulary developed by the W3C's Data on the Web Working Group.

Property

Quality annotation

URI

dqv:hasQualityAnnotation

Range

oa:Annotation

Definition

A statement related to quality of the Dataset, including rating, quality certificate, feedback that can be associated to datasets or distributions.

Comment

The information may include quality aspects such as accuracy, reliability, comparability, coherence, relevance, timeliness etc.

Usage note

The annotation requires the provision of information about the motivation of the annotation (oa:motivation), and an explicit link to the resource being annotated (oa:hasTarget) together with either a link to a resource that contains the annotation (oa:hasBody) or text filed (oa:bodyText).
The approach undertaken by HealthDCAT-AP is also aligned with the StatDCAT Application Profile and the MobilityDCAT-AP published in April 2024.
Usage note for quality description in MobilityDCAT-AP
This property MAY describe any quality aspects regarding the delivered content, in particular methods, metrics/indicators and results of a quality assessment in the responsibility of the Rights Holder (see property dct:rightsHolder). This information SHOULD assist data consumers in determining the value of data, before actually accessing and processing it. Thus, the information SHOULD be publicly available. Furthermore, it can be helpful for validation processes by 3rd parties, e.g., a National Body in context of EU Delegated Regulations.
To summarise, HealthDCAT-AP enables health data holders to provide the necessary information to generate the quality labels defined by the EHDS Regulation. However, how to assess technical quality still needs to be determined. And methods for collecting user feedback also need to be streamlined. Health Data Access Bodies are responsible for producing and reporting these labels by category, as outlined in the EHDS Regulation. The process for generating overall quality and utility labels according to a common framework also remains to be defined. This represents a broader challenge for the EU Data Spaces as they work to develop a comprehensive quality framework. A standardised approach to describe the quality of datasets across EU data spaces should be adopted. HealthDCAT-AP recommends using the Data Quality Vocabulary annotation property (dqv:hasQualityAnnotation) to provide information on quality, aligning with the approach taken by StatDCAT-AP and MobilityDCAT-AP.

Here's an example of how to include a quality certificate in a HealthDCAT-AP metadata record using the property dqv:hasQualityAnnotation. The oa:hasBody property points to an HTTP URI that serves as a direct link to the quality certificate, enabling users to access the certificate's full details and verify the dataset's quality standards.

dqv:hasQualityAnnotation [

a dqv:QualityCertificate ;

oa:hasTarget //...mydataset> ;

oa:hasBody //acertificateserver.eu/mycertificate> ;

oa:motivatedBy dqv:qualityAssessment];


For example, as illustrated in the vocabulary DQV examples (vocab-dqv-examples):

 

<https://acertificateserver.eu/mycertificate>

dqv:hasQualityMeasurement[a dqv:QualityMeasurement ;

dqv:isMeasurementOf :

[a dqv:Metric ; skos:definition "Ratio between the number of objects represented in the csv and the number of  objects expected to be represented according to the declared dataset scope."@en ;

dqv:expectedDataType xsd:double ;

dqv:inDimension [ a dqv:Dimension ;

skos:prefLabel "Completeness"@en ;

skos:definition "Completeness refers to the degree to which all required information is present in a particular dataset."@en ;

dqv:inCategory :intrinsicDimensions    

];

];

dqv:value "0.5"^^xsd:double];

...

 
A quality certificate relying on the Data Quality Vocabulary (DQV) can encompass a range of detailed information, creating a comprehensive assessment of data quality. This may include:

  • a quality label,
  • a utility label,
  • specific quality dimensions (such as accuracy and completeness),
  • associated metrics,
  • and user feedback.
Real-world example:
The dataset 'European Quality of Life Time Series, 2007 and 2011' has achieved Platinum (Expert level) certification from the Open Data Institute (ODI).

The Open Data Institute facilitates an auto-certification process by retrieving DCAT metadata records. Although the certificate is not available in RDF format (and thus lacks the RDF-based DQV vocabulary structure), it is still accessible in a machine-readable JSON format (see example below). The quality certificate generated by the ODI service can be incorporated into HealthDCAT-AP using the dqv:hasQualityAnnotation property as a dqv:QualityCertificate.

The service implemented by ODI offers significant advantages, enabling any data holder to automatically generate a data quality certificate based on their dataset description. Data holders can enrich the certificate by adding additional information, such as technical quality metrics and user feedback reported, for instance, by the Health Data Access Body (HDAB). The HDAB can then use the generated certificate to create a graphical representation, as outlined in Article 2 (ae) of the EHDS Regulation, by accessing its machine-readable format. This flexibility allows the HDAB to customise the appearance of the graphic in its data catalogue. Furthermore, HealthDCAT-AP's approach supports data holders in subscribing to various quality and utility frameworks, enhancing the adaptability and interoperability of their datasets. These frameworks could be designed by the HDAB, a Member State, the EHDS, the common EU Data Spaces, or any relevant organisation (e.g., EOSC).
Example of an ODI dataset quality certificate at the 'Platinum' level.