Compliance

Quality Management System

varvis® is developed under the EN ISO 13845 certified quality management system. The standard ensures that the medical device consistently meets customer expectations of quality, safety, and performance.


Versioning Scheme

Version numbers of the software including its sub-modules follow semantic versioning, i.e. a numeric schema major.minor.patch and the following semantics:

  • Major versions reflect substantial changes *; i.e. changes that have impact in terms of the intended purpose of the device or the device performance and/or safety

  • Minor versions reflect non-substantial changes in the features and functionality of the software

  • Patch versions reflect bug fixes and maintenance releases

* 'Substantial changes' refer to paragraph 2.4, Annex IX, IVDR


Safety and performance information

In the following, performance and safety information is provided for the entire device.

Highlights of analytical performance

The analytical performance evaluation demonstrates the device's adherence to state-of-the-art NGS requirements through several core technical strengths:

  • Breadth of detection: The device is evaluated for both small-scale DNA changes (SNVs/Indels) and large-scale structural variations (CNVs, STRs, SVs) across WGS, WES, and NGS panels. This ensures comprehensive coverage for complex genetic conditions.

  • Platform and framework versatility: Performance is maintained across both short-read (high-throughput) and long-read (high-resolution) technologies. Support for both GRCh37 and hg38 ensures continuity with legacy data while remaining aligned with current genomic standards.

  • High-accuracy germline & somatic analysis: The device achieves high detection rates across the whole genome and exome. In somatic contexts, the integration of Unique Molecular Identifiers (UMIs) enables a refined Limit of Detection (LoD), allowing for the reliable detection of rare sub-clonal mutations and circulating tumor DNA (ctDNA) by suppressing technical noise and sequencing artifacts.

  • MLPA-equivalent CNV detection: The device demonstrates high analytical performance in short-read germline CNV discovery, supporting its role as a sensitive alternative to MLPA for first-line copy number diagnostics.

  • In silico CNV analysis: CNV performance was evaluated using both MLPA-confirmed samples and a systematic in silico approach. While results were acceptably high for both methodologies, the in silico approach allows for a large-scale evaluation of both duplications and deletions across diverse genomic coordinates, ensuring statistically robust assessment of CNV calling accuracy.

  • Long-read optimization: For STR and SV discovery, the device delivers performance comparable to international (GiaB) benchmarks and is consistently superior to legacy tools designed for short-read data.

  • Analytical stability: The device shows consistent variant calling performance within and between runs across variant types and genomic contexts.

  • Conservative performance reporting: By systematically reporting the minimum observed performance and strictly accounting for no calls (positions failing quality thresholds), the evaluation ensures that the reported metrics are derived solely from high-confidence data, providing a dependable performance baseline.

Overview of analytical performance evaluation: scope and methodology

To ensure the accurate and reliable detection of genetic variants, the medical device underwent a comprehensive analytical evaluation. The assessment covered a broad spectrum of genomic variants, ranging from small DNA changes to complex structural variations, across multiple genomic contexts.  

The evaluation utilized a combination of three primary data types to ensure technical robustness: 

  • Public reference data: Benchmark datasets, such as Genome in a Bottle (GiaB), were used to establish baseline accuracy against internationally recognized high-confidence standards.

  • Real-world customer data: Samples from clinical/laboratory settings were used to confirm performance across diverse sequencing runs, with ground truth confirmed via orthogonal validation (e.g., Sanger sequencing, MLPA (Multiplex Ligation-dependent Probe Amplification), and FLA (Fragment Length Analysis).

  • In silico data: Synthetic / artificial data were utilized to test against computationally generated / established ground truth in genomic settings where physical sample data are limited.

All performance testing adhered to international standards for Next-Generation Sequencing (NGS) data processing. The results confirm that the device consistently achieves state-of-the-art accuracy and aligns with global benchmark performance data, for both legacy (GRCh37) and modern (hg38) genomic reference frameworks, across various types of complex genetic data. While analytical performance is equivalent across builds, the use of hg38 is recommended to enhance mappability and increase the number of samples acceptable for analysis. The performance of NGS-based tests depends on laboratory factors such as library prep, sequencing depth, and coverage uniformity. It is recommended that each laboratory performs an assay-specific validation, according to applicable guidelines, using established reference materials in conjunction with the device.
 

Broad scope of performance evaluation

The following table outlines the specific variant types and genomic contexts assessed during the performance evaluation. This comprehensive scope ensures the device’s variant calling performance is accurate, reliable, and robust across diverse applications and sequencing technologies. 

Variant type

Application
(Analysis type)

Sequencing method

Analyte
(Assay scope)

Reference genome

SNVs & Indels

(Single Nucleotide Variants, Insertions, Deletions)

Germline & Somatic

Short-read &
Long-read

WGS, WES, Targeted Panels

hg38, GRCh37

CNVs

(Copy Number Variations)

Germline

Short-read

WGS, WES, Targeted Panels

hg38, GRCh37

 

STRs

(Short Tandem Repeats)

Germline

Long-read

WGS

hg38

SVs

(Structural Variants)

Germline

Long-read

WGS

hg38

Technical confusion matrix for categorizing device’s calls 

To quantify accuracy, the device’s findings were compared against a defined ground truth – the gold-standard reference that represents the actual genetic state of the sample. Performance is measured primarily within truth regions, which are genomic areas where the sequence is known with high certainty. 

The evaluation utilized three primary sources to establish ground truth:

  • Public consensus benchmarks: Highly characterized reference datasets, such as Genome in a Bottle (GiaB) consortium call sets.

  • Orthogonal confirmation: Truth sets validated through independent laboratory methods.

  • In silico data: Computationally generated datasets used to establish ground truth for variants where physical reference materials are limited or unavailable.

These ground truth sources allow every call made by the device to be categorized into a technical confusion matrix:
 

 

Ground truth
Reference Positive

Ground truth
Reference Negative

Total

Call Positive

TP (True Positive)

FP (False Positive)

TP+FP

Call Negative

FN (False Negative)

TN (True Negative)

FN+TN

No/Invalid Call

E

F

E+F

Total

TP+FN+E

FP+TN+F

N

Definitions:

  • True Positives (TP): Variants correctly identified by the device.

  • True Negatives (TN): Genomic positions correctly identified as wild-type (no variant).

  • False Negatives (FN): Variants present in the ground truth that the device missed.

  • False Positives (FP): Wild-type positions incorrectly identified as variants.

No calls (E, F): Positions where the device could not confidently assign a genotype because quality thresholds were not met.
 

Performance metrics & definitions

Raw counts from the technical confusion matrix are used to calculate standardized performance metrics in alignment with the 2018 FDA final guidance (1,2) and established guidelines from professional associations and scientific societies, including ACMG, ESHG, GfH, and CLSI (3-7). These metrics quantify the device’s accuracy and stability in detecting genetic variants compared to established international benchmarks, ensuring results meet the stringent thresholds required for NGS analysis.
 

Metric

Alias

Formula

Importance

Positive Percent Agreement
(PPA)

Sensitivity

TP/(TP+FN)

Measures the ability to detect real variants within defined truth regions.

Technical Positive Predictive Value
(TPPV)

Precision

TP/(TP+FP)

Measures confidence that a Positive call is a true variant.

Limit of Detection
(LoD)

Limit

-

For somatic analysis, defines the lowest variant allele frequency (VAF) or concentration at which variants are consistently and reliably detected by the device.

Repeatability

Within-run

Coefficient of variation (CoV)

Measures deterministic consistency in variant calling performance when processing the same raw data multiple times under identical conditions.

Reproducibility

Between-run

Coefficient of variation (CoV)

Measures robustness in variant calling performance when processing replicates under varying conditions, e.g., different runs, days, or operators.

Acceptance criteria

To ensure full transparency and demonstrate adherence to state-of-the-art requirements, each result table presented in this report displays the relevant acceptance criteria alongside the observed performance. This format allows for a direct comparison.
 

Summary of analytical performance results

The following tables summarize the device’s analytical performance for calling germline and somatic SNVs and Indels using both reference genomes hg38 and GRCh37. These results represent the minimum performance observed across all applicable performance tests, ensuring that the reported values reflect a conservative estimate of the device's variant calling performance.

Statistical conventions

To provide a conservative and reliable assessment, the following statistical conventions are applied:

  • Point estimates (point est.): Represent the lowest performance measured across all applicable performance tests, ensuring reported values reflect a conservative baseline.

  • Lower bound of the 95% confidence interval (95% LCL): Demonstrates statistical reliability, calculated with the Wilson Score method (8).

  • Acceptance criterion (acc. criterion): Represents the predefined threshold established by the manufacturer in alignment with state-of-the-art NGS standards.

Performance is continuously assessed against these criteria using both the metrics’ point estimates and 95% LCLs. The analytical performance results consistently meet or exceed the predefined acceptance criteria across all evaluated categories, demonstrating the device’s accurate, reliable, and robust variant calling performance across the full spectrum of intended use cases / evaluated analyses.
 

Germline analysis

Analyte
(Assay scope)

Sequencing method

Variant type

Ground truth

PPA (Sensitivity) [%]

TPPV (Precision) [%]

Acc. criterion

Point est.
[95% LCL]

Acc. criterion

Point est.
[95% LCL]

srWGS

Illumina compatible

SNV

GiaB

>=97.0

99.3 [99.3]

>=83.0

99.2 [99.2]

 

Illumina compatible

Indel

GiaB

>=83.0

96.8 [96.8]

>=56.0

95.4 [95.4]

 

Illumina compatible

CNV

CNV

MLPA

in silico

>=97.0

>=97.0

100.0 [99.0]

98.9 [98.6]

>=80.0

>=80.0

100.0 [99.0]

87.7 [87.0]

lrWGS

PacBio

ONT SUP

SNV

SNV

GiaB

GiaB

>=97.0

>=97.0

99.9 [99.9]

99.9 [99.9]

>=83.0

>=83.0

99.9 [99.9]

99.9 [99.9]

 

PacBio

ONT SUP

Indel

Indel

GiaB

GiaB

>=83.0

>=83.0

98.9 [98.9]

90.5 [90.4]

>=56.0

>=56.0

99.0 [99.0]

94.1 [94.0]

 

PacBio

ONT SUP

STR

STR

GiaB

GiaB

>=85.0

>=85.0

96.5 [96.4]

92.9 [91.5]

>=80.0

>=80.0

94.3 [94.1]

94.3 [93.0]

 

PacBio

ONT SUP

SV

SV

GiaB

GiaB

>=86.0

>=86.0

95.8 [95.5]

92.7 [92.3]

>=85.0

>=85.0

99.0 [98.9]

98.8 [98.6]

srWES

Illumina compatible

SNV

GiaB

>=97.0

97.9 [97.8]

>=83.0

98.7 [98.6]

 

Illumina compatible

Indel

GiaB

>=83.0

88.8 [87.9]

>=56.0

82.9 [81.9]

 

Illumina compatible

CNV

CNV

MLPA

in silico

>=94.0

>=94.0

100.0 [93.0]

99.7 [98.9]

>=80.0

>=80.0

96.2 [87.2]

96.5 [94.8]

srPanel

Illumina compatible

SNV

GiaB

>=90.0

91.6 [90.3]

>=83.0

99.1 [98.5]

 

Illumina compatible

Indel

GiaB

>=45.0

71.9 [64.0]

>=45.0

68.0 [60.1]

 

Illumina compatible

CNV

CNV

CNV

ICR96 [9]

MLPA

in silico

>=94.0

>=94.0

>=94.0

98.4 [96.0]

100.0 [97.4]

100.0 [99.5]

>=80.0

>=80.0

>=80.0

81.7 [77.0]

100.0 [97.4]

99.6 [98.8]

lrPanel

PacBio

SNV

GiaB

>=90

98.4 [97.7]

>=83.0

99.1 [98.5]

 

PacBio

Indel

GiaB

>=45.0

93.0 [89.7]

>=45.0

92.7 [89.9]

Somatic analysis

Following representative results utilize Unique Molecular Identifiers (UMIs), where specified, which allow the device to track original DNA fragments and suppress technical noise at low allele frequencies (VAF). Samples were sequenced on Illumina sequencers. For somatic SNV/Indel analysis, GiaB benchmark datasets served as ground truth.
 

Sample type (Analyte)

Method / library prep

Variant type

LoD (%)

PPA (Sensitivity) [%]

TPPV (Precision) [%]

Acc. criterion

Point est.
[95% LCL]

Acc. criterion

Point est.
[95% LCL]

srWGS

w/o UMI

SNV

15.0

>=95.0

97.7 [97.7]

>=95.0

99.3 [99.2]

 

w/o UMI

Indel

15.0

>=75.0

79.0 [78.6]

>=90.0

93.8 [93.5]

srPanel

w/o UMI

SNV

8.0

>=90.0

100.0 [99.5]

>=90.0

100.0 [99.5]

 

w/o UMI

Indel

8.0

>=75.0

91.1 [83.4]

>=80.0

82.0 [73.3]

 

w/ UMI

SNV

0.25

>=90.0

99.1 [96.7]

>=90.0

93.3 [82.1]

 

w/ UMI

Indel

0.25

>=75.0

97.7 [94.8]

>=80.0

93.3 [82.1]

Repeatability & reproducibility analysis

The table below summarizes the technical stability of the device’s variant calling performance, quantified by the Coefficient of Variation (CoV) of the PPA and TPPV. These results demonstrate consistent performance across both germline and somatic contexts, utilizing aggregated data from short-read and long-read sequencing platforms.

To ensure technical robustness, the CoV is reported as a maximum threshold; all observed performance data fell below these predefined limits across varying wet-lab (library preparation) and dry-lab (computational environment) conditions. The use of CoV as a summary metric allows for an effective and transparent assessment of stability across the multiple genomic contexts and variant types evaluated.
 

Variant type

Metric

Repeatability
(Within-run)

Reproducibility
(Between-run)

SNV/Indel

CoV

< 0.01

< 0.05

CNV

CoV

< 0.01

< 0.15

STR

CoV

< 0.01

< 0.02

SV

CoV

< 0.01

< 0.02

HPO Similarity Score 

The analytical performance of the HPOSimScore, a ranking technique designed to assist in NGS-based variant detection analysis by prioritizing candidate genes, was evaluated using 8 150 anonymized patient cases alongside a large-scale simulation experiment. The HPOSimScore reduces the vast pool of detected genomic variants by comparing the phenotypic abnormalities of a patient against the HPO terms associated with every HPO-project gene, assigning a similarity score to each patient-gene pair to bring the True Causal Gene (TCG) to light. Comparative benchmarking demonstrates that the HPOSimScore outperforms the standard baseline (HPOMatchScore) by placing the known True Causal Gene (TCG) more frequently on top ranks, particularly under sub-optimal or challenging conditions where less than 50% of the patient terms are directly associated with the TCG (representing approximately 55% of real-world cases). When processing three to five HPO terms per patient, the software ranks the TCG within the top 50 in 31% of cases overall, rising to a peak sensitivity of 82% under optimal data conditions (where 100% of the patient’s HPO terms are associated with the TCG, representing approximately 20% of real-world cases).
 

References 

[1] FDA, “Considerations for Design, Development, and Analytical Validation of Next Generation Sequencing (NGS) - Based In Vitro Diagnostics (IVDs) Intended to Aid in the Diagnosis of Suspected Germline Diseases”, U.S. Department of Health and Human Services, Food and Drug Administration, Center for Devices and Radiological Health, Center for Biologics Evaluation and Research, April 13, 2018. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/considerations-design-development-and-analytical-validation-next-generation-sequencing-ngs-based.

[2] F. Luh and Y. Yen, “FDA guidance for next generation sequencing-based testing: balancing regulation and innovation in precision medicine”, NPJ Genom Med. 2018 Oct 3, 3:28. doi: 10.1038/s41525-018-0067-2.

[3] C. Rehder et al., “Next-generation sequencing for constitutional variants in the clinical laboratory, 2021 revision: a technical standard of the American College of Medical Genetics and Genomics (ACMG)”, Genet Med 23, 1399–1415 (2021). doi: 10.1038/s41436-021-01139-4.

[4] E. Souche et al., “Recommendations for whole genome sequencing in diagnostics for rare diseases”, Eur J Hum Genet 30, 1017-1021 (2022). doi: 10.1038/s41431-022-01113-x.

[5] P. Bauer, “S1 Leitlinie: Molekulargenetische Diagnostik mit Hochdurchsatz-Verfahren der Keimbahn, beispielsweise mit Next-Generation Sequencing”, medgen 30, 278–292 (2018). doi: 10.1007/s11825-018-0189-z.

[6] G. Matthijs et al., “Guidelines for diagnostic next-generation sequencing”, Eur J Hum Genet 24, 2-5 (2016). doi: 10.1038/ejhg.2015.226.

[7] CLSI, “EP17-A2: Evaluation of Detection Capability for Clinical Laboratory Measurement Procedures: Approved Guideline–Second Edition.” CLSI document EP17-A2. Wayne, PA: Clinical and Laboratory Standards Institute, 2012.

[8] R. G. Newcombe, “Improved confidence intervals for the difference between binomial proportions based on paired data”, Statist. Med. 17, 2635-2650 (1998). doi: 10.1002/(SICI)1097-0258(19981130)17:22<2635::AID-SIM954>3.0.CO;2-C.

[9] S. Mahamdallie et al., “The ICR96 exon CNV validation series: a resource for orthogonal assessment of exon CNV calling in NGS data”, Wellcome Open Res. 2, 35 (2017). doi: 10.12688/wellcomeopenres.11689.1.