This Spring, CDISC announced a new initiative, to develop, in collaboration with Microsoft, open source software and machine-executable conformance rules for validation of implementations of a number of its
foundational standards: the CORE project.
CORE stands for "CDISC Open Rules Engine". A very good webinar was given last week, explaining the principles and phases of the project. You can find the recording here.
In order to explain the "why" of the project, we need a bit of history ...
Until about 5 years ago, CDISC did not publish conformance rules for its most used standards: SDTM, SEND, ADaM, ODM and Define-XML.As far as I could find out, the first CDISC conformance rules were published in 2017 for SDTMIG-3.2. The probable reason for this is that there was a "leave it to the vendors to implement" policy at CDISC before that time.
Several private initiatives and companies filled the gap, including our own company: already 15 years ago, we developed software to validate ODM files against the standard, the ODM Checker. The software has regularly been updated and is still available at no cost to CDISC members. It has also been extended for Define-XML. These were also integrated into commercial products such as our popular SDTM-ETL mapping software and Define-XML Designer. We did however never publish the rules as "open source", as we did not want to pretend that our interpretation of the ODM/Define-XML standard is the ultimate correct one.
For SDTM and later also for SEND and ADaM, other companies developed software for conformance checking, some first as open source, later more and more as closed source. The conformance rules, with these companies own interpretations of the Implementation Guides, were published
as Excel files at the best, and only contained a (sometimes vague) description of each of the rules, or even only the error message that is generated. Some of such rules are even completely wrong, such as "Original
Units (--ORRESU) should not be NULL, when Result or Finding in Original Units (--ORRES) is provided". In no case at all, the rules were published as "machine-executable".
One company even managed to sell its solution to the FDA and later also to the PMDA and NMPA, essentially developing the rules on behalf of these regulatory authorities. Although that software is known
for its many false positives, it is used by almost every pharma company. Another vendor however managed to find its place in the market by specializing on SEND, later extending to SDTM.
About 5 years ago, CDISC started publishing its own conformance rules, first for the SDTMIG, and later also for the SENDIG and for ADaM. Also here, the conformance rules were published in the form of Excel files, with the rules themselves coming as a text description, i.e. without machine-executable code. Recently, also conformance rules for Define-XML v.2.1 were published by CDISC. Also here, no machine-executable code was published.
For both Define-XML 2.0 as well as 2.1 such machine-executable rules however exist in the form of a Schematron, a technology used by several vendors (including ourselves), but these machine-executable rules are propriety, and are used in different free or commercial software products.
This leads to a discussion about technology for conformance rules (if you don't like technology discussions, you can skip this part). CDISC ODM and Define-XML are based on XML,
a worldwide open standard from the World Wide Web consortium W3C, who also, together with ISO, took care of developing technologies and standards for checking the conformance of XML files against a given standard: XML-Schema Schematron and XQuery. The ODM and Define-XML teams have always published their ODM and Define-XML specifications together with an
XML-Schema, meaning that 60-80% of the rules are already exactly described in a language that is both human-readable (well, for IT people at least), as well as machine-executable. The remaining 20-40% then can then be implemented
using Schematron and/or XQuery, the latter especially when information is in separate files. Also this has been done by us and by other vendors.
The problem with this approach however is that these technologies are limited to XML. Although there was a believe in the past that data exchange would become an "XML-only world", this has not come
true. At one side, the format for submissions to regulatory authorities is still SAS Transport 5,
a completely outdated "punch card" format from the IBM mainframe time, and mandated by FDA, PMDA and NMPA, this although CDISC has a much better format (which provides a perfect match with Define-XML), known as Dataset-XML. This modern format even allows to develop "smart review tools", such as the open source "Smart Submission Dataset Viewer".
On the other hand, other formats become very popular, especially JSON and, for linked data using RDF, the "Turtle" format and JSON-LD format. One could think that the latter two would not be suitable for CDISC's submission standards (SDTM, SEND, ADaM), as these represent tabular data. One should however not forget that also data in these "tables"
are essentially linked data, but that the relations are implicit, only described in Implementation Guides. And as these relations are implicit, in order to check these relations, one needs ... conformance rules. Regarding
JSON, it's usage has overtaken XML for use with APIs and in RESTful web services.
The still mandated use of SAS Transport 5, a format even discouraged by the US Library of the Congress adds another one to the list of
formats used in clinical research.
Given this variety of exchange formats, one can think about whether it is possible to have a single expression language for developing conformance rules that is both human-readable (i.e.
really "open") and machine-executable, and that can be used with all these modern as well as the outdated SAS Transport 5 format.
I have been looking for such as expression language for many years, but failed miserably to find one. This will surely be one of the main challenges of the CORE project.
The idea of publishing completely open conformance rules for the CDISC submission standards that are as well human-readable as well as machine-executable is not new - it has even been realized by the "OpenRules for CDISC Standards" initiative.
The rules have been developed using the XQuery language, a W3C standard, and some of them even use the CDISC Library API. So,
essentially, they could serve as a candidate for a reference implementation if it were not that XQuery only works for XML (and thus only for CDISC's Dataset-XML),
but is useless for use with SAS Transport 5, as well as for modern JSON, which is the first choice for use with APIs.
"Open Rules for CDISC Standards" was developed many years ago with the expectation that FDA would soon replace SAS Transport 5 by CDISC's own Dataset-XML, but this has not happened. With JSON
strongly coming up, it has become clear that the sole use of XQuery for describing human-readable rules that also are machine-executable, is not an option anymore.
One of the great starting principles of the "Open Rules for CDISC Standards" was that the rules themselves (in XQuery) are completely separated from any execution engine: implementers can choose
between many different computer languages (Java, C#, Python, ...) that read the rules and then execute them. Separation between rules and execution engine will very probably also be one of the major design principles of the
CDISC CORE project.
For the development of machine-executable conformance rules in CORE, CDISC will start with SDTMIG 3.4 and SDTM 2.0 in the first phase, which will be an "MVP" phase. No, MVP doesn't mean "most valuable player" here, it means "Minimum Viable Product". In later phases, machine-executable conformance rules for the other standards, on the long term maybe even including Therapeutic Area User Guides (TAUGs).
The execution engine will be developed by the CORE team which is a collaboration between Microsoft, CDISC, and industry, and run in the cloud using Azure. CDISC members will be able to obtain an evaluation Azure account, which is then essentially acting as a private cloud for a CORE implementation. Implementers, such as pharma companies, CROs, service providers, can later choose to use the open source code and spin up instances in other cloud environments. They will also be able to add own (e.g. company-specific) rules to their execution engine (whether a Microsoft or other one) and/or to develop their own implementation, either open source or closed source. So, one can indeed think as CORE providing a "reference implementation" that anyone can use, extend, or just use as "the reference", i.e. the outcomes of any conformance checking must be identical to that of the reference implementation, even when completely different technology is used. The CDISC Library will be the single source of truth for both the CDISC standards and the CDISC rules since the rules will be made available in the Library. The CORE Engine will retrieve the rules and standards from the Library using an API.
This means that in the MVP phase, involvement of vendors will be relatively low, but as of phase 1, it is expected that there will be a lot of involvement from vendors, either directly working together with CDISC and Microsoft (as we envisage to do), or just go their own way, using the open source. As everything will be open source, vendors can also choose between offering products that use a cloud execution engine, or create a solution that uses local production environments (e.g. desktop applications).
This is a big project. There will be quite a lot of CDISC teams involved, for example a QA team, a number of conformance rules development teams (Conformance rules for SDTMIG 3.4 / SDTM 2.0 are expected to be published in Autumn 2021), the CDISC Library team, and a software engineering team, which I presume will consist of a mix of CDISC and Microsoft people. And of course, the overall architecture must be developed and project management must be taken care of by a CORE Leadership team. More details can be found on the slides of the webinar recording.
Executable rules will be metadata driven (of course!), but it has not been decided yet what programming language will be used to make them machine-executable. Personally, I consider this as a critical part of the project, as the people developing the rules (standards specialists, e.g. SDTMIG specialists, mostly volunteers) usually are not programmers (no, I do not expect that the rules will be developed in SAS, this would be a major design error, as that would not be vendor-neutral), and the programmers (with good knowledge of Java, C#, Python ...) usually do not have a good knowledge about the SDTM standard and their Implementation Guides. So a lot of communication (and documentation of that communication) will be necessary between these two groups: the last we want to have is that CORE (based) software produces false positive warnings and errors ...
Another critical part will surely be QA and testing: what rule is applicable when, and are all the possible scenarios covered? Testing will require a huge amount of test data, covering every possible use of the standard.
All this work cannot be done by CDISC and Microsoft people alone. Therefore, CDISC is seeking for volunteers out of the CDISC community for the different teams involved in the project, in the MVP phase probably
mostly SDTM specialists for development of very precise rules: only when a rule is very precisely defined, it can be made machine-executable (see the FDA rule on -ORRESU ...). So the webinar also contained a "call for
participation". CDISC would like to have volunteers from the community that can at least spend 20% of their time in the next 9 months, and ideally, beyond that period. A kick-off meeting is already planned for September
9th, so there is not much time to lose.
The call for participation can be found on the CORE website under "participate" and contains a list of what teams and roles there
are. This will be your starting point if you want to participate.
A very interesting part of every CDISC webinar is the Q&A. In this case, not every question that came in could be answered (but some bundling took place). CDISC however promised that all question will
be answered and will be posted on the CDISC website. A few highlights:
- the reporting format will be anything the user wants. This is possible as there will be a rich set of API methods available. An Excel report interface will surely be provided.
- it is not decided yet what programming language will be used. Will it be a 3th or 4th generation language (Java, C#, Python, ...) or a meta-language that can be interpreted and translated to
source code in one of these languages? As my own attempts failed to find such a language that is independent of the data format, I am of course very curious ...
Important however is that the rules implementation will
be metadata driven, and that it needs to be relatively easy for members of the CDISC community (sponsors, CROs, ...) to develop their own specific rules as machine-executable rules and implement them in exactly the same way
as the CDISC rules themselves.
- it is intended to also include rules developed by regulatory authorities (FDA, PMDA, NMPA, ...), so the scope is not limited to CDISC rules alone.
- as the CORE software will be enriched by a large number of API methods, it will be easy to integrate CORE into third-party applications. CDISC will carefully listen to the vendor community to find out which
API methods are necessary.
- although there of course a number of deliverables and time lines, the conformance rules will never be complete as long as new standards are developed. When new standards and versions are published, it is envisaged that they immediately come with their own set of machine-executable CORE rules, which can then be implemented immediately.
- upon the question "will the regulatory agencies adopt the CORE rules?", the answer essentially was "we don't know, but we strongly hope so". One should not forget that in some cases, these
agencies rules deviate considerably from CDISC rules, and that currently, some of these agency rules are just wrong (e.g. the FDA rule "Original Units (--ORRESU) should not be NULL, when Result or Finding in Original
Units (--ORRES) is provided"), ambiguous, or even incorrectly implemented in software. However, rules from the regulatory agencies are surely within the scope of the project.
- another question was about how these conformance rules differ from already existing applications and vendors". Peter van Reusel answered it very exactly: "not much". The difference is however
that for the first time, the implementation will not only be fully transparent, but also independent of the execution engine. Also, the rules themselves will be maintained by the CDISC community, and that with each new version
of a standard, the corresponding conformance rules, in an open, machine-readable form, will be immediately available. Venkata Maguluri (Pfizer) of the webinar's panel also added that these rules will not allow any "wiggle room" anymore in their interpretation. Personally, I consider this "wiggle
room" (sometimes even "cleft room") as one of the major problems of the way the rules are currently described and published (as "text" in Excel format),especially with the regulatory authorities published rules having severe quality problems.
- the period between publication of a new standard version and adoption by regulatory authorities will be used to ensure that the conformance rules also work for the agencies. Also, CDISC will provide the agencies
with technical support regarding the implementation of the execution engines.
Last but not least, what does this all mean for our "Open Rules for CDISC Standards" project?
We are very enthusiastic about
the CORE project at CDISC. Although much larger in scope and volume, the basic principles have a lot of overlap with our own "Open Rules for CDISC Standards": in both cases, the rules are fully transparent, human-
as well as machine-executable, and separate from the execution engine. Both use or will use the CDISC Library as "the single source of truth". The major thinking error we made when starting with "Open Rules
for CDISC Standards" many years ago, was that we expected that it would soon be an "all-XML world" for data exchange, and especially that FDA (and then followed by PMDA/NMPA) would soon move away from SAS Transport
5 and start requesting modern (CDISC's own) Dataset-XML format for electronic submissions. None of these have come true: FDA still requires outdated SAS Transport 5, and instead of becoming an "all-XML world",
JSON has become a very important player, especially in combination with APIs and RESTful web services (which I consider as the future, also for submissions). Also RDF has become important as a methodology that makes implicit
SDTM/SEND/ADaM relations explicit instead of completely hidden in PDF or HTML files.
However, I hope "Open Rules for CDISC Standards" becomes a source of inspiration, and maybe even a good number of rules implemented
as XQuery can directly be translated into machine-executable CORE rules.
Not everything is clear yet, e.g. the expression (machine) language for defining the rules, and that works for any modern data format as well as outdated SAS Transport 5, has not been decided yet. This is where I failed miserably myself. Of course, Microsoft has much more resources and a lot of brilliant people to find out what that expression language should be.
Also, I do have some serious concerns that in the MVP phase, for SDTM, one will limit to an implementation that only works for SAS Transport 5, and that implementation for other formats will only be done at a later stage. That would be a wrong signal to the FDA not to look for any other alternative, modern format, and provide them with the excuse that they cannot move to a modern format, as CORE only supports SAS Transport 5, although that is essentially not true. In my opinion, interfaces and APIs can take care that a large range of import formats can be supported, and that should already go into the requirements, even in the MVP phase. Maybe I can provide some input or technical support for that part of the project: we should not forget that developing APIs and implementing them as RESTful web services is relatively easy for JSON and XML, but that no RESTful web service has ever been developed yet that supports SAS Transport 5. I expect that to be more difficult than for JSON and XML.
Another concern regarding possible (Microsoft) vendor lock-in has already been taken away during the web conference: it will be possible to use any other cloud provider or to develop local (e.g. desktop) applications.
CORE is a very ambitious project. It is even considerably larger than the CDISC Library project. The huge success of the latter however makes me confident that also CORE will be a huge success.
And please, do not forget to watch the recording of the webinar if you did not attend the webinar already: you can find it here.
