This is a follow up on my prior posts "Adding Custom Rules to CDISC CORE - first experiences" and "Generating custom CORE Rules starting from CDISC SDTM Dataset Specializations".
Introduction
CDISC Open Rules Engine (CORE) is now well established as the best validation software for validation of submission SDTM, SEND and ADaM files. Also FDA reviewers are more and more switching to CORE, as the existing classic validation software they used before is notoriously known for its false positives, forcing them to "dive" in the Reviewers Guide to find out whether these false positives are documented.
CORE has the advantage over the classic validation tools that it is completely open source, whereas the classic tool is "black box", i.e. one does not know how each of the rules has been implemented.
One can only guess.
Also, all the rules implemented in CORE are completely open (as either YAML or JSON) and can be inspected by everyone.
One of the advantages of CORE that is not so well known yet, is that everyone, every sponsor, every CRO or service provider, can develop their own, additional rules, and execute them in the CORE engine. Such "custom" rules can be either written in YAML or JSON, usually YAML being the easier choice.
Custom Rules
Suppose we have a trial where subjects older than 65 years are not allowed to be enrolled. To check whether such a subject is present in our DM dataset, we can develop a simple rule in YAML as:
In the existing CORE implementation (in this example using the Windows executables), the rule can then be executed using the "-lr" ("local-rules") parameter, like:
.\core.exe validate -s sdtmig -v 3-4 -lr D:\CDISC_Standards\CDISC_CORE_extensions\AGE_65.yaml
-d D:\MetaDataSubmissionGuide_2_0_SASXPTFiles ...
with the parameter value of "-lr" the path to the (here YAML) file with the "custom rule, and the value of the "-d" parameter the path to the directory with the files to be validated.
The
report then e.g. looks like:
The "-lr" parameter can not only take simple file as parameter value, but also a directory containing "custom" rules, like:
.\core.exe validate -s sdtmig -v 3-4 -lr D:\CDISC_Standards\CDISC_CORE_extensions
allowing to execute several custom rules in one run.
This can be especially interesting for CROs or service providers that work with different sponsors, studies, and thus, different sets of rules.
Custom Standard Rules
Another possibility is to really "add" sets of custom rules to the CORE engine itself for reuse. We call them "custom standards". Even nicer, it is possible to version such "custom standards", thus making maintenance easy.
The CORE CLI reference explains this pretty well, but of course, an example is always nicer.
First of all, one must add all the rules for a single "custom standard" and "version" in a single directory, usually as YAML files (but also JSON can be used). You will also need a (small) JSON file just containing a list of the rule names, like:
stating that this is for "standard" with name "sponsor_1_rules" with version is "1-0" (best to use a dash here) followed by the list of rule names. The latter than correspond
to files "Rule_1.yaml", "Rule_2.yaml" etc..
Of course, one can right a little program to automate this step (as we did ...).
These custom rules can then be added to the CORE engine "cache" using the command:
.\core.exe update-cache --custom-standard D:\sponsor1_CORE\sponsor_1_rules.json ...
with the last part of the command pointing to the JSON file with the list, as above.
At the end, you will also need to add your API key using the "--apikey" parameter.
To check whether the set of rules has really been added, use:
.\core.exe list-rule-sets --custom
which returns a list of all custom rule sets in uppercase. For using a set, one must however use the lower-case values as defined in the JSON file.
To use the "custom standard", one needs to provide the keyword "-cs" ("custom-standard") in the validation command, like:
.\core.exe validate -cs -s sponsor1_rules -v 1-0 -d ....
stating that version 1-0 of the "custom standard" "sponsor1_rules" must be executed on the files in the directory provided after the "-d" parameter.
Sometimes, it is also useful to get a list of the rules within a "custom standard", although these are also already listed in the validation report. To do so, one can use:
.\core.exe list-rules --custom-rules -s sponsor1_rules -v 1-0
It is even possible to ask for a single rule within a "custom standard", like:
.\core.exe list-rules --custom-rules -s sponsor1_rules -v 1-0 -r Rule_1
where one can leave out "-s sponsor1_rules -v 1-0" when one wants to obtain all rules named "Rule_1" within any if the custom standards and versions.
Using the python version of CORE
All the above assumes that the Windows executables of CORE are used. One can of course also use the native Python (source) code, even a Mac or Linux version.
For the use with Python, just replace
".\core.exe" with "python core.py"
Rules generated from the CDISC "SDTM Dataset Specializations"
CDISC "Biomedical Concepts" are used more and more, especially in the generation of a digital protocol as USDM together with the "SDTM Dataset Specializations". The latter are essentially "SDTM implementations" of the Biomedical Concepts. For example, the SDTM Dataset Specialization for DIABP states that it must be mapped to the VS domain, with VSORRES expected to be an integer, and VSORRESU expected to be either "mmHg" or "cmHg", and VSPOS being limited to "PRONE", "SEMI-RECUMBENT", "SITTING", "STANDING", and " SUPINE".
Essentially, one can also consider this a set of rules. As one sees, one "SDTM Dataset Specialization" can then lead to several rules.
We found a way to automatically create a large set of rules (over 2000) from the "SDTM Dataset Specializations", in YAML format, which we then loaded as a "custom standard", with the name "cdisc_dataset_specializations" version 1-0. Later, a new release of these SDTM Dataset Specializations was published by CDISC, so we did the exercise again, and thus created and loaded a version 1-1, which now has over 2600 rules.
If you would like to obtain a copy of these, with or without consultancy services, please let us know.
Implementation and integration in (existing) software
Many of us will not prefer to always do CORE validation using CLI (command line interface). The latter is of course ideal for "batch" implementation, e.g. in "pipelines", for example, when SDTM datasets regularly are received and then automatically and immediately validated without human intervention.
A great advantage of CORE is that one can easily integrate it into existing software, or build a Graphical User Interface (GUI) on top of it. For the first (integration), we did so in our well-known SDTM-ETL mapping software. This means that when using SDTM-ETL, one can immediately validate mapping results, not only against the CDISC rules and FDA business rules, but also against the rules derived from the "SDTM Dataset Specializations", and even against own custom rules that have not been loaded into the CORE engine.
We have also integrated CORE, together with the rules derived from the "SDTM Dataset Specializations" in our open source "Smart Submission Dataset Viewer". We have build the GUI in such a way that the user can choose between either the "CDISC rules" or the "SDTM Dataset Specializations rules" (doing both in one run is technically not possible) or "own rules". Below are a few screenshots with explanations.
Example 1:
This screenshot comes from the open source "Smart Submission Dataset Viewer", with integrated CORE engine, including the custom rules derived from the CDISC SDTM Dataset Specializations.
where violations are highlighted in the "table view". This is made possible by having the CORE engine generating the result output as JSON, and the GUI picking up this output and using it to render the cells that violate a rule with a separate background color, together with a tooltip explaining what the violation is.
Example 2:
Also a screenshot from the "Smart Submission Dataset Viewer" showing how the user can select rules, in this case the "custom rules" generated from the CDISC SDTM Dataset Specializations. This can be done just by selecting single rules, or using the "Filter" button to select sets of rules by specific criteria. For further details, one can download the documentation file "Smart_Submission_Dataset_Viewer_CDISC_CORE.pdf" from the SourceForge site.
Example 3:
Most of these features have also been implemented in our popular SDTM-ETL [] mapping tool.
In the newest version, which will also support the new SDTMIG-4.0, it is even possible to perform validate
individual mapping scripts using CORE. Below is a screenshot demonstrating this.
Conclusions
As "open source", CORE is not only superior to existing validation tools for CDISC submissions, but also has the advantage that companies can easily add their own validation rules, bundle them into "custom
standards" and add them to the CORE engine.
Furthermore, as we have shown here, CORE can rather easily be integrated into existing software, or a GUI being build on top of it.
If you want to know more about how we can help companies with the deployment and integration of CORE, or need help with the development of "custom rules", always feel free to contact us.
As always, comments to this article are highly appreciated!
