CDISC Validation: PointCross – Pinnacle21 v.3 comparison: Part 2: Validation features

PointCross

When first using the PointCross "Validation" part (we will discuss the other features in one of the next blogs), one sees that one can drag-and-drop submission files just as in Pinnacle21. There is no separate field for a define.xml file, so if one wants to use one (which I strongly recommend: it is the "sponsor's truth") one also needs to drop it in the same area. I soon found out that for the submission files, it first looks like only XPT files are treated by the software – there does not seem to be support (yet) for the more modern Dataset-XML format. PointCross claims that also SAS7BDAT files can be handled, as well as CSV files, but I haven't tried that yet. Nice is that one can also drag-and-drop a "zip" file with all the files needed.
When a define.xml is in the set (it must really be named "define.xml" - lowercase), a checkbox "" is enabled, allowing to do a separate define.xml validation, for which a separate report is then generated. This validation is based on a set of PMDA rules.

For the validation options, one can choose between FDA and PMDA rules, CDISC rules are not included. CDISC-CT version can also be selected. It (of course) depends on whether SDTM, SEND or ADaM is selected. For SDTM, one can choose between SDTM-IGs versions 3.1.2, 3.1.3 and 3.2 (3.3 is not implemented yet). For SEND-IG, there is the choice between 3.0, 3.1 and SEND-DART 1.1. For ADaM-IG, only version 1.0 is supported. There is no option to select individual rules to be executed, it is really an "all or nothing". Selecting individual rules to be executed would be a great option, especially during pre-submission generation of datasets, e.g. even before study completion, as one can then concentrate on single issues.

Furthermore, one can select different versions of UNII and of Med-RT. The software currently comes with versions 2018-10-25 for UNII and 2018-11-05 for Med-RT only.  Also different versions of MedDRA and SNOMED-CT can be selected, when such are installed. Such files do not come with the software, as licenses are needed that do not allow redistribution.
When looking how and where the CDISC controlled terminology files are installed, we found out that these are in a binary format (with Lucene indexing for faster access), so that we cannot check whether they have been manipulated (as Pinnacle21 did in the past), or that the original, as by CDISC published version is used.
MedDRA, and SNOMED-CT files can be added by a button click, they will then be indexed using Lucene and stored into a local database.

For the report, there are no options. Only Excel reports can be produced, nothing like XML (would be great for reuse, transformations etc.) or any other formats. Also, the reports always go to a "Desktop" directory, one cannot choose for or select another directory, at least not from the graphical user interface.
A nice feature is that one can select an older report as a "base version" when doing a validation. The new report will then have an additional tab with a "dif" between the two reports.

PointCross also announced "Auto-Update" for as well the CT files as well for the software itself. We will test these as soon as new versions of the software and/or become available.

A feature that we will discuss later is to generate a Word (template) of a "Reviewers Guide" starting from the validation report.

As far as I could see, PointCross does not have a CLI (Command Line Interface) allowing to start the software and populate the fields from within another software or to run the validator in batch mode.

PointCross MySEND is not open source.

 

Pinnacle21

A first surprise in v.3.0.0 was that only "FDA" and "PMDA" validation is now possible: the options "SDTM 3.2" and "SDTM MD 1.0" are not available anymore. These two were Pinnacle's own interpretations of the SDTM-IG 3.2 and the corresponding "Medical Devices" IG. In 3.0.0, these are not present anymore.

For ADaM, one can now choose between FDA ADaM-IG 1.0, FDA ADaM-IG 1.1, and PMDA ADaM-IG 1.0. Especially the FDA ADAM-IG 1.1 was really necessary, as the FDA is currently switching from ADaM 1.0 to 1.1.

Also new is that when one selects for a Define-XML validation, one also needs to state for which standard (SDTM, SEND or ADaM), and whether FDA or PMDA must be used. This is a bit strange, as essentially, the submission standard for the define.xml is used for, is provided in the define.xml itself. Also, I do not see how there could be differences between FDA and PMDA regarding define.xml rules. Maybe it has to do with which Trial Summary (TS) parameters are expected by the FDA and by the PMDA and need to be provided in a codelist. We will look into this later.

Support is provided for as well XPT format, CDISCDataset-XML format, as for "poor man's" CSV format. In the case of Dataset-XML and CSV, a define.xml MUST be provided, in the case of XPT it is optional. Essentially, in my opinion, a define.xml should always be provided, as it is "the sponsor's truth". We will later discuss this again when testing the execution of the rules.

As for PointCross, it is not possible to select individual rules for execution during validation. This means that when does already have a well-developed define.xml file (e.g. as a specification already at or before the start of the study), and not all submission files are ready yet, one can get a large number of errors.  

Support is provided for all most recent CDISC controlled terminology (starting from 2016-2017 depending on the CDISC standard). The CT files are located in the "My Documents" directories on the local PC (so probably cannot be centralized when a shared drive or file server is used), with one directory per CT version. Pinnacle21 seems to have moved away from the practice to manipulate the CT files (hurray!) – only the "NullFlavor" codelist has been extended (using the correct mechanism with 'def:ExtendedValue="Yes"'). What I liked a lot is that all "xxTESTCD" ("test code") codelists come as "CodeListItem" codelists, i.e. that for each test code, also the test name is provided in a "Decode" child element. This is a very good practice which unfortunately is not applied anymore by the CDISC-CT team. Interesting is also that the CT files contain "ValueList" definitions, probably (very smart!) to allow for validation of subsets of controlled terminology.

Also here, there is no option to control where the reports are generated and in which format. They all go into a subdirectory of "My Documents" and as Excel. 

Though I had expected this as a new feature, Pinnacle21 does not seem to have implemented the results of the "Phuse Reviewers Guide in XML" (to which they contributed) yet, allowing to auto-generate a partial reviewers guide in XML format from the validation report, which in its turn can then be extended and transformed to e.g. PDF.

What is also nice is that there are "hyperlink buttons" for helping the user to install the MedDRA and SNOMED-CT files. So the user must not take care that these files are correctly copied to the correct directory, this is taken care of by the software itself when the local (or shared) locations are provided. At least the MedDRA files are just copied "as is". No attempt is made to index them (as far as I could find out) for faster access as in the case of MySEND.
Auto-update of CT files has been announced by Pinnacle21. We will see how this works when new CDISC-CT becomes available. I am not sure whether there is an auto-update for the software itself. I hope so, and I also hope that bug fixes and new sub-releases become available much sooner and faster, and not every 2-3 years. However, users should be able to switch off auto-update, so that they can work with a stable version. This is often a requirement at pharma companies.

Pinnacle21 3.0.0 comes with a CLI (Command Line Interface), probably allowing to start the software from within other software, and populate the fields with information, or to run the validator in batch mode. Some information can already be found here, and in a PharmaSUG article which can be found here. As soon as I can, I will try it out and share my experiences with all of you.

Very good news is that Pinnacle21 validator is once again "open source". The 3.0.0 code can be found on BitBucket (the source of v.2.2.0 can still be found here). In the past, I had severe problems with getting the source of the even older version 2.1.3 compiled at all (but maybe I am a lousy software developer!). When I have the time, I will try to compile the source code for v.3.0.0. Unfortunately, the Pinnacle21 license does not allow to  redistribute software or code that was changed ("adapted material"). In my opinion, this is against the spirit of open source.

 

Conclusion for "features"

Both software packages are relatively similar concerning the features provided. Both are limited to FDA and PMDA rules, none has support for CDISC developed rules. Pinnacle21 has support for Dataset-XML and CSV formats, which is not the case for PointCross, which only supports XPT format.
PointCross has Med-RT files included, whereas Med-RT is not at all supported in Pinnacle21.
CDISC-CT is provided by both, and auto-updates will be provided by both. 

I cannot judge yet about deploying the software packages on a shared drive or a file server for shared use within a company (e.g. on Citrix). The use of "My Documents" and "Desktop" directories might limit that. Some of my colleagues in Germany, Switzerland and Italy are trying to accomplish this right now for both packages. When I know more, I will publish more information about this.

Next time, I will probably report about first tests performed using both packages.

 

Also read:
Part 1: Installation
Part 3: hot topics