So I started evaluating this set of rules, and especially the changes as published by Pinnacle21.
Unfortunately, once again, the set of has been published as a worksheet, which is machine-readable in some way, but not machine-interpretable. The worksheet does even not contain pseudo-code for the rules execution, as e.g. was published by the SDTM team when they published their "SDTMIG Conformance Rules" early 2017.
As in prior versions, one must sometimes just guess what is being tested anyway. For example rule SD1321/FDAB001:
What is being tested here?
The "business rule" is identical to the one of P21 rule SD1097, so what is the difference? One can only guess from the "FDA Validator" message, which has, in brackets: "Missing SUPPAE". So, if this is indeed what is being tested, the rule should say: "SUPPAE must be present in the submission". It doesn't.
Interesting is also that the line with "SD1321" is marked as "new" (in yellow), and as the line with "SD1097" isn't but states that it is the same FDA rule, it looks as this is just an additional validation test for the "same", already existing (February 2017) rule. The same two lines also appear in the FDA worksheet. But was really is or must be tested, remains in the dark. Probably only one company knows this …
Another example of a new rule that is (i.m.o.) completely unclear is rule SD2006/FDAB003:
What is being tested here? Is it being tested whether a MedDRA code appears in a SUPPQUAL domain? If so, how can that be tested anyway? By the "QNAME"? Or by information in the define.xml? No indication at all is provided.
Interesting is that it is now also indicated when the rule is also applicable to SEND, and to which version of SEND. That is of course a good thing.
Worse is that some rules present in the February 2017 version that are nonsense have not been removed. For example the rule SD0026/FDAB012:
Stating that a unit MUST be provided in --ORRESU when --ORRES is populated.
THIS RULE IS NONSENSE!
I know thousands of tests in the given domains (EG, LB, VS, …) for which there is no unit. For example, what is the unit for "NEGATIVE"? Even for tests where the result is a number, like pH, there is no unit. In the prior version of the P21 validation software, it was tried to add some "logic" for when a unit is necessary, but this often ended in disaster, with a lot of false positives.
At least for LB and VS, whether a test has a unit or not could be retrieved from the LOINC code e.g. using a RESTful web service, as has been made available by us, the National Library of Medicine, and LOINC itself. LOINC coding is however still not a requirement in SDTM. Also, the validation software used by the FDA does not use any RESTful web services at all.
There is unfortunately a lot of such rules in the new version, where one has refused to learn from the lessons in the past, and just copied the "nonsense", "un-understandable" or "expectation" rule from the prior version.
Interesting is also that the FDA worksheet now also contains the CDISC rules, assigns a number/code to them, and even provides error messages for them, although all of this is not in the competence of the FDA at all. This is of course also to blame to CDISC, as CDISC never made a serious attempt to provide or publish a "reference implementation", i.e. an implementation of the CDISC rules to which all vendors must comply, meaning that vendors implementations must exactly provide the same results as the reference implementation. This is e.g. how Java library implementations are developed. More about "reference implementations" in the next section.
We suppose however that adding the CDISC rules to the FDA rules (with new IDs) is a further attempt of the company that we suppose created the rules in assignment of the FDA, to "monopolize" the implementation of the rules: the "application logic" in "pseudo code" as published by CDISC does not even appear in the FDA worksheet. They will simply use their own interpretation.
A recent article from Pinnacle21 lso reveals some interesting information regarding the future of the "Community" version of the validation software. It states "Last 5 columns, highlight new rules in the next release of Community" and more important: "If you are a Pinnacle 21 Enterprise user, all FDA Validator Rules are already available to you. For Community users, these new rules will become available in 3.0 release scheduled for January." So it looks as there WILL be a new "Community" version. It was promised to us for November (2017), but nothing happened. I was already afraid that there would never be a new "Community" version anymore, as Pinnacle21 is pushing companies very hard to purchase the (expensive) "Enterprise" version (which I do understand - they are not a charity).
Reference implementations
When a standard is being established, it has become usual to also provide a "reference implementation". This is an implementation in software that fulfills the minimal requirements to
adhere to the standard and to which all other (e.g. third party) implementations must comply, i.e. give the same results for all test cases as the reference implementation.
For example, the Java "Jersey" library for RESTful web services is the reference implementation of the standardized JAX-RS API to which all third party libraries implementing RESTful web services must comply.
Unfortunately, CDISC did not provide reference implementations for SDTM/SEND rules in the past, as it did not want to go into software development. The problems was that it did not think about formalizing "rules" from the SDTM-IG in the first place! It is the virtue of OpenCDISC (later renamed to Pinnacle21 as CDISC did not want that its name was used) to have developed such rules, however as their own interpretation of the SDTM-IG. It allowed to validate SDTM (and later also SEND and ADaM) submissions. No wonder that the FDA was highly interested in this. However, we still must take into account that what the FDA is using, is an interpretation of the SDTM-IG by a single company, and not the CDISC interpretation. CDISC recognized having lost control of its own "rules" pretty late, and finally also started publishing validation rules, first for SDTM, and later also for ADaM, containing "pseudo code" for the validation checks. Real "reference implementations" however to which vendors must comply have however not been published by CDISC.
Such a "reference implementation" for CDISC/FDA/PMDA rules could be the current "Open Rules for CDISC Standards" project within CDISC. It aims to provide rules in a machine-readable (and -executable) language that is at the same time very well human-readable, so they are completely open. Such rules are NOT embedded in software, they come independent of any software, but can be used by software in a vendor-neutral way, i.e. any vendor can use these machine-readable rules in its software, or develop software that generates exactly the same results for each possible test case. These rules can be implemented using Java, C#, Python … software languages. So really language and operating system independent. A few CDISC volunteers already develop a good number of such machine-readable rules. You can find their results here.
Conclusion
The new version of the FDA rules for SDTM and SEND are completely along the lines that were followed in the past. No corrections were done for rules that are completely wrong (i.e. nonsense), can essentially not be implemented in software, or are expectations rather than rules. The rules have been developed and published in such a way that only one vendor (probably the company that supposedly developed the rules) can implement them in software (not vendor-neutral). Including the CDISC owned rules into the "FDA rules" publication gives a good oversight, but gives the impression that one wants to establish a monopoly on the software implementation of all the rules, either from FDA or from CDISC. The way these rules have been developed and published i.m.o. violates the requirement that it must be vendor-neutral. But we are already used to that. Also the XPT format is not really vendor-neutral (the TS-140 "specification" is very hard to implement in non-SAS software), and i.m.o. also discriminates Spanish-speaking patients in the US (it does not support "Spanish" characters).
A good alternative can be the future results of the "Open Rules for CDISC Standards" project (though already a lot has been done), but this would require the FDA to switch to a modern transport format (like XML or JSON).
