SDTM --STRESN: why we need UCUM - Part 1

In the last weeks and months, I had a lot (e-mail) discussions with Clem McDonald (Chief Health Data Standards Officer at NIH/NLM), Daniel Vreeman (Regenstrief Institute, known from LOINC and NLM), and CDISC representatives. Clem has been a pioneer in medical informatics: he was already working in the field when I was still at high school and never heard about computers.

It was all about LOINC and UCUM.

 
CDISC has been late in endorsing LOINC. SDTM always had LBLOINC as a permissible variable, but essentially no sponsor was using it as it was (and still is) … permissible.
Everything changed when the FDA announced to start requiring LOINC coding forlaboratory results for new studies starting from 2020. This triggered CDISC, FDA and Regenstrief to regularly sit together and let to the document "Recommendations for the Submissionof LOINC Codes in Regulatory Applications to the U.S. Food and Drug Administration".  

Also, CDISC is now finalizing the development of a mapping between the most used LOINC codes and CDISC-CT for LBTESTCD, LBTEST, LBSPEC. As soon as it is published, we will make a RESTful web service available for it, allowing sponsors to automatically populate LBTESTCD, LBTEST and LBSPEC from the LOINC code. In my opinion however, sponsors should NOT need to populate LBTESTCD and such when LBLOINC is populated, as all the necessary background information is in the LOINC code itself, and can easily be retrieved and visualized using one of the many RESTful web services for LOINC.

A similar discussion is currently developing around the use of UCUM. UCUM means "Universal Coding System for Units of Measure" and has also been developed by and is maintained at the Regenstrief Institute. It has a strong relationship with LOINC, as the LOINC database contains a "preferred UCUM unit" for each code representing a quantitative test. For example it recommends the UCUM unit "mg/dL" for "glucose in urine, mass concentration" (one can find the complete information here), and "mm[Hg]" for "intravascular blood pressure, supine" (LOINC code 8461-6).

Important to note here is that UCUM is a notation, whereas CDISC [UNIT] codelist is a list (no systematics).

When comparing, one immediately sees that some of the UCUM units are identical to CDISC units, but this is mostly not the case. Whereas the CDISC [UNIT] codelist is limited in size (the CDISC CT version 2018-09-28 has 692 terms), the number of units in UCUM is in principle infinite as UCUM is a notation.  

The recent discussion between me, NLM, Regenstrief and CDISC was about allowing UCUM notation in CDISC SDTM submissions. If you now submit a blood pressure with "mm[Hg]" as the unit (e.g. in VSORRESU), validation software used by the FDA will throw an error, as "mm[Hg]" is not in the CDISC [UNIT] codelist, and the SDTM-IG expects you to map this to even when the blood pressure came from an electronic health record (EHR) where UCUM notation is almost always used. The same applies to the lab unit "international units", with CDISC unit "IU", whereas the UCUM notation is "[IU]". From the SDTM-IG (3.2 and 3.3):

"The variable, LBORRESU uses the UNIT codelist. This means that sponsors should be submitting a term from the column "CDISC Submission Value" in the published Controlled Terminology List that is maintained for CDISC by NCI EVS. When sponsors have units that are not in this column, they should first check to see if their unit is mathematically synonymous with an existing unit and submit their lab values using that unit".

This means that the SDTM forces us to "translate" "[IU]" to "IU" in an SDTM submission when the data point comes from an EHR, which is not only tedious, but also error prone.

During the discussion with Clem McDonald, he asked me to come with one or more use cases where the use of UCUM notation in SDTM would be advantageous for FDA reviewers, as this could be a "tipping point" for allowing UCUM notation in SDTM. I didn't have to think long …

The --STRESN variables (Numeric Result/Finding in Standard Units) is such a use case. These variables are meant to "standardize" results for the same test (this although SDTM has no good mechanism to define what "the same test" is) to a single unit. For example, when (for the same study), one local lab delivers the resuls in "mg/dL", another one on "pg/cL" and yet another one in "mg/cL", these all have to be recalculated for LBSTRESN to a single unit, e.g. "mg/dL". How is this done?

In the classic way, using CDISC units, converting –ORRES to --STRESN requires a lot of programming for the SDTM developer. This is not only tedious work, but also extremely error prone. The reason is that CDISC-CT does not deliver the conversion factors like between "mg/dL", "pg/cL" and "mg/cL". It all has to be programmed. So the programmer need to find them out him/her-self and program them all (there can be hundreds of them).

If we know that all our "source" units (from –ORRESU) are in UCUM notation, this can fully be automated, as through the UCUM "ucum-essence.xml",  conversions between units for the same "property" (a concentration in this case), automated conversion is easily possible.

The National Library of Medicine (NLM) even has aRESTful web service for it.

So, the program that generates the SDTM data sets can simply call that service to perform these conversions. This means hours or even days of less programming effort for the SDTM programmer, and much much less errors.

In the above case (mg/dL), "this is our lucky day"! In the given case, the UCUM notation overlaps with the CDISC list, so we could also use the NLM RESTful web service to automate the conversion. For the other example ("IU" versus "[IU]") we cannot use it.
Interesting in this case is that the CDISC-CT did a big effort to try to cover a large number of units in their list in which "IU" is present: there are over 60 of them, from "100 IU/mL" to "uUI/mL/kg". However, if one would need to cover all suitable combinations, CDISC-CT would need to add thousands of "IU" units to their list.

When using UCUM however, there is no need for all of this, as UCUM is a notation. So "u[UI]/mL/kg is just as valid as "p[UI]/mL/kg" as "m[UI]/dL/dg" and all other possible combinations of "[UI], "L" and "g" or better said, of all possible combinations of "International Units" and volume and mass. So also "d[UI]/cm3/[lb_av]" (the last "[lb_av]" is the symbol for "pounds") is a valid UCUM unit! And every value in "d[UI]/cm3/[lb_av]" can be automatically be converted and standardized to e.g. "u[UI]/mL/kg". 

But how do we know whether a provided unit is in UCUM notation? Also here, there is a RESTful web service provided by the NLM to validate whether a given unit is a valid UCUM notation unit.

 

Not very many sponsors have asked yet for better CDISC support for LOINC and UCUM. One of them is SHIRE. Veena Nataraj and Diane Piper gave a presentation at the 2018 CDISC International Interchange titled "Delivering LOINC Codes in Future – Bridging Gaps within clinical lifecycle". You can find their paper here, and

their slides here.Some interesting statements from their paper are: 

concerning LBTESTCD, LBTEST, LBSPEC, …:

"Using CDISC conventions, all of these attributes are required to uniquely and accurately identify a given test and its results. This can be challenging, and the variability of how these attributes are managed and implemented make it difficult to determine if a given test conducted in one study is comparable to a test in another study, even within studies conducted by a single Sponsor company, never mind across the industry. In contrast, LOINC incorporates all of the SDTM attributes as well as others to properly and uniquely identify a test in a single term".

Corresponding to my statement (already years ago) that only a LOINC code can uniquely define a lab test .

Continued in Part 2 ...

 

New FDA Validation Rules - October 2018

The FDA has recently updated its SDTM and SEND validation rules which can be downloaded as an Excel file (!) from the FDA website. Immediately after, Pinnacle21 published an overview (also in Excel) of all the changes relative to the prior version (February 2017). So I wonder whether these rules were developed by the FDA itself, or that it was outsourced to a specific software company.

So I started evaluating this set of rules, and especially the changes as published by Pinnacle21.

Unfortunately, once again, the set of has been published as a worksheet, which is machine-readable in some way, but not machine-interpretable. The worksheet does even not contain pseudo-code for the rules execution, as e.g. was published by the SDTM team when they published their "SDTMIG Conformance Rules" early 2017.

As in prior versions, one must sometimes just guess what is being tested anyway. For example rule SD1321/FDAB001:



What is being tested here?
The "business rule" is identical to the one of P21 rule SD1097, so what is the difference? One can only guess from the "FDA Validator" message, which has, in brackets: "Missing SUPPAE". So, if this is indeed what is being tested, the rule should say: "SUPPAE must be present in the submission". It doesn't.

Interesting is also that the line with "SD1321" is marked as "new" (in yellow), and as the line with "SD1097" isn't but states that it is the same FDA rule, it looks as this is just an additional validation test for the "same", already existing (February 2017) rule. The same two lines also appear in the FDA worksheet. But was really is or must be tested, remains in the dark. Probably only one company knows this …

Another example of a new rule that is (i.m.o.) completely unclear is rule SD2006/FDAB003:



What is being tested here? Is it being tested whether a MedDRA code appears in a SUPPQUAL domain? If so, how can that be tested anyway? By the "QNAME"? Or by information in the define.xml? No indication at all is provided.

Interesting is that it is now also indicated when the rule is also applicable to SEND, and to which version of SEND. That is of course a good thing.
Worse is that some rules present in the February 2017 version that are nonsense have not been removed. For example the rule SD0026/FDAB012:



Stating that a unit MUST be provided in --ORRESU when --ORRES is populated.

THIS RULE IS NONSENSE!

I know thousands of tests in the given domains (EG, LB, VS, …) for which there is no unit. For example, what is the unit for "NEGATIVE"? Even for tests where the result is a number, like pH, there is no unit. In the prior version of the P21 validation software, it was tried to add some "logic" for when a unit is necessary, but this often ended in disaster, with a lot of false positives.

At least for LB and VS, whether a test has a unit or not could be retrieved from the LOINC code e.g. using a RESTful web service, as has been made available by us, the National Library of Medicine, and LOINC itself. LOINC coding is however still not a requirement in SDTM.  Also, the validation software used by the FDA does not use any RESTful web services at all.

There is unfortunately a lot of such rules in the new version, where one has refused to learn from the lessons in the past, and just copied the "nonsense", "un-understandable" or "expectation" rule from the prior version.

Interesting is also that the FDA worksheet now also contains the CDISC rules, assigns a number/code to them, and even provides error messages for them, although all of this is not in the competence of the FDA at all. This is of course also to blame to CDISC, as CDISC never made a serious attempt to provide or publish a "reference implementation", i.e. an implementation of the CDISC rules to which all vendors must comply, meaning that vendors implementations must exactly provide the same results as the reference implementation. This is e.g. how Java library implementations are developed. More about "reference implementations" in the next section.

We suppose however that adding the CDISC rules to the FDA rules (with new IDs) is a further attempt of the company that we suppose created the rules in assignment of the FDA, to "monopolize" the implementation of the rules: the "application logic" in "pseudo code" as published by CDISC does not even appear in the FDA worksheet. They will simply use their own interpretation.

A recent article from Pinnacle21 lso reveals some interesting information regarding the future of the "Community" version of the validation software. It states "Last 5 columns, highlight new rules in the next release of Community" and more important: "If you are a Pinnacle 21 Enterprise user, all FDA Validator Rules are already available to you. For Community users, these new rules will become available in 3.0 release scheduled for January." So it looks as there WILL be a new "Community" version. It was promised to us for November (2017), but nothing happened. I was already afraid that there would never be a new "Community" version anymore, as Pinnacle21 is pushing companies very hard to purchase the (expensive) "Enterprise" version (which I do understand - they are not a charity).

Reference implementations

When a standard is being established, it has become usual to also provide a "reference implementation". This is an implementation in software that fulfills the minimal requirements to
adhere to the standard and to which all other (e.g. third party) implementations must comply, i.e. give the same results for all test cases as the reference implementation.
For example, the Java "Jersey" library for RESTful web services is the reference implementation of the standardized JAX-RS API to which all third party libraries implementing RESTful web services must comply.

Unfortunately, CDISC did not provide reference implementations for SDTM/SEND rules in the past, as it did not want to go into software development. The problems was that it did not think about formalizing "rules" from the SDTM-IG in the first place! It is the virtue of OpenCDISC (later renamed to Pinnacle21 as CDISC did not want that its name was used) to have developed such rules, however as their own interpretation of the SDTM-IG. It allowed to validate SDTM (and later also SEND and ADaM) submissions. No wonder that the FDA was highly interested in this. However, we still must take into account that what the FDA is using, is an interpretation of the SDTM-IG by a single company, and not the CDISC interpretation. CDISC recognized having lost control of its own "rules" pretty late, and finally also started publishing validation rules, first for SDTM, and later also for ADaM, containing "pseudo code" for the validation checks. Real "reference implementations" however to which vendors must comply have however not been published by CDISC.

Such a "reference implementation" for CDISC/FDA/PMDA rules could be the current "Open Rules for CDISC Standards" project within CDISC. It aims to provide rules in a machine-readable (and -executable) language that is at the same time very well human-readable, so they are completely open. Such rules are NOT embedded in software, they come independent of any software, but can be used by software in a vendor-neutral way, i.e. any vendor can use these machine-readable rules in its software, or develop software that generates exactly the same results for each possible test case. These rules can be implemented using Java, C#, Python … software languages. So really language and operating system independent. A few CDISC volunteers already develop a good number of such machine-readable rules. You can find their results here.

Conclusion

The new version of the FDA rules for SDTM and SEND are completely along the lines that were followed in the past. No corrections were done for rules that are completely wrong (i.e. nonsense), can essentially not be implemented in software, or are expectations rather than rules. The rules have been developed and published in such a way that only one vendor (probably the company that supposedly developed the rules) can implement them in software (not vendor-neutral). Including the CDISC owned rules into the "FDA rules" publication gives a good oversight, but gives the impression that one wants to establish a monopoly on the software implementation of all the rules, either from FDA or from CDISC. The way these rules have been developed and published i.m.o. violates the requirement that it must be vendor-neutral. But we are already used to that. Also the XPT format is not really vendor-neutral (the TS-140 "specification" is very hard to implement in non-SAS software), and i.m.o. also discriminates Spanish-speaking patients in the US (it does not support "Spanish" characters).


A good alternative can be the future results of the "Open Rules for CDISC Standards" project (though already a lot has been done), but this would require the FDA to switch to a modern transport format (like XML or JSON).