Discussion of the statistics behind the NEDU study on the redistribution of decompression stop time

[Joseph]

Hi,

There was a suggestion on the CCR Explorers Facebook group that I post here a link to an article I had written a few months ago, as it might be of interest to some. CLICK HERE FOR THE ARTICLE.

I’m happy to discuss any questions that arise about the actual topic of the article, which is an overview/discussion of the statistical analysis of the data, but please note the disclaimer below.

DISCLAIMER:
Note that this article was written only with a view to help those divers who are interested in better understanding the statistical analysis of the NEDU study but do not have a mathematical and statistical background. I’d like for any discussion not to stray out of topic and into tangential points. Kindly keep the focus of the article in mind. Needless to say, please keep any discussion civilised.

I hope you enjoy the article and find something of value in it!

Happy & safe diving to all,

Joseph

 

[Randy Thornton]

Joseph, thank you very much for posting your article. For those of us who don't have a serious background in statistical analysis, this is very useful in understanding the significance of the data! Thanks again!

 

[Tony_Q]

Hello Joseph,

Thank you very much for the analysis, very informative.

Regards,
Tony

 

[UWSojourner]

Nice write-up Joseph. One thing you made clear was that you'd have liked to see the two-sided test. I think your concern is that with a two-sided test we're only ~91% sure the two profiles differ in their efficiency (it fails a two-sided 5% threshold). You also say that "pending further studies" you're more guarded about the way you're responding (hopefully that's a fair interpretation).

My question relates to how you factor the other studies (see this post) into your thinking? Since they all confirm the direction of the NEDU study, which of the following would be more convincing to you: (a) an original NEDU study that had met the 5% threshold of a two-sided Fischer test, but no other information was available, or (b) the existing NEDU study that meets the 5% threshold of the one-sided test and the several other studies that have reinforced the direction of the NEDU findings?

Your application of the NEDU results (i.e. being comfortable with a low GF between 30 and 45) certainly seems like a reasonable personal response to me. Just wondering how you work the other studies into your thinking.

 

[Joseph]

My question relates to how you factor the other studies (see this post) into your thinking? Since they all confirm the direction of the NEDU study, which of the following would be more convincing to you: (a) an original NEDU study that had met the 5% threshold of a two-sided Fischer test, but no other information was available, or (b) the existing NEDU study that meets the 5% threshold of the one-sided test and the several other studies that have reinforced the direction of the NEDU findings?

Your application of the NEDU results (i.e. being comfortable with a low GF between 30 and 45) certainly seems like a reasonable personal response to me. Just wondering how you work the other studies into your thinking.

Hi UWSojourner,

Thanks for the input!

I'm aware of those other studies; something to note and be careful about (especially when comparing those studies with the NEDU study) is that their endpoint wasn't clinical DCS, but that's not to say they don't add value to the discussion / provide further insight (seemingly pointing towards the same direction). The fact that the NEDU study's endpoint was DCS makes it a unique study in this regard (and also very difficult for it to be repeated).

As I say in the text, I already take into account the result of the NEDU study, even though it's a one-sided Fisher result (which is not ideal); to quote directly: "For my diving, I do take into account the findings of the NEDU study. Given the support – on the basis of a one-sided Fisher’s exact test – for the hypothesis that dive profiles generated by a bubble model are less efficient than those generated by a gas content model, I’m still using a dissolved-gas model, but I am aware of the limitations of this study." The limitations, which, as with any study, are important to appreciate, are described at length in the article, so I think there's no need to repeat them in detail here. Just briefly, one of them is the fact that, ideally, the statement of the null hypothesis would read as in the 'Two-sided Framework' box, whereby one strives to reject the statement that both approaches are equally efficient, as opposed to testing in one direction, which was mandated by ethical concerns.

As a result, I've adopted a low GF of 30-45 as a careful approach that seems to work well for my diving, and stay on the lookout for further published research results.

Kind regards,

Joseph

 

[Tony_Q]

Hello Joseph,

If we were able to find (or build) a database of actual dive profiles with decompression injury outcomes (accepting the under or over reporting error potential for self-assessing decompression injury outcomes for a given dive) and wished to test how well existing decompression models predicted such outcomes, is there anything (based upon your experience looking at the NEDU or other studies) which you would suggest as a preferred method/s for determining the statistical significance of any observed variations in number of injury outcomes for different sub groups of divers?

Would the type of approach of using an exact Fisher Test (one and/or two sided as used in your analysis of the NEDU study) be practical or would the number of potential permutations (multiple decompression models e.g bubble, dissolved tissue etc; and intra- model variations e.g combinations of different GF factors within a dissolved tissue model etc) mean that a different statistical approach would be needed.

From a practical perspective, do you believe that it would ever be feasible to obtain meaningful data from such raw field data, or that statistically significant variations (to the extent they may exist) in predictive capability of the various decompression risk models can only really be obtained by controlled studies (like NEDU) requiring deliberate/substantive risk of DCS injury to divers testing profiles at each models limits of where they predict injury likely to occur.

What I am basically trying to understand is what are the practical options for evaluating effectiveness of any existing or new decompression risk model to a high level of statistical significance. Is it a NEDU 2 Study or is there some other smarter, safer way (from a study participant diver risk perspective) to achieve a similar analytical outcome.

Thanks,
Tony

 

[Joseph]

Hi Tony,

A Fisher test is used for associations between two categorical variables; for the use you are describing, which is a complex problem, a different treatment would be appropriate.

The main problem I see with such "user-reported” data is its reliability. Specifically, I see issues with how DCS would be identified if there is no proper, medical examination. Sure, extreme cases are easy to self-diagnose, but what about those cases where divers do not really identify symptoms correctly?

A second issue is the extent to which divers actually follow the ascent profile prescribed by a given algorithm. In principle, such a database could be filtered such that only ascent profiles that match the prescribed ones (that should have been followed) are retained, but one would need to have the dive log and precise (algorithm) details for every dive in the database.

I’m all for having the ability to pore over hundreds / thousands of dive profiles, but it’s not an easy problem to tackle due to these (and other) issues. Reliability, I think, is the main concern.

Joseph

 

[rjack]

Hi Tony,

A Fisher test is used for associations between two categorical variables; for the use you are describing, which is a complex problem, a different treatment would be appropriate.

The main problem I see with such "user-reported” data is its reliability. Specifically, I see issues with how DCS would be identified if there is no proper, medical examination. Sure, extreme cases are easy to self-diagnose, but what about those cases where divers do not really identify symptoms correctly?

A second issue is the extent to which divers actually follow the ascent profile prescribed by a given algorithm. In principle, such a database could be filtered such that only ascent profiles that match the prescribed ones (that should have been followed) are retained, but one would need to have the dive log and precise (algorithm) details for every dive in the database.

I’m all for having the ability to pore over hundreds / thousands of dive profiles, but it’s not an easy problem to tackle due to these (and other) issues. Reliability, I think, is the main concern.

Joseph

I'm curious why you didn't identify all the other biological variables at play here. Sex, age, BMI, smoking status, prior DCS, other prior injuries, repetitive dive status, hydration etc. I would guess that absent any ability to control for the (unknown or poorly known) variation that these variables introduce, you are going to have a very hard time demonstrating statistical significance attributable to the decompression model selected, unless you somehow found a large number and a high rate of DCS cases in one model that are almost absent from the other. i.e. subtle differences are going to be lost in the noise of other variables. Perhaps something like principle component analysis could reduce the mess of variables to a few vectors. I would be very interested to know if 2 competing deco models were in the same vector or in different vectors.

Nevertheless, having a large number of (successful) dives with two different models does not, by itself, create a lot of statistical power. The more successful dives you have, the smaller the rate becomes, the more individual DCS cases influence that calculated rate, and the harder it becomes to demonstrate that changes in rate are due to changes in the model. Flooding the dataset with successful dives (and a ton of extra variables) is really counterproductive from a power perspective.

And of course until the NEDU study, subtle differences between bubble and dissolved models were actually lost in the noise. You might say they are still a bit lost due to the absence of a two-way test.

Personally I am happy with the one-way test, in combination with all the other lines of evidence, and with the understanding that VPM and buddle models are actually allowing the supersaturation across each compartment to rise to a given "allowable" level. Which leads to an overall higher sum/integral surfacing saturation across all compartments. That total gas burden leading to a slight increase in DCS risk for a given deco time makes biological sense, so I don't need a two-way test as nail-in-the-coffin evidence. Especially since it would be unethical to hurt people to prove it that way.

 

[UWSojourner]

Personally I am happy with the one-way test, in combination with all the other lines of evidence

I feel the same way. Even if you used a 2-sided test, there is only about a 1 in 11 chance that the results were a statistical fluke and the DCS rates from the two profiles were in fact the same.

Assuming the NEDU study was just a statistical fluke, you would expect contrary information to be coming out from other research. But as has been pointed out by Dr. Mitchell (see this post), so far all credible research is pointing in the same direction as the NEDU trials. So the odds of a statistical fluke have narrowed considerably I think. I'd far rather have one study at 91% confidence and several others confirming the general findings of the study than have a single study at 95% confidence awaiting confirming evidence.

 

[emmbee]

Hi,

Nice write up.

A few random thoughts...

I'm not mad keen on the way you're rewriting the objective of the tests into null hypothesis and alternative hypothesis. It's not what the test actually does. And it confuses things. For instance you say, for the midpoint analysis, that the null hypothesis is "A deep stops profile is as efficient as, or more efficient than, a shallow stops profile" and the alternative "A deep stops profile is less efficient than a shallow stops profile". If the hypotheses are mutually exclusive, the p-value is either 0 or 1. But it's not. In fact, what you're saying is a much (much!) stronger proposition: one of them is correct and the other one is wrong. But a Fisher test can not do that.

While the two-tailed test is required in most cases, there is a class of problems where a one-tailed test is perfectly appropriate: when, should the null hypothesis be correct and there is no difference between the two options, we'd still be fine endorsing the one that gave the better result during the experiment. Basically, when we just need it to not be worse than the other. In this case, a one-tailed test is sufficient. For instance if a medicine was cheaper than the other and _appears_ more effective, it doesn't matter if the null hypothesis is correct and they're both as effective. This is the case here. All else being equal, the profile that gets the diver out of the water faster, and using less gas to boot, is the better one. So I have no problem with the use of a one-tailed test.

Last, I'm not convinced by the case you're making about the two-tailed test, for two reasons:

One, when you reject an alternative hypothesis as "not statistically significant", you in effect assert that there is evidence supporting the null hypothesis at the significance level. Usually this is the case. In particle physics, there's the standard model, supported by a mountain of sigma 3+ experiments. If it predicts nothing of note should happen in the experiment, dismissing a sigma 2 oddity is reasonable. In medicine, the default impact of swallowing something is: nothing happens. The standard there is 0.05. In the present case, however, we don't have that, not that I know of. I'm not aware of anything supporting the idea that the null hypothesis is a valid default for this experiment at any significance level.

In fact, and that's point two, we know that the null hypothesis is wrong. We know that the dive profile impacts DCS. At what significance level, I don't know, but the null hypothesis itself is flaky at best. It's used because that's what the Fisher test demands, but it's important to note that it's not, on the face of it, a reasonable explanation. To me this justifies a higher statistical significance level, and I'm happy with 8.7%.

Cheers,

Matthieu

 

[Joseph]

Hi rjack,

I'm curious why you didn't identify all the other biological variables at play here. Sex, age, BMI, smoking status, prior DCS, other prior injuries, repetitive dive status, hydration etc.

Merely because the variables are indeed many, and the principal point of my previous post was not an attempt to list them all

I would guess that absent any ability to control for the (unknown or poorly known) variation that these variables introduce, you are going to have a very hard time demonstrating statistical significance attributable to the decompression model selected,

I agree that it is not an easy problem. However, that’s where more sophisticated analysis comes in.

Personally I am happy with the one-way test, in combination with all the other lines of evidence

Well, as I say in the article, I myself have converged to Bühlmann ZHL-16C + GFs. This article is simply spelling out aspects of statistics (pertaining to this study) that are not necessarily discussed often, or are a matter of some confusion.

Joseph

 

[Joseph]

Hi UWSojourner,

Assuming the NEDU study was just a statistical fluke, you would expect contrary information to be coming out from other research. But as has been pointed out by Dr. Mitchell (see this post), so far all credible research is pointing in the same direction as the NEDU trials.

Yes, I referred to this in a previous post. The results would indeed seem to point the same way. One should also note that the experiments are not directly comparable; as I said in my previous message, the NEDU study had clinical DCS as its endpoint whereas the others did not. Indeed, this is what makes the NEDU study unique.

I'd far rather have one study at 91% confidence and several others confirming the general findings of the study than have a single study at 95% confidence awaiting confirming evidence.

The present article merely points out what the NEDU study finds; it’s a discussion of that study. (In other words, its point was not to delve into each and every other study... that would make for quite a long article, and I believe the present one is long enough already )

Joseph

 

[Joseph]

Hi Matthieu,

Nice write up.

Thanks

Hi,
A few random thoughts...

I'm not mad keen on the way you're rewriting the objective of the tests into null hypothesis and alternative hypothesis. It's not what the test actually does.

It most certainly is what it does; the Fisher test (an exact test) is based on null hypothesis testing; it’s not a matter of rewriting anything at all but merely stating the basic foundation to which the test itself relates.

For instance you say, for the midpoint analysis, that the null hypothesis is "A deep stops profile is as efficient as, or more efficient than, a shallow stops profile" and the alternative "A deep stops profile is less efficient than a shallow stops profile".

Yes, that is correct.

If the hypotheses are mutually exclusive, the p-value is either 0 or 1. But it's not. In fact, what you're saying is a much (much!) stronger proposition: one of them is correct and the other one is wrong. But a Fisher test can not do that.

I think you’re a bit confused about the premises of null hypothesis testing. The p-value can assume any value between 0 and 1. In null-hypothesis testing, you try and reject the null hypothesis (which states there is no relationship between two given phenomena), and in the process, if you do reject it (by finding a p-value less than an adopted threshold), then you've effectively garnered support for the alternative (which obviously can’t be the same as the null by definition).

While the two-tailed test is required in most cases, there is a class of problems where a one-tailed test is perfectly appropriate:

Sure - and furthermore, note what I said in the article itself (I quote): "Given the context within which the study was carried out, namely the fact that the U.S. Navy would depart from continuing to use shallow-stop profiles ONLY in case of the “finding of significantly lower PDCS [probability of DCS] for the bubble model schedule [deep-stops profiles] than the gas content model schedule [shallow-stops profile]” the choice of a one-sided test is appropriate, as the authors point out. However, one has to keep this context and associated implications in mind when interpreting this study."

So I think you missed the point here.

One, when you reject an alternative hypothesis as "not statistically significant", you in effect assert that there is evidence supporting the null hypothesis at the significance level.

Again, there is some confusion here which seems to point to a misunderstanding of (a) the premises of null hypothesis testing, and (b) what this kind of testing does. We do not set out to “reject an alternative hypothesis”; that’s not at all what we do in such testing. We strive to reject the null hypothesis, and in rejecting it we build support for the alternative; with a statistical test (such as the Fisher test) we calculate a p-value that we want to be lower than an adopted threshold we’re happy with (e.g. 0.05). All we can say is that we can reject the null hypothesis (and therefore have support for the alternative hypothesis) if the p-value < 0.05. If the p-value is >/=0.05, we do not have a statistically significant result and we cannot reject the null hypothesis. That is all we can say. Reframing this as a process whereby we try to reject the alternative hypothesis is incorrect, and a fairly common mistake.

In medicine, the default impact of swallowing something is: nothing happens. The standard there is 0.05. In the present case, however, we don't have that, not that I know of.

0.05 is an adopted threshold (which the p-value has to be smaller than). Once you adopt that threshold (as is done here) you simply want your p-value to be smaller than it. The third sentence above is unclear.

I'm not aware of anything supporting the idea that the null hypothesis is a valid default for this experiment at any significance level.

Anything that you do with the type of testing we’re talking about here relates back to the null hypothesis, it's the very backbone.

In fact, and that's point two, we know that the null hypothesis is wrong. We know that the dive profile impacts DCS. At what significance level, I don't know, but the null hypothesis itself is flaky at best. It's used because that's what the Fisher test demands, but it's important to note that it's not, on the face of it, a reasonable explanation. To me this justifies a higher statistical significance level, and I'm happy with 8.7%.

You never know "that the null hypothesis is wrong” before you collect data. If you did know, then there would be no need for the experiment to begin with. The whole point is to collect data and try to reject the null hypothesis.

Hope that clears up some of the confusion.

Joseph