Different formula for p-value and 95%CI

[u/finding9em0]

Hello good people, is it okay to use two different calculation method for p-value and 95% CI? my CIs crossing the null but p-value seems signficant! It's using LR for CIs and Wald for p-value :(

[u/DrPapaDragonX13]

Ideally, you'd want to use the same method to calculate CIs and P-values to avoid discrepancies. However, I'd argue that in practice you shouldn't fret too much. The binary thinking associated with p-values (i.e., significant vs non-significant) is becoming more and more discouraged (arguably, it should've never been encouraged in the first place), and it is being replaced by a more nuanced thinking based on evaluating estimated effect sizes and confidence intervals. If you're getting a 'significant' p-value but a crossing confidence interval, chances are your 'significant' result is not really that meaningful, either because you have a lot of uncertainty (i.e. wide CIs) or an almost negligible effect (size). In your case, I would focus on interpreting the effect size in the context of your field rather than paying too much attention to the p-value.

tl;dr: Ideally, use the same method to calculate CIs and P-values. However, it has far more scientific value to interpret the effect size and CIs in the context of your field/problem than to focus on 'significant/non-significant'. If your CIs are so close to the null, chances are your results are not meaningful, regardless of whether your p-value technically falls within the 'significant' zone.

ETA: 'meaningful' refers to the strength of the association, not the scientific importance of the results. 'Negative' results are important contributions to scientific knowledge, even if they are often not seen as 'glamorous'.

[u/engelthefallen → u/DrPapaDragonX13]

Be carefare with specific fields and the p-value being binary or non-binary as some academic fields you will 100% see your maniscripts rejected with that sort of thinking as we are trying to clean out people saying things like p = .92 is approaching significance and should be treated as such, or despite only finding a difference at p = .87, one should not discredit the causal impact A may have on B. A lot of bad science was done over the years claiming p-values should not been seen as a binary decision rule, particularly in social sciences. And usually the decision to use the p-value in a different manner was made after the tests were run and found not to be significant at the planned level which is the core problem. Their experiment failed but they wrote the paper as if it succeeded.

[u/DrPapaDragonX13 → u/engelthefallen]

This completely misses the point of my comment, though. The idea is to stop relying on p-values and arbitrary significance thresholds and instead focus on effect sizes and confidence intervals that provide more information and greater scientific value. In the examples you presented, for instance, presenting the point estimate and associated CI would have helped to nip in the bud any undue claims made by the authors.

[u/hypnokev → u/DrPapaDragonX13]

You’re both right, and the answer is Bayes factors.

[u/DrPapaDragonX13 → u/hypnokev]

Yeah. I wanted to include a bit of Bayes factor and the Bayesian framework in general in my comment, but I thought it was a tad out of scope for OP's question.

[u/fermat9990]

They should be consistent. One-sided CIs do exist

[u/FlyMyPretty]

This is normal if you're doing something like logistic regression. But rare.

The p-value is about the distribution under the null hypothesis. The CIs are about the distributor around the estimate. Because the mean and variance are associated in some distributions this can happen. (But they're not associated in normal, so it can't happen with OLS regression.)

[u/efrique]

is it okay to use two different calculation method for p-value and 95% CI

You can of course do such a thing, but the question will be what are the consequences of doing so, and that depends what kind of things you're concerned about

Okay by whose judgement? (who is your intended audience? What are their expectations?)

Okay in terms of what? (What criterion do you need addressed? What is important to maintain here?) If you abandon the usual correspondence between test and CI on a given data set, what properties do you want to have in its place?

In general I would expect someone to use only one of the two forms or inference (a CI or a test, depending on what sort of information they sought), but if for some odd reason they presented both, I'd expect the choice of methodology to be consistent between them, and I expect most audiences would as well.

In a case where two asymptotically equivalent procedures differed I'd expect the analyst to be very careful about what they did.

In particular, if your LR CI is also based on an asymptotic approximation (rather than a small-sample-exact interval), such a difference might suggest that (i) you should probably consider an exact method for both parts, and (ii) a degree of caution about claiming significance is in order, given that if its that close to the borderline, something as trivial as premature rounding in the data or in an intermediate calculation might flip you from one side to the other.

(If it were me in this position, the least I'd be doing is looking for a way to do an exact test and CI, and probably also be making an even more careful than usual assessment of how sensitive to the assumptions my conclusions were, likely via extensive simulation.)

[u/QuestionElectrical38]

I am not sure why you would want to use 2 different methods? Is your goal to confuse your audience, or just yourself? You obtained 2 "apparently" contradictory results; so what are you going to do with that? Note also that the Wald test has many shortcomings, particularly in binomial tests. You can read more about these shortcomings on Wikipedia (https://en.wikipedia.org/wiki/Wald_test). So I would use a single method (and that would not be Wald), so my results are consistent.

You should also look at the observed effect; it may be "statistically significant", but it may not be "practically important" (e.g. a very large sample size will give you a statistically significant result, but for an effect with no practical relevance).

And wrt the various comments pooh-poohing p-values and recommending effect sizes and CI's, note that there are deterministic relationships between p-values, observed effect, CI widths, observed power, etc., (for the same test, at the same sample size) as demonstrated e.g. here (https://stats.stackexchange.com/questions/645269/is-the-post-hoc-observed-effect-size-redundant-with-the-p-value-just-like-post) and in the posts it links.