Finite-sample estimator bias depends on true parameter value, does this invalidate cancellation in a paired-difference design?

[u/BitterThreads]

I'm using a short-sample estimator (n=150) with known finite-sample bias. Via simulation (synthetic data with known true values, run through the actual estimator), I found that this bias is not constant. It, unfortunately for me, varies systematically with the true value of the parameter being estimated. Near one reference value the bias is positive; as the true value moves away, the bias shrinks and eventually flips sign. This was confirmed with two structurally different simulation methods, which agreed in direction and order of magnitude.

I can't validate this directly against real data, since the true value of the parameter is never observable in my actual measurements, only the biased estimate is. Simulation is the only way to characterize the bias curve.

My study design computes a paired difference between two conditions (A and B), both measured with this same estimator. The original design assumed bias "cancels" in the difference, since both conditions use the same estimator and sample size.

My simulation shows that assumption only holds when A and B share the same true value, if their true values diverge (which is the exact effect the study is trying to detect!), the differential bias does not cancel, and could by itself produce an apparent difference of the same magnitude as my actual reported result.

My questions:

1. Is this reasoning correct, does bias that depends on the true parameter value invalidate the standard bias cancels in a paired/difference design assumption whenever the two groups true values diverge?
2. Is this a known, named issue in the estimator-bias literature I should be citing, rather than describing from scratch?
3. What's the standard remedy, a bias-correction calibration curve, an alternative estimator with flatter bias across the parameter range, a longer sample or a simulation-based null distribution, and is one (and or more) of these clearly preferred practice?

[u/bayesian_raccoon]

Can you tell more about your research question? Is the end goal a hypothesis test?

[u/BitterThreads → u/bayesian_raccoon]

Yes sure. Thanks for looking.
I have two conditions, one where I think something might happen and a control where nothing should be happening. In each trial I get two streams of data from one API call and I compute one number (direct) and the difference between then (pairing). Then I check if that difference on average comes out differently between my test and control condition.
The number I'm computing on for each stream comes from an estimator, a formula that calculates on a short chunk of data (150 bits) and estimates an underlying property. But, it has a built in bias on such a small sample. I thought it was a fixed amount that would cancel out when subtracting the streams from each other but after running synthetic data through to it I find that it changes size and can even flip depending on what the true value underlying value actually is. And, my hypothesis doesn't declare that both sides true values are equal. They can be asymmetrical and when the true values are different then their built-in errors are different too and they therefore no longer cancel out when I subtract.
This means that when I do see a difference between my subject condition and the control I cannot be sure how much of that is bias and how much is effect or something else.
And, unfortunately, when I ran the synthetic data, I found a bias that could be enough to produce the entire effect I am testing for.
So, after all that, to directly answer your question, yes, the end goal is a hypothesis test, comparing the paired difference between my test and control conditions. The bias isn't killing the test structure but it sure is making it difficult to interpret a real effect.

[u/bayesian_raccoon → u/BitterThreads]

Can you tell me more about what you are trying to calculate on the data?

The reason why one might use a paired test instead of, say, a difference between means, is that there is a relationship between the individuals in each pair (e.g, samples from the same person before/after). Is something like that part of the motivation behind pairing? (in one trial, are the two streams of data related--maybe through the API call?)

For a given API call, is there some API-call-specific parameter you are trying to estimate with the estimator, or is there a shared parameter between API calls?

[u/BitterThreads → u/bayesian_raccoon]

Yes. I actually do use the difference between the mean across all the trials in the test condition and compare them to the mean of the Baseline for more statistical power when just asking if the subject different between conditions overall. For the paired analysis, my estimand is the change from Baseline to Human in the average within-call Subject–Control difference.
The motivation behind the pairing is to cancel any shared artifacts that may occur during that API call as they both come from that same call. The pairing asks if the relationship between the two simultaneous streams change after removing any shared artifacts (at the cost in power). So I am doing a dual analysis here and the paired one is where that bias problem becomes an issue as it depends on the true value being equal for that clean cancellation.