6

I have been conducting an exploratory research study and am beginning to analyze the data for it. This study is quantative and is not predominantly hypothesis-driven, however there are a few questions the collaborators want to investigate - as there has to be a starting point for analysis.

Most of the research questions and trends in the data appear to be null findings. To be clear, I am not suggesting this study should be framed as supporting a "null hypothesis", since there are no robust and testable predictions to be made. This is non-experimental work which does not involve manipulation of any variables. Rather, it is an observational design.

Nevertheless, I am having a hard time finding patterns in the inferential analysis of this study. There are a couple of descriptive findings that could be reported, however this does not reveal many meaningful insights into the trends in the results. While there are several variables that can be examined, I am left primarily with null findings.

Since exploratory and observational work fundamentally differs in terms of how flexibly results can be disseminated, is it advisable to simply report null findings? The eventual goal would be to publish this work and it could provide guidance for future studies - as little prior work has been done on this subject. I would hate for this all to go to waste, so would like some insight on how to proceed with this matter.

Improve this question
2
  • 2
    A null result can mean either of two things: 1) The effect you thought is there isn't 2) There isn't enough data to tell what's going on. To disambiguate between the two, build confidence intervals and then ask yourself: if the truth is at one or the other ends of this interval, would that be substantively important?
    – Nathan Wycoff
    Commented Jan 19 at 22:38
  • There is definitely enough data at this point and nearing target sample will be thousands. There is at least a handful of effects that is there that isn't. However not all variables have been explored independently in much detail. There were correlations made for each alomg with confidence intervals. I am not sure which variables confidence intervals are worth looking into at this point.
    – Neurostar
    Commented Jan 20 at 15:46

3 Answers 3

Reset to default
5

As stated, a null result (i.e., p>0.05) doesn't mean the null hypothesis is correct, it just means there isn't enough data or evidence to reject it.

Publishing negative results remains tricky. In my experience it depends on the context of the negative result you are publishing. If the goal is to show there isn't a correlation between things you have good reason to expect a correlation between, it might be okay, especially if it's in a story where you do have some interesting significant results. But reviewers may question the experimental design, saying you were asking the wrong questions.

Otherwise, there is always the preprint route, leaving a collection of negative results out there but non-peer reviewed. This seems to be becoming more common, and is especially useful for PhD candidates or early career researchers.

Improve this answer
4
  • Thank you and good point. In some cases, null findings are useful for overturning previous held theories or findings with that as the main focus. However, since this is an observational study that has very limited work, I think it is necessary to show some sort of effect. But I am interested in the 'preprint route' you mentioned. Can you expand more on this? Is the suggestion not to publish in a journal?
    – Neurostar
    Commented Jan 20 at 15:39
  • 1
    The preprint route shouldn't be the goal. It can be useful in getting results out there that are either not worth the effort to polish to publication level, or if you need to show what you've been up to for future labs. Whilst preprints are starting to get cited more frequently, it's still up for debate on when and whether this is appropriate.
    – dthorbur
    Commented Jan 20 at 16:14
  • 1
    Null findings don't necessarily overturn previously held theories. You'd have to convincingly show either that result is not replicable (null results would be appropriate here), or show an alternative result that is significant and disputes said theory (null results wouldn't be enough). But again, it can be more nuanced if you are publishing a story that has implications on a theory but doesn't directly contradict it.
    – dthorbur
    Commented Jan 20 at 16:16
  • Finding gaps in previously held theories is usually the starting point for beginning to formulate alternative studies that support competing theories. Regardless, this study is mostly exploratory and the literature is quite scant in this research, so do not think it would be that revolutionary. I am a big fan of preprints and have gained a lot of traction with this. However, I do not see that as the main goal. It could end up suggesting a file drawer problem.
    – Neurostar
    Commented Jan 21 at 8:35
1

When you publish an exploratory analysis of an observational dataset, it can be important to talk about the analyses you have conducted, and those that did not reject the null it for no other reason than if you asked a question of the dataset, your readers are also likely to have that same question.

You might phrase that "We wondered whether X was true, but on conducting test Y with the dataset could find no support for this hypothesis. We acknowledge that a failure to reject the null hypothesis is not evidence in favour of the null [and indeed, the data trends towards supporting the hypothesis, even if not significantly] or [however, the data does not even trend in that direction irrespective of significance levels](Supplementary figure 50b)".

However, you probably wouldn't write an entire paper around this. Ideally you would publish some positive findings, and just mention these things in passing. You you could sell this as a data paper, where the real real value is in providing the dataset to other researchers (this might be more or less doable, depending on your field).

Improve this answer
1
  • That is an interesting thought. Examining all variables and simply reporting their trends regardless of whether it attained significance. I am not sure about the data paper part, maybe you can expand on that or provide some examples of papers like that. However, if I were to report all null findings is it even worth conducting the analyses to begin with? There are at least a handful of unimportant findings that are non-significant Maybe there is a different criteria for what should be included in an exploratory paper other than positive vs. null results.
    – Neurostar
    Commented Jan 22 at 7:06
1

Short answer: Yes.

Longer answer: As a statistician who sometimes reviews papers, I think it is very important to report findings that are not significant (and I would use that phrase rather than "null").

It is also important for journals to publish such findings.

One big reason for this is what is known as the "file drawer problem" or "publication bias". Briefly, this means that, if only significant results are published, readers in the field will get a false idea of what is actually going on and anyone who attempts a meta-analysis will have biased input (in fact, part of a good meta-analysis is trying to assess and maybe adjust for this.

For more see e.g. Wikipedia

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.