Three thoughts. First, I totally agree with the idea of training for reviewers. The challenge is that this demands reviewers take the time to do so and some will balk at that. In the geosciences we get some review instructions with the manuscript that provide guidelines. I do wonder how many reviewers read them carefully. Second, as both a recipient and provider of reviews I have often wished for the opportunity to provide or receive feedback. As an author I have found some reviews are a blessing and others best ignored. Feedback to reviewers from both authors and editors would also go a long way to training reviewers. Sometimes I would really like to more explicitly thank a reviewer. for their insights. I suspect those reviewers who would object to getting feedback might self-select out which might not be all bad. I sometimes think I have provided a particularly helpful review, but how am I to know? Third and perhaps more philosophic - I would argue there is a continuum between objectivity and subjectivity that it would be helpful for all to be reminded of. Science can't be purely objective, but it can be more more objective than alternatives, which is useful and part of the tremendous success it has had in understanding the world around us. Thanks so much for your post. It is especially nice to hear it from an editor's perspective.
Thank you for sharing your thoughts! By peer reviewer training I mean early training as a grad student and postdoc. Perhaps a way to get a certificate based on completing a course or workshop. What we have now is learning by reading reviews from other peers. We either replicate bad habits or decide to do better, but it is not a good system and perpetuates bad behavior.
Journals like Frontiers enable the exchanges between reviewers and authors that you describe. Some people like that, but I don’t really. It is just too much work to go through all of this back and forth. Reviewers are not coauthors, and I think it is too much to ask to put in so much work to improve the publications of others. Some people feel differently and are willing to engage in that process. Good for them!
I don’t think there can be a continuum between objectivity and subjectivity. Those are binary theoretical concepts. In practice, it is almost impossible to take a completely objective measurement of anything. And you are right that any scientific publication can vary from purely subjective to containing a lot of objective information. So, yes, a continuum, but that’s because there are all these layers of subjectivity around the kernels of objectivity without which it is not science at all.
Also I have never understood those scientists who don't like to celebrate other scientists success stories. I am all for celebration, even if its someone who scoped me, as that indicates that I am on the right track.
Second also, I think you have a good point with incorporate learning how to do a good peer review in scientific training. Which I should also include how to write a constructive peer review report, after all, peer review is there to make the manuscript better.
Thanks! In my experience, graduate students get excited when reading about new breakthroughs, but older professors tend to wag their heads and mutter about "extraordinary findings". It's not a common character trait to want to celebrate other people's success, in science or elsewhere. Fortunately, there are a few people like you!
Thanks so much for this thoughtful essay about the best/worst way to keep published science on track. As somebody (too?) deeply embedded in that process of peer-review myself, I can only fully agree. Agreeing with one nagging thought, though, can I open a can of worms, please? You wrote a particularly interesting sentence: “Generative AI is trained on online and published content, so it reflects all the human bias in the world.” Asking for a good friend (of course), let me pose this. What if we combine all the human biases? Do we get objectivity? As much as all the different colors of light combine to bright, clear white light? Will generative AI eventually become the ultimate judge of objectivity? I told my (imaginary) good friend, who posed this question in the first place, not to ask such things, but as a good (imaginary good?) friend, here I do.
Whow, that is a great question! I don’t like it when people say that, but this time I mean it. Wouldn’t you end up with the world’s universal bias if you combined all human expression and knowledge in AI? It’s like mixing all the colors of the rainbow, but in different proportions, which won’t give you white.
You can clearly train AI to recognize and reduce bias. For example, https://www.latimer.ai/ is trained that way to reduce cultural bias. And Grok can recognize bias and faithfully spout it out to users. So, perhaps one could train AI to eliminate all known bias and only leave the bias that we are not yet aware of. But then how do you eliminate that? Perhaps ask AI to identify unknown bias? I should try that and see if Claude melts down while searching for something that exists but that nobody is aware of.
We have to peer review science before we can select for the systems that have the potential for comprehension. Right now, science has a mechanistic foundation, so there is no reason to find anything non mechanical down the road. So, peer review for plant science or anything else having to do with Life is just a practice of diminishing returns. Thank you.
You are right about the limitations of science, but I wouldn’t call it a mechanistic foundation. A lot of science documents mechanisms, causes and effects, but that is not the foundation of science. Science is rooted in objectivity. Something that can’t be measured or documented objectively may still exist but it is outside the realm of science. Consciousness may be an example for that, and there is a hot debate about whether or not science can potentially explain it.
I advocated in previous posts for using approaches other than science to understand the world, but we can’t change science toward being something other than a search for objective truth. Because then it is not science anymore.
There are other ways to understand life, and anyone is free to pursue those ways while the scientists do their thing. Science has an amazing track record so far, despite all its limitations. That does not mean that everybody must be a scientist. It would be a terrible world if we couldn’t explore the world in other ways and were all forced to be objective all the time.
Thank you. I understand exactly what you're saying and I see where you are coming from. I don't agree though, as I don't confine science the the scientific method or purely objectivist views. But yes, that is what I mean by the mechanistic foundation. So, I think we actually, probably, agree about what we want, but we see the situation a bit differently. In our efforts to understand each other, we are doing science. And that, to me, is the most important thing: that we don't silo science into institutional-only knowledge, which is a very dangerous proposition.
This article was pia-reviewed ✅️
Thanks for peering into it!
Three thoughts. First, I totally agree with the idea of training for reviewers. The challenge is that this demands reviewers take the time to do so and some will balk at that. In the geosciences we get some review instructions with the manuscript that provide guidelines. I do wonder how many reviewers read them carefully. Second, as both a recipient and provider of reviews I have often wished for the opportunity to provide or receive feedback. As an author I have found some reviews are a blessing and others best ignored. Feedback to reviewers from both authors and editors would also go a long way to training reviewers. Sometimes I would really like to more explicitly thank a reviewer. for their insights. I suspect those reviewers who would object to getting feedback might self-select out which might not be all bad. I sometimes think I have provided a particularly helpful review, but how am I to know? Third and perhaps more philosophic - I would argue there is a continuum between objectivity and subjectivity that it would be helpful for all to be reminded of. Science can't be purely objective, but it can be more more objective than alternatives, which is useful and part of the tremendous success it has had in understanding the world around us. Thanks so much for your post. It is especially nice to hear it from an editor's perspective.
Thank you for sharing your thoughts! By peer reviewer training I mean early training as a grad student and postdoc. Perhaps a way to get a certificate based on completing a course or workshop. What we have now is learning by reading reviews from other peers. We either replicate bad habits or decide to do better, but it is not a good system and perpetuates bad behavior.
Journals like Frontiers enable the exchanges between reviewers and authors that you describe. Some people like that, but I don’t really. It is just too much work to go through all of this back and forth. Reviewers are not coauthors, and I think it is too much to ask to put in so much work to improve the publications of others. Some people feel differently and are willing to engage in that process. Good for them!
I don’t think there can be a continuum between objectivity and subjectivity. Those are binary theoretical concepts. In practice, it is almost impossible to take a completely objective measurement of anything. And you are right that any scientific publication can vary from purely subjective to containing a lot of objective information. So, yes, a continuum, but that’s because there are all these layers of subjectivity around the kernels of objectivity without which it is not science at all.
Nice peak behind the curtain of peer review.
Also I have never understood those scientists who don't like to celebrate other scientists success stories. I am all for celebration, even if its someone who scoped me, as that indicates that I am on the right track.
Second also, I think you have a good point with incorporate learning how to do a good peer review in scientific training. Which I should also include how to write a constructive peer review report, after all, peer review is there to make the manuscript better.
Thanks! In my experience, graduate students get excited when reading about new breakthroughs, but older professors tend to wag their heads and mutter about "extraordinary findings". It's not a common character trait to want to celebrate other people's success, in science or elsewhere. Fortunately, there are a few people like you!
Thanks so much for this thoughtful essay about the best/worst way to keep published science on track. As somebody (too?) deeply embedded in that process of peer-review myself, I can only fully agree. Agreeing with one nagging thought, though, can I open a can of worms, please? You wrote a particularly interesting sentence: “Generative AI is trained on online and published content, so it reflects all the human bias in the world.” Asking for a good friend (of course), let me pose this. What if we combine all the human biases? Do we get objectivity? As much as all the different colors of light combine to bright, clear white light? Will generative AI eventually become the ultimate judge of objectivity? I told my (imaginary) good friend, who posed this question in the first place, not to ask such things, but as a good (imaginary good?) friend, here I do.
Whow, that is a great question! I don’t like it when people say that, but this time I mean it. Wouldn’t you end up with the world’s universal bias if you combined all human expression and knowledge in AI? It’s like mixing all the colors of the rainbow, but in different proportions, which won’t give you white.
You can clearly train AI to recognize and reduce bias. For example, https://www.latimer.ai/ is trained that way to reduce cultural bias. And Grok can recognize bias and faithfully spout it out to users. So, perhaps one could train AI to eliminate all known bias and only leave the bias that we are not yet aware of. But then how do you eliminate that? Perhaps ask AI to identify unknown bias? I should try that and see if Claude melts down while searching for something that exists but that nobody is aware of.
We have to peer review science before we can select for the systems that have the potential for comprehension. Right now, science has a mechanistic foundation, so there is no reason to find anything non mechanical down the road. So, peer review for plant science or anything else having to do with Life is just a practice of diminishing returns. Thank you.
You are right about the limitations of science, but I wouldn’t call it a mechanistic foundation. A lot of science documents mechanisms, causes and effects, but that is not the foundation of science. Science is rooted in objectivity. Something that can’t be measured or documented objectively may still exist but it is outside the realm of science. Consciousness may be an example for that, and there is a hot debate about whether or not science can potentially explain it.
I advocated in previous posts for using approaches other than science to understand the world, but we can’t change science toward being something other than a search for objective truth. Because then it is not science anymore.
There are other ways to understand life, and anyone is free to pursue those ways while the scientists do their thing. Science has an amazing track record so far, despite all its limitations. That does not mean that everybody must be a scientist. It would be a terrible world if we couldn’t explore the world in other ways and were all forced to be objective all the time.
Thank you. I understand exactly what you're saying and I see where you are coming from. I don't agree though, as I don't confine science the the scientific method or purely objectivist views. But yes, that is what I mean by the mechanistic foundation. So, I think we actually, probably, agree about what we want, but we see the situation a bit differently. In our efforts to understand each other, we are doing science. And that, to me, is the most important thing: that we don't silo science into institutional-only knowledge, which is a very dangerous proposition.