Hello,
Today I'd like to discuss responsible science. Once one a while, we can notice a piece of news discussing an issue related to the irrelevant output of a bunch of scientific papers. Even though they are peer-reviewed.
However, only a few of them have a reviewer who is willing to re-do all the described tests. It is pretty straight forward: someone else's experiment for the second, third, or fourth time isn't nearly as exciting as running your research for the first time, but studies like this are showing us why we can no longer avoid it.
What is your opinion about the peer review process in science?
https://science.howstuffworks.com/innovation/science-questions/database-18000-retracted-scientific-papers-now-online.htm
https://www.blog.pythonlibrary.org/2019/10/13/thousands-of-scientific-papers-may-be-invalid-due-to-misunderstanding-python/
https://www.sciencealert.com/a-bug-in-fmri-software-could-invalidate-decades-of-brain-research-scientists-discover
Interesting! You just have to remember that only the one who does nothing is wrong. Something that seemed real yesterday today may be fiction. It is unfortunate that, for example, 15 years of brain research may be unnecessary, and on the other hand, maybe these 15 years of research led to the discovery that something is wrong. Discovering an error is also a finding. It's good that databases with erroneous discoveries are created, because we can verify something in them. The only question is whether discovering the error is true or not? The paradox of the liars quoted in E = MC ^ 2 is recalled. If a liar lies telling the truth or not? Is the discovery that the previous one is wrong true or not? Who and how will confirm the thesis denied? ...
ReplyDeleteThank you for raising such an interesting topic. Honestly, I never thought that so many articles could be questioned or disproved because of a bug in the program or not understanding the library. This shows how important a critical approach to our own research and what we publish is. Sometimes you need to check or repeat something ten times to make sure that our results are correct. Even so, someone may accuse us of error or misinterpretation. In my opinion, the fact that every article goes through the hands of several reviewers is very good. Unfortunately, as it was written in the cited articles, not everyone has time to thoroughly analyze someone's results. I have heard about situations where work is on someone's desk for months before reading and reviewing it. I have also heard of cases when someone decides to verify someone's article and during their research receives completely different results. It happens that scientists sometimes withhold certain information to not lose a research grant when it turns out that the tested material is unstable under typical conditions. Sometimes, someone's work only goes through because someone has a known name in the environment. It's hard to say what it really looks like. Do some reviewers not want to delve into a given work? Don't they have time to check everything carefully? Or maybe the topic is so easily published that everything that is new easily passes through the hands of reviewers? To undermine someone's research, you need a lot of evidence, compare the results with other publications, often you just don't have the energy or the inclination to do so. However, I am happy that there are inquisitive people and even if they reject someone's work several times pointing out their mistakes, improving it we can gain experience and confidence that our work will definitely have a better contribution to learning.
ReplyDeleteThat's true that not everyone has time to analyze someone's results thoroughly. In the end, lack of the same results could expose the reviewer to be incompetent in the field (even if it's far away from the truth), so nobody wants to be perceived in an incompetent way.
DeleteThanks to the author for the article.
ReplyDeleteToday, information has become several times more, the necessary information and unnecessary. Because of the "information age", a lot of scientific papers have also appeared, with varying degrees of importance. But this does not mean that this information is unnecessary, perhaps some article may prompt a potential scientist to discover something. I think it’s very difficult to filter scientific papers, because of subjectivity, because it’s difficult to develop uniform standards. I think scientifics works should not be deleted, but we need to develop a good method for filtering and sorting scientific papers .
In my opinion the peer review process in science is very complicated and I suppose that I do not have enough knowledge about it. Anyway, the review process depends on the reviewer. In some cases two reviewers can make totally different reviews on the same article, first of the reviews being positive, while the other claiming, that the article is of zero scientific value. I understand that two persons can have different point of view but I think that the mentioned situation cannot appear.
ReplyDeleteI think that it is very difficult to do good review. I mean that it requires a lot of time because if somebody writes in the article that something was based on something and gives some references to improve the conclusions, the reviewers often does not check what the references are. One can find examples of the articles being good at technical point of view published in renowned journals but not providing any serious scientific impact.
The situations similar to the one described in one of your recommended articles happen sometimes, when the presented research only repeats some scientific works conducted in the past or the results can contain errors. The scientists as human beings also make mistakes sometimes.
The process of verifying the results contained in the article is definitely needed. However, it is not so simple. We need experts in every field who can be very narrow, who will devote their valuable time to checking the work of others. How it works at this point shows that the results are not checked in depth. The articles that you quoted are a great example of how a misuse of a function or error in software can remain unnoticed for many years. In my work I cooperate with biologists and unfortunately they are often unable to obtain identical results by repeating the experiments from the articles step by step. I think it would be easier in computer science, but we have to remember how many fields of research science has.
ReplyDeleteRecently I read a very interesting article about repeating well-known scientific experiments. This problem is especially visible in complex research. Where the results are affected by many variables. In addition, these factors should be taken into account when conducting an experiment. The more dependent factors influencing the results of the study, the greater the discrepancy in the results of the study, even when the protocol regime is maintained. This effect is particularly visible in social or medical research, and especially in psychological research. There has now been a so-called replication crisis. It is a problem of the repetition of experimental results: will the results obtained by one group of scientists also be repeated to other psychological teams? If not, this may mean that the effect announced does not exist at all and was the result of a case or methodological error, and in extreme situations even deception. Recently there has been an attempt to repeat 28 high-profile psychological tests. Each of them was attended by more than 7,000 people from different cultures (the original ones were on average 112 people). Similar results, although slightly weaker than the original ones, were obtained in only half of the experiments. And what is even more ridiculous, such research was also repeated. Because in the next one we managed to replicate only 36 out of 100 experiments. So even repeating the comparison gave different results. In 2016, the scientific weekly "Nature" asked more than 1,500 scientists whether they had failed in trying to repeat an experiment conducted by another scientist and by themselves. The results differed according to the field the researchers represented. The most reproducible results were in mathematics and chemistry, followed by physics and finally psychology.
ReplyDeleteIn my opinion, the problem of repetitiveness in computer science is marginal. Something just works or doesn't work. But only and exclusively with the methodological regime in place. And in my opinion only it should be evaluated. If we examine, for example, the effectiveness of a classifier, we can examine it in an infinite number of ways. But in such a case, the data on the basis of which we test them may be a problem. In such a case, it is necessary to pay attention to which measurement tool they were taken from and who or what was tested.
Finally. On the basis of my experience I claim that often reviewers have no idea about the subject they have to review. Additionally, they often do not read their papers thoroughly and make wrong decisions.
It's outstandingly interesting about those psychological tests. I have to read about them because I've always perceived them as a field that is well discovered and repetitive.
DeleteThank you for interesting article. This topic is very important for us researchers. We are new in some research area and search for articles that could help us in our research. We read article and think that the methodology is correct because its published in most famous journal. I was shocked when I saw to the website with so many retracted articles. I think it's a big problem and we need to find a solution to avoid situation that our article is retracted. In my opinion the reviewers in some case have less knowledge about the research topic presented in article. Our solution presented in article should be repeatable but in many case its difficult to reproduce it ex. author used a hardware that is rare in many countries or used specified laboratory to examined problem. Sometimes a bug in library that we used in our software and we don't know about it could disqualified our work. Wa are humans and we make mistakes. We should try to write article that present our solution is easy to reproduce way. The article should be accepted if our solution is checked by specialists and our methodology and solution are correct. Sometimes reviewers know author's of the article and could accept it despite some mistakes.
ReplyDeleteIt's quite ironic that even in STEM it's so difficult to reproduce work of researchers. Even though there ar initiatives promoting sharing a source code together with a research paper like https://paperswithcode.com/ in most of a cases it's still very difficult to verify findings in peer review process due to the unavailability of required hardware or simply the lack of time.
ReplyDeleteIt's especially unfortunate when we consider the conflict of interests between scientific community which would like to have bug-free code and 100% verified experiments results and individual researches who would like to have their papers published as fast as possible (also in large quantity).
But peer review process is as it was for years and still there are advances in science so I guess it works after all.
Thank you for your opinion. BTW: I've never heard about https://paperswithcode.com/ - it's simply brilliant. Thank you for that link because, in my opinion, it's an excellent perspective for clearer insights into the results in IT.
DeleteUnfortunately we are not capable of repeating every research multiple times just because it can be reviewed – mostly when some research are lasting few months or even few years. Of course it mainly depends on the sort of experiment that is conducted and what is the purpose of the experiment, what it will do. I can agree that in data science field, when we adhere to the methodology and earlier check it’s correctness, there is no use in repeating months of experiments just to gain the same effect. It is better to improve research and then compare the results. Usually research are in some ways confirmed or abolished by later similar experiments – it is the basis of nowadays science, to use and improve science discovers.
ReplyDeleteI've heard about the Python sorting thing. It's quite nasty, as in typical conditions, the code seems to work OK and only fails when executed on different operating system or after some seemingly innocent file operations.
ReplyDeleteThere is a growing trend in some science branches to supply not only raw data, in addition to publication, but also whole source code used to process the data. This might have two opposite outcomes - on one hand, it is easier to reproduce computations, but using the same implementation of algorithms, so having the same potential errors as authors. On the other hand, the source code itself can also be reviewed, and errors found.
And yes, there might be a problem of badly placed incentives, that is too much pressure on new and original research, and too little of reproducing the experiments, validating claims. Just like when weaving the fabric, both warp and weft are required.
I could not agree more about the source code. It's a pity that such a small amount of articles in the IT lacks *any* line of the computer code, even if it strongly depends on it.
DeleteThank you for all your comments. I've found them incredibly insightful. I'm glad that this subject is important for our scientific community.
ReplyDeleteThank you for raising such important and interesting topic. The peer review process is a significant proocedure of reasearch verification, but has its limitations. First of all it is conducted by human specialists, that may have different knowledge and opinion about the research field of the analyzed paper. Being human also means that the reviewer could have a tough day or is tired what may influence their accuracy or could be prone to making mistakes. Moreover, some research requires substanial amount of resources or time, which makes any trial of repeating the experiment almost impossible. I wish that peer review process could reveil all the inaccuracies in the reaserch, but I am affraid it is not possible.
ReplyDelete