Add your Insight
15 November 2020
Registered Report Replication Studies
We are delighted to announce a special issue of the Computer Science Education journal which will exclusively feature Registered Report replication studies in any area of computer science education.
Replication is at the heart of science: replicating the results of other researchers can give us increased confidence in the effects we observe, or provide useful caveats on when these effects occur. It allows us to build future research on a reliable foundation of knowledge. Replication is applicable to many styles of research: quantitative, qualitative and mixed method studies can all be replicated. Sometimes replication is not valued highly enough, with novel results deemed more appealing. This special issue aims to redress the balance by only accepting replications.
Registered reports are a new way of structuring the research and publication process. Peer review is moved from the end of the process -- after analysis -- to before data collection has begun. This means that comments on the research design can still be addressed before the study has taken place. It also means that the accept/reject decision is made in principle before the data has been collected, which removes worry on the researchers’ part as to whether the study will result in a publication, and removes any pressure to present novel results.
This style of publication and reviewing is new for the field of Computer Science education. Therefore we make a call for papers and a call for reviewers (applying for both is acceptable and encouraged) below. If you have any questions about either aspect, please do not hesitate to get in touch with the special issue editors.
Call for Papers
This special issue does not have a specific topic focus. All topics that are usually of interest to the Computer Science Education journal's aims and scope are suitable for submission to this special issue. This explicitly includes qualitative, quantitative and mixed methods work. The only restrictions are that the submissions must be replications and submitted as a Registered Report.
Submissions must be replications of existing work
Submissions must propose to replicate an existing study that has been described in previous publication(s). This may range from a very close replication -- such as using identical materials and methods to collect new data -- to a replication that improves an existing method to investigate the same question.
For example, a submission may propose to use a programming education technique that produced an effect at a university to see if it works in an adult education setting. Or a submission may use the same open-ended survey questions as an existing study, on a similar population but develop their own coding scheme during analysis.
For purposes of this special issue, replications conducted by the authors of the original study will not be considered. Ideally, the submitting authors will contact the authors of the original paper to request sharing the materials, coding schemes, analysis scripts and other useful information to support the replication. We do not require nor recommend including the authors of the original paper as co-authors, and instead include them in the acknowledgements. If you believe there is a strong reason to include an author of the original study as a co-author, please contact the special issue editors.
Publication will be done in a Registered Report style
Registered reports differ from standard publications as to when peer review is performed. Submissions to this special issue must be made before data acquisition has begun (see submission instructions below). Reviewers will then examine and judge the research plan. The decision for each paper will either be a reject, or an acceptance in-principle with suggested and required changes to the research plan. If researchers follow the original plan plus the required changes, and the write-up is of reasonable quality, the paper will be accepted into the special issue regardless of the result of the replication.
To facilitate review of the registered report, submissions will be handled in a two stage process. The first stage requires authors to submit their research plan prior to data collection.
An effective stage 1 manuscript will consist of:
- a structured abstract (250 word max). The journal requires a structured abstract for papers; for the stage 1 submission you will supply background, objective and method, but leave the findings and implications blank.
- an introduction, in which the authors clearly justify the replication of the original study and the specific aspects they will add or modify to shed more light on the phenomenon studied. This section should also include hypotheses about how the outcomes of the replication will compare to the original study.
- a literature review, containing a summary of the relevant literature, especially the literature which has been published since the publication of the original paper.
- a method section, defining the type of replication that will be used (see below), clearly distinguishing between the materials/approaches of the original study and the additions of the replication study.
- a plan of analyses, outlining the original analyses and the newly planned analyses. Quantitative replications should also include a power analysis in this section.
- risks to the study, including:
- whether the funding/means to conduct the study is already secured or is pending,
- the current status of the ethics application, and
- any likely barriers to completing the study before the end of 2021 (see timeline below).
Reviewers will examine the stage 1 manuscript and provide feedback on the soundness of the replication plan. Authors of manuscripts initially selected for inclusion in the special issue based on stage 1 reviews will be provided reviewer feedback, along with clear indications of what would need to be addressed prior to data collection. We encourage authors to then upload their adjusted stage 1 submission (incorporating reviewers’ comments) as a PDF on the OSF website, for the benefit of later readers.
Once data collection and analysis has been completed, authors will submit a revised stage 2 manuscript that presents the study in full. Reviewers will consider the details of the study relative to the registered plans and stage 1 feedback to inform the editors' final decision.
Because of the novelty of the replication and Registered Report aspects, we are also maintaining a live Google document with further details and an FAQ section which will be updated as new questions from prospective authors and reviewers arise.
Call for Reviewers
The essence of reviewing remains the same with Registered Reports as without: reviewers must evaluate the design and rationale of the research study. The difference is that rather than retroactively spotting flaws in what has been done, the reviewer must foresee flaws in what is proposed to be done. For example, if a study proposes to evaluate a programming education intervention in two classes (one control, one intervention), a reviewer might request that the authors have a mechanism in place to check for (and act upon) differences in the class’s prior experience of programming. If a study proposes to use interviews with prompts and a single rater doing the coding, a reviewer might request multiple coders and a plan for what happens if the raters cannot agree.
In anticipation of these differences in reviewing, the editors of the special issue will provide additional resources and training for reviewers of the special issue. This will take the form of written guidance, and a live webinar (with opportunity for questions). We hope this training and the overall reviewing experience will be a useful benefit to reviewers who take part. Due to the novel nature of the reviewing, we do not intend to use the journal’s standard reviewing pool exclusively, and instead ask for reviewers to volunteer specifically for this special issue. Even if you have reviewed for CSE in the past, please do actively indicate your interest in reviewing for the special issue by filling out the reviewer interest form here: https://forms.gle/eHmobeUvY4oyDJE86 We welcome prospective authors taking part as reviewers, and authors may find the training beneficial for writing their registered report.
We anticipate reviewing will take place in November and December 2020.
Looking to Publish your Research?
We aim to make publishing with Taylor & Francis a rewarding experience for all our authors. Please visit our Author Services website for more information and guidance, and do contact us if there is anything we can help with!
How to Submit
Stage 1 manuscripts should be submitted through the journal's standard submission site. See details above for what to include in an effective stage 1 manuscript.
Authors should be sure to select the "Registered Report Replication Studies" special issue title when entering their initial manuscript metadata.
- November 15th 2020: Submission of the stage 1 manuscript due (see above)
- February 1st 2021: Decisions returned to authors with individualised deadlines for each accepted paper (based on data collection schedule)
- Late 2021/Early 2022: Full papers submitted to editors for approval and publication.
View the latest tweets from EducationArena