OpenACJ

A browser-based platform for adaptive comparative judgement research.

OpenACJ

Developed by Jeffrey Buckley

Launch app View source code

About the app

OpenACJ is a browser-based platform for conducting adaptive comparative judgement research. It supports the collection of comparative judgements in which judges make a series of binary decisions between pieces of work.

The app is designed to support open-ended educational assessment, particularly where work is difficult to evaluate using conventional rubrics alone. It can be used to organise judgement sessions, support judge participation, and generate data for modelling comparative judgement outcomes.

OpenACJ is currently hosted through Hugging Face Spaces and uses Google Sheets to support the underlying data workflow. The app can be launched directly in a browser, and its source code is available through the Hugging Face repository.

Typical uses

OpenACJ may be useful for researchers interested in:

• assessing open-ended student work;
• comparing design portfolios, artefacts, or project outcomes;
• conducting adaptive comparative judgement studies;
• exploring judgement reliability and ranking outcomes;
• supporting research on assessment in technology education and related fields.

Supporting files

These files can be used to guide you through using the OpenACJ app.

Note on Google Account Integration: Connecting your Google Account is a one-time setup. Once configured, you will not have to repeat this process for any future OpenACJ sessions!

File	Description
Google Account Instructions	Presentation with step-by-step instructions for setting up your Google Account for use with OpenACJ.
OpenACJ Session Setup Guide	Presentation detailing how to configure and run an OpenACJ session.
Portfolio Links Template	Blank spreadsheet template for preparing your portfolio data to upload into OpenACJ.
Judge List Template	Blank spreadsheet template for preparing your list of judges to upload into OpenACJ.

Related publications

Development

Buckley, J., & Zhu, C. (2025). Investigating the impact of comparison difficulty and judge fatigue on the feasibility of comparative judgement for technology education. The 42nd International Pupils’ Attitudes Towards Technology Conference Proceedings 2025. Montreal, Canada: Université de Sherbrooke.

Buckley, J. (2024). An open-source adaptive comparative judgement app for technology education research and practice: Alpha version. Journal of Technology Education, 36(1), 58–82. https://doi.org/10.21061/jte.v36i1.a.4

Buckley, J., & Lin, T.-J. (2024). Simulating adaptive comparative judgement sessions to determine the effect of judge disagreement on reliability. The 41st International Pupils’ Attitudes Towards Technology Conference Proceedings 2024. Nanjing, China: Nanjing Normal University.

Buckley, J., Seery, N., & Kimbell, R. (2023). Modelling approaches to combining and comparing independent adaptive comparative judgement ranks. The 40th International Pupils’ Attitudes Towards Technology Conference Proceedings 2023. Liverpool, UK: Liverpool John Moores University.

Buckley, J., & Canty, D. (2022). Assessing performance: The technical challenges in developing and validating “ACJ-Steady State”. PATT2022: PATT on the Edge: Technology, Innovation, and Education. Newfoundland & Labrador, Canada: Memorial University.

Use and application

Zhu, C., Buckley, J., Klapwijk, R., Spandaw, J., & de Vries, M. (2025). A holistic look at creativity: Evaluating pupils’ creative design ideation and prototypes through comparative judgment. International Journal of Technology and Design Education. https://doi.org/10.1007/s10798-025-10027-w