How to conduct online experiments

💡 Why would we want to conduct online experiments? It’s better to think thoroughly about the pros and cons before you start learning the technical details.

-Pros: more subjects from a more general pool; almost no time constraint; easy data collection across multiple sessions…

-Cons: limited control over subjects regarding how they interact with the task, and you may even encounter frauds/bots; poor internet connection could lead to data loss; imprecise time control and measurement; if your experiment needs to create a sense of social connection among strangers, it may be more challenging online.

While some of the cons can be improved through experiment designs or technical tricks, others are more intrinsic and unavoidable. Here are some articles for further reading:

Response rate differences between web and alternative data collection methods for public health research: a systematic review of the literature

Using online, crowdsourcing platforms for data collection in personality disorder research: The example of Amazon’s Mechanical Turk

Crowdsourcing research: Data collection with Amazon’s Mechanical Turk

Online psychology experiments: everything you need to know

Prolific.ac—A subject pool for online experiments

Introduction

You’ll need a similar process for running online experiments compared to in-person experiments.

  • Recruit people.

From wherever you recruit people for in-person experiments —>

  1. Continue using the resources you used for in-person experiments, but you will not see your participants in person. You’ll probably just send them a web link. You won’t get a more diverse pool of participants by using this method, but you may trust your pool more.
  2. Use large-scale recruitment platforms
  • The task is played by participants.

Instead of using MATLAB/Python to run a task on a local computer —>

Participants open the web link and complete the task on their own. The task is playable in a web browser.

  • Save data and pay participants.

This is quite intuitive for in-person experiments. You have a save_data function in your experiment program, and you pay the participants in cash. —>

The data needs to be saved from participants’ browsers to somewhere you can download, and you usually pay them through the recruitment platforms or manually by gift card or bank transfer.

Recruit people

The most widely used recruitment platforms are Amazon Mechanical Turk and Prolific. Unfortunately, I don’t have personal experience using these platforms, as I use a local recruitment platform run by UZH and ETH. However, here are some great resources I saved for my possible future use:

Meanwhile, there is a tool called psiTurk developed by cognitive scientists to help run experiments on Mturk.

Run task in a web browser

Programming language

To code a task that is playable in a browser, you will usually need JavaScript, HTML, and CSS. To put it simply, JavaScript is for the parts with “animations” and where participants take actions, basically your task itself; HTML is for integrating the elements in a webpage, including your task; CSS is for making your webpage look nicer.

Sometimes you may need to use other programming languages (for example, Python) to achieve specific functions in your task. This is doable, but it means that you will need a server to host your experiment.

It is unusual to need game engines to code your experiment, which typically require C# or C++.

(I personally used codecademy to learn Javascript and the basics of web development.)

Programming library/toolbox

  • jsPsych: a JavaScript framework for creating behavioral experiments. It provides useful functions for common psychology experiment components.
  • Phaser: a JavaScript framework for coding games. I personally used this library because I think games and experiment tasks have a lot in common, and they provide many useful example codes for various functions (although they may look unnecessarily fancy for an experiment). The downside is that you need to code some psychology experiment components from scratch.
  • PsychoPy/PsychoJS: You can build an experiment from their GUI and export to JavaScript. **But you need to check the features supported by PsychoJS to ensure the components you need will work online.
  • nodeGame: JavaScript for large-scale, online, multiplayer, real-time games and experiments.
  • Minno: It builds and runs experiments with time-sensitive tasks.
  • oTree: Python based framework ****for web-based interactive tasks.

There are also some toolboxes which require no coding, and some are even one-stop experiment platforms:

Usually, the flexibility of a programming toolbox is negatively correlated with its complexity. Therefore, there is no single correct answer for the choice of toolbox. What programming toolbox you should use highly depends on the specific needs of your task.

Survey

Sometimes you don’t need to have a playable task but only a survey, and usually you may need a survey as a supplement to your task to collect some user information, show instructions and run quizzes. Here are some survey websites:

And surveys can have advanced functions, for example:

SMARTRIQS: A Simple Method Allowing Real-Time Respondent Interaction in Qualtrics Surveys

Server

As I mentioned before, participants will open a web link to play the task. But where does the link come from? In other words, where is your task hosted, and where is the data saved?

This is basically the function of a web server. If you use platforms like Mturk or Prolific, you don’t necessarily need a server, as long as your code can be run on their platform and this is also where the data is saved. However, if you want more control over your experiment and data, you will need a server to host your experiment.

  • Pavlovia: this is what I used for one experiment. It has a document for running tasks coded with jsPsych and lab.js. For tasks coded from Phaser, you can use the code snippet from here for data writing.
  • Heroku: support multiple languages.
  • MindProbe
  • A server provided by IT department in your university. This is what I used for another task.

Here comes another question: do I need a server to locally test my experiment? Yes, I used MAMP.

Graphic design (optional)

Here are some free online resources for graphic design of your task.

Examples

Here are some code examples:

The programming libraries’ websites usually have their own examples as well.

Save data and pay participants

If you use Mturk or Pavlovia, they have their own documents about saving data. Meanwhile, the recruitment platforms usually have a solution for paying participants. If you use your own server, you’ll need to have a chunk of code transferring the data to server.

Other resources

Resources for Running Developmental Studies Online

Acknowledgement: I learned a lot of the information in this blog from the great tutorials by my previous lab mates and current colleagues. Special thanks to Qixiu Fu and Ili Ma ❤️