Chapter 7 Piloting

In this video we are going to talk about the importance of piloting. You might think that since people have already done these kinds of tests in the past, then you don’t have to do piloting yourself. This is simply not true, because you need to make sure that the community whom you are working with agree on this project and feel comfortable and give you feedback.

What do we mean by community? Community means having at least 2 families who come from the population that you are going to sample, so this means not your friends or family. You might want to start by asking them, but if they do not belong to the community you are targeting, then their answers might be different, if not opposite to your target community, thus misleading.

So, something you need to bring up with the community you are targeting is how they feel about the clothing: you need to ask them if they would be ok with their child wearing this or that type of clothing with hardware in it. Also, you need to ask them if they are open with the device you intend to use: suppose you are not sure whether you want to use a device or another, for example Olympus or USB. What we found out is that usually families tend to have strong preferences about the device they prefer their child to wear, so that might also help you decide. For example, in my personal experience [ALEX], what I found is that some families don’t like the USB device because they feel like it can be easily lost and the child can play with it resulting in an additional worry for the parents, which you want to avoid; while other families have a preference for the USB device, especially when used for extended periods of time and during the night, because they are afraid the child would get tired by going around with bigger devices, and this happens especially when children are very small, so the parents think it’s weird to put a 300g device on them.

You also want to ask them how they want you to analyse their recordings. Sometimes you have a preference, whether you want to use automated analysis or manual analysis, so that’s fine. But it’s always very useful to talk about it with the families and have their impressions on what it means when each of these methods are used. So, for instance, what we found is that for some communities, they do not like the idea of human annotation, because they are not sure of what the people who are going to listen to the recordings are going to think of them or how they are going to treat these data. So if this is your preferred choice, and the family tells you this, then that’s something you’ll want to address. So you will need to specify how the people who are going to listen to the recordings, are going to treat these data, that they are going to do it with respect and they are going to be cognisant of the unique situation each family is in.

At this point we also want to remind you something that we are going to touch upon the ethics and legal series: if you are recording in a setting where you are supposed to report anything illegal to the law (e.g drug consumption or anything involving violence), if you are mandated by the law to report anything illegal, what you want to do with piloting is to mention this to the families and ask them what they think about this and what they want to do about this. Since you will have to specify this anyway when making people sign the consent to you study, your risk is to have a biased sample of the population you are targeting.

Some of the complaints and worries we got about annotation are concerned with automated annotation, but they are very very few and usually they come from very informed parents that come from the tech industry. So, if you are collecting data from families like that, parents might know the use of automated analysis for purposes that are close to surveillance or that are being used in discriminatory manners (e.g face recognition, since algorithm are trained with faces taken from the internet, which is biased with faces of whit people, these algorithms might work against people who are not white). This is something that some parents are aware of and that you want to bring up in your piloting phase.

To cite a couple more examples: in the piloting stage you don’t want to just discuss about logistics but you also want to take a moment to discuss all the stages of you data collection and analysis with the representatives of the community you are working with because you want to make sure you won’t have surprises down the line.

When you are going through all the stages, you want to point out also what the goals of your study are and you want to have impressions of what these goals mean to them. Sometimes when you have a very narrow, technical goal, parents might not really care about it, and that’s fine. You just want to check if they have any strong feelings about it. When talking about goals with the families, you’ll also want to discuss about what could happen in the context of a reuse and this is something that we will go through in another series of videos but we think that these data but we think that these data are really valuable so they shouldn’t be used for a single purpose, right? People have gone through a lot of effort to collect them, so they should’ just be thrown away after one use. So this is for you a good possibility to talk to parents about donating data and reusability of data. You can also mention other purposes for their data, make examples of what would be interesting to look into, and see if parents are ok with data being reused in different settings.

Another thing you want to check is that the families have other areas or research goals they want you to be thinking about (e.g something that happens very frequently is that these communities are interested in researches about bilingual or multilingual children development, they are maybe speaking a language that is not spoken by many and they are afraid their children will lose it, so they would like you think about ways to make sure their language isn’t lost). This is thus the natural time to look into this, so that you can already foresee it in your request. Finally, here is the time for you to ask parents what is reasonable feedback they would like to have, whether they are interested in having some of your resulting data and in which form.

7.1 Resources

Cychosz, M., Romeo, R., Soderstrom, M., Scaff, C., Ganek, H., Cristia, A., … & Weisleder, A. (2020). Longform recordings of everyday life: Ethics for best practices. Behavior research methods, 52(5), 1951-1969. pdf
Levin, H. I., Egger, D., Andres, L., Johnson, M., Bearman, S. K., & de Barbaro, K. (2021). Sensing everyday activity: Parent perceptions and feasibility. Infant Behavior and Development, 62, 101511. pdf