Researching your own ‘backyard’: on bias and ethical dilemmas

This is a post particularly for those in the social sciences and humanities who may be doing a form of ethnographic research within the context in which they work or study – in other words, doing ‘insider research’ to use Paul Trowler’s term. Researching a context with which one is intimately familiar and in which one has a vested interest can create possible bias and ethical dilemmas which need to be considered by researchers in these situations. The last thing you want, in presenting your completed research, is for your findings to be called into question or invalidated because you have not accounted clearly enough for issues of insider bias, and your own vested interests.

Insider bias and vested interests

In the article cited in this post, Trowler considers issues of bias in data generation. Bias in research can be defined as having only part of the ‘truth’ in your data but treating that part as a whole, ignoring other possibilities or answers because you are prejudiced towards the ones that best represent your interests or investment. If you are working in a context with which you are familiar, especially your own department or faculty, or an organisation in which you have worked or do work, you will have a vested interest in that context. Either you want everyone and everything to look amazing, or perhaps you are unhappy about certain aspects of the ways in which they work and you want your research to show problems and struggles so you have a basis for your unhappiness. Either way, you have to acknowledge going in that you cannot be anything but biased about this research.

bias blindspot

However, acknowledging that you are biased, and detailing what that bias might entail for readers and examiners, does not undermine your position as researcher. By making yourself aware of potential blindspots in your research design – for example the participants you have chosen, or the cases you are including and excluding from your dataset (and why) – you can better head off possible challenges to the validity of your data later on, and you can strengthen your research design choices. Be honest with yourself: there is a balance to strike here between being pragmatic and strategic in choosing research participants, sites, or cases that will be accessible and that will yield the data you need to make your argument, and between choosing too neatly and risking one-sided or myopic data generation. Why these participants, these cases, these sites? Are there others that you know less well that you could include to balance out the familiarity, and increase the validity of your eventual findings? If not, how might you maintain awareness of your ‘insiderness’ and account for this in analysis and discussion later on?

You need to account for these decisions and questions in your methodology, and discuss what it means for your study that you are doing insider research, and that this does imply particular forms of bias. I don’t think you can get away from being biased in these cases, but you can think through how this may affect your data generation processes, and your analysis as well, and share this thinking with your readers frankly and reflexively.

Insider bias and ‘intuitive analysis’

Another point Trowler makes concerns insider ‘intuition’ when analysing the data you have generated and selected for your study. You may be analysing a policy process you were part of, or meetings you sat in on, or projects you were involved in. You have insider knowledge of what was said, the tone of the conversations, background knowledge (and perhaps even gossip) about participants – in other words, you have a kind of cultivated ‘intuition’ about your data set that you reader will not be privy too. Accounting for bias here is crucial, because if you cannot see it, you may rely too much on this insider intuition in analysing your data, and too much of the language of description you are using to convey your theorised findings will be tacit and hidden from the reader. They will then struggle to understand fully on what basis you are claiming that X is an example of poor management, or that Y means that the department is doing well in these particular areas.

ideas

It is thus vital that you get feedback here on whether it is clear to your reader why you are making particular claims, and whether they can see and understand the basis on which you are making such claims. Do they understand your ‘external language of description’ or ‘translation device’ to use Bernstein’s and Maton’s terms respectively? If they do not, you may be relying too much on your insider view of your case or participants, and may need to find a way to step back, and try to see the data you are looking at as more strange and less familiar. Getting help from a supervisor or critical friend who can ask you questions, and expose and critique possible points of bias is a useful way to re-interrogate your data with fresher eyes.

Ethical dilemmas

An ethical dilemma is defined as ‘a choice between two options, both of which will bring a negative result based on society and personal guidelines’. In research, this definition could be nuanced to suggest that an ethical dilemma presents itself when you have to make a decision to protect the interests of your research or the interests of your participants or study site. For example, in an interview with a senior manager you learn information that may be better off staying private and confidential, yet would also add an important and insightful dimension to your findings. What do you do? A participant in your study asks you for help, but to help might be to prejudice that participant’s responses in a later survey or interview, possibly skewing your data. Yet it is your job to help them. Study first, or job first? These are the kinds of dilemmas that can arise when you do research in the same spaces in which you work, and with people you work with and have other responsibilities to outside of your research.

Cheating-clients,-ethical-dilemmas

As researchers we have a duty to be as truthful and ethical in our research as possible. We are working to create and add to knowledge, not to simply maintain the status quo. In your study this may mean being carefully but resolutely critical, reflective and challenging, rather than only saying the palatable or easy things to say. This work is always going to present difficulties and dilemmas, but accounting as far as possible for your own bias and vested interests, and for your own relevant insider knowledge, can create space in your study for the development of your own reflexivity as a researcher, and can bolster rather than undermine the validity and veracity of your findings.

Trowler, P. (2011) Researching your own institution: Higher Education, British Educational Research Association online resource. Available online at [http://www.bera.ac.uk/files/2011/06/researching_your_own_institution_higher_education.pdf]

Advertisements

Iterativity in data analysis: part 1

This post is a 2-parter and follows on from last week’s post about generating data.

The one thing I did not know, at all, during my PhD was that qualitative data analysis is a lot more complex, messy and difficult than it looks. I had never done a study of this magnitude or duration before, so I had never worked with this much data before. I had written papers, and done some analysis of much smaller and less messy data sets, so I was not a c0mplete novice, but I must say I was quite taken aback by the mountain of data I found I had once the data generation was complete. What to do now? Where to start? Help!

The first thing I did, on my supervisor’s advice, was get a license for Nvivo10 and uploaded all my documents, interview and video recordings and field notes into its clever little software brain so that I could organise the data into folders, and so that I could start reading and coding it. This was invaluable. Software that enables you to store, organise and code your data is a must, I think, for a study as large and long as a PhD. This is not an advert for Nvivo so I won’t get into all its features, and I am sure that other free and paid-for qualitative data analysis packages like Atlas Tii or the Coding Analysis Toolkit from UMass would do the job just as well. However, I will say that being able to keep everything in one place, and being able to put similar chunks of text into different folders without mixing koki colours or scribbling all over paper to the point of confusion was really useful. I felt organised, and that made a big difference to my mental ability to cope with the data analysis and sense-making process.

The second thing I did was keep very detailed notes in my research journal on my process as it unfolded. This was essential as I needed to narrate my analysis process to my readers in as much detail as possible in my methodology chapter. If a researcher cannot tell you how they ended up with the insights and conclusions they did, it is much harder to trust their research or believe what they are asking you to. I wanted to be believable and convincing – I think all researchers do. Bernstein (2000) wrote about needed two ‘languages of description (LoD)’ in research: the internal (InLoD) which is essentially where you create a theoretical framework for your study that coheres and explains how you are going to understand your problem in a more abstract way; and the external (ExLoD) where you analyse and explain the data using that framework, outlining clearly the process of bringing theory to data and discovering answers to your questions. The stronger and clearer the InLod and ExLoD, the greater chance other researchers then have of using, adapting, learning from your study, and building on it in their own work. When too much of your process of organising, coding, recoding, reading, analysing, connecting the data is hidden from the reader, or tacit in your writing about it, there is a real risk that your research can become isolated. By this I mean that no one will be able to replicate your study, or adapt your tools or framework to their own study while referencing yours, and therefore your research cannot be readily be built on or incorporated into a greater understanding of the problems you are interested in solving (and the possible solutions).

This was the first reason for keeping detailed notes. The second was to trace what I was doing, and what worked and what did not so that I could learn from mistakes and refine my process for future research projects. As I had never worked with a data set this large or varied before, I really didn’t know what to do, and the couple of qualitative research ‘textbooks’ I looked at were quite mechanical or overly instrumental in their approach, which didn’t make complete sense to me. I wanted a more ‘ground-up’ process, which I felt would increase the validity and reliability of my eventual claims. I also wanted to be surprised by my data, as much as I wanted to find what I thought I was looking for. The theory I was using further required that I not just ‘apply’ theory to data (which really can limit your analysis and even lead to erroneous conclusions), but rather engage in an open, multiple and iterative reading of the data in successive stages. Detailed notes were key in keeping track of what I was doing, what confused me, what made sense and so on. Doing this consciously has made me feel more confident in taking on similarly sized research projects in future, and I feel I can keep building and learning from this foundation.

This post is a more conceptual musing about the nature of qualitative data analysis and lays the groundwork for next week’s post, where I’ll get into some of the ‘tools’ or approaches I took in actually doing my analysis. Stay tuned… 🙂

 

Data: collecting, gathering or generating?

I’m thinking about data again – mostly because I am still in the process of collecting/gathering/generating it for my postdoctoral research. I had a conversation with a colleague at a conference I went to recently who talks about ‘generating’ his data – colleagues of mine in my PhD group use this term too – but the default term I use when I am not thinking about it is still ‘collecting’ data. I’m sure this is true for many PhD scholars and even established researchers. I don’t think this is a simple issue of synonyms. I think the term we use can also indicate a stance towards our research, and how we understand our ethical roles as researchers.

Collect (as other PhD bloggers and methods scholars have said) implies a kind of linear, value-free (or at least value-light) approach to data. The data is out there – you just need to go and find it and collect it up. Then you can analyse it and tell your readers what it all means. Collect doesn’t really capture adequately, for me, the ethical dilemmas that can arise, large and small, when you are working in the ‘field’. And one has to ask: is the data just there to be collected up? Does the data pre-exist the study we have framed, the questions we are asking, and the conceptual and analytical lenses we are peering through? I don’t think it does. Scientists in labs don’t just ‘collect’ pre-existing data – experiments often create data. In the social sciences I think the process looks quite different – we don’t have a lab and test tubes etc – but even if we are observing teaching or reading documents, we are not collecting – we are creating. Gathering seems like a less deterministic type of word than collecting, but it has, for me, the same implications. I used this word in my dissertation, and if I could go back I would change it now, having thought some more about all of this.

Generating seems like a better word to use. It implies ‘making’ and ‘creating’ the data – not out of nothing, though; it can carry within it the notions of agency of the researcher as well as the research participants,  and notions of the kinds of values, gazes, lenses, and interests that the parties to the research bring to bear on the process. When we generate data we do so with a particular sense in mind of what we might want to find or see. We have a question we are asking and need to try and answer as fully as possible, and we have already (most of the time) developed a theoretical or conceptual gaze or framework through we we are looking at the data and the study as a whole. We bring particular interests to bear, too. If, as in my study, you are doing research in your own university, with people who are also your colleagues in other parts of your and their working life, there are very particular interests and concerns involved that impact not just on what data you decide to generate, but also how you look at it and write about it later on. You don’t want to offend these colleagues, or uncover issues that might make them look bad or make them uncomfortable. BUT, you also have a responsibility, ethically, to protect not just yourself but also the research you are doing. Uncomfortable data can also be very important data to talk about – it can push and stretch us in our learning and growth even as it discomforts us. But this is not an easy issue, and it has to be thought about carefully when we decide what to look at, how and why.

These kinds of considerations, as one example, definitely influence a researcher’s approach to generating, reading and analysing their data, and it can help to have a term for this part of the research process that captures at least some of the complexity of doing empirical work. For now, I am going to go with others on this and use ‘generating’. Collecting and gathering are too ‘thin’ and capture very little if any of the values, interests, gazes and so forth that researchers and research participants can bring to bear on a study. Making and creating – well, these are synonyms for generating, but at the moment my thinking is that they make it sound too much like we are pulling the data out of nothing, and this is not the case either. The data is not there to be gathered up, nor is it completely absent prior to us doing the research. In generating data, we look at different sources – people, documents, situations – but we bring to bear our own vested interests, values, aims, questions, frameworks and gazes in order to make of what we see something different and hopefully a bit new. We exercise our agency as researchers, not just alone, but in relation to our data as well. Being aware of this, and making this a conscious rather than mechanical or instrumental ‘collection’ process can have a marked impact, for the better I think, on how ethically and responsibly we generate data, analyse it and write about down the line.

Fieldwork: to participate or not to participate…

This is my last post about fieldwork. This final one is about observations, and whether and how to participate or not participate in what you are observing. In my case I was observing classroom teaching, but I think these comments could also apply to tutorials, meetings, workshops – any kind of encounter where there is an opportunity for you to be present, watching and taking notes, and in some cases also participating.

I have read a little bit about participant and non-participant observations, and the relative pros and cons of each. I chose non-participant observation, and in the spirit of this blog, I want to add my own voice on what it was actually like to sit in lectures for 14 weeks and not participate very much at all, what eventually tipped me from being totally quiet to venturing a little participation, and why I think non-participant observation can be a challenge.

I decided not to participate in the classes I was sitting in on for one big reason and a couple of smaller ones. The big reason was that I had majored in the one subject I was including in my study (Politics), and I have worked for 4 years with lecturers teaching the other particular course  (Law) so I have come to know a fair bit about the knowledge and I find it very interesting. I was worried, in short, that if I participated I would ask too many questions or make comments that would in some way silence the voices of legitimate students or perhaps lead to the lecturer and I engaging in a conversation or debate in class that might exclude students. I have been in higher education as a student and tutor for a long time and these students are by-and-large in their first year of study. I felt I had no right, really, to come into their classroom and take up their time with their lecturers. 

One smaller reasons were that I thought I would be able to capture more accurate and objective fieldnotes if I was not too involved in the course. The more you participate, I reasoned, the more you perhaps want to agree with the lecturer, or the less you want to make a note of things that could be negative or less flattering, so your fieldnote data can be skewed or incomplete. I think that this ties in with my first post on fieldwork, where I talked about the Trowler and Williams’ articles on doing research in your ‘backyard’ and the possibility of finding out knowledge that can put you as a researcher in a tricky position in relation to your participants or your university/organisation. I felt that participating might tip my own personal scales in a too-subjective direction. I can’t here go into a full conversation about whether research like mine can be called fully objective (suffice to say it can’t be because there is always some researcher bias in qualitative studies like mine), but I will say that I was trying to record, as verbatim and as faithfully as possible everything that went on in the lectures without trying to pre-judge or pre-organise my data into categories or decide what had to stay and what could go, and I felt that being too involved in the lectures would hinder and further bias this process.

Another smaller reason was a little more vain: I simply wanted to be invisible. I didn’t want to call attention to myself because  after all my years of studying and teaching, I still get palpitations when I have to speak up in class or ask a question in a meeting where people will look at me. So I liked the quietness of non-participant observation, even though I had introduced myself to the whole class at the start of lectures and they knew who I was and why I was there.

However, being that quiet, especially when I really had a question to ask or an answer to a question posed by the lecturer, was really difficult. At times my notes record this frustration: ‘I really want to join in the discussion. So hard not to comment’. I felt, especially in Politics, that I had some useful thoughts to share, but I resisted the urge to call out answers because I felt it was unfair. I did this course when I was an undergrad so answering would have felt a bit like cheating on a test. Right at the end of both courses, though, I gave up resisting and I asked a couple of questions in Law lectures, and at least raised my hand to vote on issues in Politics although I did not ask questions there. I was nervous about doing that, but the lecturers included me as a student and did not offer any special treatment which allayed at least my worries about taking over a student space.

This year, I am participating a little – as a very-semi-participant observer – in my post-doc data gathering. I am doing it partly to try out a different way of doing observations like this, and partly because I have learned that limited and careful participation does not necessarily lead to the issues I was concerned about, like skewing my data or distracting the lecturer or muscling in on students’ space. But I do think if you are going to be a participant observer you have to be careful and keep a record of your participation in your fieldnotes. You need at all times to be the researcher first and the participant second. You need to check with those you are observing if it is okay, and to what extent you could or should participate.

It is in many ways easier and less fraught to stay silent in the background and just watch and make notes, but participating can be more fun even if it brings possible complications with it. It’s up to the researcher considering the situations in which data are being gathered to decide what will work best. Be pragmatic, take careful notes and be open, and don’t forget to tell your readers why you made the decisions you did when you get to your methodology chapter!

Fieldwork: making and transcribing field notes

This is the second post on fieldwork: this one is specifically about field notes – some thoughts on how to write them and how to transcribe them. I am still working this out, so it’s a thinking process in progress.

For my PhD, I gathered data largely from sitting in on lecturers’ classes and watching them teach, scribbling furiously during each one hour lecture. As you can imagine, over 14 weeks in two courses this ends up being a rather thick pile of notes. In my case it amounted to 5 and a half notebooks full of notes (over 500 A5 pages). These all had to be transcribed and organised so as to make sense of of them, and so that I could put them into NVivo10 to analyse them as part of my larger data set. Also, they needed to be typed up so I could copy and paste relevant pieces into my chapters as needed. I procrastinated a lot about doing the transcription. It’s not my favourite activity as a researcher. But I couldn’t have someone else do it because these notes were something I really needed to read several times, understand and sift through. I think, in hindsight and in agreement with Pat Thomson, that tedious as it is all researchers should try and do as much of their own transcription as possible because of how involved it enables you to become with your data. It makes the analysis process more enjoyable and productive too.

I thought what I would do in this post is list some of the things I did, why and what I learnt along the way in the hopes that you may find it useful if you, too, are gathering some of your data this way.

1. I handwrote my notes for two reasons: the first is that in the one class students were not allowed to use laptops because the lecturers wanted them paying attention as part of their training for the profession they will eventually enter, and the second is because I write faster than I type and writing is quieter. I tend to bash my keyboard a bit, and I did not want to distract other students or stand out too much by using equipment they were not allowed to use. A further plus with handwriting that I learned along the way was that I could easily copy diagrams lecturers put up on slides or drew on the boards, and I could represent what they were saying pictorially or non-linearly as well, which was often quicker and easier and made my notes feel more authentic to me.

IMG00075-20130612-0930

2. Pat Thomson wrote in a blog post a while back that she takes lots of field notes, and that she tries to capture verbatim what is going on as much as she can. I tried to do that too, and often was able to succeed in bits in one of the two courses because both lecturers talked fairly slowly and deliberately and paused for students to ask questions or take notes, so I could keep up. This was harder in the  other course where the lecturers spoke quite fast. I am having this issue now, too, in my post-doc research where one lecturer in particular is a very fast talker. However, I am much more comfortable with my theoretical framework now, so I can note particular kinds of phrases or instructions or comments that he says because I know I will use them in my analysis later on. However, I want to avoid being too selective in my hearing, because I don’t want to pre-empt the data or tell it what to tell me. I want to be surprised by it too, and find things I am not necessarily anticipating. Thus, I try now as I did last year to write as much as I can of what is going on in the moment, and can then sift the notes later and reorganise them during transcription.

3. I developed a shorthand: lecturers’ initials for the lecturers involved, like CA or BM (not their real initials). I also used S for student and tried to keep track of students’ questions or inputs where I could hear them clearly (the classes were large and often noisy). So I would have S1, S2 and so on engaging the lecturer in conversation or debate.

4. I didn’t transcribe everything. By being such a procrastinator about the transcription of the field notes, I ended up transcribing them while I was also beginning to analyse the data, so in a way this worked out well because it meant I had a sense of what I needed and what was just additional information that was unimportant, like comments made about admin issues, or comments I wrote about the lecturer telling the students off for not coming to tutorials the week before. Thus, I did not need to transcribe every word I wrote in my notes, and what I did was to read through my notes first, quickly, to remind myself of what I had written, and noted things I could excise. I learnt that it is okay to cut bits of your notes out and just not transcribe them. You won’t analyse ALL your data.

5. I had to do some reorganising when I transcribed my notes. Field notes are very in the moment – you are just trying to keep up and get it all down as faithfully and fully as possible, and you don’t really have time to think. When you go back to transcribe, though, you do have time and you can see how you can transcribe your sometimes chaotic and messy notes to impose a little more order, often needed in data analysis, and also how you can represent your pictures and scribbles in words so that you know what you mean, and can show readers what you mean if you use those examples in your chapters. I think that you need to be careful with reorganising, though, because you don’t want to rewrite history and make things that were chaotic seem simple, or things that were challenging seem easy. You would be skewing or tampering too much with your data and this would be unethical. It may also rob you of some potentially interesting findings. However, a little reorganisation that makes the notes easier to read and easier to represent to a reader, while staying true to the original scribbles, may sometimes be necessary.

I think the biggest thing I learned, and am still learning, is that it is an ongoing process of learning how to write these notes well, and how to collect rich and interesting data in ways that will be usable and make sense to me later. Stop every few notes and look back – reflect on what is working and what is not, and try to use that reflection to improve your taking of field notes. Capturing them can be tedious but field notes can also give you many-faceted and rich data for later probing and analysis.