English for
Academic Studies II
LCS1088
Weeks 10-12
Research skills
- Constructing and administering a small-scale research study
- Processing and reporting the data for a small-scale research study
Overview
These two classes will provide you with a brief overview of the key things to consider when designing a small-scale research study using a questionnaire as a research tool.
It is often assumed that anyone with some common sense can design and process a questionnaire. In fact, many people who use questionnaires to collect and process data are unaware that to do this well requires careful design and planning. This is why many research studies start with an exciting research question, but are flawed by a badly designed or poorly processed questionnaire. Good research cannot be built on poorly collected data!
These two lessons will help you to avoid many common pitfalls.
How well are you doing?
| I know how to construct a good questionnaire | | | |
| I know how to administer a good questionnaire | | | |
| I know how to process and report the data I collect from a questionnaire | | | |
| A. Constructing and administering a small-scale research study |
| 1. WHAT ARE QUESTIONNAIRES? WHAT DO THEY MEASURE? WHY USE THEM? |
| 1.1. What are questionnaires? |
| 1.2 What do questionnaires measure? |
| 1.3 Why use questionnaires? Why not? |
| 2. CONSTRUCTING THE QUESTIONNAIRE |
| 2.1 Selecting a research focus |
| 2.2 General features of a good questionnaire |
| 2.3 Questionnaire items: Closed items and open-ended items |
| 2.4 Multi-item measures vs. single-item measures |
| 2.5 How to write good items |
| 2.6 Grouping and ordering items |
| 3. ADMINISTERING THE QUESTIONNAIRE |
| 3.1 Selecting the sample |
| 3.2 Strategies to increase the quality and quantity of responses |
| 3.3 Ethical considerations |
| 3.4 Checklist for administering your questionnaire |
| REVIEW: Constructing and administering a small-scale research study |
| B. Processing and reporting data for a small-scale research study |
| 1. PROCESSING YOUR DATA |
| 1.1 Code and input your data |
| 1.2 Generate descriptive statistics |
| 1.3 Analyse the content of any qualitative items |
| 2. SUMMARIZING AND REPORTING QUESTIONNAIRE DATA |
| 2.1 Some general guidelines for summarizing and reporting questionnaire data |
| 2.2 Reporting how your study was designed |
| 2.3 How to make your data digestible and reader-friendly |
| REVIEW: Processing and reporting data for a small-scale research study |
A. Constructing and administering a small-scale research study
Objectives
By the end of this section, you should be able to:
- develop an appropriate research question for a small-scale research study
- design and construct a questionnaire combining quantitative and qualitative items appropriate to the research question
- identify an appropriate group of target respondents
- administer the questionnaire effectively.
1. WHAT ARE QUESTIONNAIRES? WHAT DO THEY MEASURE?
WHY USE THEM?
1.1. What are questionnaires?
The word questionnaire is often misleading because many questionnaires do not contain real questions that end with a question mark. Instead, most contain a series of statements, to which the respondents react. ‘Questions’ are usually called items. There are two broad types of item:
| Quantitative items | Quantitative is related to quantity, i.e. items that can easily be processed and reported using numbers. Often called closed or structured items. Most structured items fall into one of these categories: Single response (e.g. From the following list please select the category which includes your household income) Multiple response (e.g. From the following list of news sources, please select all that you regularly read) Scaled questions (e.g. The Chief Executive is doing a good job – followed by Strongly Agree to Strongly Disagree) |
| Qualitative items | Qualitative is related to quality, i.e. items that explore the respondent’s values. Often called open-ended items. Examples: One thing I liked about this workshop is ………………………….. What did you find most useful about the workshop? |
We will focus on questionnaires consisting mostly of quantitative items with a small number of qualitative items (often just one or two). This is typical for a small-scale research study.
1.2 What do questionnaires measure?
Questionnaires can be used to collect three types of information about the respondent:
| Facts | Usually used only to find out who the respondents are, e.g. age, sex, education level etc. |
| Behaviour | How do people behave? What do they do? How often do they do things, or how often have they done things in the past? |
| Attitudes | What do people think about things? What are their opinions and beliefs? |
Many questionnaires collect information about all three of the above.
Behaviour and attitudes cannot be truly measured, but they can be estimated. How well we can estimate them depends on how well we design, conduct, administer and process our questionnaire.
1.3 Why use questionnaires? Why not?
TASK: Advantages and disadvantages of using questionnaires
What are some of the advantages and disadvantages of using questionnaires? Discuss with your group and make a list below:
| ADVANTAGES OF QUESTIONNAIRES | DISADVANTAGES OF QUESTIONNAIRES |
Quantitative or qualitative?
Both quantitative and qualitative items have advantages. Can you think of any? Discuss with your group and make a list below:
| ADVANTAGES OF QUANTITATIVE ITEMS | ADVANTAGES OF QUALITATIVE ITEMS |
| …and a disadvantage: | …and a disadvantage: |
2. CONSTRUCTING THE QUESTIONNAIRE
Questionnaires are easy to do quickly and badly. Constructing a questionnaire that reveals useful data which we can use to draw conclusions and make decisions requires more a more careful approach.
2.1 Selecting a research focus
First, you need a research focus, i.e. what do you want to know? Let’s consider these words one by one:
(i) a research focus
Look again at the definition of research on p.1 of these notes. What do you consider the key words in this definition? What exactly is research? What isn’t it?
(ii) a research focus
| TOO WIDE | In a small-scale research study, a common problem is to focus too widely. Example: What do Hong Kong people think about the Hong Kong-Macau bridge? This focus is too wide in two ways: a) Hong Kong people: you would need to ask a lot of Hong Kong people to get a good estimate (maybe 1,000 or more). b) What do they think?: a rather vague research question. It’s fine for everyday conversation, but as a research question it is too imprecise. What exactly does this mean? |
| TOO NARROW | Another common problem in a small-scale research study is to focus too narrowly. Example: Do students of PolyU SPEED class LCS1088-201 have a favourable opinion of the Hong Kong-Macau bridge? This is a well-crafted question: it’s tightly defined and will probably result in a clear yes/no answer. However, it has no relevance or interest for anyone not in class LCS1088-201, so why ask it? |
Suggest a more appropriate focus for the above research question.1
When you know what your research focus is, you need to phrase it as a question.
Example:
Are young people in FT education between the ages of 19 and 22 in favour of building the Hong Kong-Macau road bridge?
Now we have a research focus that is neither too wide nor too narrow, expressed as a simple question which we can attempt to answer using a questionnaire.
TASK: Selecting an appropriate focus
Choose one of the following and turn it into a research question with an appropriate focus:
I’m interested in how young people choose their bank.
I’m interested in whether there’s a generation gap between young people and older people.
I’m interested in how much young people know about the NGO Oxfam.
2.2 General features of a good questionnaire
(i) Main parts
A good questionnaire will include the following main parts:
| Title | Make sure respondents have a clear understanding of what the questionnaire is about. It may not be necessary to reveal your exact research question, but be clear about what you want to find out. |
| Instructions | Both general instructions and more specific ones where needed. |
| Questionnaire items | See 2.3 |
| Additional information | For example, contact name; how to return questionnaires; a promise to send a summary of findings if the respondent is interested; an invitation to volunteer for interview? |
| Thank you | Treat your respondents with the respect they deserve. A short ‘thank you’ is easy, but will earn you much goodwill. |
(ii) Length
How many minutes would you be willing to spend answering a questionnaire designed by someone you don’t know?
| < 5 | 5 -10 | 10 -15 | 15-20 | > 20 |
(iii) Privacy
Are there any circumstances when you would prefer not to provide your name?
If so, what are they?
2.3 Questionnaire items
For an excellent overview of the types of questionnaire items that you can design, see this page:
Source: https://www.outsource2india.com/kpo/articles/questionnaire-types-of-questions.asp Retrieved on 23 February 2016.
(i) Closed-ended items
Close-ended (or simply closed) items provide respondents with ready-made options to choose as a response:
Examples:
Simplified Chinese characters should be taught more widely in Hong Kong.
| Strongly agree | Agree | Neither agree nor disagree | Disagree | Strongly disagree |
Studying for an honours degree is:
difficult ___ : ___ : ___ : ___ : ___ : ___ easy
useless ___ : ___ : ___ : ___ : ___ : ___ useful
TIP: See the link above for a more comprehensive overview of closed item types. |
(ii) Open-ended items
Open-ended items do not provide respondents with ready-made options to choose as a response: e.g. What is your opinion of questionnaires?
TASK: Advantages and disadvantages of open-ended items
Think of one possible advantage and one possible disadvantage of open-ended items:
| One advantage of open-ended items is …………………………………………………………. One disadvantage of open-ended items is ……………………………………………………… |
Open-ended items often work best when they contain some guidance. Here are four ways you can guide respondents towards the kind of answer that might be useful for you:
| OPEN-QUESTION TYPE | EXAMPLES | |
| (i) Specific open questions | What is your main source of news? Where were you born? | |
| (ii) Clarification questions | If you rated the workshop as poor or very poor, please briefly explain why. If you answered None of the above, please specify: | |
| (iii) Sentence completion items | One thing I liked about this workshop is ………………………….. One thing I didn’t like about this workshop is…………………….. I found this workshop ……………………………………………… | |
| (iv) Short-answer questions | Keep these as focused as possible, dealing with one concept or idea at a time: | |
| | What was your opinion of the workshop? | |
| | What did you find most useful about the workshop? What were the most effective aspects of this workshop? What were the least effective aspects of this workshop? How could this workshop be further improved? etc. |
2.4 Multi-item measures vs. single-item measures
Quickly (i.e. poorly) constructed questionnaires typically ask just one question about each piece of information the researcher wants to learn about.
Example:
Let’s say we want to know to what extent people’s consumption decisions are influenced by others. We could simply include this item:
I frequently gather information from friends and family about a product before I buy.
| Strongly agree | Agree | Neither agree nor disagree | Disagree | Strongly disagree |
However, your precise choice of words can have a significant impact on responses. Questions that look similar actually produce very different results. For example, changing forbid to not allow can affect how some people respond. So, the following two items might elicit different responses depending on the choice of verb:
Do you think that Hong Kong should forbid people to wave the colonial Hong Kong flag in public?
Do you think that Hong Kong should not allow people to wave the colonial Hong Kong flag in public?
For this reason, it is usually best to look at a problem from multiple angles by using variations of the same item. This spreads the burden of proof across multiple questions and can provide a more reliable and valid response. Several variations of the same item is called a multi-item measure. It is particularly important for eliciting information about respondents’ awareness, beliefs, attitudes, opinions, values etc.
Example:
Remember that we want to know to what extent people’s consumption decisions are influenced by others. Instead of asking just one question to measure this, we can include variations of the same question2:
- I frequently gather information from friends and family about a product before I buy.
- I often consult other people to help choose the best alternative available from a product class.
- To make sure I buy the right product or brand, I often observe what others are buying.
- If I have little experience with a product, I often ask my friends about the product.
We can then sum all responses. When we do this, any idiosyncratic (= strange or unusual) responses should be averaged out and no individual item carries an excessive load.
TIP: A possible disadvantage of using multi-item measures is that respondents may react negatively if they feel that your questionnaire is trying to trick them or test their honesty. To avoid this, skilful design of multi-item measures is crucial.
TASK: Designing a multi-item measure
Let’s say that you want to investigate people’s perceptions of media freedom in Hong Kong. Here is one item you could include:
The media in Hong Kong is genuinely free.
| Always | Usually | Sometimes | Occasionally | Never |
Now use the space below to suggest some possible variations on this item:
2.5 How to write good items
To write good items, you need:
- Some creativity
- A lot of common sense.
Bad questions, on the other hand,distort or prevent communication between the researcher and the respondent. Let’s look at these in more detail.
Five guidelines to writing good items
| 1 | Use simple language On balance, would you say that the volume of tourism experienced by Hong Kong over the past decade is beneficial for the Hong Kong economy, taking into account such factors as employment, income, balance of payments and inflation? Short questions are good questions. Always use simple, clear natural language. |
| 2 | Ask one question at a time How do you rate your ability to communicate in Chinese and English? 1 2 3 4 5 Do you brush your teeth after all three meals? YES NO Smartone offers the fastest and most economical mobile data service. YES NO If respondents answer at all, you won’t know what the answer means. Instead, break up these items. For example: Do you brush your teeth after breakfast? YES NO Do you brush your teeth after lunch? YES NO Do you brush your teeth after dinner? YES NO |
| 3 | Avoid negatively-worded items whenever possible Mass tourism does not benefit the average Hong Kong person. YES NO DON’T KNOW Mass tourism is unhelpful for the Hong Kong economy. YES NO DON’T KNOW Negatively worded items can lead to inconsistencies: Respondents may respond differently to negatively worded items (see 2.4), or they may accidentally agree with a negative statement when they meant to disagree. Researchers may process negatively-worded items incorrectly (e.g. a ‘5’ for a positively-worded item is usually a ‘1’ for a negatively-worded item.) It can be difficult to completely avoid negatively-worded items. Just be aware that negatively-worded items may not be very reliable, so use them sparingly (= only occasionally). |
| 4 | Ensure that items are phrased neutrally Do you agree that law and order is desirable? YES NO DON’T KNOW Do you think that violence is bad? YES NO DON’T KNOW The respondent cannot reasonably answer NO to the above questions. No item should lead the respondent towards giving the answer that the researcher wants to hear. Such items are called leading (= biased / loaded) items as they lead the respondent in a direction that s/he might not otherwise go. Here is another example: Do you agree that the recent performance of your fixed line, broadband mninternet service brings you inconvenience in your daily life? In this example, the questionnaire writer appears to want the respondent to agree. Such a question would NOT be appropriate at honours degree level. Remember: you can lead a respondent to the answer, but that won’t help you make good decisions from the data! Read more at: http://researchaccess.com/2013/07/leading-questions/ |
| 5 | Avoid vague or subjective items Do you learn vocabulary easily? YES NO DON’T KNOW How frequently do you eat meat? 1 2 3 4 5 6 The problem here is that the meaning of easy and frequently might vary between people. How could you rephrase the above items? |
TIP: Pilot your questionnaire if you can. Get feedback from the pilot version and look for ways to improve it. It’s crucial that your data is reliable, i.e. it reflects what people really think or feel. If you don’t pilot your questionnaire, poorly written items may result in poor quality data.
2.6 Grouping and ordering items
When you have written your items, you need to decide on their order. Item sequence is an important factor because it can affect how people respond to your questionnaire. Here are some guidelines.
| 1 | Clear and orderly structure You want the respondents to sense that your questionnaire is well-organized. If they sense it is unpredictable or random, this may affect how they respond. Aim to present your items in a logically connected way with well-signposted sections and sub-sections, and clear numbering. Short linking sentences such as this one might be useful: In this section, we would like to know about your opinions towards….. |
| 2 | Opening questions First impressions are always important, so choose your first few opening questions carefully. You can create a pleasant first impression by ensuring that your first few items are interesting, fairly simple, but still focused. Avoid asking anything threatening or sensitive here. You need to build trust first. |
| 3 | Personal information questions Name, address, sex, marital status, number of children etc. : These questions are best placed at the end of the questionnaire. Novice questionnaire designers often place them at the beginning, where they may reduce interest and motivation, and increase resistance – especially if the respondents consider some of this information as personal and sensitive. Instead, aim to win your respondents’ trust first. |
| 4 | Open ended questions at the end Again, the evidence shows that people are more likely to complete a questionnaire and tackle open-ended questions if the open-ended questions are placed at the end. |
3. ADMINISTERING THE QUESTIONNAIRE
Administration procedures matter! How you administer a questionnaire can have a real impact on the quality of responses that you receive.
3.1 Selecting the sample
(i) How should I select people?
The key thing is to make sure that your questionnaire is relevant to the people responding to it. Paying attention to questionnaire relevance will increase the number of responses and the quality of the data they contain.
Relevance starts with a good understanding of who you need to survey to meet your research objectives. You can use screening questions to ensure that only those people who fit your preferred target audience actually respond to the survey. For example, if you only want to survey IT professionals, you might start by asking:
Do you consider yourself an IT professional?
Those who reply ‘Yes’ can then continue. There is no limit to the number of screening questions, but practical considerations limit them to 3 – 5 per questionnaire.
You can also use skip questions to increase the relevance for respondents. A typical questionnaire may include items that are intended only for certain sub-groups, IT managers for example. Skip questions allow respondents to skip certain items that are not relevant for them. Online survey tools make this very easy, but it is also possible in print and phone surveys.
For LCS1088, you will probably use convenience sampling (i.e. you will survey whoever is available and meets some basic criteria). Professional researchers may select a sample using more sophisticated techniques based on specific demographic characteristics.
(ii) How large should the sample be?
People who ask this question usually mean What’s the smallest sample I can get away with? There are no clear answers to this question. Here are some important considerations:
- When you select your sample size, it is advisable to leave a margin to allow for unplanned circumstances – people who don’t complete your questionnaire, incorrectly completed questionnaires etc.
- It can be useful to survey enough people to produce a normal distribution. When you have an approximately normal distribution, it becomes possible to make generalizations from your data about the wider target population (who did not complete your questionnaire). When there is enough data to make such generalizations about a wider population, your results are said to be statistically significant. For this, you typically need a sample size of about 30 or more.
- The more scientific the sampling procedure, the smaller the sample can be. This is why professionally conducted opinion polls can produce accurate predictions from samples as small as 0.1% of the population. The survey research literature suggests a ‘magic sampling fraction’ of between 1% and 10% of the target population.
3.2 Strategies to increase the quality and quantity of responses
| THE BAD NEWS: | It’s difficult to get respondents to spend enough time and effort completing questionnaires. |
| THE GOOD NEWS: | Most people like expressing their opinions and don’t mind answering questions – provided that they think that a) it’s for a good cause; and b) their opinion matters. |
With the above good news in mind, here are some tips to help you persuade respondents give truthful and thoughtful answers. These apply to a wide range of research contexts, so you will need to decide which ones are applicable and/or achievable in any specific context.
Advance notice
Announce the questionnaire a few days in advance if possible. This will help to generate a positive atmosphere and increases the professional feel of the survey. This in turn promotes positive participant attitudes.
Behaviour of the survey administrator
How you dress, how you introduce yourself, your own keenness and motivation and your overall professionalism: these factors can influence how seriously people take your survey.
Communicating the purpose and significance of your survey
People are usually happy to answer questions provided that they understand they are being asked. So, tell respondents what you are trying to find out. It may make them more helpful and more likely to respond appropriately.
Emphasising confidentiality
If you suspect that participants may be concerned about confidentiality of their responses, be clear about how you will handle the data. This may persuade some participants to provide more open, honest and full answers.
Questionnaire instructions
Of course, these should be clear. But you could go further: people tend not to read written instructions. (This is a general experience in educational psychology). So, it is advisable to read the instructions aloud while the participant(s) read silently.
Style and layout
Ensure that the questionnaire is attractive to the eye. Always remember that every detail of your questionnaire should be designed to encourage people to continue to cooperate.
Promising feedback
It shouldn’t all be about you. Promising feedback is something that YOU can do for your participants. Not everybody will want this, but it’s a nice gesture towards people who are giving up their time for you. A simple way is to include a box for people to tick if they would like to receive more information. If anonymity is important, you can collect contact details separately.
3.3 Ethical considerations
Finally, three key principles to ensure that you approach your research ethically:
| Principle 1: | No harm should come to anyone as a result of taking part in your survey. |
| Principle 2: | Always respect the respondent’s right to privacy. Respondents always have the right to refuse to answer any items without explaining why. You cannot publish any information about identifiable people without their permission. |
| Principle 3: | It is your moral and professional (and sometimes even legal) obligation to maintain the level of confidentiality that you initially promised. Don’t promise a higher level of confidentiality than you can achieve. |
3.4 Checklist for administering your questionnaire
Whether you brief people verbally or in writing, here is a checklist of things to remember. If you follow these steps, you are likely to receive more and better quality responses:
| What to include | DONE |
| Introduce your study. Explain its purpose and potential usefulness | |
| Explain why the particular participants have been selected | |
| Assure confidentiality | |
| Say how long it usually takes to complete it | |
| “Any questions?” | |
| “Thank you!” |
REVIEW: Constructing & administering a small-scale research study
Use this checklist of the key steps involved in constructing and administering a small-scale research study. In the following class, we will turn our attention to processing and reporting the data you collect.
| Focus | ||
| – Appropriate research question | | |
| Questionnaire design | ||
| – Relevant questions | | |
| – Multi-item scales | | |
| – Clear layout | | |
| – Clear instructions | | |
| Administration | ||
| – Relevant respondents | | |
| – Large enough sample size | | |
If you have indicated for any of the above steps, make sure you revise the appropriate step(s) before distributing your questionnaire.
B. Processing and reporting data for a small-scale research study
Objectives
By the end of this section, you should be able to:
- process information of a small-scale research study to produce basic descriptive statistics to describe your data set
- summarise the design and methodology of the study, including any possible limitations.
- summarise and report the findings of the study using an appropriate mix of figures/tables and text.
- present any conclusions that can legitimately be drawn.
1. PROCESSING YOUR DATA
Congratulations. You have constructed and administered a questionnaire to answer a research question that interests you. You now have piles of data. What now?
Basic data processing is very simple and involves descriptive statistics. When people use descriptive statistics, they talk about the data they have, i.e. the data collected from the respondents. To really dig deep into your data and explore what it means, you will need a working knowledge of inferential statistics. When people use inferential statistics, they talk about data they don’t have, i.e. they use what they know about the people who took part in the study and attempt to generalize about larger groups of people. Inferential statistics is beyond the scope of LCS1088. Instead, we will limit our focus to descriptive statistics, i.e. summarizing what we actually know about the sample of people we surveyed.
1.1. Code and input your data
Depending on the focus of your study and the nature of your questionnaire, you may need to assign an identity code to each questionnaire, for example by numbering each questionnaire from 1 to 30.
You will need to convert any alphabetic information into numerical information (e.g. strongly agree = 5; strongly disagree = 1)
TIP: Negative & positive scoring of multi-item measures
Beware! When you process data from multi-item measures, remember to ensure that it is consistent. In this example, ‘Never’ for item 1 is consistent with ‘Always’ in item 2:
1. In the past four weeks, how often have you felt very nervous?
| Always | Usually | Sometimes | Occasionally | Never |
2. In the past four weeks, how often have you felt calm and peaceful?
| Always | Usually | Sometimes | Occasionally | Never |
You are now ready to input your data in Excel or similar software. Mistakes happen surprisingly often when you inputting data, so make sure you check it for any coding or input errors.
1.2 Generate descriptive statistics
Descriptive statistics generally include:
- Overall sample size
- Sample sizes of important subgroups (e.g. male/female)
- Averages (usually the meanbut the median or mode might be appropriate if the data does not follow a normal distribution)
- The standard deviation (sometimes called the variance)
- Maximum and minimum values.
1.3 Analyse content of any qualitative items:
Finally, you will need to decide how to process any qualitative, open-ended items included in your survey. Including a small number of qualitative items is often a good idea (see 2.2 below), provided that you process it effectively. This usually involves coding it in some way to transform it into quantitative data.
2. SUMMARIZING AND REPORTING QUESTIONNAIRE DATA
2.1 Some general guidelines for summarizing and reporting questionnaire data
Two common issues in reporting the findings of small-scale research studies are:
(i) How much to generalize
Big claims can usually be made only by conducting big studies. In a small-scale study, you will probably not have enough data to make claims about the population beyond the participants themselves. This will probably be a limitation of your study (see 2.2 below). This is not a problem provided that you don’t make claims about the wider population which your study cannot support. In a small-scale study, the best you can usually do is to suggest more, larger studies in the future to investigate any apparent trends more scientifically.
(ii) Detachment from real life
In a small-scale study, you cannot usually generalize to make the data more meaningful (see above). As a result, small-scale studies can be rather detached from real life and hence rather thin and dull if they rely entirely on the quantitativeresults of a questionnaire. One way to enliven your data is to include some qualitativeinformation. One or two open-ended questions in your questionnaire will enable you to collect some quotations, which can help to retain or restore a more human perspective when reported. You will, however, have to decide how to process this data (see 1.3 above).
2.2 Reporting how your study was designed
Your findings should be accompanied by a concise but detailed summaryof your methodology, including any possible limitations of your study. This is important because it establishes your credibility as a researcher. A reader might ask, “Can I trust the findings of this study?” Here are two things you can do to build trust with your readers:
(i) Show that your study was appropriately designed. A well-designed study is said to have construct validity, i.e. it is capable of measuring what it is designed to measure.
(ii) Be honest about its limitations. There is no perfect study: ALL studies have limitations and readers must decide themselves how valuable the findings are. This is especially important if you attempt to make any generalizations about a wider group of people other than those who took part in your study.
TASK: Identifying key information about a real study
Read pp.98-99 about the design of the following study (a hardcopy should be available):
Sung, C. C. M. (2014). Hong Kong university students’ perceptions of their identities in English as a Lingua Franca contexts: An exploratory study. Journal of Asian Pacific Communication, 24(1), 94-112. doi: 10.1075/japc.24.1.06sun
As you read, use this checklist to identify what information the author includes:
| REPORTING HOW YOUR STUDY WAS DESIGNED | / | EXAMPLE(S) |
| PARTICIPANTS (i.e. the sample) | ||
| Description of the sample: the exact details will depend on the focus of the study but normally include at least the following: | ||
| Total number (possibly with some justification for the sample size and the total number of all eligible people) | ||
| Age | ||
| Gender | ||
| Ethnicity | ||
| Additional details (Depends on the study, but might include native tongue, job types, social class etc.) | ||
| Sampling method used for selecting the participants | ||
| If the sample consisted of different groups, similarities and differences between them. |
| QUESTIONNAIRE | ||
| Description of / justification for the main content areas covered | ||
| Factual description of the questionnaire (with the questionnaire possibly attached as an Appendix) (e.g. Number of parts, number of items, types of items, scoring procedures) | ||
| Justification of why some potentially important areas were left out | ||
| Details of how the questionnaire was piloted | ||
| Any available data about the reliability and validity of the questionnaire. | ||
| For multi-item scales: how many items on each scale | ||
| Details about how confidentiality/anonymity was handled. | ||
| Procedures used to administer the questionnaire | ||
| Length of time required to complete the questionnaire | ||
| Duration of the complete survey (i.e. from (date) to (date)) | ||
| Questionnaire return rate | ||
| PROCESSING THE DATA | ||
| How the data was processed | ||
| List of all variables derived from the questionnaire data | ||
| LIMITATIONS | ||
| Any circumstances (foreseen or unforeseen) that may have affected the results in a systematic way. | ||
| Problems related to sample size and representativeness | ||
| Any potential biases of the sample (composition, selection procedure, non-participation, dropout rate etc.) | ||
| Biases stemming from missing data | ||
| Problems with the research design |
Compare your answers with a partner.
TIP: Using the checklist above
The exact information included about how a study was designed will vary slightly from study to study. As you have seen, not everything on the checklist above was included in the example paper. Use the checklist above as guideline, not as a definitive list. Ask yourself this question: “What do I need to include to ensure that my small-scale study is credible?”
TASK: Redrafting a participant description section
- Read the participant description section below for a SPEED undergraduate Integrated Study project about the relationship between Facebook usage and self-esteem. Then look at the tutor’s actual comments that follow it.
- Brainstorm possible additional information to satisfy the tutor’s comments.
- Re-draft the text to improve both a) the content, and b) the accuracy of the language.
Method
i) Participants
Undergraduates (n=154) from HKCC and SPEED Hung Hom Bay campus have completed the questionnaires. The response rate is 97.4%. Four male participants are excluded from the study as they never use Facebook. With the population size of 150 participants who have completed the questionnaire, 46% of the participants are female while 53% are male. Participant ages ranged from 18-24. Participant taking part in the current study must have a Facebook account. Participant who fails to meet the requirement will be excluded from the current study.
Tutor’s actual comments:
- Elaborate and state clearly the characteristics of the participants
- How were participants recruited?
- Any other exclusion criteria?
2.3 How to make your data digestible and reader-friendly
Questionnaires often provide a lot of data. Deciding how best to present this data to your audience is an essential skill for researchers. Here are two pieces of advice to ensure that your data is both digestible and reader-friendly:
(i) Whatever can be presented in figures and tables should be
Wherever possible, find ways to visualize your data instead of writing about it. The impact on your audience will be greater. Figures such as bar charts, line charts, pie charts (see below), flow diagrams etc. are especially effective. Tables of data can also be effective.
(ii) The text should comment on and interpret the data, not describe it
When you present your data in figures or tables, it is not necessary to describe it again in words.
Compare:
| | For the statement ‘My workload is reasonable’, 8.2% of respondents strongly agreed that their workload was reasonable, 39.3% agreed with the statement, 18.7% were neutral, 23.7% somewhat disagreed and 10.1% strongly disagreed. The mean score was 2.88. |
| | For the statement ‘My workload is reasonable’, it was notable that about one-third (33.8%) of the respondents disagreed or strongly disagreed. Although this was only a small-scale study, this may indicate that a significant number of PolyU SPEED students struggle with their academic workload. |
Notice how the second version does not waste words repeating what the reader has already seen in the pie chart. Instead, it helps the reader to understand what might be important about the data, to understand what the data might be telling us.
So, when you provide a figure or table, aim to interpret it for your readers: you might comment on key trends or other highlights; or you might discuss any implications of the data. In this way, you add value.
Here is a three-step technique to help you:
Step 1: Introduce the figure or table: e.g. Table 1 below presents the….
Step 2: Provide the figure or table
Step 3: Comment on the figure or table. E.g. What is important? What is unexpected?
Example:
Table 1 below presents the most recent figures for new applications to PCSSA. [STEP 1]
Table 1: Number of new applications to the PCSSA scheme [STEP 2]
| Year | No. of new applications |
| 2011-12 | 2985 |
| 2012-13 | 2660 |
| 2013-14 | 2378 |
| 2014-15 | 2241 |
| 2015-16 | 2134 |
Source:Examination of Estimates of Expenditure, 2015
| According to the government’s own figures, the number of new applications has been falling steadily for several years. This falling trend is a cause for concern and calls into question the effectiveness of the scheme. | [STEP 3] |
** Step 3 is commonly missing in undergraduate writing. **
TASK: Redrafting poorly presented data
Using the three-step technique above, re-draft the text which supports this simple table:
| Questionnaire | n | Mean | Standard deviation |
| Time spent on Facebook | 150 | 3.32 | 0.84 |
| Self-esteem self-rating | 150 | 3.55 | 0.73 |
Table 2: Descriptive statistics
From Table 2, it shows that the mean score of the Facebook intensity questionnaire is 3.32, with a standard deviation of 0.84. This means that there are considerable differences between participants regarding the amount of time that they spend on Facebook.
For the self-esteem questionnaire, the mean is 3.55 with a standard deviation of 0.84. This means that the distribution of participants’ scores is not even.
| Comments: The commentary reports information which readers can see for themselves in the table. Useful commentary on the importance of the high standard deviation – although no indication of the scale! (Note: When you re-draft it, assume that the scale is 1-5) Self-evident, word-wasting commentary: the distribution of participants’ scores is not even. Transfer of mother-tongue pattern: From Table 2, it shows that…. |
Re-draft:
…………………………………………………………………………………………………………
…………………………………………………………………………………………………………
…………………………………………………………………………………………………………
…………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………
…………………………………………………………………………………………………………
…………………………………………………………………………………………………………
…………………………………………………………………………………………………………
REVIEW: Processing and reporting on a small-scale research study
Use this checklist of the key steps involved in processing and reporting a small-scale research study:
| Processing | ||
| – Code and input data | | |
| – Generate descriptive statistics | | |
| – Analyse content of any qualitative questions | | |
| Reporting | ||
| – Summarise study design and methodology | | |
| – Describe any limitations | | |
| – Summarise and report findings using an appropriate mix of figures/tables and text. | | |
| – Present any conclusions that can legitimately be drawn | | |
If you have indicated for any of the above steps, make sure you revise the appropriate step(s) before distributing your questionnaire.
1 In section 3.1 we will further explore how to select an appropriate sample of people to answer your questionnaire.
2 Source: Bearden, W.O., Netemeyer, R. G., and Teel J.E. (1989), Measurement of Consumer Susceptibility to Interpersonal Influence. Journal of Consumer Research, 15 (March), 473-481. Retrieved from http://surveyanalysis.org/wiki/Multi-Item_Scale#cite_note-1 on 22 February 2016