Survey research and design in psychology/Assessment

Overview

1. There are 9 online quizzes (5% each; total 45%), a data collection and entry exercise (10%) and a lab report (45%).
2. All assessment is optional. Non-completed assessments will be awarded 0.
3. A final overall mark of 50% or higher is required to Pass the unit. Partial final marks will be rounded up.
4. Summary of assessment items:

Quizzes

1. Topics: There are 9 online quizzes, each worth 5% (total 45%), with the following topics:
   1. Survey research
   2. Survey design
   3. Descriptives and graphing
   4. Correlation
   5. Exploratory factor analysis
   6. Psychometrics
   7. MLR 1
   8. MLR 2
   9. Power and effect size
2. Availability:
   1. Quizzes will be available until Mon 09:00 Week 14.
   2. Moodle quizzes are only available to enrolled University of Canberra students.
3. Late submissions: Not accepted
4. Assessed knowledge: Quizzes assess knowledge of concepts covered in lectures, tutorials, and readings.
5. Quiz style:
   1. Quizzes typically consist of 10 multiple choice questions drawn randomly from a larger test bank.
   2. Some questions have more than one correct answer. In order to get full marks, all correct the answers must be selected and no incorrect answers must be selected. For example, a question has 5 answer choices, 2 of which are correct (worth 50% each) and 3 of which are incorrect (worth -33% each). The mark awarded will be the total of the marks for each of the answers selected. However, there are no negative total marks for a question, so the lowest mark that can be awarded for a question is 0.
   3. Some quizzes are harder than others (e.g., the lowest mean is typically for Quiz 05: Exploratory factor analysis).
6. Time limits: Quiz time limits vary between 10 and 20 minutes, depending on the complexity of the questions. If an attempt goes over time, 0 will be awarded for the quiz. It is recommended to submit at least 5 to 10 seconds before the maximum allowable time. A timer will be shown in the top left corner. To help keep track of time consider:
   1. Working from bottom to top because the timer may be more easily viewable towards the end of the quiz
   2. Using a separate stop watch to keep track of time
7. Academic integrity: Quizzes are to be completed independently (i.e., without assistance of others), in your own time.
8. Open book: Quizzes are open book - recommended resource materials include lecture, tutorial, and reading notes.
9. Number of attempts: Each quiz can only be attempted once.
10. Feedback: Answers to the quiz questions are shown immediately after quiz submission, but are no longer accessible once the feedback window is closed. Take any notes for study purposes prior to closing the feedback screen.
11. Bonus marks: Bonus marks are awarded to anyone who identifies errors or significant improvements to any quiz questions - email details to the unit convener.
12. Practice quizzes: Some practice quizzes are available:
    1. Quiz 0 (a practice Moodle quiz) can be attempted as many times as you like. Use this quiz to test and make sure that you are familiar with the Moodle quiz system works on the computer you are using.
    2. Several practice quizzes are available on Wikiversity, including:
       1. Research types
       2. Research purposes
       3. Correlation
       4. Exploratory factor analysis
       5. Psychometric instrument development
       6. Multiple linear regression
    3. Pearson - see External link sections on the lecture pages

Data collection and entry

This assessment exercise provides hands-on involvement in survey data collection and data entry. It requires collecting data using a survey, entering data, submitting an electronic data file, and submitting the hard copy surveys in an accurate, timely fashion as part of a larger class project. The specific steps are to:

1. Collect 5 cases of survey data by following the survey administration guidelines.
2. Enter the survey data by following the data entry guidelines.
3. Submit:
   1. Electronic data file via Moodle by Monday 09:00 AM of Week 4 - no late submissions accepted
   2. Hard copy surveys as a single bundle with a cover sheet by Monday 09:00 of Week 6 - no late submissions accepted - see hard copy survey guidelines

Important note: In order to earn any marks, it is necessary to submit both electronic and hard copy data on time. No marks will be provided if either the electronic or hard copy submissions are missing or late.

Lab report

Overview

1. Present an independently-developed APA style lab report which examines:
   1. the psychometric properties of a multi-item, multi-factorial measure (either university student satisfaction or time management) using exploratory factor analysis and reliability analysis (guided by a research question)
   2. an explanatory model involving at least three predictors and one dependent variable using multiple linear regression (guided by hypotheses - one per predictor).
2. Use the aggregated data file from the data collection and entry assessment exercise, based on the TUSSTMQ9.
3. For further information about requirements, see the general marking criteria and the detailed marking criteria which explains what to include for each section.

Data download, screening, & recoding

1. Data download: TUSSTMQ9_2016_All_Unscreened.sav
2. Data screening
3. Data recoding

Marking process

1. Lab reports will be evaluated according to the general and detailed marking criteria.
2. For reports submitted by the original due date, marks and feedback should be available three weeks of submission. Marking of reports submitted late or with extension may take longer than three weeks from the date of submission.
3. Availability of marks and feedback will be notified via Moodle.
4. If you don't understand or disagree with a mark and/or feedback, then please see this suggested marking dispute process.

General marking criteria

Within each of the lab report sections, marking will reflect these general criteria:

1. 70%: Quality of execution of the task
   (e.g., review literature and develop hypotheses, describe method, summarise results, discuss findings)
2. 10%: Quality of written expression
   (structure and headings, flow, sentence and paragraph structure, spelling, and grammar)
3. 10%: Contribution to a cohesive, meaningful report
   (i.e., a story that makes sense - with sections that are not disjointed)
4. 10%: APA style

Detailed marking criteria

Detailed requirements and suggestions for each of the sections of the lab report:

Section overview

Summary of the marking criteria for each section:

The word count ranges per section are suggestions only. The only restriction is an upper limit on the overall word count.

Cover sheet

1. p. 0
2. To download - go to Lab report cover sheet, then click File - Save As

Title page

1. p. 1
2. Example
3. Do not include your name (so that the report can be blind marked)
4. Title
   1. ~10-15 words
   2. Does the title convey the content of the report?
   3. Is the title succinct, yet specific to the study (e.g., mentions key constructs)?
   4. How catchy or memorable is the title?

Abstract

1. p. 2
2. ~160-200 words.
3. Purpose of the study
4. Summarise the method, avoiding excessive detail
5. Summarise key findings
6. Summarise theoretical and methodological implications and/or conclusions.

Introduction

1. pp. 3-
2. Introduce the topic and concisely explain the study's purpose.
3. Provide a critical overview of relevant past research and identify key issues to be addressed in this study.
4. Only review constructs which are analysed in the Results - e.g., see possible topics.
5. Use citations to key background literature - eReserve offers some starting references, but use of additional references is recommended.
6. Provide logically-derived and clearly-stated research question(s) and/or hypotheses (can be null and/or alternative). The derivation of the questions and hypotheses should be supported by theoretical argument and citations.
   1. Firstly, ask a research question about the underlying factor structure of one of the multidimensional constructs in the TUSSTMQ9 (i.e., either university student satisfaction or time management). (→ Exploratory factor analysis (EFA)) e.g.,
      1. How many distinct dimensions (factors) of X are there, what are they, and which items best represent these factors?
   2. Secondly, make a hypothesis about whether at least three independent variables (IVs) predict a dependent variable (DV) (→ Multiple linear regression (MLR)) (choose any suitable variables, or composite of several variables, from the TUSSTMQ9) e.g.,
      1. It is hypothesised that IV1, IV2, and IV3 will each positively predict DV.
   3. Thirdly (for G students), provide a hypothesis or research question that can be addressed through qualitative analysis of responses to one or both of the open-ended questions.
7. Sub-headings may be used (optional)

Method

1. Clearly explain how the study was conducted in sufficient detail to allow a replication study, but without extraneous detail.
2. Key marking criteria: Is the study replicable? Is sufficient detail provided for a "naive person" (say, someone in Japan in 20 years time) to be able to fully replicate the study?

Participants (5%)

1. Provide a one to two paragraph descriptive overview of the participants in the sample.
2. Consider which of the available data could be summarised in order to provide an insightful description.
3. Advanced option: You may wish to compare the sampled data with population statistics for UC students e.g., UC at a glance, Annual Reports.

Measures (5%)

1. Briefly summarise the survey instrumentation used to collect the data which is analysed in the Results. There is no point in describing measures which are not used in the study. Use a table to help present the information which should include:
   1. The response format (measurement scale), including the meaning/direction of high/low scores
   2. The proposed factors (e.g., label, definitions and example items)

Procedure (5%)

1. Sampling:
   1. What was the target population and the sampling frame?
   2. What sampling technique was used?
2. Administration:
   1. Briefly summarise and reference the Survey administration guidelines
   2. Where and how did you collect data?
   3. How long did participants take?
   4. Refusal rate? (for the surveys you administered)
   5. Procedural anomalies? (e.g., were there any unanticipated responses or unplanned occurrences?)

Results

1. The approach to analysis should proceed through three basic steps:
   1. Screen the data - Summarise how the data was screened
   2. Psychometric instrument development - Conduct an EFA of a multidimensional construct (either the university satisfaction items or the time management items). Then, for each factor, provide reliability analysis (internal consistency - Cronbach's alpha), composite scores and descriptive statistics, and report about the correlations between factors.
   3. Multiple linear regression - Conduct at least one MLR with at least three IVs to address one hypothesis per IV
2. Communicate the depth of your understanding by using your own words; avoid writing results in a robotic (mindless) manner (e.g., paraphrasing sample write-ups)
3. Qualitative analysis (for 6667 (G) students only)
4. Most statistics should be rounded to two decimal places - unless there is particularly useful information communicated by including a third decimal place.
5. Additional analyses may be presented. However, it is quite possible to gain maximum marks by conducting one of each of the required analyses. If additional analyses are presented, then they must be clearly related to the research question and hypothesis(es).
6. In marking, some account will be taken of the scope of the analysis undertaken. Where a more advanced analysis is appropriate (given the research questions(s) and/or hypothesis(es)) and is well conducted, this could represent higher quality work than a simpler analysis. However, there's much also to be said for parsimony (keep it simple, get it right), and focusing on doing a good job of fulfilling the minimum requirements. The best reports are usually not the most complex ones. If in doubt, go with analyses which meet the minimum criteria, which relate to the research question and/or hypotheses, and which you are confident about accurately conducting, interpreting and presenting.

Data screening (5%)

1. Summarise what was done to check and correct the data - see data screening

Psychometric instrument development (20%)

1. Report the results of at least one EFA of a multidimensional survey construct (either university student satisfaction or time management)
   1. The minimum requirement is to report one EFA. However, it may be of interest to examine the factor structure of more than construct in order to develop other composite scores for further analysis. In this case, present one EFA in full and summarise the results of the other EFA perhaps with relevant output in an appendix.
2. Indicate the type of EFA used (Type of extraction? Type of rotation?)
3. Explain the extent to which EFA assumptions were met (sample size (incl. cases:variables ratio), linearity (e.g., check at least some scatterplots, particularly for bivariate outliers or non-linear relations), factorability of correlation matrix)
4. Provide the inter-item correlations for the final model(s) (in an Appendix) - see FAQ for suggestions about how to do this
5. Focus on the final model but summarise the steps taken to get there (e.g., How many factors were extracted initially? What models/factors structures were examined? Was the expected structure evident? Which factors were retained and dropped and why?)
6. % of variance explained (for the initial and final model(s)
7. Label and describe each factor
8. Summarise which items were retained and/or dropped and the reasons why
9. Table of factor loadings and communalities (for the final model)
10. Reliability analysis (Internal consistency - (Cronbach's alpha)) for each factor
11. Calculation of Composite scores to represent each factor
12. Descriptive statistics for the composite scores
13. Correlations between composite scores

Multiple linear regression (20%)

1. Report the results of at least one MLR with at least three predictors - can use any variables in, or derived from, the supplied data set (if they meet the assumptions for MLR)
2. Reiterate the purpose (research question and/or hypotheses) of the MLR
3. Mention the type of MLR
4. Describe the IVs and DVs, and any manipulations of the variables (e.g., recoding or creating an interaction term)
5. Explain the extent to which assumptions were met (e.g., multicollinearity, multivariate outliers)
6. Present the correlations between the items (can be part of the MLR coefficients table - see sample write-ups for examples). Demonstrate understanding of the directions of any relationships (e.g., if there is a positive correlation between X and Gender, what does this mean? Are higher values of X associated with males or females?)
7. Report amount of variance explained (R² and Adjusted R² (and the R² change at each step if a hierarchical MLR is being conducted), with inferential tests (F(df), p)
8. Report significance, size, direction and relative contribution of each IV, based on a table showing the MLR coefficients, including B for intercept & IVs and Beta (β), t, p, and possibly the zero-order (r), partial correlations (r_p), and semi-partial correlations squared (sr²) for each IV and explain the direction and size of the results.

Discussion

1. Build on the introduction to explain the results and what they mean in a balanced manner.
2. Demonstrate breadth and depth of understanding of the results and their implications. Avoid merely summarising the results without providing additional critical commentary.
3. Critically review the strengths and weaknesses of the study's methodology and make practical suggestions for how it could be improved, e.g.,
   1. Validity and reliability of the measures?
   2. Statistical power?
   3. Appropriateness of the sampling technique?
   4. Generalisability of the findings?
4. Provide tangible recommendations for future research and practical implications.

References

1. Is the reference list complete (i.e., none missing and all cited)?
2. Does the lab report make effective use of a core set of relevant, high-quality, peer-reviewed, citations? This involves citing and meaningfully discussing appropriate references (as opposed to just dumping citations without explanation), particularly in the Introduction and Discussion.
3. Reference the instrumentation and the survey administration guidelines - do not copy the guidelines into the Appendices.
4. Use APA style, including for electronic sources. Include DOIs where relevant.

Appendices

1. Appendices are for additional detail which is relevant to understanding the main body, but which would break the flow of the main report.
2. Copies of statistical output can be provided in the appendices, but this is not required. If you do, it may help the marker to diagnose problems with results.
3. Appendix content does not need to follow APA style (e.g., Raw statistical output or other annotations can be provided to demonstrate how the results were arrived at).
4. Appendices should be well organised, with clear and effective labeling.
5. All Appendices should be referred to in the main body.

Style

1. Follow APA style except:
   1. add a cover sheet
   2. use single-spacing (for electronic submission; double-spacing is a relic from paper submission days which allowed room for hand-written comments)
   3. do not include author name on the title page (for blind marking)
   4. formatting of any appendices does not need to follow APA style
2. Example (owl.english.purdue.edu)
3. The most important aspects of APA style for this lab report are:
   1. Times New Roman 12 pt font
   2. Running head with page numbers
   3. Left-justify body text
   4. Captioning and layout of tables
   5. Citations and referencing (including for electronic references and use of DOIs)
   6. Symbols, abbreviations, and formatting of statistical symbols (including M, SD, skew, kurt, N, n, F, t, p, r, R, R², sr²)

Word count

1. There is no minimum word count.
2. Maximum word count:
   1. 2800 words + 10% (for undergraduates)
   2. 3300 words + 10% (for graduates)
3. Count everything using a word processor from the beginning of the Introduction to end of Discussion including all text, headings, footnotes, citations, tables and figures etcetera, but not the sections after the Discussion (e.g., References and Appendices).
4. Penalty for exceeding maximum word count: Markers will ignore words beyond the maximum (i.e., most likely resulting in a reduced mark for the Discussion).
5. Word counts provided in the section overview are suggestions only.

Sample write-ups

It is necessary to demonstrate independent thinking and writing in order to satisfy the learning outcomes. In other words, avoid plagiarising from these samples or other guides. These sample write-ups are provided to give ideas. Also seek out other examples. Then write up your results in your own words.

Write-ups:

1. Lab report from a previous participant: Not everything in this example lab report is correct - in fact, there is a lot of "red ink". This has been done on purpose to illustrate how the report could have been even better.
   1. Example 2007 HD lab report (.pdf) - Marking and feedback sheet (.xls)
   2. Example 2013 HD lab report (.pdf)
2. Sample write-ups for specific analyses:
   1. Exploratory factor analysis (.doc)
   2. MLR (.doc)
   3. The textbooks provide examples of APA style write-ups for specific analyses.

Graduate participants

Graduate participants in this unit are required to produce a more advanced lab report which incorporates a qualitative analysis within a 3,300 word report. The additional requirements are:

1. 1. Introduction: Develop an extra research question and/or hypothesis which sets up a qualitative analysis.
   2. Method: Mention the open-ended question(s) in the Measures section.
   3. Results:
      1. Present an additional analysis, a qualitative analysis of responses to at least one of the the open-ended questions. The analysis could be approached qualitatively (e.g., using thematic analysis) or quantitatively (e.g., using multiple response frequency analysis).
      2. Explain the data coding and/or interpretative process
      3. Present a thick description of the data (if treating qualitatively) or descriptive results (frequencies/percentages via multiple response analysis, possibly with an accompanying figure, if treating quantitatively), with the themes illustrated by representative quotes.
   4. The weighting for the Results section (45%) will be retained, with the sub-section weighting adjusted to: 0% Data screening; 15% Psychometric instrument development; 15% Multiple linear regression; 15% Qualitative analysis

Submission

1. Submit the lab report electronically via the unit's Moodle site - via the Assessment module and then Lab report submission.
2. Follow the lab report guidelines.
3. Acceptable file types: .doc, .docx, .odt, and .rtf
4. Name the file with your student ID number (use the format u######.*** where ###### is the student number and *** is the file extension (e.g., u613374.doc)).
5. Submit a single document which includes the cover sheet, lab report, and any appendices. Multiple file attachments are not accepted. Maximum file size: 20 MB.
6. Use the "Add submission" button or drag and drop the file; if it doesn't display, make sure to login first. Once the correct file has uploaded, click on "Save changes".
7. The submitted file will then be sent to URKUND to be analysed. A report may take ~24 hours. Availability of the report will be notified via email. Further information on URKUND is available from http://learnonline.canberra.edu.au/course/view.php?id=1344&section=4.
8. You may wish to modify the report and re-upload by clicking on “Edit submission”.
9. Click on “Submit assignment” and then "Continue" to submit the assignment. Until this point, the assignment has not been submitted and will not be marked; after this point, the assignment is submitted and can no longer be changed other than by a lecturer manually reverting the submission.
10. An automatic confirmation that the file has been submitted will be sent to the student email address.
11. Do not submit hard copies.
12. Do not send backup email copies.

Late submissions

1. The standard UC policy applies (i.e., a 5% penalty per day (including weekends); no submissions accepted after 7 days.

Academic integrity

1. Emerging academics have a responsibility to uphold University standards on ethical scholarship. Good scholarship involves building on the work of others and use of others’ work must be acknowledged with proper attribution made. Cheating, plagiarism, and falsification of data are dishonest practices that contravene academic values. The Academic Skills Centre provides opportunities to enhance student understanding of academic integrity.
2. Participants are expected to submit independent work on the assessment items.
3. Note the University of Canberra policy on plagiarism.

Extensions

1. Participants are expected to work on the assessment items throughout semester. Extensions will only be granted in exceptional circumstances. Early communication of problems is strongly advised. Participants should assess within the first few weeks of semester whether they have a reasonable likelihood of being able to complete the unit and should consider withdrawing by the census date if not keeping up.
2. Extensions will not be granted for:
   1. Workload (e.g., study load and/or paid or voluntary work load) - such problems should be anticipated and withdrawal from the unit is recommended if workload is a problem.
   2. Technical problems (e.g., hard drive crashes, internet access problems, corrupted disks, viruses) - it is strongly recommended that you keep multiple and regular backups of lab report drafts, data, syntax, and output files.
   3. Undocumented issues.
3. Extension requests should be submitted via email to the unit convener from your UC student email address and include:
   1. Your first and last name
   2. Which unit and assessment item(s) an extension is requested for
   3. Length of extension requested
   4. Reason for the extension request
   5. Attached documentary evidence which specifies the period of incapacity/disruption. Appropriate evidence could consist of:
      1. A medical certificate, including:
         1. Signed by a registered medical, dental, or health practitioner
         2. Registered provider number and provider’s contact details
         3. Duration of incapacity to study (must be during the study period prior to the due date)
         4. Date(s) of consultation
      2. A death notice or other appropriate documentation for bereavement.
      3. A Reasonable Adjustment Plan from Inclusion and Welfare at the University of Canberra