Chapter 8 Rubrics
Experimental designcraft assessment framework
There are at least two primary modes of assessment (Kennedy et al. 2008). Formative assessment can happen during the learning process (Bennett 2011). This active process of engagement with content and doing experiments is critical to becoming an effective life-long learner and successful scientist. In practicing experimental design and doing experiments professionally, this can take the of form of notes, sketches, photographs of the process or experiment at different steps, flowcharts, field and lab notebooks, code, and discussion with collaborators. This process of learning can include feedback from the team (in this course the teaching assistant, the instructor, or peers examining the same challenge). It can be enabled by testing how well one has advanced in achieving specific outcomes. For instance, share your meta-data with a peer and explore whether the individual can understand the meaning of the data and the process of experimentation that supported the collection or reuse of data. Summative assessment can happen at the end of key benchmarks in a learning cycle or at the completion of logical stopping points within the learning process that generated concrete products for review and grading (Taras 2005). In this designcraft process of actively exploring experimental design, this can include production of data with meta-data, a lab report describing the deeper dive for one of the field experiments, and a lab report describing the design process of data reuse from one the examples provided. The process of formative asessment (steps along the way) and summative assessment (final products) should support one another to consolidate learning (Harlen and James 1997).
A rubric is a scoring tool that enables fair, transparent and replicable grading in summative evaluation (Timmerman et al. 2011). Checklists are useful for formative self or peer assessment in the steps along the way to final products. In designcraft for experiments, this applies to the published data with meta-data and lab reports. In the formal offering of these labs for the course ‘SC/BIOL 3250 4.00 Experimental design for environmental and evolutionary biology’ at York University, the lab component is worth 50% of the final grade.
Rubrics
Formative = checklist for you to mentally tick of as you do work.
Summative = graded by educator to assess work and provide marks.
Summative marking key for published datasets
This is the marking key you are looking for. This key is used for the field datasets to ensure that you can improve and learn from the process. Reminder, metadata are the ‘data about the data’ - i.e., the descriptions of what each column header means, a list of the key concepts, and a description of the terms within each specific variable. Sometimes, it is really useful to make a short meta-data table to describe the data and upload that file with the observational or experimental data too (Lortie et al. 2022).
| item | description | criteria | score |
|---|---|---|---|
| 1 | data | tidy (rows and columns logical and organized), clear labels, no errors | 2 |
| 2 | data | observations meaningful, accuracy, sufficient (i.e. sufficient replication based on the principles discussed in labs & lecture relevant to the design you used) | 2 |
| 3 | meta-data | every variable or column clearly described for the reader, units not within cells, time of day or understanding of format of data clearly described | 2 |
| 4 | meta-data | description ensures the process of observation be repeated by another, like a mini-methods to a paper, either as table and/or paragraph | 2 |
| 5 | open science | details sufficient for open science (include Figshare url in the PDF you submit for grading), include clear title to data, location, and details in a short paragraph and/or metadata table so that someone else can understand the experimental data you collected, ensure that the data with metadata are a coherent, collective package of evidence that can be used to enable synthesis science as a whole | 2 |
Summative assessment marking key for field lab report
This is the marking key for the field lab report. Single or double spaced, 12 point font, at least 1 inch margins (the default). PDF format only. Lab reports must also be submitted to turnitin.com. We strongly prefer captions for the figure and figure at the end of the report.
Facets journal is Canada’s first and only multidisciplinary open access science journal. Follow the instructions proposed for a research article for this journal. Please keep the length of your article between 2000 to 5000 words.
| page | concept | description | value |
|---|---|---|---|
| 1 | Title & abstract | Title of experiment (like you would see for a peer-reviewed scientific publication, meaningful, and captures the purpose of experiment you did), your name, ID, contact details, Abstract that describes experiment | 5 |
| 2-3 | Introduction | Sets the context, explains why study needs to be done, state hypothesis and predictions (i.e. the purpose that you defined and why you collected this specific data using this design) | 5 |
| 4 | Methods | Described well enough for someone else to replicate design, explain the relevant details, consider subheadings in Methods such as site, design, and statistics | 5 |
| 5 | Results | Clear text should be able to stand alone including description of statistics, consider presenting main findings simply and mentioning covariates and confounding only very simply | 5 |
| 6-7 | Discussion | Restate findings in brief and then propose significance of the work and/or link to the published work to date, support, reject, extends what we know | 5 |
| 8 | Literature cited | At least 5 recent papers (typically defined as published in a peer-reviewed journal within last 5 years) on topic and 2 on design decisions, any standard citation format such as APA acceptable, the two paper on design decisions can be from general how-to papers or from specific, similar published studies to your lab experiment, we will check for the dates of publications and that there are at least 7 in total that seem reasonable | 0 |
| 9 | Figure legend | Figure legend describing what the figure shows, plus put at end of the report before the figure, legend is a short description of what the figure is all about | 2 |
| 10 | Figure | Figure at the end of report, on a single page, you need a plot (or table, we can be flexible, but with summary data not raw data) showing the mean, sample size, or if a figure, showing the trend or difference that you tested with statistics | 3 |
Summative assessment marking key for data-design lab report
This is the marking key for the data-design short report. Single spaced, 12 point font, at least 1 inch margins (the default). PDF format only. Lab reports must also be submitted to turnitin.com.
Facets journal is Canada’s first and only multidisciplinary open access science journal. Follow the instructions proposed for a note for this journal. Please keep article at 1400 words or less.
| page | concept | description | value |
|---|---|---|---|
| 1 | Title | Title of experiment (like you would see for a peer-reviewed scientific publication, meaningful, and captures the purpose of experiment you did), your name, ID, contact details, no abstract needed for this super short paper | 0 |
| 2 | Introduction | Single paragraph stating background, hypothesis, and your prediction(s) | 3 |
| 3 | Results | Single brief paragraph stating findings (including basic summary statistics) and single figure with figure legend (legend and figure at the end of paper) | 5 |
| 4 | Conclusions | Single paragraph stating conclusion and implications of what you tested with these data | 2 |
| 5 | Lit cited | A total of 3 references, recent published within the last 5 years in a peer-reviewed journal | 0 |