| Steps | Lead Assessor | Co–Rater(s) |
| 1 | Create New Observation on your tablet. | Create New Observation on your tablet. |
| 2 | Complete Observation. | Complete Observation. |
| 3 | With your completed observation open, tap Tools then Inter-rater Reliability Tool. | With your completed observation open, tap Tools then Inter-rater Reliability Tool. |
| 4 | Select “I am the Lead Assessor for this reliability visit”. | Select “I am a co-rater for this reliability visit”. |
| 5 | Give the code that appears on the screen to your co-rater(s). | Enter the code given to you by the lead assessor. |
| 6 | When you are sure co-rater(s) has entered code, tap Done. | Once code is entered, tap Done. |
| 7 | A message will appear telling you that the system will make a copy of your assessment a nd the consensus observation will now open automatically- tap OK. You will notice that this new observation is a different color and has Assessor Scores in the lower left corner. This will be the observation you work with for the remainder of the reliability. | Do nothing. |
| 8 | Check the box next to each co-rater doing the reliability with you, then tap Next. | Do nothing. |
| 9 | Under the appropriate column, enter the score for each co-rater on the IR Reliability Worksheet. You must include items with a score of NA. | Read off your scores from the IR Reliability Worksheet to the lead assessor. |
| 10 | If your organization excludes any subscales or items for IR (ex., Parents and Staff) be sure to mark No in the “Include in Average?” column for the item(s ). | Tap Close. |
| 11 | When all co-rater scores have been entered completely, tap Close. | Do nothing. |
| 12 | Start the consensus discussion with your co-raters. When you have come to a decision, you will change the consensus observation you have open, to match the a greed upon score (add/change any observation notes as needed). Remember this consensus observation starts out as a copy of your completed observation. Check the “Consensus Reached” box for each item, once you have agreed upon the score. | Use your complete observation to come to consensus with lead assessor and other co- raters. Your Reliability Observation is now Read-Only, you cannot make changes to it. |
| 13 | A blue square icon next to each item in the item index indicates that consensus has been reached. A blue square icon next to the date and name of the facility in the item index indicates you have reached consensus for the observation. | Use your complete observation to come to consensus with lead assessor and other co-raters . |
| 14 | With your consensus observation open, tap Tools then Inter-rater Reliability Tool. Each participant’s overall reliability will be listed at the bottom of the screen. | Close your observation. |
| 15 | Create Reliability Report by tapping “Run Report”. You can save this report in PDF format to email or print out for the participants. | Get a copy of the Reliability Report from your lead assessor. |
| 16 | Run Flag Missing Justifications and Spell check on Consensus Observation. | Do nothing. |
| 17 | Close Consensus Observation. | Do nothing. |
| 18 | Upload you Reliability Observation (this assessment is now Read-Only, you cannot change it). | Upload your Reliability Observation. |
| 19 | Upload the Consensus Observation. | Do nothing. |
| 20 | Your supervisor/anchor will provide feedback or finalize your consensus observation. You will now treat this observation as you would any other completed observation. | Do Nothing – your part is complete! |
© Copyright Branagh Group
