Integration using third-party scoring
A high level technical walkthrough of how a third-party scoring service can be integrated into a typical Learnosity-powered assessment system
Tyrell Corporation provides a service for the assessment of English language proficiency. They offer branded testing services to universities. These services are used to test a university applicant's ability to speak English as part of their application process.
The test environment is personalized, so the look and feel of the test experience matches the university brand. The assessment comprises of three twenty-minute parts.
- Interactive - These are multiple choice and fill-in-the-blanks type questions
- Audio - Here the student provides audio recorded answers
- Essay - In part three, the student writes long-text essays
At the end of the process a simple percentage grade is reported back to the college.
Tyrell Corporation's Needs
The existing version of Tyrell's assessment system uses auto-scored multiple-choice and fill-in-the-blanks questions. In the next iteration, they want to introduce audio and essays as they built a proprietary essay scoring system and established a partnership with a third-party company that specializes in analysing and understanding audio. They would like to:
- Have a single authoring environment where they can create all three groups of questions; multiple-choice, essay and audio
- Present these questions in a white-labelled assessment player which they can personalize for each university
- Have a robust and reliable cross-platform means of capturing audio
- Auto-grade the multiple-choice and fill-in-the-blanks question types
- Have a means of storing all grades in one place so they can generate a single unified report
Tyrell can meet all of their requirements using the Learnosity APIs and hosted services. We will walk through an example where Tyrell has used:
- The Author Site
- This Learnosity-hosted authoring environment will allow Tyrell to author questions in a user-friendly instinctual manner
- Requires zero development time, just log in and start creating Items straight away
- Provides the ability to author 65 different question types including essay and audio
- Provides the ability to create questions where the stimulus and response are both audio
- Provides the ability to create timed guided workflows with instructions presented as pre-recorded auto-playing audio
- Saves all content to Tyrell’s Learnosity-hosted Itembank
- Used to present the assessment to the student
- The Items API pulls items from Tyrell's Learnosity-hosted Itembank
- Along with CSS styling and other customisations, will render a branded version of the assessment in the client's webpage
- Saves the student's work, and submits it to the Learnosity servers for scoring
- A server to server API which gives the client access to all of their raw data
- Used to push scores from the Audio and Essay questions in to Learnosity servers to facilitate unified reporting
- Automatically builds a series of standard report types based on the results of Tyrell’s auto-scored assessments
- Reports will be augmented with data from the Audio and Essay question types
- Embed these reports in Tyrell's system with just a few lines of code
The Assessment Experience
The assessment experience is presented on a website hosted on Tyrell's servers. Each client gets their own subdomain, such as nyu.tyrell.com. On this website, university applicants can log in and take the tests. The assessment is handled by the Learnosity Items API.
User management and user authentication is handled by Tyrell. When the logged-in users are ready to begin their assessment they will click a link which will take them to a webpage where the Learnosity Items API displays the appropriate activity for them. It will take the ID of the activity and the ID of the logged in user as two of its parameters. No personally identifying information is used.
As the student works their way through the three activities, their student responses are saved by Learnosity. The data, including essay answers, are stored in the Learnosity Student Response database, while mp3 audio files are stored in one of our Amazon S3 storage buckets.
The User ID, Session ID and Activity ID can later be used to generate a range of reports for individuals or groups.
Now it is time to consider the integration of Tyrell's in-house essay scoring and 3rd-party audio scoring. This is done in a back-end processing system that Tyrell will code as part of this integration project. The Learnosity Data API will be used to retrieve the student's essay responses from the Learnosity Student Response Database, these will be scored in Tyrell's system and the scores will be pushed back to Learnosity so they can be used in reports.
The Learnosity Data API is also used to retrieve the URLs of the MP3 audio recordings of the students responses and pass these to the third party audio scoring service. When scored they are sent back to Tyrell and then are written to the Learnosity database, again using the Data API.
Let's recap what has been created so far. Tyrell has used Learnosity's Author Site to create and store all the Items required for their language assessments. They just have to store the Activity References on their side. When it's time to present those activities to their users, they just need to pass a few relevant IDs from their side along with the Activity Reference for Learnosity to render the assessment experience.
When all three parts of the assessment are completed a Tyrell back-end process will find scores for the essay type questions and the audio questions and will then write those scores into the Learnosity Student Response Database using our Data API.
The Analytics Experience
Finally, we will deal with the analytics part. Tyrell currently provides a simple report showing the percentage grade for the multiple choice assessment. They want to maintain this and have it include all of the essay types and audio questions. This can be done with Learnosity's Reports API. Like the current report this will be made available via a webpage on the Tyrell system.
The IDs which Tyrell saved during the Assessment experience, can now be passed to the Reports API.
In this final diagram we see the entire system:
By using this workflow, Tyrell Corporation was able to create 65 different question types including audio workflows, render them as a branded assessment for each client, integrate third-party scoring and produce detailed unified reporting.
Tyrell Corporation's step-by-step tasks
Here are the steps Tyrell needs to follow to integrate this plan:
- Create the items needed for their assessments. Assemble these into three ‘activities’. Learnosity Authoring team will provide training for this
- Add a webpage where their logged-in users can take an assessment. On this page initialize the Items API using the required IDs, which includes User ID and Activity ID from the Tyrell database
- Build a backend processing system to perform these 4 steps:
- Use the Data API to retrieve Student Essays from Learnosity
- Score the essays using their in-house scoring engine and write the score back to Learnosity
- Use the Data API to retrieve Student MP3 recordings from Learnosity
- Score the MP3s using their third-party scoring engine and write the score back in to Learnosity
- Add webpages where the teacher can see reports