Termbank
  1. C
    1. Checking Daemon
      Exercises System
    2. Celery Worker
      Checking Daemon
    3. Content Graph
      Content Courses
    4. Content Page
      Content
    5. Course
      Courses
    6. Course Instance
      Courses
    7. Course Prefix
      Courses System
  2. E
    1. Embedded Content
      Content Exercises
    2. Enrollment
      Courses System
  3. F
    1. Feedback
      Content Feedback
    2. File
      Media
    3. File Upload Exercise
      Exercises
    4. Front Page
      Content Courses
  4. H
    1. Hint
      Exercises
  5. I
    1. Instance
      Course Instance
    2. Image
      Media
  6. L
    1. Lecture Page
      Content
    2. Legacy Checker
  7. M
    1. Media File
      File
    2. Markup
      Content
    3. Media
      Media
  8. P
    1. PySenpai
  9. R
    1. Regex
    2. Repeated Exercise Generator
    3. Responsible Teacher
      Courses System
    4. Revision
      System
  10. S
    1. Slug
      System
    2. Staff
      Courses System
    3. Statistics
      Exercises
  11. T
    1. Teacher Toolbox
      System
    2. Term
      Content
    3. Textfield Exercise
    4. Triggerable Highlight
      Exercises
Completed: / exercises

Authoring Content

This section of the guide described how to author basic content in more detail. Basic content is considered to include lecture pages and the simpler exercise types.

Lecture Pages

Lecture pages are managed from the Lectures section of the admin site. You will see a list of lecture pages that you have access to. This includes pages you have edited (there is at least one
revision
with your account as the editor) and pages that are linked to a course where you are the responsible teacher or in the
staff
group.
Although they are called lecture pages, they are just as useful for bundling together exercises around a theme. Overall, lecture page simply refers to any page that is available in the course's table of contents. Do note that while it is possible to have other types of pages linked directly from the table of contents, the recommended way of building material is to contain all other types of contents in lecture pages.
There are not many guidelines to creating lecture pages. The most important companion is the -- WARNING: BROKEN LINK --Lovelace markup guide.

Embedding Content

Typically lecture pages will embed all kinds of content be it
other pages
or
media
, or
term
definitions. When you save a content page, the existence of all of these will be checked and the submission form will be considered invalid if any are missing. It is therefore a good idea to write your lecture page offline in a text editor, and create the embedded objects as you go, or before copying the content to Lovelace. Writing offline is recommended for other reasons as well, as the admin site text edit box is not the best tool in existence.
When you embed content pages that have headings, these headings will be included in the table of contents sidebar. Also, if these content pages are exercises, the heading in the TOC will be accompanied with a symbol that shows whether the student has already answered the exercise or not. Therefore including a heading in each exercise is recommended.

Adding Feedback

You can attach any number of
feedback
questions to a lecture page. These will be shown at the bottom of the page. The feedback statistics can be seen from the
teacher toolbox
at the top of the page. You can learn more about Lovelace's feedback system from -- WARNING: BROKEN LINK --the feedback guide.

Exercises

This section describes the simpler exercise types, namely: checkbox exercises, multiple choice exercises and text field exercises. All of these are created entirely through the admin site and do not require any files. They are also checked on the spot instead of being sent to separate the
checking daemon
. More complex exercise types are mentioned for completion, but they each have their own guide dedicated to them.

Common Features

When editing exercises, there are two different text boxes to fill: content and question. Of these, content should contain majority of the exercise text. Question is optional. If omitted, a default question text will be shown instead. The relations of different fields in the admin interface to the layout seen by the student are shown below.
Exercise layout
  1. Main content - this is where the content field content goes to
  2. Question line(s) - contents of the question field will be here
  3. Answer box - in this case it's a text box
  4. Status line
  5. Evaluation box - will show up when 'Send answer' is pressed and evaluation is ready
  6. Hint box - will also show up if an incorrect answer is sent
  7. Status indicator + link to view previous answers
  8. Teacher toolbox
    , only visible to staff members
All exercises also have a field called default points. This determines how many points this exercise is worth when completed by the student. Note that by default this is not shown anywhere in the exercise layout. If you want students to know how many points the exercise is worth, you should write it in the description somewhere. The value in this field is used by the course scoring algorithm.
For all exercise types you can define the correct answer, and also various
hints
that are connected to incorrect answers. The way to give answer-specific hints is described separately for each exercise type. You can also set hints that are automatically given after a certain amount of tries. This can be done under the Configurable hints section of any exercise admin form. Finally, just like with lectures, you can attach
feedback
questions to any exercise. Attaching at least a comment box is generally a good idea - this will allow students to report bugs or other problems with the exercise on the spot.
Currently there is no way to limit how many times an exercise can be answered.

Multiple Choice Exercises

Multiple choice exercises are the most basic exercise type, where students must choose the correct answer from a number of options. Their advantage is that they are both easy to create and quick for students to answer. However there is only so much you can do with them. Typical use cases are questions like "Which of the following statements is true?" or selecting the correct answer for a problem.
After typing the exercise description and question, you can create the answer options. Each option consists of the following information:
The hint text is useful for explaining why a certain answer is wrong. The comment box on the other hand is mostly useful in exercises that have multiple correct answers and can, for instance, have some value statement about what's good/bad about that particular choice compared to the other correct choices.

Checkbox Exercises

Checkbox exercises are very similar to multiple choice exercises, but instead of choosing one answer, students can choose any number of answers. The exercise will be correct if the student chose all of the correct answers and none of the incorrect ones. Typical use cases are along the lines of "Which of the following belong to the described category?"
The interface to edit checkbox exercises is identical to multiple choice exercises.

Textfield Exercises

In textfield exercises, students give their reply into a text box. The correctness of the answer is checked against regular expression(s). This makes them generally useful for questions that have (somewhat) precise answers, e.g. mathematical problems or single lines of code. The advantage over choice exercises is that students have to form the answer on their own instead of just choosing from options. By far the most common use case is to get students to review how to write lines of code that do one specific thing.

Regular Expressions

Textfield exercises are only as good as their regular expressions are. As a teacher you have to explicitly define all the correct answers with your regular expression(s). This can lead to frustration with students who come up with their creative solutions that happen to do the right thing, but were not covered by the possible correct answers. You can also define regular expressions for incorrect answers, and attach hints to them - these hints will be shown when a student enters an incorrect answer matching the regular expression.
Lovelace uses Python regular expressions. Understanding the possibilities and limitations of regular expressions is key in creating good textfield exercises. Textfield exercises currently run regular expression matching in single line mode. If your answer contains multiple lines, they must be written into the regular expression with \n. Do note that multiline answers can quickly get out of hand, especially if there's more than one correct way to order the lines (you need to match each permutation separately!). You can define multiple correct regular expressions. Sometimes this can be a bit clearer than trying to jam everything together.
One of the more important symbols is \s* which should be put into regular expressions with abandon. It makes it so that extra white spaces do not interfere with matching the correct answer.

Hints and Hint Coverage

When writing regular expressions for hints, try to be as permissive about everything else as you can while being very explicit about the error the hint concerns. If your hints are accidentally triggered without proper cause, they can be more confusing to students than helpful. In some cases it's useful to put .* to basically ignore everything before the part where you expect the error to be. Another useful thing to know is that the correct answer matches, hints are never matched. This means that you can have a hint that is a broader version of the correct answer without having it trigger when the student enters the correct answer.
A good way to review your hints is to open the exercise
statistics
where you can see all answers given by students and whether they matched a hint or not. Unless the exercise is frozen, matches will always be recalculated against the latest version when you load the stats page. This takes a while, but allows you to easily see by refreshing if your changed/added hint regular expression caught the types of answers you wanted it to. Do note that actual student evaluations do not change when you alter regular expressions - an answer's correctness is evaluated and fixed at the time it's given.

Textfield Demonstration

This exercise is a short demonstration of how textfield exercises are rendered and how they work. It also shows a recommended layout for exercises. This one is a simple Python question about creative list slicing.
Learning goals: See how textfield exercises work
Preparations:
Create a list of numbers in the Python interpreter, e.g.
In [1]: numbers = [1, 6, 3, 7, 2, 8, 2, 1, 6, 7]
Answer specification:
We want to print the values from odd indices in the list, backwards. The resulting print should be [7, 1, 8, 7, 6]. If you don't know the answer try typing "halp" in the text box.
Type your answer below
Warning: You have not logged in. You cannot answer.

Repeated Template Exercises

Repeated template exercises are a variant of textfield exercises with two major differences: the student has to complete a series of questions instead of one question, and the question templates can be parametricized. They are ideal for creating review exercises or exercises that improve the students' routine. The exercises have a penalty regime where a student has to start over if they make a mistake. The downside of repeated template exercises is their need for a software backend. That is why this topic is covered in -- WARNING: BROKEN LINK --a separate guide.

File Upload Exercises

File upload exercises are at the heart of Lovelace - they're what it has been built around after all. As the title suggests, these are exercises where the student uploads a (code) file for evaluation. The evaluation is done by custom checking programs. This is both the strength and weakness of file upload exercises. On the one hand, pretty much anything can be checked with as much detail as one wishes; on the other hand, teachers have to develop these checking programs for each exercise. Being the most complex topic, file upload exercises have their own set of guides. This page is a good starting place.

Terms

Terms are a good way to create a knowledge bank that is readily available to students. All terms linked to a
course instance
can always be accessed from the term bank sidebar. You can also enrich your material with hover-on keywords that will show popups with the associated term definition. This guide uses terms so you have probably already seen them in action. A good way of creating terms is to create them as you write content. This way you will also have your hoverable keywords in place. Since you cannot save content with references to non-existent terms this is also a good way to ensure you've written descriptions for all of your terms.
In the terms list of the admin site, you can use the right hand filtering box to only see the terms of one course. The term edit form itself is relatively straightforward. Basic terms only need a name, a description and the course they're attached to. The name given here is displayed as-is in the term bank. Therefore, unlike media, you should probably not use any prefix or
slug-like
naming. Term names are also not unique because of high probability of overlap between courses. Instead, name + course pairs are unique.
Terms can also be given tags and aliases. Tags are shown in the term bank after the term name. Aliases are shown in the term bank as plain words with an arrow to the term itself (which is hoverable). You can input multiples of both by separating them with commas.
A rendered term alias in the term bank
Tags can be seen in use by looking at this course's term bank.

Term Tabs

You can add tabs to term definitions. They are useful for separating examples etc. from the definition body. Each term tab has a name and a description. Term popup boxes do have size limits. On the one hand this is a reason to split long term descriptions into multiple tabs; on the other hand it also places a limit of roughly 3 tabs per term. Click "Add another term tab" under Term Tabs in the admin interface to add a term tab.

Term Links

You can also add links that will be displayed at the bottom of the term popup box regardless of which term tab is being viewed. Links consist of link URL and link name. For internal links, page
slugs
should be used. You can still link to a specific heading in the page by adding the heading anchor after #. Note that the anchor is also slugified. For instance, to link to this particular section of the guide, you would put this into the URL field: 2-sisällön-luominen#term-links. For external links you need to give the full http:// etc.
Example term with both two tabs and a link
?
The checking daemon is a separate multi-threaded program that is invoked whenever Lovelace needs to execute code on the command line. The most common use case is to evaluate student programs by running checking programs. When a task is sent to the checker daemon, copies of all required files are put into a temporary directory where the test will then run. The daemon also does necessary security operations to prevent malicious code from doing any actual harm.
Content graphs are objects that connect content pages to a course instance's table of contents. Content graphs have several context attributes which define how the content is linked to this particular course instance. A content graph's ordinal number and parent node affect how it is displayed in the table of contents. You can also set a deadline which will be applied to all exercises contained within the linked content page. Content graphs also define which revision of the content to show - this is used when courses are archived.
In Lovelace, content page refers to learning objects that have text content written using a markup language. All types of content pages are treated similarly inside the system and they are interchangeable. Content pages include lecture pages, and all exercise types.
  1. Description
  2. Relations
In Lovelace, course refers to an abstract root course, not any specific instance of instruction. Courses are used for tying together actual instance of instruction (called course instances in Lovelace). In that sense they are like courses in the study guide, while course instances are like courses in WebOodi. The most important attrbutes of a course are its responsible teacher and its staff group - these define which users have access to edit content that is linked to the course.
  1. Description
  2. Relations
  3. Cloning and Archiving
In Lovelace, a course instance refers to an actual instace of instruction of a course. It's comparable to a course in WebOodi. Students can enroll to a course instance. Almost everything is managed by instance - student enrollments, learning objects, student answers, feedback etc. This way teachers can easily treat each instance of instruction separately. Course instances can also be archived through a process called freezing.
Course prefixes are recommended because content page and media names in Lovelace are unique across all courses. You should decide a prefix for each course and use that for all learning objects that are not included in the course table of contents. The prefix will also make it easier to manage learning objects of multiple courses - especially for your friendly superuser who sees everyhing in the admin interface...
  1. Description
  2. Examples
Embedded content refers to learning objects that have been embedded to other learning objects through links written in the content of the parent object. Embedded content can be other content pages or media. When saving a content page, all embedded objects that are linked must exist. A link to embedded content is a reference that ties together course instance, embedded content and the parent content.
Enrollment is the method which connects students to course instances. All students taking a course should enroll to it. Enrollment is used for course scoring and (once implemented) access to course content. Enrollments are either automatically accepted, or need to be accepted through the enrollment management interface.
Lovelace has a built-in feedback system. You can attach any number of feedback questions to any content page, allowing you to get either targeted feedback about single exercises, or more general feedback about entire lecture pages. Unlike almost everything else, feedback questions are currently not owned by any particular course. However, feedback answers are always tied to the page the feedback is for, and also to the course instance where the feedback was given.
  1. Description
  2. Archiving
  3. Embedding
In Lovelace file normally refers to a media file, managed under Files in the admin site. A file has a handle, actual file contents (in both languages) and a download name. The file handle is how the file is referened throughout the system. If a media file is modified by uploading a new version of the file, all references will by default fetch the latest version. The download name is the name that is displayed as the file header when it's embedded, and also as the default name in the download dialog. Files are linked to content through reference objects - one reference per course instance.
Media files are currently stored in the public media folder along with images - they can be addressed directly via URL.
  1. Description
  2. Legacy Checkers
File upload exercises are at the heart of Lovelace. They are exercises where students return one or more code files that are then evaluated by a checking program. File upload exercises can be evaluated with anything that can be run from the Linux command line, but usually a bit more sophisticated tools should be used (e.g. PySenpai). File upload exercises have a JSON format for evaluations returned by checking programs. This evaluation can include messages, hints and highlight triggers - these will ideally help the student figure out problems with their code.
Front page of a course instance is shown at the instance's index page, below the course table of contents. Front page is linked to a course instance just like any other page, but it uses the special ordinar number of 0 which excludes it from the table of contents. Any page can act as the course front page.
Hints are messages that are displayed to students in various cases of answering incorrectly. Hints can be given upon making incorrect choices in choice-type exercises, and they can also be given after a certain number of attempts. In textfield exercises you can define any number of catches for incorrect answers, and attach hints to each. Hints are shown in a hint box in the exercise layout - this box will become visible if there is at least one hint to show.
  1. Description
  2. Archiving
  3. Embedding
Images in Lovelace are managed as media objects similar to files. They have a handle that is used for referencing, and the file itself separately. Images should be always included by using reference. This way if the image is updated, all references to it always show the latest version.
Images stored on disc are accessible directly through URL.
Lecture pages are content pages that do not have any exercise capabilities attached to them. A course instance's table of contents usually consists entirely of lecture pages. Other types of content pages (i.e. exercises) are usually embedded within lecture pages.
Legacy checker is a name for checkers that were used in previous versions of Lovelace and its predecessor Raippa. They test the student submission against a reference, comparing their outputs. If the outputs match (exactly), the submission passes. Otherwise differences in output are highlighted. It is possible to use wrapper programs to alter the outputs, or output different things (e.g. testing return values of individual functions). Legacy checkers should generally be avoided because they are very limiting and often frustrating for students. Legacy checking is still occasionally useful for comparing compiler outputs etc.
Lovelace uses its own wiki style markup for writing content. Beyond basic formatting features, the markup is also used to embed content pages and media, mark highlightable sections in text and create hover-activated term definition popups.
In Lovelace, media refers to embeddable files etc. These come in there categories: images, files and video links. Like content pages, media objects are managed by reference using handles. Unlike other types of files, media files are publicly accessible to anyone who can guess the URL.
PySenpai is a library/framework for creating file upload exercise checking programs. It uses a callback-based architecture to create a consistent and highly customizable testing process. On the one hand it provides reasonable defaults for basic checking programs making them relatively straightforward to implement. On the other hand it also supports much more complex checking programs. Currently PySenpai supports Python, C, Y86 Assembly and Matlab.
Regular expression's are a necessary evil in creating textfield and repeated template exercises. Lovelace uses Python regular expressions in single line mode.
A generator acts as a backend for repeated template exercises, and provides the random values and their corresponding answers to the frontend. Generators can be written in any programming language that can be executed on the Lovelace server. Generators need to return a JSON document by printing it to stdout.
Responsible teacher is the primary teacher in charge of a course. Certain actions are available only to responsible teachers. These actions include managing enrollments and course instances.
Lovelace uses Django Reversion to keep track of version history for all learning objects. This can be sometimes useful if you need to restore a previous version after mucking something up. However the primary purpose is to have access to historical copies of learning objects for archiving purposes. When a course instance is archived, it uses the revision attribute of all its references to set which historical version should be fetched when the learning object is shown. Student answers also include the revision number of the exercise that was active at the time of saving the answer.
Slug is the lingo word for names used in urls. Slugs are automatically generated for courses, course instances and content pages. Slugs are all-lowercase with all non-alphanumeric characters replaced with dashes. Similar naming scheme is recommended for other types of learning objects as well although they do not use generated slugs.
Staff members are basically your TAs. Staff members can see pages hidden from normal users and they can edit and create content (within the confines of the courses they have been assigned to). They can also view answer statistics and evaluate student answers in manually evaluated exercises. Staff members are assigned to courses via staff group.
Lovelace has answer statistics for all exercises. Statistics are collected per instance, and allow you to review how many times an exercise has been answered, what's the success rate etc. All of this can be helpful in identifying where students either have difficulties, or the exercise itself is badly designed. For some types of exercises, there's also more detailed information about answers that have been given. Statistics can be accessed from the left hand toolbox for each exercise.
Teacher toolbox is located on the left hand side of each exercise. It has options to view statistcs, view feedback about the exercise and edit the exercise. For file upload exercises there is also an option to download all answers as a zip file. Do note that this takes some time.
  1. Description
  2. Examples
Terms are keywords that are linked to descriptions within your course. They will be collected into the course term bank, and the keyword can also be used to make term hint popups on any content page. Terms can include multiple tabs and links to pages that are relevant to the term. For instance, this term has a tab for examples, and a link to the page about terms.
Textfield exercises are exercises where the student gives their answer by writing into a text box. This answer is evaluated against predefined answers that can be either correct (accepting the exercise) or incorrect (giving a related hint). Almost always these answers are defined as regular expressions - exact matching is simply far too strict.
  1. Description
  2. Markup
  3. Triggering
Triggerable highlights can be used in content pages to mark passages that can be highlighted by triggers from file upload exercise evaluation responses. When a highlight is triggered the passage will be highlighted. This feature is useful for drawing student attention to things they may have missed. Exercises can trigger highlights in their own description, or in their parent page. It is usually a good idea to use exercise specific prefixes for highlight trigger names.