Learning outcomes and material¶

During this exercise students will learn how to implement a RESTful client using either Python or embedding the client in a website.

Slides can be downloaded from:

Exercise4.pdf

Implementing Hypermedia Clients¶

In this final exercise we're visiting the other side of the API puzzle: the clients. We've been discussing the advantages

hypermedia

has for client developers for quite a bit. Now it's time to show how to actually implement these mythical clients. This exercise material goes through two examples: one fully automated machine client (Python script), and a browser client that generates a user interface for human users (Javascript with jQuery). We will show you how to make clients that are robust against API changes.

Because this exercise's number is even, we're using the MusicMeta API in this exercise. Jk, the real reason is that we're much more familiar with that API's hypermedia representations. We also envisioned some clients in exercise 2 so we have don't have the create a new idea from scratch.

API Clients with Python¶

Our first client is a submission script that manages its local MP3 files and compares their metadata against ones stored in the API. If local files have data that the API is missing, it automatically adds that data. If there's a conflict it just notifies a human user about it and asks their opinion - this is not an AI course after all.

Learning goals: Using Python's requests library to make

HTTP requests

Preparations¶

Another exercise, another Python module. We're using Requests for making API calls. Installation into your virtual environment as usual.

pip install requests

Just for the fun of it and showing off more API development tools, we're not going to give you the server code; instead we ask that you use the mockup server from Apiary to test the client. For this purpose we're giving you an update to the

API Blueprint

you filled at the end of exercise 2 - the tracks group documentation, which you can paste into your existing documentation. It also fixes some errors that were discovered when trying the client against the mockup server, so be sure to update your albums section as well. We also added the entry point as a resource so that it can be fetched by the client.

In order for this to work you need to combine it with the artist resources you defined in the last task of exercise 2.

musicmeta.md

You can find the address for your mockup server in the Inspector tab when viewing your Apiary document.

Do note that technically there's still errors in some of the example responses for 400 status codes because we've incorrectly split some strings to make the examples more readable (the ones with JSON validation errors produced by jsonschema). These don't matter for testing, but if you want everything to work fully you should remove the newlines. Finally, a note about the mockup server: it uses the Accept header of the request to determine which response it will send. This means it will send the first error response for PUT, POST and DELETE requests because the success responses do not have response body and therefore also don't have a content type. You can fix this in two ways:

use "application/vnd.mason+json" as the content type for the 201 and 204 responses even though there is no body, e.g. + Response 204 (application/vnd.mason+json)
either do not use the Accept header in testing, or add */* to the Accept header value (separate with comma).

Of course as the mockup only operates on the data that's included in its examples, your requests must match those examples to succeed. In particular you cannot test creating a new resource and then requesting that resource.

Using Requests¶

The basic use of Requests is quite simple and very similar to using Flask's test client. The biggest obvious difference is that now we're actually making a

HTTP request

. Like the test client, Requests also has a function for each

HTTP method

. These functions also take similar arguments: URL as a mandatory first argument, then keyword arguments like headers, params and data (for

headers

query parameters

and

request body

respectively). For example, to get artists collection (sub your own Apiary mockup URL to SERVER_URL):

In [1]: import requests
In [2]: SERVER_URL = "http://private-xxxxx-yourapiname.apiary-mock.com")
In [3]: resp = requests.get(SERVER_URL + "/api/artists/")
In [4]: body = resp.json()

Sending a POST request doesn't actually do anything to the mockup server, and regardless of what data you send, it will reply with the canned Location header. It does however suffice for testing the client:

In [5]: import json
In [6]: data = {"name": "Mono", "location": "JP"}
In [7]: resp = reqests.post(SERVER_URL + "/api/artists/", data=json.dumps(data))
In [8]: resp.headers["Location"]
Out[8]: '/api/artists/mono/'

You can view the requests you sent to the mockup server from the same Inspector tab where you found its address. Do note that it compares what you send to the examples, and will say (for both requests above) that they are incorrect (they're missing headers, and Apiary thinks all four artist fields are mandatory).

*Some example requests sent while compiling this material*

However as you can see from the console, these requests still got the responses we expected, so it's good enough for client testing. If you try to send the POST request to an actual API server with content type validation (which happens when the server tries to use request.json) you will get rejected. Setting the header is shown in the PUT example here:

In [9]: resp = requests.post(SERVER_URL + "/api/artists/scandal/",
   ...:     data=json.dumps(data),
   ...:     headers={"Content-type": "application/json"}
   ...: )

Because this starts to be a mouthful, we should parametrize making requests in our client into a function. But not yet.

Request Deleted¶

We didn't show you how to send a DELETE request. As you might have guessed, this means we're going to ask you how to do it.

Learning goals: How to write a HTTP request using requests, a DELETE request in particular.

Assuming that the

host part

of the URL is stored in a constant called SERVER_URL, write a code line that deletes the album Hypnagogia by Evoken. Check the MusicMeta apiary documentation if you don't remember how.

Remember the artist name should be in lower case, while the album is in uppercase.

Often when making requests using

hypermedia

controls the client should use the method included in the control element. When doing this, using the request function is more convenient than using the method specific ones. Assuming we have the control as a dictionary called crtl:

In [10]: resp = requests.request(ctrl["method"], SERVER_URL + ctrl["href"])

Using Requests Sessions¶

Our intended client is expected to call the API a lot. Requests offers sessions which can help improve the performance of the client by reusing TCP connections. It can also set persistent

headers

which is helpful in sending the Accept header, as well as authentication tokens for APIs that use them. Sessions should be used as context managers using with statement to ensure that the session is closed.

In [1]: import requests
In [2]: SERVER_URL = "http://private-xxxxx-yourapiname.apiary-mock.com"
In [3]: with requests.Session() as s:
   ...:     s.headers.update({"Accept": "application/vnd.mason+json"})
   ...:     resp = s.get(SERVER_URL + "/api/artists/")

With this setup, when using the session object to send

HTTP requests

, all the session headers are automatically included. Any headers defined in the request method call are added on top of the session headers (taking precedence in case of conflict). Note: if you want to avoid the issue with the mock server's response selection, you can use "application/vnd.mason+json, */*" as the Accept header value here as one way to do it.

Basic Client Actions¶

The client code we're about to see makes some relatively sane assumptions about the API. First of all, it works with the assumption that

link relations

that are promised in the API

resource

state diagram are present in the representations sent by the API. Furthermore it trusts that the API will not send broken

hypermedia

controls or

JSON schema

. It will also have issues if new mandatory fields are added for POST and PUT requests (but we'll make it easy to update in this regard).

We're not going to show the full code here, only the parts that actually interact with the API. Furthermore, while the client was tested with actual MP3 files, it might be easier for you to simply fake the tag data by creating a data class with necessary attributes, e.g. (only in Python 3.7. or newer)

from dataclasses import dataclass

@dataclass
class Tag:
    title: str
    album: str
    track: int
    year: str
    disc: int
    disc_total: int

In older Python versions you need to make a normal class and write the __init__ method yourself (data classes implement this kind of __init__ automatically).

class Tag:

    def __init__(self, title, album, track, year, disc=1, disc_total=1):
        self.title = title
        self.album = album
        self.track = track
        self.disc = disc
        self.disc_total = disc_total
        self.year = year

Learning goals: How to navigate an API with an automated client, and send requests. Taking advantage of hypermedia to implement dynamic clients.

Client Workflow¶

The submission script works by going through the local collection with the following order of processing:

check first artist
check first album by first artist
check each track on first album
check second album by first artist

and so on, creating artists, albums and tracks as needed. It also compares data and submits differences. It trusts the local curator more than the API, always submitting the local side as the correct version. However when it doesn't have data for some field, it uses the API side value. Since MP3 files don't have metadata about artists, it uses "TBA" for the location field (because it is mandatory).

GETting What You Need¶

The key principles of navigating a

hypermedia

API are:

start at the entry point
follow the link relations that will lead to your goal

This way your client doesn't give two hoots even if the API changed its

URIs

arbitrarily on a daily basis, as long as the resource state diagram remains unchanged. Our submission script needs to start at the artist collection. However, instead of starting the script with a GET to /api/artists/, it should start digging at the entry point /api/ and find the correct URI for the collection it's looking for by looking at the "href" attribute of the "mumeta:artists-all" control.

With this in mind, this is how the client should start its interaction with the API:

    with requests.Session() as s:
        s.headers.update({"Accept": "application/vnd.mason+json"})
        resp = s.get(API_URL + "/api/")
        if resp.status_code != 200:
            print("Unable to access API.")
        else:
            body = resp.json()
            artists_href = body["@controls"]["mumeta:artists-all"]["href"]

This is the only time the entry point is visited. From now on we'll be navigating with the link relations of resource representations (starting with the artist collection's representation). With the artist collection at hand, we can start to check artists from the local collection one by one.

def check_artist(s, name, artists_href):
    resp = s.get(API_URL + artists_href)
    body = resp.json()
    artist_href = find_artist_href(name, body["items"])
    if artist_href is None:
        artist_href = create_artist(s, name, body["@controls"]["mumeta:add-artist"])

    resp = s.get(API_URL + artist_href)
    body = resp.json()
    albums_href = body["@controls"]["mumeta:albums-by"]["href"]

We've chosen to fetch the artist collection anew for each artist, for the off chance that the artist we're checking is added by another client while we were processing the previous one. The first order of things is to go through the "items" attribute and look if the artist is there. Remembering the non-uniqueness issue with artist names, our script falls back to the human user to make a decision in the event of finding more than one artist with the same name. Doing comparisons in lowercase avoids capitalization inconsistencies.

def find_artist_href(name, collection):
    name = name.lower()
    hits = []
    for item in collection:
        if item["name"].lower() == name:
            hits.append(item)
    if len(hits) == 1:
        return hits[0]["@controls"]["self"]["href"]
    elif len(hits) >= 2:
        return prompt_artist_choice(hits)
    else:
        return None

Assuming we find the artist, we can now use the item's "self" link relation to proceed into the artist resource. This is only an intermediate step that is needed (according to the state diagram) in order to find the "mumeta:albums-by" control for this artist. This is the resource we need for checking the artist's albums. We have skipped exception handling because we trust the API to adhere to its own documentation (also for the sake of brevity).

Rat in the Maze¶

Hypermedia can also be used for dumb things like making mazes that can be navigated using link relations. Solving one with an automated client is good exercise, so we decided to do some cheese hiding into a rather large maze...

Learning goals: Making a

hypermedia

client that chains together GET requests using

link relations

to find what it needs.

What to do

We have an API server running in https://pwpcourse.eu.pythonanywhere.com. Its media type is Mason. The server only has one resource, which only supports GET, and an entry point at /api/. The resource is a room, and it has two attributes:

"handle": string, the room's unique identifier
"content": string, possible values: "" and "cheese"

In addition it can have up to four directional controls which represent transitions from one room to another. Corner rooms only have two, rooms along the edge only have three. The controls belong to the maze namespace.

"maze:north": head towards the wall; you probably know nothing
"maze:south": head to warmer climates
"maze:east": go towards the sunrise
"maze:west": go west where the skies are blue

In addition the entry point has one control from the maze namespace: "maze:entrance" which leads to the first room.

Your quest is to find the cheese. The "maze" is just a square grid, but it has a lot of rooms. Unless you feel like clicking through thousands of links manually, using an automated machine client is recommended. When you find the cheese, remember to print out the room's handle - it's your answer to this task.

Your client needs some minimal intelligence so that it goes through the rooms in some logical order (e.g. one row at a time). Please make sure your client doesn't get stuck in infinite loops. Your client is sending actual network traffic - don't flood the API.

Anna palautetta

Koitko tämän tehtävän hyödylliseksi oppimisen kannalta?

Kommentteja tehtävästä?

Schematic POSTing¶

When something doesn't exist, the submission script needs to obviously send it to the API. We're skipping ahead a bit to creating albums and tracks. For both the data comes from MP3 tags (for albums we take the first track's tag as the source). The POST

request body

for both can also be composed in a similar manner thanks to

JSON schema

included in the

hypermedia

control. The basic idea is to go through properties in the schema and for each property:

find the corresponding local value (i.e. MP3 tag field)
convert the value into the correct format using the property's "type" and related fields (like "pattern" and "format" for strings)
add the value to the message body using the property name

In the event that a corresponding value is not found, the client can check whether that property is required. If it's not required, it can be safely skipped. Otherwise the client needs to figure out (or ask a human user) how to determine the correct value. We've chosen not to implement this part in the example though. It would be relevant only if the API added new attributes to its resources.

As a reminder of what it looks like, here's the "mumeta:add-album" control from the the albums collection resource:

"mumeta:add-album": {
    "href": "/api/artists/scandal/albums/",
    "title": "Add a new album for this artist",
    "encoding": "json",
    "method": "POST",
    "schema": {
        "type": "object",
        "properties": {
            "title": {
                "description": "Album title",
                "type": "string"
            },
            "release": {
                "description": "Release date",
                "type": "string",
                "pattern": "^[0-9]{4}-[01][0-9]-[0-3][0-9]$"
            },
            "genre": {
                "description": "Album's genre(s)",
                "type": "string"
            },
            "discs": {
                "description": "Number of discs",
                "type": "integer",
                "default": 1
            }
        },
        "required": ["title", "release"]
    }
}

As it turns out, we only need one function for constructing POST requests for both albums and tracks:

def create_with_mapping(s, tag, ctrl, mapping):
    body = {}
    schema = ctrl["schema"]
    for name, props in schema["properties"].items():
        local_name = mapping[name]
        value = getattr(tag, local_name)
        if value is not None:
            value = convert_value(value, props)
            body[name] = value

    resp = submit_data(s, ctrl, body)
    if resp.status_code == 201:
        return resp.headers["Location"]
    else:
        raise APIError(resp.status_code, resp.content)

In this function, tag is an object. In real use it's an instance of tinytag.TinyTag, but can also be instance of the class Tag we showed earlier. The ctrl parameter is a dictionary picked from the

resource's

controls (e.g. "mumeta:add-album"). The mapping parameter is a dictionary with API side resource attribute names as keys and the corresponding MP3 tag fields as values. The knowledge of what goes where comes from reading the API's resource

profiles

. As an implementaion note, we're also using the getattr function which is how object's attributes can be accessed using strings in Python (as opposed to normally being accessed as e.g. tag.album).

The mapping dictionary for albums looks like this, where keys are API side names and values are names used in the tag objects.

API_TAG_ALBUM_MAPPING = {
    "title": "album",
    "discs": "disc_total",
    "genre": "genre",
    "release": "year",
}

Since all values are not stored in the same type or format as they are required to be in the request, the convert_value function (shown below) takes care of conversion:

def convert_value(value, schema_props):
    if schema_props["type"] == "integer":
        value = int(value)
    elif schema_props["type"] == "string":
        if schema_props.get("format") == "date":
            value = make_iso_format_date(value)
        elif schema_props.get("format") == "time":
            value = make_iso_format_time(value)
    return value

Finally, notice how we have put submit_data as its own function? What's great about this function is that it works for all POST and PUT requests in the client. It looks like this:

def submit_data(s, ctrl, data):
    resp = s.request(
        ctrl["method"],
        API_URL + ctrl["href"],
        data=json.dumps(data),
        headers = {"Content-type": "application/json"}
    )
    return resp

Overall this solution is very dynamic. The client makes almost every decision using information it obtained from the API. The only thing we had to hardcode was the mapping of resource attribute names to MP3 tag field names. Everything else regarding how to construct the request is derived from the hypermedia control: what values to send; in what type/format; where to send the request and which HTTP method to use. Not only is this code resistant to changes in the API, it is also very reusable.

Of course if the control has "schemaUrl" instead of "schema", the additional step of obtaining the schema from the provided URL is needed, but is very simple to add.

To PUT or Not¶

When using dynamic code like the above example, editing a resource with PUT is a staggeringly similar act to creating a new one with POST. The bigger part of editing is actually figuring out if it's needed. Once again the core of this operation is the

schema

. One reason to use the schema instead of the resource representation's attributes is that the attributes can contain derived attributes that should not be submitted in a PUT request (e.g. album resource does have "artist" attribute, but the value cannot be changed).

In order to decide whether it should send a PUT request, the client needs to compare its local data against the data it obtained from the API regarding an album or a track. For comparisons to make sense, we need to once again figure out what are the corresponding local values, and convert them into the same type/format. This process is very similar to what we did in the create_with_mapping function above, and in fact most of its code can be put into a new function called compare_with_mapping:

def compare_with_mapping(s, tag, body, schema, mapping):
    edit = {}
    change = False
    for field, props in schema["properties"].items():
        api_value = body[field]
        local_name = mapping[field]
        tag_value = getattr(tag, local_name)
        if tag_value is not None:
            tag_value = convert_value(tag_value, props)
            if tag_value != api_value:
                change = True
                edit[field] = tag_value
                continue
        edit[field] = api_value

    if change:
        try:
            ctrl = body["@controls"]["edit"]
        except KeyError:
            resp = s.get(API_URL + body["@controls"]["self"]["href"])
            body = resp.json()
            ctrl = body["@controls"]["edit"]
        submit_data(s, ctrl, edit)

Overall this process looks very similar. There's just the added step of checking whether a field needs to be updated, and marking the change flag as True the first time a difference is discovered. Note also that for albums we're doing this comparison for the album resource, but for tracks we are actually doing it to track data that's in the album resource's "items" listing. This way we don't need to GET each individual track unless it needs to be updated. When this happens, we actually need to first GET the track and then find the edit control from there. This explains why we're not directly passing a control to this method, and also why finding the control at the end has the extra step if an edit control is not directly attached to the object we're comparing.

Fun fact: if at a later stage the API developer chooses to add the edit control to each track item in the album resource, this code would find that, making the extra step unnecessary. Sometimes clients can apply logic to find a control that's not immediately available. Following the self link relation of an item in a collection is a good guess about where to find additional controls related to that item.

A final reminder about PUT: remember that it must send the entire representation, not just the fields that have changed. The API should use the request body to replace the resource entirely. That is why we're always adding the API side value to fields when we don't have a new value for that field.

The Schemanator - Terminal Edition¶

Generic hypermedia clients are built on the idea that it's possible to fully interact with an API by generating a user interface using information provided by hypermedia controls. One part of this is prompting values for POST and PUT requests from the user by generating a form or similar from either a template or schema provided by the API. In this task you will do a simple command line version of a generic data prompt function.

Learning goals: How to make prompts and convert data based on

JSON schema

, fetching a schema from a schema URL, submitting a POST/PUT request.

Before you begin:

You can use the submit_data function from the submission script example as-is. The only difference is that the checker in task will provide complete URLs as the "href" attribute in controls. Therefore you should either remove the API_URL constant or set it to "" before submitting your answer.

You should also download the latest version of the Sensorhub API and

run it

. This will help you in testing.

sensorhub.py

Function to implement: prompt_from_schema

Parameters:

requests session object - you must use this object to make any requests
Mason hypermedia control as a dictionary

The function should obtain the schema in one of two ways:

if the control has a "schema" attribute, use that schema
otherwise send a GET request to the URL specified in "schemaUrl" attribute and read the schema from the response with .json()

Once it has a schema, it should ask the user to input all required values (using description as the input prompt), and convert these into correct types as specified in the schema. The types used in this exercise are "number", "integer" and "string". String formats or patterns are not used. Once the function has compiled the request data, it should send it to the API server using attributes from the hypermedia control. You can use the submit_data function for this.

You do not need to implement typechecking for the user's inputs - the checker will only give you valid inputs. You do need to prompt inputs in the order they appear in the "required" list though (inputs from the checker are given in this order). Use the builtin input function for making these prompts.

Testing your program

You can test this program against any API that uses Mason as its

media type

, you just have to check whether the API uses full URLs or omits the

host part

. All of our examples omit it, which means you need to use the API_URL constant for the host part. Just remember to change it to an empty string before submitting your answer. Best way to do this is to set it to empty string at the beginning, and change its value to your test target in the main program portion of your module (as seen below).

Here's a main program that you can use to test against the Sensorhub API, to create a sensor.

if __name__ == "__main__":
    API_URL = "http://localhost:5000"
    with requests.Session() as s:
        resp = s.get(API_URL + "/api/sensors/")
        body = resp.json()
        prompt_from_schema(s, body["@controls"]["senhub:add-sensor"])

About the checker

This checker does not run an API server, but rather just uses objects that emulate just enough of the behavior of requests.Session and requests.Response to make it possible to check your function. They do not provide the full set of methods and attributes of these objects so try to stick to the basics shown in the examples (i.e. get, request, put and post for the Session and json and status_code for Response).

Anna palautetta

Koitko tämän tehtävän hyödylliseksi oppimisen kannalta?

Kommentteja tehtävästä?

Closing Remarks and Full Example¶

Although this was a specific example, it should give you a good idea about how to approach client development in general when accessing a

hypermedia

API: minimize assumptions and allow the API resource representations to guide your client. When you need to hardcode logic, always base it on information from

profiles

. Always avoid working around the API - workarounds often rely on features that are not officially supported by the API, and may stop working at any time when the API is updated. Having a client that adjusts itself to the API is also respectful towards the API developer, making the job of maintaining the API much easier when there aren't clients out there relying on ancient/unintended features.

Here's the full example. If you want to run it without modifications, you need to actually have MP3 files with tag data that matches your Apiary documentation's examples. The submission script doesn't currently support VA albums.

mumeta_submit.py

API Clients with Javascript¶

In this section we're going through some considerations about making browser clients with Javascript and jQuery. A typical example of such a client is a graphical user interface to the API for human users. We're only commenting some implementation details of the code. If you need to learn Javascript basics, please refer to other sources (here are some). It's not a hard language to pick up if you already know Python, although it has a a lot of caveats. One of the principal components of browser side scripts is

DOM

manipulation. Because jQuery does DOM manipulation a lot better than Javascript by itself, and is therefore an almost unseparable component despite technically being an external library. In fact the example in this material is almost entirely written using jQuery.

Because Javascript is not expected to be as familiar as Python in our degree program, we're not jumping immediately into making as dynamic code as possible. Rather we'll show the basic actions without taking full advantage of

hypermedia

Learning goals: Learn the basics of implementing a browser-based API client for human users. Making

Ajax

-requests with jQuery. Manipulating DOM with jQuery.

About Javascript Style¶

In our examples, we're following this style guide and expect you to do the same. We also encourage the use of JSLint. Javascript can be a devious language to debug as is - it's best to not make it any harder with poor code quality. When using JSLint for scripts that use jQuery, remember to add $ and document to the global variables box at the bottom to avoid a legion of "undeclared '$'" errors.

It is also recommended to run Javascript in strict mode. This makes it raise more errors which makes overall debugging and maintenance easier. You can run a script in strict mode by including the following line at the beginning of your file:

"use strict";

Server Side Preparations¶

For this example, we're going to serve the client from the same server application as the API itself. It's actually just one view that serves a static HTML page and a couple of script files (jQuery and our code) - maybe CSS if you're feeling designer-y - we're not. Serving everything from the same server also sidesteps the need for

cross origin resource sharing (CORS)

definition in the API side. In order to follow this example, download jQuery and create the following structure for your project's static folder:

static
├── css 
├── html
├── profiles
└── scripts

Drop jquery.js into the scripts folder, and the following HTML file into the html folder. There's nothing particularly amazing about this HTML file, it just defines a couple of div elements where the client script will place elements as it receives

resource

representations from the API. You can also take this short CSS file to avoid 404 reports when loading the page. Also either name your own script file admin.js or change the reference in the HTML file appropriately.

admin.html

admin.css

Then put these lines into your single file application after resource

routing

, or inside the create_app function in __init__.py if you followed the more elaborate project strcuture guide.

@app.route("/admin/")
def admin_site():
    return app.send_static_file("html/admin.html")

After saving and starting your server, you should be able to visit /admin/ in localhost and see the rendered contents of the HTML file (which is not a whole lot).

In case you didn't already have it, here's the latest version of Sensorhub API single file version:

sensorhub.py

Making Ajax Calls¶

Ajax

historically stood for Asynchronous Javascript And Xml and was spelled AJAX. Since then XML has been largely dropped, and presently Ajax is mostly used as a term rather than acronym. However, its asynchronous nature persists. Due to this nature, Ajax is non-blocking: after a request is sent with Ajax, the client side script will continue to run. Without this, browsing web pages that use Ajax would be plagued by constant freezing (and yes, constant, web pages these days do excessive amounts of Ajax calls). It also means, from the programming perspective, that your script cannot expect the response to be immediately available after making an Ajax request.

In the Python client above we always did blocking API calls - the script actually paused to wait for the response, and when it got one, it was stored into the resp variable. With non-blocking calls the response cannot be directly stored into a variable when making the call, because the response is not ready at the time. Instead, the code must register a

callback

that will handle the response when it's ready. In most cases at least one other callback with be registered for handling errors. All of this needs to be registered somehow. With jQuery, all of these are collected into an object that's given to the ajax fuction. The same in practice, for getting the entry point:

$.ajax({
    url: "/api/",
    success: function (body, status, jqxhr) {
        console.log("RESPONSE (" + status + ")");
        console.log(body);
    },
    error: function (jqxhr, type, error) {
        console.log("ERROR (" + type + ") - " + error);
    }
});

$ is a prefix that's used for all jQuery functions (and it's also a callable by itself, don't worry about it just now though). The $.ajax function takes a settings object as its argument. Here is a complete list or properties that can be included in this object. We're not going to use all of them. In the example we defined callbacks as

anonymous functions

. It's perfectly legal to assign existing functions instead:

$.ajax({
    url: "/api/",
    success: handleEntry,
    error: handleError
});

Another thing that is entirely legal is to leave out function parameters if you don't need them, as long as the ones you don't need are at the end. E.g. we could leave jqxhr (jQuery XmlHttpRequest object) out because it's not used (you need it if you want to access headers). Javascript is dirty that way (and in many other ways). Also legal is to give $.ajax the url as the first argument, and rest of the settings as second argument (the joys of function overloading).

Regardless of how they are defined, one of these functions will be called once the server responds to our Ajax call. In case of success, the first parameter will get the

response body

as a compiled Javascript object. What usually happens after is some kind of

DOM

manipulation where data from the response is placed into the DOM for the user to see.

There is another way to register callbacks by using something called a Promise object. The code below has identical functionality:

$.ajax("/api/")
    .done(function (body, status, jqxhr) {
        console.log("RESPONSE (" + status + ")");
        console.log(body);
    })
    .fail(function (jqxhr, type, error) {
        console.log("ERROR (" + type + ") - " + error);
    });

For this exercise we're asking you to use the first method, i.e. pass the callbacks in the settings object. All the examples will also use it. As a practical reason, the current checkers cannot deal with the method that uses Promise object. They also do not support giving the URL in a separate argument - include it in the settings. In real life both approaches are equally valid. Promise objects are newer tech however, and introduce new options that are not possible with the old way. One such example is the ability to add multiple callbacks to the same event.

Basic DOM Manipulation¶

Although it also makes Ajax calls cleaner, the primary purpose of jQuery is to make

DOM

manipulation a lot cleaner. Document Object Model is a programming interface that allows Javascript to modify the contents of the document (i.e. web page) "on the fly" (i.e. without reloading the page). It presents the HTML as a tree structure where

elements

can be addressed either directly through their own attributes (e.g. class or id), or through relationships such as parent, children, next and previous. Basic selection with jQuery uses the same syntax as you would use with CSS when setting styles:

$("table")                      // selects all table elements
$(".resulttable")               // selects any elements where class="resulttable"
$("#sensorlist")                // selects the element where id="sensorlist"
$(".resulttable tbody tr")      // selects all rows from all tables where class="resulttable"

As you can see, jQuery itself can be called. This will return a query object which has methods that can be used to perform operations on all of the selected elements at once. Generally it's best to use id when attempting to select only one specific element, and class when selecting a group of related elements. Element types are usually used as seen in the last example, i.e. selecting all elements of one type within another selection. There's a wide variety of selectors to choose from but you can usually get pretty far with the basics.

Generally the best way to test selections and manipulation is to use your browser's Javascript console. In Chrome you can open it with Ctrl + Shift + j while in Firefox Ctrl + Shift + k gives a similar view (using j does also give a console but in a separate window). For testing the commands from now on you can use the HTML page served in your API server's "/admin/" URL.

Once you have a selection, there's another world of possibilities regarding how to manipulate it. One of the basic manipulation methods is html which can be used to set the HTML inside the selected element to whatever is in the arguemnt (note: if arguments are not given this method becomes a getter and will return the HTML inside the element instead of setting it). Here's one way to set the contents of an error message area (HTML div element with the error class) in the UI:

$("div.notification").html("<p class='error'>Guru meditation error</p>");

Another simple common operation is to clear everything inside an element, such as clearing a table when we receive a different set of data to show. This is done with empty. Generally when manipulating tables, it's common to select the <tbody> element unless you want to change the table's headers. Here we're only clearing the data rows.

$(".resulttable tbody").empty();

Also regarding tables, the last very common thing we'll do is to append elements inside another element - usually a table, a form or a list (i.e. <ol> or <ul>). This is done with the append method.

$(".resulttable tbody").append("<tr><td>placeholder-sensor</td><td>placeholder-model</td></tr>");

Other things that are done commonly but not so much in this material are changing elements' attributes with attr or their appearance with css. These are related to things like hiding/showing and activating/deactivating elements in the UI, and obviously a lot more.

Rendering Data Collections¶

Now that we know how to make Ajax calls and how to insert data into the HTML page, we can look into some basic client operations. First is showing the items of a collection resource by putting them inside a table. But first let's do just the part the fills the data into the table.

For this purpose we need to make an Ajax call to "/api/sensors/" and fill the results into the HTML table inside the success

callback

. This is shown below using a named functions as handlers. The bit at the bottom is what calls this function once the HTML document has been completely loaded. Because doing the Ajax request itself is always the same, it's a good idea to make the function that renders the results into the HTML page a parameter:

function renderError(jqxhr) {
    let msg = jqxhr.responseJSON["@error"]["@message"];
    $("div.notification").html("<p class='error'>" + msg + "</p>");
}

function getResource(href, renderFunction) {
    $.ajax({
        url: href,
        success: renderFunction,
        error: renderError
    });
}

$(document).ready(function () {
    getResource("http://localhost:5000/api/sensors/", renderSensors);
});

For the first iteration of this client, we use a resource specific rendering function. This means the function is coded with knowledge of what the table headers are, and what columns are in the data. In this case it's also simplest to just construct the entire row's worth of HTML as a string and stuff it into the relevant part of the table, using html for the header (since we want to replace whatever was there) and append for the data. With these decision the implementation is quite simple, divided into two functions:

function sensorRow(item) {
    return "<tr><td>" + item.name +
            "</td><td>" + item.model +
            "</td><td>" + item.location + "</td></tr>";
}

function renderSensors(body) {
    $(".resulttable thead").html(
        "<tr><th>Name</th><th>Model</th><th>Location</th></tr>"
    );
    let tbody = $(".resulttable tbody");
    body.items.forEach(function (item) {
        tbody.append(sensorRow(item));
    });
}

Rendering Forms¶

Showing data in the client is one thing - being able to submit is another. This will be done by using HTML forms that call a Javascript function instead of submitting directly to the server with POST. In the first iteration we will once again make resource specific functions which means form fields are hard-coded. We will still use the

schema

to mark which fields are mandatory, and retrieve the field descriptions.

function renderSensorForm(ctrl) {
    let form = $("<form>");
    let name = ctrl.schema.properties.name;
    let model = ctrl.schema.properties.model;
    form.attr("action", ctrl.href);
    form.attr("method", ctrl.method);
    form.submit(submitSensor);
    form.append("<label>" + name.description + "</label>");
    form.append("<input type='text' name='name'>");
    form.append("<label>" + model.description + "</label>");
    form.append("<input type='text' name='model'>");
    ctrl.schema.required.forEach(function (property) {
        $("input[name='" + property + "']").attr("required", true);
    });
    form.append("<input type='submit' name='submit' value='Submit'>");
    $("div.form").html(form);
}

Here we are using attr to set the various attributes of the form and the required attribute for mandatory input fields, We also use submit to set a function that will be called when the form is submitted (i.e. when the user presses the submit button). We're also using the syntax for selecting elements using arbitrary attributes ("element[attribute='value']"). All that's left is to implement the functions that actually send the data to the server. First let's talk about this one:

function submitSensor(event) {
    event.preventDefault();

    let data = {};
    let form = $("div.form form");
    data.name = $("input[name='name']").val();
    data.model = $("input[name='model']").val();
    sendData(form.attr("action"), form.attr("method"), data, getSubmittedSensor);
}

The most important line is at the very beginning. Calling event.preventDefault makes the browser skip the default behavior it would normally do with this event. In this case that would be to send the form contents as form encoded data to the address specified by the form element's action attribute. Our API doesn't take form encoded requests, so this would cause a 415 error. It's important to put this line in all functions that replace a default behavior. Another example is the anchor element which when clicked normally follows the associated URL - again something that we don't want because our entire client is built on the idea that we never leave the page. The sendData function has been generalized to work for all requests that have a request body:

function sendData(href, method, item, postProcessor) {
    $.ajax({
        url: href,
        type: method,
        data: JSON.stringify(item),
        contentType: "application/json",
        processData: false,
        success: postProcessor,
        error: renderError
    });
}

Here we have enabled post-processing with a callback argument. In the case of a POST request, the post-processing for successful addition would be to append the sensor's data into the sensor table. The options we have are:

use local data: read values from the form and put them into the table
refetch the sensor collection
fetch the new sensor using the location header and insert the data into the table

None of these are without problems. Local data is no good if resources have attributes that are generated by the API server, and local data does't have controls. Refetching the entire collection avoids any inconsistencies but is a rather heavy operation. The last option can also be problematic, especially if the sensor comes with bunch of data that we don't need for this view. It's still best in this scenario, so we will chain immediately into another Ajax call that uses the generic getResource from earlier, coupled with the function that appends the sensor into the table after the GET request.

function appendSensorRow(body) {
    $(".resulttable tbody").append(sensorRow(body));
}

function getSubmittedSensor(data, status, jqxhr) {
    let href = jqxhr.getResponseHeader("Location");
    getResource(href, appendSensorRow);
}

Here we also see a use for the rather mysterious jqxhr object - getting the Location header.

Showing Individual Sensors¶

Finally we're going to need navigational links to traverse between

resources

. For this example, we're only going to travel back and forth between the sensor collection and individual sensors. The first order of business is to enrich each sensor in the table with its "self"

link relation

, allowing us to get details about each. This necessitates a change to our sensorRow function:

function sensorRow(item) {
    let link = "<a href='" +
                item["@controls"].self.href +
                "' onClick='followLink(event, this, renderSensor)'>show</a>";

    return "<tr><td>" + item.name +
            "</td><td>" + item.model +
            "</td><td>" + item.location +
            "</td><td>" + link + "</td></tr>";
}

The anchor tag definition looks a bit messy but it should be clear that we're getting the URL from the control. We're also hooking up the tag's onClick to a function that will override its normal behavior. Important difference to earlier: when setting an event handler with jQuery's method (e.g. submit), only the function's name is written; but when using HTML attributes, the function entire call is written. This function is quite simple:

function followLink(event, a, renderer) {
    event.preventDefault();
    getResource($(a).attr("href"), renderer);
}

Once again the most important thing is to prevent the default behavior. After this we'll handle the transition in our own way: by GETting the sensor's resource, and using a new rendering function to insert it into the

DOM

. When showing a resource that can be edited, a good way to do that is to put the data into a form. This way editing is really straightforward. Attributes that cannot be submitted this way can be put into read-only fields. Because we already have a fuction that renders a similar form for creating sensors, we can reuse it, and apply changes to the form afterward:

function renderSensor(body) {
    $(".resulttable thead").empty();
    $(".resulttable tbody").empty();
    renderSensorForm(body["@controls"].edit);
    $("input[name='name']").val(body.name);
    $("input[name='model']").val(body.model);
    $("form input[type='submit']").before(
        "<label>Location</label>" +
        "<input type='text' name='location' value='" +
        body.location + "' readonly>"
    );
}

We're now using the before method to insert the location field (which is not in the

schema

) as a read-only field into the form before the submit button. Because the submission was made in a rather dynamic way, we can actually already edit the sensor. There will be an error in processing the results however, because the post-processing function tries to follow the Location header after submission. The header is not there though, because this is an edit. We could device a more elaborate way to deal with this, but for now let's just add a simple if statement to the function:

function getSubmittedSensor(data, status, jqxhr) {
    let href = jqxhr.getResponseHeader("Location");
    if (href) {
        getResource(href, appendSensorRow);
    }
}

We should actually consider fetching the resource again or returning to the sensor collection. Why? Because changing the name of a sensor changes its

URI

. But instead of focusing too much on this, let's just finish this example by adding one navigational control to the navigation div: the "collection" link relation. This means adding another somewhat messy anchor tag by putting the following snippet somewhere inside renderSensor:

    $("div.navigation").html(
        "<a href='" +
        body["@controls"].collection.href +
        "' onClick='followLink(event, this, renderSensors)'>collection</a>"
    );

And the following line into renderSensors to clear the navigation div when we go back to it:

    $("div.navigation").empty();

Full Example¶

You can download the full example from below:

admin.js

The Paginator: Measurement Day¶

Pagination is often used with large datasets and something you have undoubtedly seen a lot in the web. It means that data is fetched in chunks of a certain size, and displayed along with controls like "next" (">") and "prev" ("<") (sometimes also "first" ("<<") and "last" (">>")). This is also used with APIs. In this exercise you will get to implement pagination for measurement data.

Learning goals: Fetching data from the API with pagination controls. Showing paginated data with

DOM

manipulation.

Before you begin:

We have made a final update to the Sensorhub API server. This update inlcudes the API side support for getting measurements with pagination, in chunks of 50. You can download it below.

app.py

Remember you need to create a database and populate with data. We have include some commands in the SensorHub API to simplify this tasks. They are explained in the Flask API project layout tutorial. If you just want to have it up and running execute the following commands:

flask init-db
flask testgen
flask run

Beyond that, you should continue where we left off with admin.js. Get the API running in port 5000, and point the browser to "http://localhost:5000/admin/". While developing, you will probably need more than a few reloads. Important: when you are testing in the browser, you must make a forced reload to the page - normal reload will keep the CSS and JS files cached and your changes won't take effect. Forced reload is performed with Shift + F5.

Do not touch any parts of the code that you haven't been instructed to.

Function to modify: renderSensor

You need to add an additional navigational control to the navigation div

element

. It should be added to the end (i.e. as second). Like others before it, this link will also call followLink when clicked, using the renderMeasurements function as its last argument. The href attribute for this link can be found from the "senhub:measurements-first" control.

New function: renderMeasurements

When the

Ajax

call returns, this function should have a measurement collection resource. This collection contains 50 measurements starting from the start index defined in

query string

(0 if omitted). Each measurement has two attributes:

time: a full ISO timestamp (string)
value: measurement value (float)

In addition the collection resource has the following controls:

"up": leads back to the sensor resource (irrelevant to this task)
"prev": retrieves the previous 50 measurements, only present if not already showing the first measurement
"next": retrieves the next 50 measurements, only present if not already showing the last measurement

Here's a list of things your function must do:

replace the tablecontrols div cotents with "prev" and "next" links if they are present in the resource representation

not the same div as navigation - check the HTML!
need to be shown in both top and bottom tablecontrol div elements!
"prev" should always be on the left

set the table headers to correspond with measurement attribute names (time on left)
display the 50 measurements in the table

At the end your UI should have three different screens: one with the list of sensors, one with information about an specific sensor and one with the list of sensor's measurements. They are shown in the following image:

About the checker:

The checker is a rather interesting monstrosity that uses NodeJS and JSDOM and some bonus hacking to make your code think it's running in a browser, and that the user is clicking links. It does not use the checker architecture used by all the other checkers (due to lack of JS support), making it less resistant to errors. If your code produces any errors, the checker will crash (or at least Lovelace will think so) - test your code properly before sending. The same will happen if your code does any console logging - remember to comment out any console.log lines.

NOTE: The checker does not work if you have "use strict"; in the file. Please comment the line out before submitting.

Anna palautetta

Koitko tämän tehtävän hyödylliseksi oppimisen kannalta?

Kommentteja tehtävästä?

Anna palautetta

Kommentteja materiaalista?