MS Health API Bot

As we make steady progress from a world where the user input to a computer or mobile device consists of flat two dimensional elements controlled by mouse and keyboard it is not difficult to imagine more natural ways to interact with these devices. We have seen speech, gesture, touch etc begin to gradually become more accepted as input modes instead of needing to learn fixed grammars and commands we are moving into a place where more natural modes of conversation are possible. The tech isn’t quite at a place where these conversations are unbounded but can be made to work where we have more specific questions that we’d like answers for. So, I was thinking a question like

“how much sleep have I had in the last two weeks?”

or

“how many steps did I take yesterday?”

where a specific answer exists and I could ask that question naturally of a computer using the language I might if I were talking to another person. In the short-term this kind of technology could be used to streamline existing customer support functionality. In the longer term this might evolve into a more general purpose human-computer interaction; more conversational.

Microsoft announced their Bot Framework and associated tools at the recent Build conference and to kick the tyres of this offering I thought I would try to answer the aforementioned questions using the Bot Framework and the Microsoft Health API using some of the code I wrote for a previous post (see /band/microsoft/ms%20band/universal/uwp/win10/windows/winrt/2015/07/13/microsoft-health-api-uwp-sample.html). The data I can get back via the health API has been collected over time using my Microsoft Band which can be used to track sleep, running, cycling, general workouts, etc.

I’m not going to delve into what the Microsoft Bot framework is as that has been covered here https://blogs.msdn.microsoft.com/uk_faculty_connection/2016/04/05/what-is-microsoft-bot-framework-overview/ – instead I’m just going to cover some of the aspects of this that I found interesting and point to the project I created to get started with bots.

Up and Running

First, to get started I installed the Bot Builder SDK and read through this documentation http://docs.botframework.com/connector/getstarted/#getting-started-in-net (note the SDK is also available using javascript + node.js). Following this guide will get you set up to run and debug your bot, from getting the Visual Studio template installed to debugging your bot using the Bot Framework Emulator which will enable you to communicate with your locally running bot. Note that it is very easy to get set up and run/debug your bot locally on your machine. The project template

 

creates an ASP.NET Web API project with a Messages controller which has an http POST action to receive messages from a user communicating with your bot.

Your bot is authenticated using an app id and app secret set up in the web.config file. If you press F5 now to run the bot your web API will be hosted locally and you will be able to connect to it using the Bot Framework Emulator (please see http://docs.botframework.com/connector/getstarted/#getting-started-in-net for installation and set up instructions).

Auth

None of the examples I had seen up to this point mentioned auth in terms of how would a bot user authenticate and authorise access to their data. Azure Active Directory auth allows web-browser-free auth using the OAuth2 client credentials flow to authorise an app but applications and APIs using Windows Live style application registration do not. Microsoft Health Cloud API uses “Microsoft account (formerly Live Id)” token-based OAuth (Open Authorization) 2.0 authentication and more specifically the OAuth 2.0 authorisation code flow which requires browser-style redirects which are not provided for by a typical Bot client. Since I specifically wanted to call the Health API this requires the user to give consent to access for the calling app - in order to support this I implemented the following:

If the app is not authorised the bot will reply with a link

User clicks the link which will take them to authorisation server passing a unique ID from the Bot Framework

The user authenticates by logging in and then gives consent through the standard OAuth consent dialog

The unique user id is passed to the redirect uri which I have implemented as part of my Bot Web API along with the Health API access token

The bot Web API stores the access token and it’s related unique user id together

Subsequent calls to the Health API can use the access token to make calls (will stop working if the access token expires as I didn’t implement refresh tokens)  

Subsequently, we have the access token which we can send in the auth header for requests to the Health API.

Natural Language Processing

At this point we can send a message to our bot; the text of that message along with metadata around it’s context such as the current user, etc. will get posted to the MessagesController’s Post. The out-of-box implementation returns the length of the text passed into it. We would like to allow the user to use natural language to ask the Bot about activities such as sleeping and running. Microsoft Research have a service (currently in beta) which can be used to better understand user questions - Language Understanding Intelligent Service (LUIS) – see https://www.luis.ai . For my simple Bot I wanted to understand the entity in a user input string which identified the activity the the user was asking about; running, sleeping, etc. and also the time period over which their request was concerned with; e.g. last 2 weeks, last year, yesterday, etc. LUIS allows you to train a machine learning model by providing examples and tagging parts of those examples. So, I created an Entity called activityType and then I labelled the activityTypes in my example sentences.Obviously the more data I give the model the better it will become. Here are some of the examples I used:

Notice also that the time durations in the above image are also highlighted. This is happening because I added the Bing pre-built entity for Date Time to my app. Imagine the myriad of different ways in which time durations and instances in time can be specified using natural language! When I query the Machine Learning model for this it should identify an activity and a time period – if for some reason a particular query isn’t recognised I can take the failing query and train the model using it so that it will work moving forwards. Once the model is trained or re-trained you can publish it either to a temporary endpoint for testing or to be hosted in your Azure subscription. The API would then be available over a HTTP, REST endpoint.

Putting it Together

So now we have the building blocks to build a fully functional MS Health Bot which will answer the original health-related queries so to pull those blocks together into a functioning Bot

The ParseUserInput method takes the string input by the user, makes a call to the LUIS model and returns a C# object which holds information about the health activity and the duration related to the question. Note that the app id’s and secrets are gained through setting environment variables – this ensures that I won’t add any secrets to the GitHub repository by mistake. In order to run this code you need to train a LUIS model and set the app id and secret into the environment variables, MSHEALTHBOT_LUIS_API_KEY and MSHEALTHBOT_LUIS_APP_ID. Also, in order to access the Health API you would need an app id and secret from an app registered here https://apps.dev.microsoft.com – this shows my app registration. Note the redirect uri

The code also relies on two other environment variables being set: MSHEALTHBOT_HEALTHAPI_CLIENTID & MSHEALTHBOT_HEALTHAPI_CLIENTSECRET which can be copied from your app registration. To set the variables from your dev machine you can use the setx command from the command prompt with the value as the last parameter:

 

This code snippet shows how the duration data from LUIS is converted into a start and an end DateTime required by the health API (I used NodaTime to parse out the durations) and then the response from the API is aggregated to find the total sleep duration. This, of course could be generalised and used to query the other activities besides sleep. Please find the source code for the Bot here https://github.com/peted70/ms-health-bot. This image shows the Bot in action: