Chris Webb's BI Blog

Analysis Services, MDX, PowerPivot, DAX and anything BI-related

Power BI Preview Review Part 2: Q&A

with 20 comments

Today the natural language query functionality in Power BI, Q&A, was enabled in my Power BI Preview tenant. This is the last major piece of functionality to be added to the Preview, and of course as soon as I heard that it was there I had to go and play with it, despite being on a slow mobile WIFI connection and having tons of emails to catch up with. So apologies to anyone who is waiting for an email from me!

First thing to point out: you can only use it on some sample workbooks provided by Microsoft, and these get added to your Power BI site when you enable Q&A. The ability to use it on your own data is coming soon but I don’t think this stopped me from getting a feel for how well it works.

With Q&A switched on my Power BI site now looks like this:

image

Notice that the top half of the page now has a ‘Popular Questions’ area which takes you to some pre-defined questions on the sample data, while in the bottom half you see the sample workbooks along with any other workbooks you have uploaded. The two sample workbooks contain data on medals won at the Olympics and sales of drinks in a bar.

Here’s what one of the sample questions looks like when you click on it:

image

At the top of the screen you have a search box where you can type your question; immediately underneath that box is how Q&A has interpreted the question. On the left-hand side you see the workbook used as the data source, and you can see alternative visualisations for the data that you can choose. On the right-hand side there are some suggested questions for this data and other hints and tips. It’s all pretty straightforward.

But enough of the descriptions – there are two big questions that need to be answered here:

  1. Does it work? That’s to say, will a real user be able to ask questions and find they data they want?
  2. Will anyone use it, or is it just demo-ware?

I’m aware these are loaded questions and I run the risk of upsetting my friends at Microsoft if I say rude things about this product. I’m also aware that I have only had a very limited amount of time to play with it; that it is still a Preview and will undoubtedly improve a lot over the next few months (however good it is now); and most importantly that I am not the target audience for this product therefore it will be impossible for me to judge how well it works – I know too much, and I am likely to be able to be able to write questions that work much better than a normal user can. I had hoped to try this out on my wife to get some real world feedback but unfortunately I’m travelling at the moment and she’s at home with the kids…

With that in mind, let me give you some idea of how well it works by showing questions I tried and the output I got:

The first question I tried to find an answer for was how many medals Australia had won for Swimming in 2012. I typed “how many medals did Australia win for swimming in 2012” and this was the result:

image

I had wanted a single value and instead I got a count of medals for swimming broken down by name; they were Australians (I assume) but it’s not exactly what I was after. I then tried “total number of medals won for swimming by Australia in 2012” and hit the jackpot:

image

My next question “list the cities where the games have been held” got the right answer first time:

image

A similar question, “total number of times the games have been held in each city”, also worked first time (but maybe because I had learned that Q&A can interpret the phrase “total number” properly):

image

I don’t know why it thought a map was appropriate here though.

One final test: I wanted to know which athletes had won medals at more than three different Olympics. I tried “who has won medals at more than three games” but got a list of countries that had won medals at more than three games, which of course was quite a long one; “athletes that have won medals at more than three Olympics” got a list athletes and the games they had won medals at. Eventually after several more tries I gave up.

Of course this last question was deliberately difficult question and entering the same text into Google didn’t give any useful results either. I would have been very surprised if I had got the correct answer. I think, though, this question gets to the heart of the reason why a lot of us were sceptical about Q&A when they first heard about it: end users won’t know which questions a product like this can reasonably answer and which ones are just too difficult, and they won’t know how to frame questions appropriately, and therefore a product that seems to suggest it can answer any question you type will always, ultimately, disappoint them. It will be very difficult to set users’ expectations appropriately.

That said, I was pleasantly surprised at its success rate and I’m looking forward to trying it on my own data. To answer my questions above:

  1. Does it work? Yes, I think it does a good job of answering any question I would have expected it to be able to answer. Furthermore, as I said, I know it will improve over time, not only because we’re still in Preview but also because this is a cloud-based solution and MS can keep tweaking after it has been released as well. I wonder if MS are capturing usage metrics and questions and using it to tune the system? I bet they are.
  2. Will anyone use it? This is a much trickier problem. I’ve never come across an end user who has asked for a product like this, but just because no-one has ever asked for something doesn’t mean they won’t love it if they get it. It’s clearly aimed at people who find every other BI tool available today too difficult (and there are plenty of them), but are these people interested in using any BI tool however easy it is to use? Will they just go and ask a colleague or an analyst or a minion to go and find the answer for them instead? Frankly, I don’t know at this point. We shall see though!

Written by Chris Webb

September 23, 2013 at 11:12 pm

Posted in Power BI

20 Responses

Subscribe to comments with RSS.

  1. Reblogged this on analyzerhakan and commented:
    Power BI Preview blog, usefull

    analyzerhakan

    September 24, 2013 at 8:24 am

  2. Hi Chris,

    As usual a very interesting article – I find the samples very demoware friendly and would really like to use the QA on a model I created and see how the users of such a model would respond to the QA feature.

    Erik

    Erik Svensen

    September 24, 2013 at 10:23 pm

  3. Hi Chris,

    I tried ‘medalist where count of olympic name is greater than 3′ and even though it seems to work, I’m not certain if that count is a distinct count… adding the word distinct to my question didn’t help…

    So, just to quickly double check the results of that, I asked ‘medalist and olympic name where medalist is like SZIVOS’ (since according to the previous results, that person has won medals in 7 different olympic games) and for that specific question, the Olympic games where in fact different…

    It is interesting what you mention because there are advantages that IT people will have over the common business user, even the most knowledgeable data steward could have problems ‘asking the right question’… or maybe we as developers should consider crazy scenarios and prepare our models to support almost any query… which made think if the ‘distinct’ issue could be solved by adding that specific measure to the model… anyways, I’m sure Microsoft’s algorithm will only get better at understanding the user with time…

    Have you had time to run questions against your own models?
    I uploaded and enabled a model successfully, but for some reason is not listed under the ‘results from’ section when asking questions related to my model…

    Alex Gonzalez

    September 24, 2013 at 11:56 pm

    • Questions phrased like ‘medalist where count of olympic name is greater than 3′ are exactly what I was talking about when I said that developers have an advantage over regular users – I see see that you are thinking in SQL!

      No, I haven’t run questions over my own models yet because it isn’t supported in the Preview. At the moment you can only ask questions against the sample models.

      Chris Webb

      September 25, 2013 at 9:22 am

      • I have been skeptical about Q&A from the start. After all, we still don’t have perfect grammar correction, text-to-speech, or language translation software because context is so difficult to interpret. Q&A is in the same boat. Of course you wouldn’t know from the highly edited videos that show it working perfect every time!

        I haven’t tried Q&A myself – I’ve abandoned my Power BI efforts for now because, even after following the provisioning video to the letter, my Sites link was never enabled.

        Colin Banfield

        October 7, 2013 at 4:10 pm

      • Chris – by any chance, do you know, if the Q&A feature of running our own data models is enabled now (Dec 5 2013)?

        Prabhu Ganeshan

        December 5, 2013 at 8:40 am

      • It wasn’t when I checked a few days ago…

        Chris Webb

        December 5, 2013 at 1:43 pm

  4. Although the question about athletes didn’t turn up in Google, did you manage to bring the results forward in Wolram Alpha? (I didn’t). The aim and structure of Wolfram Alpha is much more akin to the Power BI “Computational Knowledge engine” than Google is.

    Koos van Strien

    September 25, 2013 at 8:00 am

  5. […] Power BI Preview Review Part 2: Q&A […]

  6. […] Power BI Preview Review Part 2: Q&A (Chris Webb) […]

  7. Wolfram Alpha says 10 to “how many medals did Australia win for swimming in 2012″ QA may be double counting team medals. http://www.wolframalpha.com/input/?i=how+many+medals+did+Australia+win+for+swimming+in+2012

    Muigai

    October 10, 2013 at 12:35 pm

  8. Natural Language Search is one of those ‘cool’ things that will help drive adoption of self service, even if ultimately you’ll prefer to use the normal capabilities of the tool. But it has to work… I once tried out a BI product that had natural language search as one of its unique selling points, where it presented the user with different search results like google does, with each result being a visualisation. So I started off like you did, with simple stuff. In fact, I tried the simplest query I could think of. It didn’t work, in my eyes- the ‘right’ set of data was 7th or 8th in the list.
    To use these things you need to a) understand all the terminology that is in the underlying model (know that it is athlete rather than sportsman) and b) the exact way to phrase it (i.e. know that total number is something the tool understands).
    In this, it is no different to any other self service tool- you need to train the users on the tool and the data at least a little before they can get the most out of it, as well as constructing a really good model that is going to be easy for people to work their way around.
    In fact, since you have to do that, why wouldn’t your users prefer a simple drag and drop tool like PowerView, that at least shows you what’s available before you create your query?

    rjback

    October 10, 2013 at 1:52 pm

  9. Has the ability to use own workbooks for Q&A been enabled yet?

    Anders Malmström

    October 29, 2013 at 10:23 am

  10. […] few months ago I posted a review of Q&A, the natural language query functionality in Power BI, based on the sample data sets that were then […]

  11. […] few months ago I posted a review of Q&A, the natural language query functionality in Power BI, based on the sample data sets that were then […]

  12. […] blogged at great length about what I think are the strengths and weaknesses of Power BI (see here, here and here) so I won’t repeat myself here. As you would expect Microsoft marketing has gone into […]

  13. […] blogged at great length about what I think are the strengths and weaknesses of Power BI (see here, here and here) so I won’t repeat myself here. As you would expect Microsoft marketing has gone into […]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 2,868 other followers

%d bloggers like this: