Chris Webb's BI Blog

Analysis Services, MDX, PowerPivot, DAX and anything BI-related

Archive for March 2014

Working with Web Services in Power Query

with 21 comments

One of many cool things about Power Query is the way that it allows you to retrieve data from web services and load it into Excel. While this is a subject that lots of people (including me) have already blogged about, the Web.Contents() function – which is the M function that you’ll need to use to call a web service in your query – has gained some new, useful functionality over the last few releases that isn’t fully documented anywhere and which is important to know about.

For this post I’ll use a real-life example of a web service that I’ve been working with recently. As you may know, I’m one of the organisers of SQLBits and a few weeks ago I was given the task of building a Power BI solution to monitor registrations. SQLBits uses RegOnline to handle registrations and they have an API that allows developers to access registration data for events. I’ve been using this API in Power Query.

To take a simple example that illustrates the new functionality in Web.Contents(), take a look at the documentation for RegOnline’s GetEvents method:
http://developer.regonline.com/getevents/

When calling this method in Power Query using Web.Contents() there are two things I need to do:

  1. Pass two query parameters to it, called filter and orderBy – although I only need to pass empty values to these parameters
  2. Handle authentication by passing an API token through the HTTP header

Here’s one way of using Web.Contents() to do this:

Web.Contents(

  "https://www.regonline.com/api/default.asmx/GetEvents", 

  [

   Query=[ #"filter"="", #"orderBy"=""], 

   Headers=[#"APIToken" = "insertAPITokenHere"]

  ])

 

In the example above, the second parameter to Web.Contents() is a record (so it has to be enclosed in square brackets) containing two options fields, Query and Headers. The Query field is itself a record, with one field for each of the two query parameters I’m passing. The Headers field is another record, with one field for each custom HTTP header I need (see this post on the Power Query forum for more details). The RegOnline API specifies that for the purposes of authentication a header is needed called APIToken which contains the API token, so the Headers option here contains one field for that header.

The problem with handling authentication like this is that I need to hard-code the RegOnline API token in the code for my Power Query query, which is not exactly secure. Therefore instead of doing this I use the ApiKeyName option to tell Power Query the name of the header that should contain the RegOnline API token:

Web.Contents(

 "https://www.regonline.com/api/default.asmx/GetEvents", 

 [

  Query=[ #"filter"="", #"orderBy"=""], 

  ApiKeyName="APIToken"

 ])

 

This way, when the Power Query query is executed for the first time, you get prompted to tell Power Query how to authenticate the GetEvents method and in the dialog you have to choose Web API and enter the API token in the Key box, as shown below:

image

After this the API token gets stored separately from the query in Power Query’s own secure credentials store and you don’t need to enter it again; this also means that the API token will not travel with the workbook if you email it to someone else, for example.

One last thing to mention (and this is something that deserves a separate blog post really) is that while Web.Contents() generates a GET request by default, you can make it generate a POST request by specifying the Content option. You can find details on how to do this on this thread started by Lee Hawthorn on the old Data Explorer forum.

Written by Chris Webb

March 26, 2014 at 2:35 pm

Posted in Power Query

Pretty Patterns With Power Query And Power Map

with one comment

Here’s something with no practical use whatsoever. Today, after I finished writing the first draft of the chapter on M of my upcoming Power Query book, I got thinking about how Power View and Power Map get all the attention because of all the eye-catching demos you can create with them. And then I thought – why bother spending time finding real data for these demos when you can generate artificial data in Power Query to create patterns? So I got to work…

As you probably know, you can create animated charts in Power Map so long as you have date-based data. I therefore created a function in Power Query to draw a circle as a series of points on a graph where each point is associated with a date; I also added data for height and colour for each point. Here’s the function definition:

let

    //declare function to draw a circle

    CircleFunction = (CircleRadius as number, StartDate as date, Reverse as logical) =>

let

    //set the radius

    radius = CircleRadius,

    //create a list of numbers from 0 to 359

    anglelist = List.Numbers(0, 359, 1),

    //function to convert degrees to radians

    radians = (a) => (a * 2 * Number.PI)/360,

    //create a list of 360 dates starting from the start date

    unordereddatelist = List.Dates(StartDate, 360,#duration(1,0,0,0)),

    //reverse the list of dates if the Reverse parameter is True

    datelist = if Reverse then List.Reverse(unordereddatelist) else unordereddatelist,

    //generate the list of points on the graph, one for each angle and date

    positionlist = List.Transform(anglelist, each 

                    {_, datelist{_}, Number.Cos(radians(_)) * radius, 

                    Number.Sin(radians(_)) * radius, Date.Month(datelist{_}), 

                    Number.Abs(Number.Cos(radians(_)))*10}),

    //convert the list of points to a table

    outputtable = Table.FromRows(positionlist, {"Angle", "Date", "x", "y", "Colour", "Size"}),

    //set data types

    ChangedType = Table.TransformColumnTypes(outputtable,

                    {{"Angle", type number}, {"Date", type date}, {"x", type number}, 

                     {"y", type number}, {"Colour", type number}, {"Size", type number}})

in

    ChangedType

in

    CircleFunction

 

I then created another Power Query query to call this function 30 times to create 30 circles with different radiuses:

let

    //generate a list of numbers from 0 to 29

    circlelist = {0..29},

    //generate a list of 30 dates starting on 1 January 2014

    datelist = List.Dates(#date(2014,1,1), 30,#duration(1,0,0,0)),

    //call the Circle() function 30 times

    tablelist = List.Transform(circlelist, each Circle(_+5, datelist{_}, Number.Mod(_,2)=0)),

    //combine the resulting tables into a single table

    positionlist = Table.Combine(tablelist)

in

    positionlist

 

And here’s the result of the query plotted on a map using Power Map:

Pretty, isn’t it? You can download the workbook with the Power Query query and the Power Map tour here.

Written by Chris Webb

March 20, 2014 at 11:34 pm

Posted in Power Map, Power Query

User-Defined Conditional Logic In M

with 9 comments

Here’s a short follow-up to my last post on conditional logic in M. After that post went live, Ehren Vox of the Power Query team made a good suggestion on Twitter: rather than hard-code the list of conditions and values inside the query, why not take those values from the Excel spreadsheet too? That way end-users can maintain the conditions and values themselves.

Here’s my Excel spreadsheet, now with two tables: one called Input, containing my input value, and one called CaseValues containing my conditions and return values.

image

And here’s my new query, a variation on the simple case statement query from my previous post, but this time using the values from the CaseValues table to drive the logic:

let

    //load input value table from worksheet

    Source = Excel.CurrentWorkbook(){[Name="Input"]}[Content],

    //get input value from that table

    InputValue = Source{0}[Input],

    //load case values from worksheet as a table

    CaseTable = Excel.CurrentWorkbook(){[Name="CaseValues"]}[Content],

    //turn that table into a list and append the else condition to the end

    CaseValues = List.Combine({Table.ToRows(CaseTable),{{InputValue, "Else condition"}}}),

    //look for the input value in the CaseValues list and return the value associated with it

    SimpleCase = List.First(List.Select(CaseValues, each _{0}=InputValue)){1}

in

    SimpleCase

The output here, once again, is the text value “Five”. Two interesting things to notice here:

  • I used the Table.ToRows() function to turn the table containing my case values into a list of lists
  • I used List.Combine() to append the else condition (a list containing two values, the input value and the text “Else condition”) onto the end of the list returned by Table.ToRows()

I’ve added this example to my original demo workbook, which can be downloaded here.

Written by Chris Webb

March 14, 2014 at 11:38 pm

Posted in Power Query

Conditional logic in Power Query

with 8 comments

Writing a simple if statement in Power Query’s M expression language is straightforward. Using an Excel table called Input that contains a single value as the starting point:

image

The following query shows how to use an if … then … else statement to test whether the value from the table is equal to 5:

let

    Source = Excel.CurrentWorkbook(){[Name="Input"]}[Content],

    InputValue = Source{0}[Input],

    IfStatement = if InputValue=5 

                    then "The number is five" 

                    else "The number is not five"

in

    IfStatement

What about more complex conditional logic? The M language doesn’t include anything like a case statement, but it is possible to write the equivalent of one quite easily.

Here’s an example of a simple case statement:

let

    Source = Excel.CurrentWorkbook(){[Name="Input"]}[Content],

    InputValue = Source{0}[Input],

    CaseValues = {

                    {1, "First"},

                    {2, "Second"},

                    {3, "Third"},

                    {4, "Fourth"},

                    {5, "Fifth"},

                    {InputValue, "Else condition"}

                },

    SimpleCase = List.First(List.Select(CaseValues, each _{0}=InputValue)){1}

in

    SimpleCase

This works as follows:

  • The CaseValues step defines a list containing six items, each of which is itself a list containing a number and some text. The number is the value to compare to the input value, and the text is what will be returned if the number does match the input value.
  • The last item in the CaseValues list contains the input value, so this will be returned where the input value matches none of the preceding values
  • The SimpleCase step uses List.Select() to filter the list in CaseValues so that only the items in the list where the input value matches the number in the list.
  • Since List.Select itself returns a list, this list is then passed to List.First() to get the first item in the list returned by List.Select (there should only be one item in the list in this particular query), and then {1} returns the text from that item. This is the output of the query.

You can write a searched case expression in a very similar way, by declaring functions that return boolean values instead of using numbers as follows:

let

    Source = Excel.CurrentWorkbook(){[Name="Input"]}[Content],

    InputValue = Source{0}[Input],

    CaseValues = {

    { (x)=>x<10, "Less than 10"},

    { (x)=>x<20, "Less than 20"},

    { (x)=>x<30, "Less than 30"},

    { (x)=>x<40, "Less than 40"},

    { (x)=>x<50, "Less than 50"},

    { (x)=>true, "Else condition"}

    },

    SimpleCase = List.First(List.Select(CaseValues, each _{0}(InputValue))){1}

in

    SimpleCase

 

In this query the CaseValues step contains a list of lists, where each item in the list consists of list containing a function and a text value. List.Select calls each function and only returns the items where the function returns true, and finally the text from the first item that List.Select returns is the output of the query.

You can download the sample workbook here.

Written by Chris Webb

March 10, 2014 at 9:00 am

Posted in Power Query

SQLBits XII

leave a comment »

In case you missed the announcement yesterday, SQLBits XII will be taking place at the International Centre, Telford, UK on July 17th-19th. SQLBits is the biggest SQL Server and Microsoft BI conference in Europe and will feature precons and sessions from some of the best-known SQL Server experts in the world (I see Brent Ozar and Brian Knight have already submitted sessions, which is cool). And apart from all the amazing technical content it’s a lot of fun – just ask anyone who’s been to a previous SQLBits!

Full details and the link to register can be found on the SQLBits website: http://sqlbits.com/ Hope to see you there…

Written by Chris Webb

March 4, 2014 at 9:50 am

Posted in Events

Allocation in Power Query, Part 2

with 2 comments

Last week’s post on allocation in Power Query caused quite a lot of interest, so I thought I would follow it up with a post that deals with a slightly more advanced (and more realistic) scenario: what happens if the contracts you are working with don’t all start on the same date?

Here’s the table of data that is the starting point for my examples:

image

I’ve made two changes:

  • I’ve added a contract name to serve as a primary key so I can uniquely identify each contract in the table. Several people asked me why I added index columns to my tables after my last post and this is why: without a way of uniquely identifying contracts I might end up aggregating values for two different contracts that happen to have the same number of months, contract amount and start date.
  • I’ve added a contract start date column which contains the date that the contract starts on, which is always the first day of a month.

Now let’s imagine that you want to make each monthly payment on the last day of the month. You need to take each contact and, for each monthly payment generate a row containing the date that is the last day of the month, containing the allocated payment amount.

Once again, having have opened the Query Editor the first step is to calculate the amount of the monthly payment using a custom column that divides Contract Amount by Months in Contract. This is shown in the Allocated Amount column:

image

Now to generate those monthly payment rows. Since this is reasonably complex I decided to declare a function to do this called EndsOfMonths inside the query, as follows:

= (StartDate, Months) =>
List.Transform(List.Numbers(1, Months), each Date.AddDays(Date.AddMonths(StartDate, _ ), -1))

This function takes the start date for contract and the number of months, and:

  • Uses List.Numbers() to create a list containing numbers from 1 to the number of months in the contract. For example if there were three months in the contract, this would return the list {1,2,3}
  • This list is then passed to List.Transform(), and for each item in the list it does the following:
    • Adds the given number of months to the start date, then
    • Subtracts one day from that date to get the payment date, which will be the last day of the month it is in

Calling this function on each row of the table in a new custom column (called Payment Date here) gives you a list of the payment dates for each contract:

image

All that you need to do then is to click on the Expand icon next to the Payment Date column header and make sure each column has the correct type, and you have your output for loading into the Excel Data Model:

image

Here’s the code for the query:

let

    //Load source data from Excel table

    Source = Excel.CurrentWorkbook(){[Name="Contract"]}[Content],

    //Add custom column for Allocated Amount

    InsertedCustom1 = Table.AddColumn(Source, "Allocated Amount", 

        each [Contract Amount]/[Months In Contract]),

    //Declare function for returning a list of payment dates

    EndsOfMonths = (StartDate, Months) => 

        List.Transform(List.Numbers(1, Months), 

            each Date.AddDays(Date.AddMonths(StartDate, _ ), -1)),

    //Call this function for each contract in a new custom column

    InsertedCustom = Table.AddColumn(InsertedCustom1, "Payment Date", 

        each EndsOfMonths([Contract Start Date], [Months In Contract]) ),

    //Expand the list

    #"Expand Payment Date" = Table.ExpandListColumn(InsertedCustom, "Payment Date"),

    //Set column data types

    ChangedType = Table.TransformColumnTypes(#"Expand Payment Date",

        {{"Contract Start Date", type date}, 

        {"Payment Date", type date}, {"Allocated Amount", type number}, 

        {"Contract Amount", type number}, {"Months In Contract", type number}})

in

    ChangedType

 
There’s one more thing to do though. Since the Contract table contains real dates, it’s a very good idea to have a separate Date table in the Excel Data Model to use with it. I’ve already blogged about how to use a function to generate a Date table in Power Query (as has Matt Masson, whose version adds some extra features) and in that function (called CreateDateTable) can be reused here. Here’s a query that returns a Date table starting at the beginning of the year of the earliest start date in the contract table and ends at the end of the year of the last payment date:
 
let

    //Aggregate the table to find the min contract start date

    //and the max payment date

    GroupedRows = Table.Group(Contract, {}, 

    {{"Min Start Date", each List.Min([Contract Start Date]), type datetime}, 

    {"Max Payment Date", each List.Max([Payment Date]), type datetime}}),

    //Find the first day of the year of the min start date    

    #"Start Date" = DateTime.Date(Date.StartOfYear(GroupedRows{0}[Min Start Date])),

    //Find the last day of the year of the max payment date

    #"End Date" = DateTime.Date(Date.EndOfYear(GroupedRows{0}[Max Payment Date])),

    //Call CreateDateTable with these parameters

    DateTable = CreateDateTable(#"Start Date", #"End Date"),

    //Change data types

    ChangedType = Table.TransformColumnTypes(DateTable,{{"MonthNumberOfYear", type number}

    , {"DayOfWeekNumber", type number}})

in

    ChangedType

 

You can now build a PivotTable to show the payments allocated over the correct ranges:

image

The sample workbook can be downloaded here.

Written by Chris Webb

March 2, 2014 at 9:50 pm

Posted in Uncategorized

Follow

Get every new post delivered to your Inbox.

Join 3,302 other followers