SQLBits X Summary

SQLBits X finished over a week ago and so it seems a bit late to be blogging about it. To be honest, though, I needed that week to recover from the conference, a cold I picked up the day before it started, and all of the other work-related stress that’s been building up over the last few months.

So, the obligatory bit of reportage. SQLBits X was indeed as big as we were hoping and behind the scenes everything went surprisingly smoothly; I guess after nine previous events we must have learned something about running a tech conference! I don’t know what the official numbers are but we packed 1400 swag bags on the Wednesday afternoon before it all started in a five-hour bag stuffing marathon, and on the Saturday evening we only had about 100 or so left, so that makes it easily 50% larger than any other SQLBits. We had a great line-up of speakers including a large number of international SQL celebrities, more than we’ve ever had before, and the parties on Thursday and Friday night were well-attended and more importantly good fun.  I don’t know how SQLBits could get any better except by getting bigger and offering even more of the same, although if we did get bigger it would probably end up killing a lot of what makes SQLBits special and push us beyond the limits of what an amateur organisation can cope with. My thanks go out to my fellow committee members Simon Sabin, Allan Mitchell, Martin Bell, James Rowland-Jones, Darren Green, Chris Testa-O’Neill, Tim Kent and Christian Bolton, as well as all of the team of helpers who gave up their time free of charge including Annette Allan, Helen Lau, and many others.

New SSAS 2012 Tabular book available for pre-order

At long last, the SSAS 2012 Tabular book that Marco Russo, Alberto Ferrari and I have been working hard on for the last few months is now available for pre-order!

You can get it from Amazon UK, Amazon US, oreilly.com and all good bookstores…

What’s new in Analysis Services 2012 Multidimensional?

Now that SQL Server 2012 has been released, I’ve just realised I’ve not seen a definitive list of what’s new in SSAS 2012 Multidimensional. In fact I’m surprised I haven’t got round to writing a blog post about this… after all, despite the fuss over Tabular, Power View and all the other cool new stuff I guess most existing SSAS users are going to be more interested in staying with the Multidimensional model when they upgrade.

So what is new? Here’s a list of everything I know about plus some relevant links:

• The infamous 4GB limit on string stores in dimensions has now been fixed. See:
  http://msdn.microsoft.com/en-us/library/gg471589(v=sql.110).aspx
  http://raphaelmsbithoughts.wordpress.com/2011/07/22/denali-ctp3-analysis-services-new-feature-scalable-string-storage/
• Distinct count performance in ROLAP mode has been improved:
  http://msdn.microsoft.com/en-us/library/bb522628(v=sql.110).aspx#bkmk_distinctCount
• SSAS now supports NUMA and more than 64 processors:
  http://msdn.microsoft.com/en-us/library/hh548655(SQL.110).aspx
  This is a very detailed, must-read blog post on the subject:
  http://blogs.msdn.com/b/psssql/archive/2012/01/31/analysis-services-thread-pool-changes-in-sql-server-2012.aspx
  All this may be why processing is much faster in some cases, as Bob Duffy notes here:
  http://blogs.prodata.ie/post/Analysis-Services-2012-Faster-Processing.aspx
• I’ve seen it mentioned in several other places that there have been 300 other improvements for performance, scalability and reliability in the SSAS engine. I’ve not seen them documented anywhere (maybe they are in a KB article) but they’re likely to be minor tweaks/bug fixes similar to what you’d get in any SP. Nevertheless if you run into the specific scenarios they address then they could be very significant.
• SSAS now supports XEvents for monitoring:
  http://msdn.microsoft.com/en-us/library/gg492139(v=sql.110).aspx
• There are some new Profiler events. BOL notes the Resource Usage and lock-related events, but the former (and I’m pretty sure the latter) were also made available to 2008 R2 users with the last SP:
  http://msdn.microsoft.com/en-us/library/bb522628(v=sql.110).aspx#bkmk_traceResource
  http://msdn.microsoft.com/en-us/library/bb522628(v=sql.110).aspx#bkmk_traceLocks
  However the really interesting new events are “Calculation Evaluation” and “Calculation Evaluation Detailed Information” which give an insight into what’s going on in the Formula Engine. I hope we get some information on how to interpret the contents of these events.
• There’s a new Analysis Services PowerShell provider and cmdlets:
  http://technet.microsoft.com/en-us/library/hh213141(v=sql.110).aspx
  http://www.databasejournal.com/features/mssql/analysis-services-powershell-provider-sqlas-in-sql-server-2012.html
• All development work is now done in Visual Studio 2010, and BIDS is now called SQL Server Data Tools (SSDT).
• The cube browser has now disappeared from SSDT, and is replaced by the control used by the SSRS query generator. This is because the old browser used the OWC control which was deprecated years ago and finally had to be replaced; it’s a massive step backwards because this control is truly awful: for example you can no longer put anything other than Measures on columns in your query. Arrggghhh! I’ve started using Excel instead, but Excel won’t always be available on some sites and it’s a hassle to move to a different app to check your figures.

Have I forgotten anything? Have you found anything changed that isn’t documented? If so, please leave a comment. I’ll update this post as and when I find/remember new stuff.

SSAS Maestros Training: July 9-13 2012, Milan, Italy

After all the activity last year, you may have been wondering what’s happened with the SSAS Maestro Program – the premier certification for Analysis Services professionals (for some more background information see here). Well, wonder no more: my fellow SSAS Maestro Marco Russo and I will be running the course again in Milan this summer on the 9th-13th July.

There are a few things that are going to be different with this run of the course which I need to point out:

• It’s no longer invitation-only so anyone can apply to attend. However we’ll only have 20 spaces in the training room and this, plus the fact that this is a very demanding course, means we’ll only be accepting people who have a lot of previous SSAS experience.
• It’s no longer free either. The cost will be €7000 plus taxes for five days of training plus the exam/coursework/labs, although people who have previously attended the Maestros course will only have to pay €5000 plus taxes. That’s fairly expensive I know, but there’s a lot of work involved in organising a one-off course like this, a lot of marking to be done afterwards and there are also a lot of overheads as well as two trainers to pay.
• The marking process will be a lot faster this time!

If you’re interested in attending please send Marco and me an email at ssasmaestro@sqlbi.com, along with a copy of your CV/resume and a few paragraphs detailing your level of SSAS knowledge. We’ll be opening registrations in mid-April and we’ll be able to let you know whether you’ve been accepted then.

Don’t say Vertipaq, say xVelocity!

Something that got a bit of attention last week on Twitter, but which may not have filtered through into the wider SSAS community yet, is the fact that Microsoft has renamed the Vertipaq engine that’s inside PowerPivot and Analysis Services 2012 Tabular. It’s now called “the xVelocity in-memory analytics engine”, and the term “xVelocity” is also going to be used to refer to the column store index feature in the SQL Server 2012 relational database. For more details see the official announcements:

http://blogs.technet.com/b/dataplatforminsider/archive/2012/03/08/introducing-xvelocity-in-memory-technologies-in-sql-server-2012-for-10-100x-performance.aspx

http://blogs.msdn.com/b/analysisservices/archive/2012/03/09/xvelocity-and-analysis-services.aspx

Personally I liked the old Vertipaq name and as I said recently, I think these umbrella marketing terms that refer to lots of different things cause confusion. But it’s only a name so there’s nothing much to get upset about – just don’t say Vertipaq any more, say xVelocity.

SQL Server 2012 RTM–and a new SSAS Tabular training course

By now you’ve probably already heard that SQL Server 2012 has been released (you can download the eval here), along with the Feature Pack (which contains some interesting stuff, including a 64-bit version of the Excel Data Mining Addin at last) and PowerPivot V2.0. I know I’m almost 24 hours late on this news but I thought I’d mention it nonetheless…

If you’re a BI professional wondering how you’re going to learn Analysis Services 2012 Tabular, then you might want to check out a new series of training courses on it that I’m teaching with my friends Marco and Alberto at various places around Europe:
http://www.ssasworkshop.com/

It’s based on the book we’ve been co-writing on the Tabular model which will hopefully be available very soon, and since we’ve been living Tabular and DAX for the last few months you can be sure that it’s the very best way to get up to speed on it. We hope to see you at one of the classes!

Natural and Unnatural Hierarchies in the SSAS 2012 Tabular Model

I’m sure you’re all aware of the difference between natural and unnatural user hierarchies in the Analysis Services Multidimensional model (if you’re confused as to what I mean when I say ‘Multidimensional model’, have a quick read of this post from a few weeks ago which explains the terminology). To recap, natural user hierarchies in Multidimensional look like this in BIDS:

There is a one-to-many set of attribute relationships between each level, so each Calendar Year has multiple Calendar Semesters but one Calendar Semester has only one Calendar Year, and they are a Good Thing as far as query performance is concerned. Unnatural user hierarchies look like this:

They work, but there aren’t one-to-many relationships between every level and query performance may be worse than with a natural user hierarchy.

But what does all this have to do with the Tabular model? In SSDT when you create a hierarchy there’s no indication as to whether it’s natural or unnatural:

…and up to recently I assumed that this was an issue that simply wasn’t relevant to Tabular. However, after a recent conversation with Marius Dumitru from the dev team and Greg Galloway I now know this isn’t the case!

If you query the MDSCHEMA_HIERARCHIES DMV as follows:

SELECT
[DIMENSION_UNIQUE_NAME], [HIERARCHY_NAME], [STRUCTURE_TYPE]
FROM $SYSTEM.MDSCHEMA_HIERARCHIES

You can see whether a hierarchy in a Tabular model is natural or unnatural:

In this case you can see that the Calendar hierarchy that I created on the DimDate table is unnatural; SSAS has determined this during processing (specifically during the Process ReCalc stage) by examining the data itself automatically.

It turns out that natural hierarchies in Tabular can result in faster query performance because certain MDX and Formula Engine code paths in the SSAS engine are still not optimised for unnatural hierarchies. I don’t have any specific examples of when this occurs at the moment but if I do find them I’ll be sure to update this post. And if anyone else using Tabular, or even PowerPivot (and I assume this is relevant to PowerPivot too) finds a good example of how changing to a natural user hierarchy improves performance please leave a comment.

In the example above, I created the Calendar hierarchy in the Tabular model by simply dragging the CalendarYear, CalendarSemester, CalendarQuarter, EnglishMonthName and FullDateAlternateKey columns underneath each other in the new hierarchy. It’s unnatural because there are only two distinct values in Calendar Semester (the semester numbers 1 and 2), four distinct values in Calendar Quarter (the quarter numbers 1 to 4) and there are only twelve distinct values in EnglishMonthName (the names of the months), so there is a many-to-many relationship between the values in all these columns. I can make it natural by creating three calculated columns that concatenate CalendarYear and CalendarSemester, CalendarYear and CalendarQuarter, and CalendarYear and EnglishMonthName as follows:

DimDate[Calendar Semester of Year] =DimDate[CalendarYear] & " " & DimDate[CalendarSemester]

DimDate[Calendar Quarter of Year] = DimDate[CalendarYear] & " Q" & DimDate[CalendarQuarter]

DimDate[Calendar Month of Year] = DimDate[CalendarYear] & " " & DimDate[EnglishMonthName]

Using these calculated columns for the Semester, Quarter and Month levels of the hierarchy as follows:

…will make the Calendar hierarchy natural:

PASS BI Virtual Chapter

Just to echo what Marco said in his blog post from last week, the PASS BI Virtual Chapter is currently looking for presenters, especially from Europe or European-friendly time zones. If you are interested please send some ideas for abstracts plus some information about yourself and your speaking experience to PASSDWBIVC@sqlpass.org. Presenting online is a great way to get speaking experience, especially if you are not able to attend conferences or user groups on a regular basis. We hope to hear from you soon!

Using the MDX Script Debugger in BIDS

Almost every time I write MDX calculations, I end up using scoped assignments – it’s by far the easiest way to control where your calculations work inside a cube. However making sure that your scoped assignments work in the places that they’re meant to work and don’t overwrite each other can be very tricky indeed (for a brief introduction to the subject and these problems, see this session I gave at SQLBits a year or so ago), so in this post I’m going to show you how you can use the MDX Script Debugger in BIDS to help you do this.

Despite its name the MDX Script Debugger doesn’t actually help you to debug individual MDX expressions or calculations. What it does is clear the MDX Script of the cube then allow you to step through each statement in the MDX Script, applying them one after the other, so you can see the cumulative effect on a query. This is only really useful when you’re working with scoped assignments because it allows you to see which cells in your query are changed with each successive assignment.

To illustrate, let’s use the Adventure Works cube. Comment out everything on the MDX Script (ie what’s on the Calculations tab) except the Calculate statement and add the following code to the bottom:

CREATE MEMBER CURRENTCUBE.MEASURES.DEMO AS 1;

SCOPE(MEASURES.DEMO);
    SCOPE([Date].[Date].MEMBERS, [Date].[Calendar Quarter].[Calendar Quarter].MEMBERS);
        THIS=2;
    END SCOPE;
    SCOPE([Date].[Calendar Year].[Calendar Year].MEMBERS);
        THIS=3;
    END SCOPE;
END SCOPE;

SCOPE([Measures].[Internet Sales Amount]);
    SCOPE([Date].[Date].MEMBERS, [Date].[Calendar Quarter].[Calendar Quarter].MEMBERS);
        THIS=2;
    END SCOPE;
    SCOPE([Date].[Calendar Year].[Calendar Year].MEMBERS);
        THIS=3;
    END SCOPE;
END SCOPE;

Then deploy your database and go to Debug menu in BIDS and click Start Debugging:

The MDX Debugger screen will then be displayed:

In the top pane you can see the Calculate statement highlighted – this is the point in the script that has been reached as you step through the code. In the bottom pane you have a browser control where you can construct a query plus four panes where you can enter your own hand-written MDX queries.

In the browser control, drag the Internet Sales Amount measure onto columns and the Calendar hierarchy from the Date dimension onto rows, until you see something like this:

You don’t see any data at the moment because the Calculate statement hasn’t been executed yet (see here for more details on what the Calculate statement does). If you hit F10 to step to the next statement in the MDX Script you’ll see the values for Internet Sales Amount appear because the Calculate statement has now been executed:

If you hit F10 again, the calculated member DEMO will be created and you can now drag it into the browser; at this point you’ll see it always returns the value 1 because none of the scoped assignments have been executed yet:

Hit F10 again until you reach the first END SCOPE statement and you’ll see the following:

You can see that MEASURES.DEMO now returns 2 for the Date, Month and Quarter level as a result of this first assignment; you can also see that only the values that have been affected by this assignment have been changed. Hit F10 some more to execute the second assignment and you’ll see that DEMO returns 3 at the Year level and the affected cells are again highlighted:

Notice how, in this case, because you’re scoping on a calculated measure only the cells you scoped on have their values changed. This is in contrast with scoped assignments on regular measures: because regular measures aggregate up, scoping on a regular measure not only affects the values in the cells you scoped on, but those values will then also be aggregated up though the cube.

To show what does happen when you scope on a regular measure, look at the next set of scoped assignments on the Internet Sales Amount measure. The first assignment scopes on the Date, Month and Quarter levels and sets their values to 2; however the Year level values now show the aggregated totals of all the quarters in the year, so if there are four quarters in a year then the year will show 4 * 2 = 8. The All level total is also similarly affected.

The final assignment sets the Year totals to 3 for the Year level; this overwrites the values that have been previously aggregated up from the Quarter level, and the Year level values are again aggregated up to the All level:

Hit F10 some more and you’ll reach the end of the MDX Script, whereupon you’ll go back to the beginning and can start all this again. Go to the Debug menu and click Stop Debugging to finish. Useful bit of functionality, isn’t it? Certainly one of the least-known features of BIDS too.

One last point – if you try to use the Debugger and hit the infamous SSAS Locale Identifier bug, check out Boyan Penev’s post here on how to solve this issue.

So, what is the BI Semantic Model?

Over six years ago now I wrote what proved to be a very popular post here on my blog called “So, what is the UDM?”. It was written in response to the widespread confusion around the time of the release of Analysis Services 2005 about what the term “UDM” actually meant. In a sense “UDM” was just another name for an Analysis Services cube, but it also represented the nebulous concept of Analysis Services being a semantic layer suitable for all your reporting needs; however people often thought it was a new product that somehow replaced Analysis Services cubes and got themselves tied in all sorts of knots as a result. Thankfully, use of the term died out pretty quickly as everyone started referring to Analysis Services as, well, Analysis Services.

Fast forward to the present day and I can see a similar amount of confusion about the term “BI Semantic Model” or BISM for many of the same reasons. What is the BI Semantic Model exactly? It is… Analysis Services 2012 plus PowerPivot. Let’s be clear though: it is not just the new Analysis Services Tabular Model, although the term BISM is all too often used as if it did mean that. It’s not even a specific bit of software. Analysis Services 2012 consists of two parts, the Multidimensional Model which is the Analysis Services of cubes and dimensions that we already had in 2008 R2 and earlier versions, and the new Tabular model which is the Analysis Services of tables, relationships, in-memory storage and column store. BISM refers to both models plus PowerPivot, or rather it refers to the way that Analysis Services and PowerPivot can be used as a semantic layer on top of other data for reporting purposes.

So what’s the point of a term like BISM then if it doesn’t refer to something tangible? Why not just call Analysis Services “Analysis Services” and PowerPivot “PowerPivot”? Well there’s certainly some conceptual stuff going on here (as outlined in the Vision and Roadmap blog post) but just as we had with the term UDM I’d say there’s also some marketing-led obfuscation here as well, for three reasons:

• A single term like BISM suggests that Analysis Services 2012 and PowerPivot are a single, cohesive product, whereas the Tabular and Multidimensional models are actually very different beasts. If you’re going to be working with Analysis Services 2012 on a project the first decision you’ll have to make is which type of model to use, and if you change your mind later you’ll have to start development again from scratch and learn a lot of new skills. I hope one day that the two models will merge again but it won’t happen soon.
• Microsoft has correctly identified that many people want to do BI but were put off by the complexity of building Multidimensional models in previous versions of Analysis Services. The simplicity of the Tabular model goes a long way to solving this problem; Tabular also replaces Report Builder models which were really a just a simple semantic layer for people who didn’t like SSAS or had valid reasons to stay with relational reporting. In order not to scare off this market a new name is necessary to avoid the negative connotations that come with “Analysis Services” and “cubes”.
• Calling something a “semantic model” suggests that it’s a nice, light, thin, easy-to implement layer on top of your relational data warehouse, with no data duplication (which is often seen as a Bad Thing) involved. In actual fact anyone who has used the Multidimensional model will know you almost always use MOLAP storage which involves all the data being copied in Analysis Services; and I suspect when people start using the Tabular model they will be using it in Vertipaq mode (where again all the data gets copied into Analysis Services) rather than in DirectQuery mode (where all queries are translated to SQL which is then run against SQL Server).

Now I’m not going to suggest that there’s anything wrong with these marketing objectives – anything that sells more Analysis Services is good for me – or that the Tabular model is bad, or anything like that. All I want to do is suggest that in the SQL Server technical community we stop using the term BISM and instead refer to Analysis Services Multidimensional, Analysis Services Tabular and PowerPivot so we’re always clear about what we’re talking about in blog posts, articles, books and so on. I don’t think using the term BISM is useful in any technical context, just as the term UDM wasn’t in the past, because it is a blanket term for several different things. I also think that so many people are confused about what the term BISM actually means that it is becoming counter-productive to use it: people will not buy into the MS BI stack if they’re not really sure what it consists of.

What does everyone think about this?