Extract Data from a Mixed Column

More and more I’m seeing examples where people are trying to extract data from a mixed column.  In other words, they have two data types in a single column, but need to find a way to extract one from the other.

Examining the issue

The sample data I’m using can be downloaded from this link.

I’m going to use Power BI Desktop for this, but the results will look identical in Excel using Power Query (except for the colour, of course.)

So let’s get started:

  • Get Data (new Query in Excel) –> From CSV –> MixedDataInColumn1.csv
  • Promote First Row as Headers

The issue can be seen in the red circles below… the report author injected the name of each vendor for the parts above their first part in the list.

image

So the issue here is how to extract the vendor name from Part No column.  The problem is that there isn’t any obvious way to do this.  We have different textual values in all columns, which could change over time.  There’s really nothing that we can test for reliably in this case.

How to Extract Data from a Mixed Column

There are actually a few different ways to extract data from a mixed column… a few of which we demonstrate in our Power Query workshop.  I’m going to show just one here.

Step 1 – Identify a column with a pattern you can exploit

The key we are really looking for is a column which has values or dates for all rows other that the one with our vendors.  In this case we actually have two: Part No and Cost.  Both have text on the Vendor lines, but what looks like values on the rest.  The challenge we have here is that we can’t always guarantee that Part No won’t have text in it.  It’s completely possible we could see a part number like TH-6715 or something. So this leaves us with the Cost column.

Step 2 – Duplicate the identified column

This next set of steps is actually the trick that lets us work this out.

  • Right click the column in question and choose Duplicate Column
  • Right click the Cost – Copy column –> Change Type –> Whole Number
  • Right click the Cost – Copy column –> Replace Errors –> null

You should now have null values where the textual values were located:

image

Step 3 – Use a little conditional logic

We now have something that we can use in order to extract the Vendor name.  So let’s build a little bit of conditional logic:

  • Add Column –> Conditional Column
  • Configure the Conditional Column as follows:

image

The only trick here is to make sure you change the Output to a column so that you can select from the list of columns.

  • Click OK
  • Right click the Vendor column –> Fill Down

The result is shown below:

image

Step 4 – Clean up

We’re now at the point of clean up which entails:

  • Filter the Cost – Copy column to remove null values
  • Delete the Cost – Copy column
  • Set the data types on all columns

The results now look as follows:

image

At this point we can commit the query and we are good to go.

Final Thoughts

This is not a new trick by any means; I’ve been using it for a long time.  The biggest key is really about identifying patterns and thinking outside the box.

It’s unfortunately very easy to get focused on the primary column we want to solve, and lose site of the others.  (Trust me, I’ve been there too.)  Sometimes though, when a column is particularly tough to deal with, we need to just step back and take a look at the bigger picture to see if there is a pattern in another column that we can exploit.  In fact, I’d say that this is probably one of the most important things to master when working with Power Query.

Visual Interactions in Power BI

In this post I’m going to explore the options for Visual Interactions in Power BI … in other words, I’m going to explore the options to control what happens to other visuals when you select one in Power BI.

Visual Interactions in Power BI – The default experience

Let’s take a quick look at a report and see what happens when we select a visual.  Here’s a simple report with 3 charts and a card:

image

And, when we click one of the visuals, it cross filters each of the others.  In the case below, I’ve clicked “Vancouver” in the “Course Attendees by City” visual, and it has cross filtered all the rest:

image

Okay, so no secret there.  The important things to remember here are that:

  1. I didn’t need to do anything to set up the linkage for the visual interactions in Power BI, and
  2. The cross filtering is applied to show the currently selected portion of the whole

But what if we didn’t want this?

The 3 Options for Visual Interactions in Power BI

There are actually three different states for visual interactions in Power BI:

  • Highlight (the default experience of cross-filtering with shading)
  • Filter (cross-filtering to show the contextual values only)
  • None (do not filter)

You can find each by selecting any visual on your report in Power BI Desktop, then go to Visual Tools –> Format –> Edit Interactions.

Let’s take a look at each of them.

Visual Interactions in Power BI – Highlight

As mentioned above, this is the default of the visual interactions in Power BI.  You don’t need to set up anything to get this behaviour.  If you monkey with it, however, you can get back to it by selecting a different visual, then clicking the little pie chart icon that appears above the visual you want to modify.

image

The only other thing I want to call out here is what happens when we select a set of data that filters all records out of another visual.  In the case below, I’ve selected Kelowna in the Course Attendees by City chart.  As you can see only four of our courses have been led in Kelowna:

image

Notice that the last three courses still show, even though we never ran them in this city.  Why?  Because the visuals indicate that we’ve led all our other courses somewhere, but obviously not in Kelowna.

Visual Interactions in Power BI – Filter

The next icon to the left of the pie chart is the Filter icon.  This toggles the visual slightly:

image

The key difference here is that the shaded portion is gone.  This gives the appearance of drilling in to the data a little more, without preserving the concept of how this data relates to the whole.

Now, check out what happens when we select Kelowna:

image

The Courses Run by Name visual no longer holds any data about the whole, allowing it to remove the irrelevant courses.  End result here is that we’re able to focus on the data that exists in this context only, without contaminating it with irrelevant data.

To be fair, most of the time I actually quite like the version with the shaded values.  But if you have a long list of data then this can certainly help trim it down so you don’t have to scroll the visuals as much.

Visual Interactions in Power BI – None

The last method we can configure for visual interactions in Power BI is to set the filter behaviour to None.  This prevents any filtering from taking place on a visual with this property set:

image

At first this looks quite similar to the Filter setting, but the key here is that the data in the Courses Run by Name visual has not been filtered at all, unlike the other two chart visuals and the card visual.  To display the effects just a bit further, the image below shows the card visual set to None, and the city filtered to Kelowna:

image

Notice that this time the Attendees card shows the true total for All Attendees.  The Courses Run by City visual, however, is filtering, as I left this in “Highlight” mode.

A key observation

I haven’t called this out yet, but should: we can set up different actions for each visual on the report.  That adds a fair amount of flexibility in order to get your report filtering working just the want you want.

A weird Visual Interaction

Before you look at the next visualization, I want you be keenly aware of this fact:

All visual interactions we set up as shown in the last image above.  I changed nothing else.

Keeping that in mind, look what happens when I click on the Creating Vibrant Dashboards course:

image

It cross filters all other visuals using the default interactions!

This is kind of a key thing to be aware of.  Just because I customized the visual interactions for the Course Attendees by City visual, it doesn’t force those relationships back the other way.  This means that I can customize how each visual affects the rest of the visuals on the page.  How cool is that?

The one missing setting

There is one setting that is badly missing, and that is the ability to persist selections when you select multiple visual filters.  What I mean by this is that I should be able to click on Vancouver, then click on the Creating Vibrant Dashboards selections to select only records that meet both of those criteria.  Alas there is no setting to do this today. You can set this up using drill down, but this means you need to think about what the user wants in advance and build it out, which is pretty tough if you have lots of potential filter combinations.

Using Aggregate to Count Visible Rows

In this post I’m going to show one of my favourite financial modeling tricks: how to use Aggregate to Count Visible Rows.

Background

Often, when I’m building models in Excel, I like to group key assumptions at the top of the worksheet in one area. This allows me to change them easily from a centralized location. The problem is that sometimes I need to collapse them to see more of the model.  Of course, you can use this trick to collapse any block of rows (or columns) in your worksheet, so it’s applicable to all kinds of uses.

Let’s take a look at the basic setup:

image

So it’s essentially a block of cells to capture key rates and stats.  No secret there.  And on the left I’ve added some outlining so that I can collapse it easily.  To do that we simply select rows 3:6 and go to Data –> Outline –> Group.

The Trick in Action

Now, check this out… I click the – on the left, and the rows collapse.

image

But check out the message in cell A7.  It wasn’t there before, but now we’ve got a nice message that not only tells you there is an area that is collapsed, it also leads the user as to how to show the rows again.

Using Aggregate to Count Visible Rows

The trick to this is using the AGGREGATE function (which works in Excel 2013 or higher).  So let’s check out how this works.

As AGGREGATE gives us back a count of rows, we will be able to test if the number of visible rows equals zero, and the react to it using an IF function.  So let’s get started.

AGGREGATE's first parameter: the Aggregation Type

=IF(AGGREGATE(

When we open the parenthesis, we are prompted for the first parameter.  There are a variety of options here, but the one I want is COUNTA(), which allows us to count the number of completed cells (either text or values):

image

Next up we put in the comma and we’re on to the second parameter.

AGGREGATE's second parameter: What to aggregate

image

Aha!  So using 5 will allow us to apply the COUNTA(), but ignore any hidden rows.  So it’s this parameter here that allows us to use AGGREGATE to count visible rows only.

AGGREGATE's third parameter: The data to aggregate

On to the next comma and now we need to select the range to count.  Now in this part we have two options.  Personally, I prefer to provide the range of the cells that will be hidden.  In truth though, you only really need to refer to a single cell in the range that will be collapsed.  Here’s what I went with:

=IF(AGGREGATE(3,5,A3:A6

Wrapping up the IF test

Perfect, and now we can just close the parenthesis and complete the test:

=IF(AGGREGATE(3,5,A3:A6)=0,

So, if the count of visible cells equals zero then… what do we want to do?

The IF test: If there are no visible rows...

This is the part that I think really makes this trick work.  I really like providing the arrow key to point to the + icon that shows up, and adding the additional wording as needed.  This allows my users to know not only that there is hidden data, but how to display it again.  So for me, that message might look like:

  • “<-- Show assumptions”
  • “<--Click to expand Revenue assumptions”

You get the idea.  For this example I’ve gone with the following:

=IF(AGGREGATE(3,5,A3:A6)=0,“<-- Show assumptions”,

The IF test: If there are visible rows...

And finally, we round it off with the messaging to provide if the count of visible rows is greater than zero (i.e. if the section is expanded).  Depending on what you want your model to do and how you want to display things for your end users, this could be something like:

  • “End of Assumptions”
  • “Total Revenue”
  • “Please insert new rows above this line”
  • “”

I think the first three are fairly self explanatory, but the last one is essentially two sets of double quotes.  Since everything between the quotes is returned to the cell as text, and there is nothing between the quotes, we get an blank cell.

The complete formula to use Aggregate to Count Visible Rows

Using that method, the finalized formula reads as follows:

=IF(AGGREGATE(3,5,A3:A6)=0,“<-- Show assumptions”,””)

Final Thoughts

My clients love this little trick. It’s fairly easy to set up, and is super useful for allowing people to hide/show the model sections that they want/need to review, without having them bogged down with all the info.

I also find it very useful when we’ve got multiple scenarios laid out on the worksheet. Say I need to look at… scenario 1 and 3 at the same time, I can compress 2 and just focus on the stuff I need to look at, avoiding scrolling up and down.

Can users see my raw data in Power BI?

One of the most important questions we have when publishing our data models and Power BI reports is Can users see my raw data?  The answer to this should virtually always be no unless you’ve explicitly stated you want this to be the case.  In this post we’re going to look at this problem and show you why this is a really serious question.

What kicked off this post?

At last week’s Power BI meetup in Vancouver, Peter Myers was demoing the feature to create a public facing web page (as I showed in this post.)  He quickly took his raw data, threw together a few visuals (without writing a single custom measure) and published the page to the web.  All in all it took less than 3 minutes for him to do this.

And naturally as he demoed to everyone that they could indeed get to the dashboard and play with it, the question of “Can users see my raw data” came up.  I confidently said no, and then it happened…. someone pointed out they could drill in to all the records that made up the data point.

It was a mic drop moment as that completely violated what we were expecting.  Part of the whole mantra of Power BI that we’ve been celebrating is that users only see what you want, and can’t get back to the raw data.

The “See Records” Feature in Power BI

Sure enough, this person was able to right click a chart and say “See Records”.  So what does that do?  It pulls up the full details of each transaction line that led to the data point:

image

Even though it doesn’t pull the records for the entire data set, it was still pretty horrifying.  With enough time, a reader could drill in and discover things about the underlying data that had never been intended to be revealed.

EDIT:  It's 2 days after the intial post date, and something has already changed here.  While See Records still shows the raw record set, it now only shows the columns that contribute to the chart, not the extra columns.  There could still be an issue, but it's not as bad as it used to be.

EDIT2: After a bit more diving in here, I discovered that it was me that changed something by hiding some of the model columns in Power BI desktop.  Hiding columns in the underlying table prevents them from appearing, but you still see all raw data points.

If you want to try this yourself, you can do so on the public version of the report I used for this post, which you can find right here.

The “See Data” Feature in Power BI

Seeing this, I immediately jumped over to the Hotel Stays report that I had published and checked my visuals.  Oddly, none of my visuals had the See Records feature.  They did however, have a feature called “See Data” which concerned me a bit at first.  But once you dig into it you’ll find it yields results like the following:

image

So basically it’s just a table view of the data that is already shown in my chart anyway.  It doesn’t expose anything new.  To be completely honest, it doesn’t add a ton of value as you don’t seem to be able to copy the table from the page or anything.  So the good news here is that no confidential data is being exposed.

Can “See Data” accidentally expose confidential data?

There is one case that I’ve found where the See Data function could actually expose confidential data, and that is when using map visuals.  On my Nights Away From Home report I used a map visual to plot the data points.  If you right click any of those data points and choose See Data, it gives you a list of all the data points.

SNAGHTML2ce4f25

Because I used a full street address to plot these on the map, I get a full list of detailed addresses. So if privacy is a big concern here I’d suggest you plot your data points using a postal code or city.  Something with less granularity. In my case I’m not totally concerned as these are all public hotels.. except for my friends locations where I had already adjusted the addresses to preserve their privacy by their request.

The Big Question: Can users see my raw data in Power BI?

So the simple answer would appear to be “Yes” they can.  But it’s actually a bit more complicated than this.  The Show Data feature is not – in my opinion – a data security risk.  The data is already shown in a visual, it’s just a different way of stating it.  So the real issue is around that Show Records option.

There are actually two big questions here…

  1. Can I disable the Show Records option?
  2. Why did my dashboard NOT have the Show Records option where Peter’s did?

As you can see, the second question actually answers the first.  It IS possible to disable the Show Records feature, but how?  As Peter and I were scratching our heads on this I decided that I needed to find out.

Controlling the ability to see my raw data in Power BI

As it turns out, the secret comes down to two things:

  • The type of visual you choose
  • The type of measure you create

I should also point out that the See Records and See Data features can be triggered from within Power BI Desktop.  This is fantastic as it means that you don’t need to publish to web to test your dashboard.  (Yay!)

The Data Set

To demo these features and issues, I’m going to use a subset of the data I had for my Nights Away From Home report:

image

To quickly explain this, I have three data columns in the table:  City, Country and StayCounter.  The first two are obviously text columns, and the last is a numeric column where each row contains the value of 1.

The last two items are DAX measures defined as follows:

  • CityCount = COUNTA(Stays[City])
  • ExplicitStayCounter = SUM(Stays[StayCounter])

Can users see my raw data in Power BI if I use a Card Visual?

To answer this, I created two card visuals.  The first was created using an Implicit measure.  I.e. I dragged the text based City field onto a card and let Power BI create an implicit “Count of City” measure:

image

The good news here is that there is no right click context menu for a Card.  (Give it a try!)

So using a Card visual means that your audience can never drill in to the underlying data.  I’m happy with that part.

Can users see my raw data in Power BI if I use a Bar Chart Visual?

Once again, I created two visuals here:

Field Red Visual (on left) Green Visual (on right)
Axis Country Country
Value City CityCount

In the case of the green visual, I have explicitly created my own measure by writing DAX.  In the case of the red visual, I have an implicit measure where Power BI has done the measure creation for me.  (Interestingly it reports the measure as Count of City which is not technically correct… it is actually a COUNTA of City, as Count can only count numeric values, where City is a text based field.)

Semantics aside, have a look at what you see for options:

SNAGHTML2ebf2ab

This was a bit shocking to me… so Power BI defaults to allowing you to drill into the underlying records… at least for implicitly generated measures that count text based entries.

Can users see my raw data in Power BI if I use an implicit SUM measure?

So the next question I had to explore is if this issue where users can see my raw data limited to only implicit measures based on text fields?  If the field was numeric and I use an implicitly created measure to count or sum it, will I have the same issue where the underlying records can be exposed?  Let’s take a look:

To avoid confusion here, I created to new visuals, but column charts this time.  Here’s how they were configured:

Field Red Visual (on left) Green Visual (on right)
Axis Country Country
Value StayCounter ExplicitStayCounter

The key difference here is that I just dragged the raw StayCounter column on to the red visual and let Power BI implicitly create the SUM measure for me.  In the case of the green visual, I wrote the DAX to say exactly what I wanted.  The two measures are mathematically equivalent which was important for an apples to apples comparison.  And here are the results:

SNAGHTML2f74c7e

(Yes, they look almost identical to the previous version, but if you check the titles – or play with the public version of the report - you’ll see that they indeed use different configurations.)

Observations and Thoughts

As you can see, the results are consistent with implicit versus explicit measures.  So if you want control of whether your audience can drill in to your underlying data records, you need to know this:

Measure Implicit – Power BI Explicit - DAX
Exposes “See Records” Yes No
Exposes “See Data” Yes Yes

This is a design issue

In my opinion, this is not good.  This basically means that the “quick and easy” way to create a report sets up to potentially expose data that should not be exposed.  And it’s not only applicable to reports that are shared publicly, it’s applicable to all reports distributed via Power BI Desktop, organizational level sharing and external sharing.  My feeling is that the Show Records feature should be disabled by default, and there should be a flag that you need to enable in order to enable this feature.

Having said this, the Power BI team has a problem.  I’d say that the vast majority of their audience do NOT want the See Records feature enabled by default, but some people rely on it.  This is one of the pain points of having something out there in the wilds, then realizing that something isn’t working the way you want.  To fix this, they’d need to released the a method to configured the visibility of the Show Records feature (probably to Power BI Desktop) then switch the default on Power BI.  Someone will no doubt complain of a loss of the feature, but as long as there is a way to fix it I’d say the risk of upsetting a customer in this manner outweighs the risk to Microsoft's entire customer base of accidentally exposing company confidential information.

My personal feeling is that I’d like to see both the See Records and See Data features configurable by the report author.  I’d like to see both off by default, with the ability to turn them on as needed.

In the mean time…

While current implementation is ultimately not what I think should happen, I’m actually happy it is consistent.  Why?

As it turns out, the reason I never saw this is that I’m in the practice of taking explicit control of my data and measures.  Since my earliest days of working with Power Pivot in Excel, I have never relied on the implicit measures, electing to always write my own using DAX.  This is true even for simple measures like SUM(Stays[StayCounter]).

Part of my reason for this is just habit, part is because I want to learn, part is because I don’t trust some of the implicit stuff that happens and don’t trust defaults.  In VBA we have a specific command to force variable declarations which is called “Option Explicit”.  I’ve always adopted that as not just a code word, but rather as a development standard.  In this case I’m fortunate that it saved my data.

If you don’t know DAX, you should learn it.  Not for this reason alone, but it is certainly another reason.

MailChimp and Power BI

This week I was playing around with MailChimp and Power BI.  If you’re not familiar with MailChimp, it’s the service that we use to send out our Monthly-ish newsletter.  I thought I’d show how incredibly easy it is to get a dashboard from this service.

Software as a Service

MailChimp is what we refer to as a “Software as a Service” (SAS) setup.  Basically what this means is that it is a subscription model where I have no need to set up my own server or host anything myself.  I simply subscribed, set up the forms on my website for you to fill out, and then use their services to build and send out the newsletter.  It’s pretty slick and they don’t even charge me until I hit 2,000 subscribers or want to set up auto replies and such.  I’m a huge fan of this model, as it lets you do a “try before you buy” test, only upgrading when you want to.

What do you need?

There are only two requirements here:

  1. A MailChimp account of your own
  2. A PowerBI.com account which you can create for free (seriously, no Pro Subscription required)

Getting Started with MailChimp and Power BI

Step 1: Connect to MailChimp

So the process begins with signing in to your Power BI.com account.  Once there, you look for this button in the bottom left corner of your screen:

image

Click that, and you’ll be taken to another screen that looks like this:

image

Click Get on the Services tab.

Now, to be fair this is still pretty new, so there’s a lot of sources that you would “hope” would be here (like Survey Monkey) which currently aren’t.  Despite this, the easiest way to find what you’re looking for is to start typing the name of the SAS source in the search pane in the top right.  It auto-filters the app list live.  Shown below is the results of “Ma” today:

image

So click the Get button at the bottom of the MailChimp.

You’ll now be asked to sign in.  If you have multiple domains under your MailChimp profile, just select one for now.  What you’ll see is that it takes you back to the portal and lets you know that it’s loading your data.

Step 2: Wait

Seriously?  Yes.  When you’ve chosen the account to log in, you’ll see that you get a new entry in the Dataset, Report and Dashboard sections of the Power BI portal:

image

See how the source is greyed out and has a star?  The star means it’s a new item, but the greyed out status means it’s loading.  It takes a while, and you won’t be able to do anything with it until it’s finished the initial load.  So go get a coffee and wait for the chimp to finish it’s trek to your portal.

Step 3: Explore the Dashboard

Wait, what?  I don’t need to build the dashboard myself?

No!  It’s already built for you.  MailChimp and Power BI together in a few easy clicks.  Check it out!

image

All fully clickable, and Q&A works like a charm:

image

What about Scheduled Refresh?

Right, I hear you.  That data set took a long time to load, so you don’t really want to have to come in and refresh it every day.  (I mean, unless you need an excuse to go for coffee…)

Well check this out.  This is shot from an account which only has a Power BI Free subscription.  I access this by clicking the … next to the data set then choosing Schedule Refresh (normally a Power BI Pro feature only):

image

Oh… and did I mention that this was created on Oct 27?  This report has already refreshed, and is scheduled to do it again with no configuration or pro license needed.  Ah the beauty of connecting to a pre-built SAS dashboard!

What about my other MailChimp Accounts?

If you have more than one domain in your MailChimp account, you might be surprised to see that when you try to add another MailChimp dashboard it just creates the same one again without prompting you to log in.  This is because it uses the cached credentials.  So how do you make this work?

Basically, you create the same report again then, once it’s finished loading, you go and edit the data source credentials on the Dataset for the second report.

To do this:

  • Go to Datasets –> Ellipses –> Schedule Refresh
  • Expand Data source credentials
  • Click Edit credentials
  • When prompted for oAuth2 credentials, click Sign In
  • Insert the username & password
  • Then choose the new data set when prompted

image

I’d also recommend going to each of the datasets, reports and dashboards and renaming them using the ellipses as well, just to keep it clear which is which.

Final Thoughts

The one thing I can’t do here is download the pbix file that was used to create this dashboard.  I’m sure it’s heavily parameterized (how else would they deploy for whomever adds it), but I’d sure love to do that.  Why?  So I could connect directly to MailChimp easily from PowerBI Desktop or Excel… maybe so that I could merge other data in with it.  (Once it’s published, I cannot get in and examine or change the M code in any way.)

Having said that, this is still pretty darned cool stuff!  I really hope other vendors jump on this train as well.  Building dashboards can be hard, and this just makes it super easy.  I’d love to see one for my Facebook account, LinkedIn and other SAS sources.

For reference, Google DOES provide a dashboard too, and it’s just as easy to set up and auto refreshes like this too.

Power BI Slicers

For those coming from the Excel 2013 world, you’ll surely want to create filters using the Power BI Slicers.  After all, you know that Slicers and Timelines are two of the sexiest filters we have for controlling filter context in Power Pivot models.  In this post we’re going to explore the slicer visual, and how to get closer to what you’re used to in Excel.

The Goal

I’m going to fall back to my last project here, and have created a single visual on a blank report page.  You probably remember this one, it’s the map of where I’ve stayed so far this year:

image

What I’d like to do now is add my slicers and timelines.  I’d like a slicers for Country and Province.  Should be easy, no?  Err… no.  Sad smile

The default Power BI Slicers

Now, Power BI Desktop has a Slicer visualization, as you can see here:

image

So what’s wrong with it?  I’ll build two to show you why I’m less than satisfied…  Here’s what I did:

  • Created a Slicer visual
  • Added the Country field
  • Created another Slicer visual
  • Added the Province field

Do these look like Excel slicers to you?  They look a lot more like check boxes to me, not slicers…

image

Now don’t get me wrong, Power BI Slicers certainly work, as you can see here where I’ve drilled in to USA:

image

So let’s look at the difference between the Power BI Slicers and the Excel slicers that my expectations are based upon:

  • The Province field hides all irrelevant items by default, unlike Excel.  I could not find a configuration option to change this.
  • The checkbox thing drives me crazy.  I hate checkboxes in my Windows files list, I don’t like them here, and again it is inconsistent with Excel.  I could not find a way to turn those off.
  • Originally I wanted to show the provinces in a two column slicer, like I can in Excel.  I could not figure out how to make this happen either.
  • Finally, I wanted to show the bubbles like an Excel slicer.  The closest I could get was the image below (WARNING! SUPER UGLY GRAPHIC AHEAD!)  Should you feel the need to create this abomination you need to select the slicer, go to the Formatting roller –> Items –> Outline –> Frame.

image

Gross.  That is just gross.  Honestly, I really don’t understand why the slicer is so different from Excel’s.  That slicer is pretty, and people are used to it.

Not happy with these, I deleted both slicers.

Is all hope lost for attractive Power BI Slicers?

Thankfully, the answer is no.  The Power BI team has given developers the ability to create and distribute their own visuals into the Power BI custom visuals gallery.  So let’s go and pull in a couple of those to fill this gap.

Locating the Custom Visuals Gallery

To be fair, the steps for this could be MUCH easier.  To get here the first time you can either just click this link or follow these steps:

  • Click the ellipses in the Visualizations gallery to import a custom visualization

image

  • Choose to Import a Custom Visual
  • Click the Learn More link

image

  • To be fair, you should probably read the page you’re taken to, as it talks about all the risks of using a custom visual.  (Remember not all custom visuals are provided by Microsoft, many are provided by 3rd parties.)
  • I scrolled straight to the bottom and clicked the link in the See Also section to go to the Power BI custom visuals gallery

You’ll be taken to the gallery, which has a lot of pretty visuals that can be imported into your project.

To make it easier to find custom visuals, I’d recommend you do a couple of things here:

  1. Bookmark this page (making it a bit easier to get back to it.)
  2. Choose to sort the gallery by Name rather than by Most Recent (which is the default)

When you click on a visual it will offer to download a pbiviz file that you can store in a folder.  You’ll want to remember the location, as you’ll need to import the visuals into every new PBI file you create.

I downloaded a specific visual here: the Chiclet Slicer which, ironically, is published by Microsoft.

Importing the Chiclet Slicer

When I returned to Power BI desktop, it’s still sitting at the Import Custom Visual dialog, which is convenient.  So I was able to just click the big yellow Import button, and select the ChicletSlicer file.  Doing so adds a new option to the Visualizations gallery:

image

I created two new Chiclet slicers, one for Country and one for Province, and was pleased to end up with the following:

image

Now that’s more like it!  Certainly needs some tweaking, but better than the past iteration.  So let’s get to that tweaking…  I changed the formatting options for each of the slicers as follows:

  • The Country slicer
    • General –> Columns –> 1
    • Header –> Off
    • Chiclets
      • Unselected Color = very light grey
    • Title –> On
      • Text = Country, Font Color = Black, Alignment = Center, Text Size = 10
  • The Province slicer
    • General –> Columns –> 2
    • Header –> Off
    • Chiclets
      • Unselected Color = very light grey
    • Title –> On
      • Text = Country, Font Color = Black, Alignment = Center, Text Size = 10

And, as you can see, the results are pretty good:

image

A couple of things that I couldn’t figure out here though:

  • I wanted to align the text in my “chiclets” to the left, like in Excel.  Can’t seem to find an option for that.
  • There is a tantalizing option in the “General” section to show disabled items “Inplace”, and an option in the chiclets to set the colour for those items.  I would have expected it to be equivalent to Excel’s “Show Disabled”, but it doesn’t seem to do that.  I have not figured out how to replicate that effect.

Final Thoughts

To be fair, there are a ton of configuration options for the Chiclet slicer, much more than I’m going to cover.  Why this slicer isn’t part of Power BI’s default install is beyond me… especially since it’s published by Microsoft.

Values Become Text After UnPivoting Other Columns

Have you ever set up a nice query to UnPivot other columns, only to find that the query data types change when you add new columns?  This post will cover why values become text after unpivoting other columns.

Background

We’ve got a nice little table called “Data” showing here.  Nothing special, it just summarizes sales by region by month, and our goal is to unpivot this so that we can use it in future Pivot Tables.  (You can download the source file here.)

SNAGHTML552a2a7

Now, you will notice that April’s sales are outside the table. This is by design, and we’ll pull it in to the table later when we want to break things.  Smile

UnPivoting Other Columns – The Hopeful Start

If you’ve been following my blog for any period of time, you’ve seen this, but let’s quickly go over how to unpivot this:

  • Select a cell in the table
  • Go to Power Query (or Data in Excel 2016) –> From Table

We’re now looking at the Power Query preview of the table:

image

Great, now to unpivot…

  • Hold down the Shift key and select the Country and Prov/State column
  • Right click the header of either of the selected columns and choose Unpivot Other Columns
  • Right click the headers of the two new columns and rename them as follows:
    • Attribute –> Month
    • Value –> Sales

Re-Pivoting from the Data Model

With the table complete, I’m going to load this to the data model and create a Pivot Table:

  • Go to Home –> Close & Load –> Close & Load To…
  • Choose to Load to the Data Model

The steps to create the Pivot depend on your version of Excel:

  • Excel 2013: Go in to Power Pivot –> Home –> PivotTable and choose a location to create it
  • Excel 2016: Click any blank cell and go to Insert –> PivotTable.  As you have no data source selected, it will default to using the data model as your source:

Iimage

With the PivotTable created, I’ve configured it as follows:

  • Rows:  Country, Prov/State
  • Columns:  Month
  • Values:  Sales

And that gives me a nice Pivot like this:

image

Let’s Break This…

Okay, so all is good so far, what’s the issue?  Now we’re going to break things.  To do that, we’re going to go back to our original data table and expand the range:

image

In the picture above, I’ve left clicked and dragged the tiny little widget in the bottom right corner of the table to the right.  The table frame is expanding, and when I let go the Apr column turns blue, indicating that it is now in the boundaries of the table.

With that done, I’m going to right click and refresh my Pivot Table, leaving me with this:

image

Huh?  Why was the sales measure removed?  And if I drag it back to the table, I get a COUNT, not a SUM of the values?  And even worse, when I try and flip it back to SUM, I’m told that you can’t?  What the heck is going on here?

image

Importance of Power Query Step Order

To cut to the chase, the issue here is that when we first created the table in the data model, the Sales column was passed as values.  But when we updated the data to include the new column, then Sales column was then passed entirely as text, not values.  Naturally, Power Pivot freaks out when you ask for the SUM of textual columns.

The big question though, is why.  So let’s look back at our query.

Our original data set

If we edit our query, we see that the steps look like this:

image

To review this quickly, here’s what happened originally

  • Source is the connection that streams in the source data with the following columns:

image

  • Changed Type set the data type for all the columns.  In this case the Country and Prov/State fields were set to text, and the Jan, Feb & Mar columns were set to whole number.  We can see this by looking at the icons in the header:

image

Note that if you don’t have these icons, you should download a newer version of Power Query, as this feature is available to you and is SUPER handy

 

  • We then selected the Country and Prov/State columns and chose to Unpivot Other Columns.  Doing so returned a table with the following headers

image

Notice that the first three columns are all textual, but Sales is showing a numeric format?  Interestingly, it’s showing a decimal format now, but it shows the numeric format because all unpivoted columns had explicitly defined numeric formats already.

The final steps we did was to rename our columns and load to the data model, but the data types have been defined, so they were sent to the data model with Sales being a numeric type.

Why Values Become Text After UnPivoting Other Columns

Okay, so now that we know what happened, let’s look at what we get when we step through the updated data set.

  • First we pulled in all the columns.  We can plainly see that we have the new Apr column:

image

  • The Changed Type step is then applied:

image

Hmm… do you see that last data type?  Something is off here…

So when we originally created this query, Power Query helpfully pulled in the data and applied data types to all the existing columns.  The problem here is two-fold:  First, the Apr column didn’t exist at the time.  The second problem is that Power Query’s M language uses hard coded names when it sets the data types.  The end effect is that upon refresh, only the original columns have data types defined, leaving the new columns with a data type of “any” (or undefined if you prefer).

  • We then unpivoted the data, but now we see a difference in the output

image

Check out that Value column.  Previously this was a decimal number, now it’s an “any” data type.  Why?  Because there were multiple data types across the columns to be unpvioted, so Power Query doesn’t know which was the correct one.  If one was legitimately text and Power Query forced a numeric format on it you’d get errors, so they err on the side of caution here.  The problem is that this has a serious effect on the end load to Power Pivot…

  • Finally, we renamed the last two columns… which works nicely, but it doesn’t change the data type:

image

Okay, so who cares, right?  There is still a number in the “any” format, so what gives?

What you get here depends on where you load your data.  If you load it to the Excel worksheet, these will all be interpreted as values.  But Power Pivot is a totally different case.  Power Pivot defaults any column defined as “any” to a Text data type, resulting in the problems we’ve already seen.

Fixing the Issue

For as long as we’ve been teaching our Power Query Workshop, we’ve advocated defining data types as the last step you should do in your query, and this is exactly the reason why.  In fact, you don’t even need to define your data types in the mid point of this one, that’s just Power Query trying to be helpful.  To fix this query, here’s what I would recommend doing:

  • Delete the existing Changed Type step
  • Select the final step in the query (Renamed Columns)
  • Set the data type for each column to Text except the Sales column, which should be Decimal Number (or currency if you prefer)

image

When this is re-loaded to the Data Model, you’ll again be able to get the values showing on the Pivot Table as Sum of Sales.

Avoiding the Issue

Now, if you don’t want Power Query automatically choosing data types for you, there is a setting to toggle this.  The only problem is that it is controlled at a Workbook level, not at a global Excel level.  So if you don’t mind setting it for every new workbook, you can do so under the Power Query settings:

image

Is Changed Type Designed in the Correct Way?

It’s a tough call to figure out the best way to handle this.  Should the data types be automatically hard coded each time you add a new column?  If the UnPivot command had injected a Changed Type step automatically, we wouldn’t have seen this issue happen.  On the other hand, if a textual value did creep in there, we’d get an error, which would show up as a blank value when loaded to Power Pivot.  Maybe that’s fine in this case, but I can certainly see where that might not be desirable.

Personally, I’d prefer to get a prompt when leaving a query if my final step wasn’t defining data types.  Something along the lines of “We noticed your final step doesn’t declare data types.  Would you like me to do this for you now (recommended)” or something similar.  I do see this as an alternate to the up-front data type declaration, but to be honest, I think it would be a more logical place.

October News and Events

It’s a busy month here at Excelguru. Instead of a technical post we wanted to catch everyone up on our October news and events!

Live Course: Master Your Excel Data October News and Events

Ken is teaching a LIVE, hands on course in Victoria, BC on Friday, October 21 from 9:00am-4:30pm. This session is great for anyone who has to import and clean up data in Excel and will change the way you work with data forever! Ken will teach you how to use Excel Tables, Pivot Tables and Power Query. Space is limited to only 20 attendees, so don't miss out on your chance to sign up. For full details and to register for the session, visit: http://www.excelguru.ca/content.php?291-Live-Course-Master-Your-Excel-Data.

October News and Events: Power BI Meet-up

The next Vancouver Power BI User Group meet-up is happening on Thursday, October 13 from 5:30-7:00pm. Scott Stauffer, Microsoft Data Platform MVP, will be presenting on How to Operationalize Power BI. Together we’ll look at some solutions that might help pass your Power BI solution over to IT to manage enterprise-wide. Dinner and soft drinks will be provided. View the full details and sign up to attend at: http://www.meetup.com/Vancouver-Power-BI-User-Group/events/234126999/.

Microsoft MVP Award Received

For the 11th straight year, Ken has received the 2016 Most Valuable Professional Award from Microsoft! The previous 10 years, Ken’s award has been in the Excel category, but this year’s award is in the Data Platform category. The new category reflects the work he’s been doing this past year with Power Query and Power BI. Congratulations Ken, your guru status remains assured.mvp_horizontal_fullcolor

Our Team Has Grown

As we mentioned the other day, Rebekah Sax has recently joined the Excelguru team. She brings with her a wealth of experience in marketing, communications, event planning and administration. Please join us in welcoming Rebekah as she helps us make new connections and continue to grow.

New team member

I’m pretty stoked to announce a big milestone for Excelguru.  That’s right, we’ve added a new team member to our company!

We’re pleased to announce that Rebekah Sax has joined our team and will be helping us with our marketing efforts.  She spent the last 15 years working at Fairwinds in a variety of roles from marketing to event planning (and more), and her broad skill set is just what we needed in order to fill some pretty big gaps in our practices. In fact, you can already see the effect.  If you remember the Excel Courses Calendar I set up on my website ages ago… it’s actually got courses listed now!

Sharing Power BI Dashboards

This is the fifth and final installment of this post series, and will demonstrate the process for publishing and sharing Power BI Dashboards (as compared to Reports.)  Key things we’ll explore here are the additional benefits of Power BI Dashboards and how to share them without embedding them in a publicly accessible environment.

Series Table of Contents

This is a portion of a series of blog posts, which build on each other to create an overall solution from Data to a Power BI dashboard.  You’ll want to follow along in order, so here’s the table of contents for the series:

Public vs Private?

So as cool as the last post was, the begin question that comes up is how to share your report with specific people, not the entire world.  The data could be confidential at some level, so this is a pretty big requirement to be able to solve.

The biggest piece of advice I would give is this area is DO NOT CREATE THE EMBED CODE as I showed in my last post.  If you do, it’s accessible if anyone discovers the URL.  We need to share in a totally different way.

Power BI Reports vs Power BI Dashboards

The secret here is that you can’t share Reports with internal users, you share Dashboards instead.  But actually that’s okay, as it adds a couple of nice things to the report when you turn it into a proper dashboard.  One is Q&A, and the other is that it stops showing the selection handles around the visuals every time you select them.  Smile

So how do you do it?

Creating a Power BI Dashboard

The answer is that you go back to your report and – up by that Refresh button that you’re never going to use – you click the “Pin Live Page” button.  That takes you to this interface:

image_thumb[21]

Notice that I have said I want to create a brand new Dashboard, and I’ve even given it a totally different name than the original report.  Once I click the Pin Live button, it creates a new dashboard that shows up in the Dashboard area.  And when you go there, you see this:

image

Notice that I’ve drilled in to Canada, and it still cross filters, but this time it doesn’t surround the chart with the little frame.  Hmm… wait… that’s actually a problem, isn’t it?  Back in the post where I showed how to create the column chart, I set it up to drill in, and now you can’t.  The good news is that you can click in the upper left where is says “Hotel Stays Page 1” and it will take you directly to the report where you CAN drill in.  (So these two things work together.)

So that is a note to keep back of mind.  Dashboards are filterable at the top level, but if you want to drill in you need to go back to the underlying Report.

Q&A… aka as “HOLY CRAP THAT’S AWESOME!!!”

I’m not generally a big fan of noise and flashing lights in anything I encounter, but if there were one feature that should have them, this is it.  Just above the visuals in the dashboard is a little area that says “Ask a question about your data”.  It’s subtle, and you could easily use Power BI for ages without ever playing with this:

image

So what’s it good for?  Natural language queries is what.  Check out what happens when I just type in “shows stays last month”:

image

Woah… pretty cool.  I never set up any calendar to do that, and it just lists the transaction listing for the previous month.  Not impressed?  How about this one:

image

I want to be really clear here… my data set does not have a field for Quarter.  In fact, it doesn’t actually even have a field for Month.

You’ll also notice that I never once in this series of posts blogged about how to set up the data set to use this feature.  That’s because I didn’t need to.  It just works out of the box once you pin your report to a dashboard.

This is one of the coolest things in this product.  Imagine pushing a data set like this to your boss and they can just query the data set to find out the questions they probably would have asked you.  Nice.  Smile  (Now, to be fair, really complicated questions are probably still going to need intervention, but still… pretty amazing.)

One more…

image

Oh, and I should also mention… once you find a really cool visual like this?  There is a pin icon in the upper right that will let you just pin it to an existing or new dashboard.

As awesome as this is though… it’s actually not the topic of this blog post, it’s just a side benefit of what we need to do in order to set up our sharing.  So with a bit of a tear in my eye, I’m going to click the Exit Q&A button and get back to it.  Winking smile

Sharing Power BI Dashboards Within the Company

When I refer to those “within the company”, I’m talking about users who are part of the same Office365 tenant.  The best indication of this is that they have the same email domain as you do.  But what I’m about to describe is equally true if the email domain is different, but the domain is registered under the same Office365 tenant.

So to share a dashboard with someone in your company, you select the dashboard, then go to the top right corner of your screen and click the Share icon:

image

That will pop up a window where you can type in the email, and even an optional message.  Since my email is all hosted in Office365, it is able to convert the email to a username for Deanna as shown below:

image

Notice also that I’m warned about her data access, and I can also disable her ability to share it with others.  This is nice, as it allows us some kind of control over what our users can actually see and do with our data.

Once I click Share, Deanna gets and email notification in her inbox and, when she logs in to Power BI, she now has my dashboard listed in her Dashboards section:

SNAGHTML2307c6f

Interestingly enough, I have been able to verify that with the Share option unchecked:

  • She cannot share the dashboard with anyone else
  • Changes made to the original dashboard update in the shared version
  • The dashboard is listed in her Dashboards section for easy access
  • All reports are drillable, and she can drill into the Report if she clicks the “Hotel Stays - Page 1” part near the top left of the dashboard.  In fact, if she wants to drill in to the country chart, she MUST do this.

Now, what about security of the raw data itself?  Well, as you can see in the image above, nothing is listed in the DataSets section for her.  This means a couple of things:

  • She cannot access the data model to see the linkage
  • There is no way to download the complete raw data set

Having said that, Q&A is enabled and – if you ask the right questions – you’ll be able to pull up the underlying data.  The right question is essentially “show <table name” and it will show you all the records for the given table.  Obviously, the bigger the data set and the more tables it has, the harder it is to make sense of it, but you CAN query it all at a raw record level, so you do want to be a bit selective about what data you include in the data set.  Now, to be fair, you can enable row level security to restrict what the user has access to, but that’s a topic for another day.

Sharing Power BI Dashboards With an External Tenant

Now, even though I covered external sharing in my last post, it was targeted at public sharing.  What about when you want to share your dashboard with specific people outside your organization.  We can totally do this, although I find the experience a bit… wanting.

Now we can certainly share a dashboard with someone external to our company.  We do it in the exact same manner as we would with someone inside the company.  Namely we do the following:

  • Select the Dashboard
  • Click the Share icon
  • Enter the email address
  • Select the sharing options as desired

image

We are helpfully warned that this user is outside our tenant, which is great.

Now, what happens next is the user is sent an email which looks like this:

image

And when we click the link, we must sign in to get access to the dashboard.  All good so far.  The dashboard cross filters and, like when shared inside the company, you can click the Hotel Stays Page 1 portion to get to the report for further drilling.  So far nothing seems off.

But now check this out… if I look at the top left corner of the screen, I see this:

SNAGHTML23bd16f

Okay, so it’s not about what I see, but rather what I don’t.  Notice the Power BI logo?  It’s directly about the title.  Where is the little icon to get to the rest of your Power BI reports?  It’s not there at all.  “No problem”, you’re thinking, “ just click the logo”.  Sure thing…

And you’re taken to a page that has this navigation structure.  Notice anything missing?  Um… like maybe the dashboard that was shared with me?  (To be clear, any reports owned by this user from within their own company would show up here.)

SNAGHTML23299d1

This is a known thing, as documented in this article by Microsoft.  And to save you the trouble of reading this, here’s the pertinent part:

When you share with people outside your organization, they get an email with a link to the shared dashboard. They have to sign in to Power BI to see the dashboard. If they don't have a Power BI account, they can create one after clicking the link.

After they sign in, they see the shared dashboard in its own browser window without the left navigation pane, not in their usual Power BI portal. They have to bookmark the link to access this dashboard in the future.

To be fair, it is an improvement over the original state where you couldn’t share with an external tenant at all.  Having said that, it’s still unfinished and hopefully it wont’ be long until Microsoft gets back to dealing with this and plumbs externally shared dashboards into the Power BI portal properly.

Final Thoughts

Listen, I complain about the things that don’t make sense to me, but overall, let’s circle back on how incredible this is overall.  Over the last five blog posts I’ve built a system which:

  • Has a spreadsheet back end to capture my hotel stays
  • Can be updated from the Excel app on my iPhone (or whatever phone you have)
  • Automatically updates
  • Is published to a public facing dashboard on the web
  • Is shared with Q&A to users both inside and outside my company

That is freaking stellar.  Period.