Some will tell you to do the Top N filter on the dataset. However this presents problems:

• You might not have access to the SQL server and so be limited in the queries you can use.
• Your query might not be aggregated and it you try and apply an aggregate filter to the dataset (eg SUM(sales) Top N= 10), Report Builder will error telling you aggregate filters are not allowed.
• Even if you can filter the dataset, you might not want to because you need the non-filtered data in another visual or table.

To filter Top N on a chart, click the chart, then in the little Chart Data box select the category (the thing you are grouping by) and click Category Group Properties:

It is here that you can add the Top N filter on the chart:

This is also the place to do sorting on the chart.

Deal line items are when you create a Deal and add Products which may be itemised in a Quote.

The goal is to get a table that can be used in Power BI that lists all the deal line items along with their respective companies and deal owners.

The Hubspot API is fairly easy to follow. This is the page dealing with Line Items: https://developers.hubspot.com/docs/guides/api/crm/commerce/line-items

In order to call the API you need to obtain a bearer token from Hubspot.

The overall plan

In Power Query we will get the line items from Hubspot and transform the response into a query table.

Then we do the same with Companies, Deals, Pipelines (the organisational units that contain Deals) and join them altogether in Power Query using the NestedJoin function.

…one slight problem

Hubspot will return 1 page for each API call which only gets us 100 rows. One of the query parameters is PageValue. So we have to do this:

• Make a PageValue parameter in Power Query to store the current page value.
• Make a function in Power Query “GetPageFunction-LineItems” that returns one page of line items taking the current page value as an arguent.
• Make a function “GetAllPagesFunction-LineItems” that iterates through the GetPage function and accumulates the data.
• Make a AllLineItemsWithDeals table that populates itself using the GetAllPages function.

AllLineItemsWithDeals

Working backwards, The AllLineItemsWithDeals table is defined like this:

let
    InitialPageValue = "1",
    InitialData = #table(type table [results.id = Int64.Type, results.properties.amount = Int64.Type, results.properties.createdate = DateTime.Type, results.properties.hs_lastmodifieddate = DateTime.Type, results.properties.hs_object_id = Int64.Type, results.properties.hs_product_id = Int64.Type, results.properties.quantity = Int64.Type, results.createdAt = DateTime.Type, results.updatedAt = DateTime.Type, results.archived = Logical.Type, paging.next.after = Int64.Type, paging.next.link = Text.Type], {}),
    AllData = #"GetAllPagesFunction-LineItems"(InitialPageValue, InitialData),
    #"Filtered Rows" = Table.SelectRows(AllData, each true),
    #"Sorted Rows" = Table.Sort(#"Filtered Rows",{{"results.id", Order.Ascending}}),
    #"Filtered Rows1" = Table.SelectRows(#"Sorted Rows", each ([getDealIDCustom] <> null)),
    #"Changed Type" = Table.TransformColumnTypes(#"Filtered Rows1",{{"getDealIDCustom", Int64.Type}})
in
    #"Changed Type"

The above creates the table the then uses Table.SelectRows to populate it using the function GetAllPagesFunction-LineItems defined as:

let
    GetAllPages = (PageValue as text, AccumulatedData as table) =>
    let
        CurrentPageData = #"GetPageFunction-LineItems"(PageValue),
        NextPageValue = if Table.IsEmpty(CurrentPageData) then null else CurrentPageData{0}[paging.next.after],
        NewAccumulatedData = Table.Combine({AccumulatedData, CurrentPageData}),
        Result = if NextPageValue = null then NewAccumulatedData else @GetAllPages(NextPageValue, NewAccumulatedData)
    in
        Result
in
    GetAllPages

This function tests if anything is in the current result of “GetPageFunction_LineItems(PageValue)”. If there is it append it to a temporary table called AccumulatedData, and runs the function again with the next PageValue (“paging.next.after” in the code). GetPageFunction-LineItems is defined below:

let
    GetPage = (PageValue as text) =>
    let
        Source = Json.Document(Web.Contents("https://api.hubapi.com/crm/v3/objects/line_items?properties=quartz_course_code&properties=name&properties=hs_recurring_billing_start_date&properties=quantity&properties=amount&limit=100", [Query=[after=PageValue], Headers=[Authorization="Bearer YOUR_HUBSPOT_TOKEN"]] )),
        #"Converted to Table" = Table.FromRecords({Source}),
        #"Expanded results" = Table.ExpandListColumn(#"Converted to Table", "results"),
        #"Expanded results1" = Table.ExpandRecordColumn(#"Expanded results", "results", {"id", "properties", "createdAt", "updatedAt", "archived"}, {"results.id", "results.properties", "results.createdAt", "results.updatedAt", "results.archived"}),
        #"Expanded results.properties" = Table.ExpandRecordColumn(#"Expanded results1", "results.properties", {"amount", "createdate", "hs_lastmodifieddate", "hs_object_id", "hs_product_id", "quantity","name","quartz_course_code", "hs_recurring_billing_start_date"}, {"results.properties.amount", "results.properties.createdate", "results.properties.hs_lastmodifieddate", "results.properties.hs_object_id", "results.properties.hs_product_id", "results.properties.quantity", "results.properties.name", "results.properties.quartz_course_code", "results.properties.hs_recurring_billing_start_date"}),
        #"Expanded paging" = if Table.HasColumns(#"Expanded results.properties", "paging") then Table.ExpandRecordColumn(#"Expanded results.properties", "paging", {"next"}, {"paging.next"}) else Table.AddColumn(#"Expanded results.properties","paging.next.after",each null),
        #"Expanded paging.next" = if Table.HasColumns(#"Expanded paging", "paging.next") then Table.ExpandRecordColumn(#"Expanded paging", "paging.next", {"after", "link"}, {"paging.next.after", "paging.next.link"}) else #"Expanded paging",
        #"Changed Type" = Table.TransformColumnTypes(#"Expanded paging.next", {{"results.id", Int64.Type}, {"results.properties.amount", Int64.Type}, {"results.properties.createdate", type datetime}, {"results.properties.hs_lastmodifieddate", type datetime}, {"results.properties.hs_object_id", Int64.Type}, {"results.properties.hs_product_id", Int64.Type}, {"results.properties.quantity", Int64.Type}, {"results.createdAt", type datetime}, {"results.updatedAt", type datetime}, {"results.archived", type logical}}), //Table.TransformColumnTypes(#"Expanded paging.next", {{"results.id", Int64.Type}, {"results.properties.amount", Int64.Type}, {"results.properties.createdate", type datetime}, {"results.properties.hs_lastmodifieddate", type datetime}, {"results.properties.hs_object_id", Int64.Type}, {"results.properties.hs_product_id", Int64.Type}, {"results.properties.quantity", Int64.Type}, {"results.createdAt", type datetime}, {"results.updatedAt", type datetime}, {"results.archived", type logical}, {"paging.next.after", Int64.Type}, {"paging.next.link", type text}}),
        #"Added Custom2" = Table.AddColumn(#"Changed Type", "getDealIDCustom", each 
            let
                InnerSource = Json.Document(Web.Contents("https://api.hubapi.com/crm/v4/objects/line_items/", [RelativePath=Number.ToText([results.id]) & "/associations/deals", Headers=[Authorization="YOUR_HUBSPOT_TOKEN"]])),
                #"Inner Converted to Table" = Table.FromRecords({InnerSource}),
                #"Inner Expanded results" = Table.ExpandListColumn(#"Inner Converted to Table", "results"),
                #"Inner Changed Type" = Table.TransformColumnTypes(#"Inner Expanded results", {{"results", type any}}),
                #"Inner Expanded results1" = Table.ExpandRecordColumn(#"Inner Changed Type", "results", {"toObjectId", "associationTypes"}, {"results.toObjectId", "results.associationTypes"}),
                #"Inner Expanded results.associationTypes" = Table.ExpandListColumn(#"Inner Expanded results1", "results.associationTypes"),
                #"Inner Expanded results.associationTypes1" = Table.ExpandRecordColumn(#"Inner Expanded results.associationTypes", "results.associationTypes", {"category", "typeId", "label"}, {"results.associationTypes.category", "results.associationTypes.typeId", "results.associationTypes.label"}), 
                InnerGetOneVal = List.First(#"Inner Expanded results.associationTypes1"[results.toObjectId])
            in
                InnerGetOneVal
        ),
        Data = #"Added Custom2"
    in
        Data
in
    GetPage

There are a couple of approaches. One could try and get the longitude and latitude from the postcodes and then calculate an as-the-crow-flies distance mathematically. You could try pasting them into co-pilot and asking it to make an estimation (this would probably work but I had many thousands of rows and co-pilot doesn’t tend to like that).

Fortunately the Google Maps Distance Matrix API offers a method which can take two postcodes and return a distance between them.

Get a Google Maps API developer account (if you don’t have one).

You will need to enter credit card details for this but Google will allow 10,000 API calls per month before charging. This gives you access to Google Maps APIs including the Distance Matrix https://console.cloud.google.com/apis/library/

Google gives an example API call in the form of a url like this:

https://maps.googleapis.com/maps/api/distancematrix/json?destinations=New%20Yok%20City%2C%20NY&origins=Washington%2C%20DC&units=imperial&key=YOUR_API_KEY

Pop this into the address bar of a browser and you can see the raw XML response of the API call:

{
   "destination_addresses" : 
   [
      "New York, NY, USA"
   ],
   "origin_addresses" : 
   [
      "Washington, DC, USA"
   ],
   "rows" : 
   [
      {
         "elements" : 
         [
            {
               "distance" : 
               {
                  "text" : "228 mi",
                  "value" : 367303
               },
               "duration" : 
               {
                  "text" : "3 hours 49 mins",
                  "value" : 13712
               },
               "status" : "OK"
            }
         ]
      }
   ],
   "status" : "OK"
}

Once you have it working, the next step is to get Excel calling this as part of a function.

Create a UDF (User Defined Function) in Excel that calls the Google Maps API

The function will have 2 arguments, Origin and Destination. I am using UK postcodes but the Origin and Destination could be any string that Google Maps will recognise (cities, places of interest etc).

Function G_DISTANCE(Origin As String, Destination As String) As String
Dim myRequest As XMLHTTP60
Dim myDomDoc As DOMDocument60
Dim distanceNode As IXMLDOMNode
G_DISTANCE = 0

Origin = Replace(Origin, " ", "%20")
Destination = Replace(Destination, " ", "%20")
Set myRequest = New XMLHTTP60
myRequest.Open "GET", "https://maps.googleapis.com/maps/api/distancematrix/json?destinations=" _
        & Destination & "&origins=" & Origin & "&units=metric&key=YOUR_API_KEY", False
myRequest.send
    
Dim response As String
response = myRequest.responseText
Dim response1Line As String
response1Line = Replace(Replace(response, vbLf, ""), "  ", "")
Dim distanceStr As String
distanceStr = Mid(response1Line, InStr(response1Line, "distance") + 22, 6)

G_DISTANCE = distanceStr

End Function

The above function takes Google’s XML response and turns into a single string. Then it identifies the part we were looking for (the bit after the word “distance”).

Invoke the function in Excel

You can now use the function in Excel to get the distance between 2 postcodes in cells A2 and B2 like so:

=G_Distance(A2,B2)

Freshly back at my desk after a week of wandering around the countryside and swimming in the sea, the thought occurred, “Do I really need to be here? Why haven’t I been replaced by a robot yet?”.

After all, we have been promised that artificial general intelligence (AGI) is around the corner for some years now, and at large scale fewer humans being needed to do the same amount of work seems inevitable. But could it do my job? I decided, for just one morning, to note down everything I did in my current role managing the IT dept of a small charity in the education sector and to ask the question “Could a robot have done this?”. In each case I would consider the answer in some (some would say tedious) detail, to try and get a sense of how round-the-corner the new utopia is.

9.00 – Catching up on emails, responding quickly to easy queries, deleting salesy ones, leaving the those that need a more considered response.

Of course I’m only seeing a fraction of what is emailed to me because the majority of the spam sent to me is pre-filtered by my email provider. Some of the easy queries could perhaps have been responded to by a bot, but if so the sender could have simply consulted a bot themselves rather than emailing me. Ergo we can suppose this task, as long as it exists, will need the human touch.

Humans 1, robots 0

9.38 – A colleague needs help with a workflow built into a proprietary application

Were this a problem with Power Automate, Co-Pilot would have easily volunteered a solution. However, in this case the software is unknown to internet-trained generative AI, which would only be able to help at a high level, and not the detailed step-by-step required by my colleague.

You could argue that this software could one day have its own AI assistant. However this is unlikely if the software is very bespoke, unless AI becomes much, much cheaper. Lets award a point to both humans and robots for this one.

Humans 2, robots 1

9.48 – An external API (Hubspot) has been updated requiring a change to a power query that is part of a symantic model in Power BI.

In order for a robot to do this it would have to

1. Read and interpret the email from Hubspot.
2. Know that this particular API is in use by a symantic model.
3. Find and read the necessary API documentation.
4. Find and open the file containing the symantic model.
5. Identify and implement a solution to the power query
6. Update the Power BI service and test.

Each of these steps seems to me well within the capabilities of AI. (Confession: AI helped me write the power query in the first place). What’s missing here is all encompassing bot that can read the email and has access to the symantic models in the Power BI service. This would have to be an AI that has access to all the cloud services necessary (email, Hubspot, Power BI) and be given the permissions necessary to make changes. In principle though, there was/is no subtask here that couldn’t be done by a bot.

So… Humans 2, Robots 2.

Conclusion

It appears there are still plenty of occasions when only a human will do, even in an ostensibly technical role. In just one morning I identified the following features where AI currently cannot replace human intervention:

1. Overall oversight. AI can read emails, research online documents and write code. But it still needs a director prompting it to do each of these subtasks and tying them together.
2. Initiative. AI with overall oversight would still need to have initiative to do anything with that oversight.
3. Permissions. AI with overall oversight and initiative will still need permission to make changes which relates to…
4. Responsibility.
5. Dealing with novelty. In common situations AI excels. For example Co-Pilot will create a complex excel formula quicker and better than any human given the right prompts. But with novel, bespoke software, it simply hasn’t had enough material to learn from.

The case is the multi-select, multiple choice question, where a learner may select x number of correct answers from y number of options.

The client particularly did not want to divulge how many correct answers there were, so out of 8 options, the learner could choose anything up to 8 answers. The client’s point of view was that telling the learner how many answers were correct was too much of a clue, and that part of what they were seeking to test was the learners’ ability to confidently distinguish the true statements from the false, rather than just guess the most plausible.

However, this can have large implications.

If we say “Choose the correct 2 options from the 8 options listed”, the learner must always choose 2 options. This means the total possible number of responses are:

8!/2!⋅(8−2)!​=28

So there are 28 ways to answer this question.

But if we don’t tell the learner there are 2 correct options they could choose 1, 2, or anything up to all 8. The total number possible responses is the total of the possible combinations. So…

8+28+56+70+56+28+8+1=255

Depending on how the marking rubric is stored, this could make quite a difference. A flat table of possible responses would require 255 rows.

Where it gets even more complicated is when we want to give partial scores for partially correct answers. Lets say the learner gets 2 marks for each correct selection. 4 marks in total are possible. But what if they choose the 2 correct options and a third incorrect one? What if they choose six? It is not obvious how such responses would be scored.

I posed this question to Co-Pilot and it simply recommended specifying the correct number of options to minimize these difficulties. So it seems there are no easy answers here!

Out of the box, Power BI seems to offer a pretty good selection of map visuals… until you try and apply them to the UK. Power BI sort-of recognises UK postcodes, but it’s no good with counties. Anyone who starts trying to visualise the geography of the UK in Power BI will soon realise they need to do a bit of work.

For shape maps that can slice up the UK (or anywhere else in the world) an out-of-the-box visual needs to be activated by going to File > Options > Preview Features and clicking “Shape map visual”. This will now appear in the visual pane like this:

Click on the visual and click Format > Map Settings and change the map type to “UK:countries” and it will look like this:

If this level of detail is all that is required then just put the country field (Northern Ireland, Scotland, England, Wales) into the Location bucket and your chosen measure in the Colour saturation bucket.

However I want to split the UK into broad regions like this:

• Northern Ireland
• Scotland
• Wales
• North West England
• North East England
• Yorkshire and the Humber
• West Midlands
• East Midlands
• East of England
• South West England
• Greater London
• South East England

Go to https://geoportal.statistics.gov.uk/

Select Boundaries > OECD / Eurostat boundaries > NUTS 1

This is the first in the list and looks okay:

Click Download and Shapefile:

In order to work with Power BI the shapefile needs to be converted to a TopoJSON file.

Go to https://mapshaper.org/

Click on Select and choose the zip download from geoportal.statistics.gov.uk

It will look like this:

The shapefile from geoportal.statistics.gov.uk uses OSGB36 projection. In order to work with Power BI this needs to be converted to WGS84 (World Geodetic System) projection.

Click Console in mapshaper

Type “info” to confirm the projection is OSGB36:

Type proj wgs84. It will look slightly odd:

Click Export and choose TopoJSON as the file format.

Back in Power BI, go back to the shape map visual and change the map settings to Custom map and upload the TopoJSON file you downloaded.

Put a UK Region field in the Location bucket and adject the Fill Colours format to appropriately show the gradations from light to dark (this will depend on the range of values for the measure). It will look like this:

There is a huge range of maps available on https://geoportal.statistics.gov.uk/ to split the UK into a shapemap, but this is a great one to start with.

My thanks to https://datawise.london/ and https://geoportal.statistics.gov.uk/ for their wonderful mapping recourses.

But if you use this Power BI will pester you to upgrade to Azure maps. Azure maps looks like this:

There are differences between the available bubble sizes in these visuals but that’s about it. There are some nice formatting options. Here’s a map with bubble sizes indicating learner achievement at awarding organisations across the UK:

There have been problems in the past with Microsoft limiting visibility of Azure maps to those logged in with the correct licence. This is no good when displaying a map for sharing more widely.

To stave off the problem I have found icon map to be the best open source, free, alternative.

Under visualisations click the three dots to get more visuals and search for “icon map”.

It looks like this:

This has all the features of Azure maps, just as many formatting options and will display fine even in publicly published Power BI reports.

Eventually you will want to have this auto refreshing and in order to do this you could install a data gateway on a machine/server. It is much simpler however, to simply change the datasource from an Excel one to a Web one, and let the schedule refresh take place entirely in the cloud.

If you go to Power Query and click on the first step, source, it will look like this:

= Excel.Workbook(File.Contents("C:\Users\DamienDevaney\Documents\YOUR_SPREADSHEET.xlsx"), null, true)

All we need to do is change it like this:

= Excel.Workbook(Web.Contents("https://YOUR_DOMAIN.sharepoint.com/sites/YOUR_SITE/Shared%20Documents/YOUR_SPREADSHEET.xlsx"), null, true)

Now when you click schedule refresh in the the PBI service it will work without any data gate. So you can share and collaborate on a spreadsheet and have it update Power BI visuals on a schedule.