Speed up M Code and/or improve techniques

p45cal

Super Moderator
Staff member
Joined
Dec 16, 2012
Messages
2,319
Reaction score
40
Points
48
Excel Version(s)
365
In the attached is a Power Query solution to getting all the combinations of 6 columns of numbers.
The original problem is from msg#27 at http://www.vbaexpress.com/forum/sho...s-(no-repeats)&p=407983&viewfull=1#post407983
To keep the file small I've deleted the results which will need refreshing; it takes several minutes (more than 10).
[I realise the results don't fit on the sheet, and it's easy enough to split that table which I'll do later]
My questions:
1. Will this benefit from the likes of Table.Buffer statements? If so where should they be?
2. The code I've used may be awful; is there a better way?

For cross checking, there is a macro solution included which takes about 5 seconds to put the 6.25 million results into an array, and a further 40 secs to write them to a sheet.

Ancillary question: Is there a way to get a vba-created array into Power Query without writing to the sheet first?
 

Attachments

  • ExcelGuru11085vbaExpress36194Book1.xlsm
    36.1 KB · Views: 17
Last edited:
A 6 week bump…
 
It took me a moment to understand the task correctly.

Then I developed 2 solutions, which are shorter from the number of steps, but not faster. In both variants I also use Table.Buffer. In the 2nd variant I have divided the query. But this has rather a negative effect on the runtime. I reduced the test set a little bit, so that the runtimes became bearable at all.

At the moment, it looks to me that Power Query does not come close to the runtime of the VBA solution in this case.
 

Attachments

  • ExcelGuru11085vbaExpress36194Book1.xlsm
    47.7 KB · Views: 12
I did some more tinkering. As a result, I have a query that expands from F to A instead of A to F and a query where I formed groups (A/B, C/D, E/F) and then connected them. The run times for both these and your original variants are pretty much identical. Whereas by enabling the "Enable fast loading of data" option in the properties of the queries, I have now arrived at about 14 seconds for all variants. I am not yet satisfied with this, but compared to before, this is a very clear improvement
 

Attachments

  • ExcelGuru11085vbaExpress36194Book1 V2.xlsm
    52.2 KB · Views: 12
Thank you for this pinarello, in some ways I'm glad to see my M-code isn't too bad (although I haven't used Table.Buffer at all). Interesting to note that you get a great increase in speed with "Enable fast loading of data". I couldn't get quite the same speed increase, but worse, when I did a refresh after changing one or two values in the source tables the first refresh still took an age!
I'm a bit disappointed that PQ takes such a long time; I'd expected it to be faster than vba.
I've not given up on this yet though…
 
Back
Top