Python threading vs multiprocess

Hi all,

I've been given access to a supercomputer! That's fun!

At first I thought I was doing something clever by importing the threading module into my python script, but I quickly discovered that all was not as it seemed. I'll cut to the nub of it.

I've written some code that rolls Yahtzees (5 of a kind with 6 sided dice). It's a nice simple test that generates CPU load. In the first instance I realised that my code running on a super computer using the Threading module was no faster than my local machine. This was a problem.

After some reading I discovered that actually, although Threading allowed some form of non-serial execution, it wasn't ever designed to do what I considered to be multi-threaded computation.

So I wrote a new version of the code to leverage the multiprocess library instead. All of a sudden we're actually doing some computation. Here are the tests I performed:

4 Threads on 4 Cores:

Multiprocess:

[xxxxx@xxxxx ~]$ cat results.txt
Number of yahtzees: 10000
Number of dice rolls: 3436750
Number of abandoned sets: 1990095
Total sets: 2000095
Running SLURM prolog script on xxxxx.cluster.local
===============================================================================
Job started on Wed Sep 11 10:08:10 BST 2019
Job ID          : 61100
Job name        : processyatzee.sh
WorkDir         : /mainfs/home/xxxx
Command         : /mainfs/home/xxxxxx/processyatzee.sh
Partition       : scavenger
Num hosts       : 1
Num cores       : 4
Num of tasks    : 4
Hosts allocated : xxxxxxx
Job Output Follows ...
===============================================================================
Writing to file
==============================================================================
Running epilogue script on xxxxxxxxx.
Submit time  : 2019-09-11T10:08:06
Start time   : 2019-09-11T10:08:10
End time     : 2019-09-11T10:08:20
Elapsed time : 00:00:10 (Timelimit=00:15:00)
Job Efficiency is: 0.00%

Threading:

[xxxxx@xxxx ~]$ cat results.txt

Number of yahtzees: 10000
Number of dice rolls: 27977719
Number of abandoned sets: 10899334
Total sets: 10909334
Running SLURM prolog script on xxxxx.cluster.local
===============================================================================
Job started on Wed Sep 11 10:07:35 BST 2019
Job ID          : 61098
Job name        : threadyahtzee.sh
WorkDir         : /mainfs/home/xxxxx
Command         : /mainfs/home/xxxxx/threadyahtzee.sh
Partition       : scavenger
Num hosts       : 1
Num cores       : 4
Num of tasks    : 4
Hosts allocated :xxxxx
Job Output Follows ...
===============================================================================
==============================================================================
Running epilogue script on xxxxxx
Submit time  : 2019-09-11T10:07:34
Start time   : 2019-09-11T10:07:35
End time     : 2019-09-11T10:08:48
Elapsed time : 00:01:13 (Timelimit=00:15:00)
Job Efficiency is: 38.01%

Job efficiency is interesting here. Suggesting that Threading is more efficient even though it took 7 times longer. Something to look into.

20 Threads on 4 Cores:

Multiprocess:

Number of yahtzees: 10000
Number of dice rolls: 648418
Number of abandoned sets: 1416784
Total sets: 1426784

Running SLURM prolog script on xxxx.cluster.local
===============================================================================
Job started on Wed Sep 11 10:10:45 BST 2019
Job ID          : 61102
Job name        : processyatzee.sh
WorkDir         : /mainfs/home/xxxxxx
Command         : /mainfs/home/xxxxx/processyatzee.sh
Partition       : scavenger
Num hosts       : 1
Num cores       : 4
Num of tasks    : 4
Hosts allocated : xxxxxxxx
Job Output Follows ...
===============================================================================
Writing to file
==============================================================================
Running epilogue script on xxxxxx.

Submit time  : 2019-09-11T10:10:44
Start time   : 2019-09-11T10:10:45
End time     : 2019-09-11T10:10:55
Elapsed time : 00:00:10 (Timelimit=00:15:00)

Job Efficiency is: 0.00%

Threading: 

Number of yahtzees: 10000
Number of dice rolls: 20623770
Number of abandoned sets: 12071981
Total sets: 12081981
Running SLURM prolog script on xxxxx.cluster.local
===============================================================================
Job started on Wed Sep 11 10:11:06 BST 2019
Job ID          : 61103
Job name        : threadyahtzee.sh
WorkDir         : /mainfs/home/xxxxxxxx
Command         : /mainfs/home/xxxxxx/threadyahtzee.sh
Partition       : scavenger
Num hosts       : 1
Num cores       : 4
Num of tasks    : 4
Hosts allocated : xxxxxx
Job Output Follows ...
===============================================================================
Running epilogue script on xxxxx.

Submit time  : 2019-09-11T10:10:50
Start time   : 2019-09-11T10:11:05
End time     : 2019-09-11T10:12:09
Elapsed time : 00:01:04 (Timelimit=00:15:00)

Job Efficiency is: 38.28%

Efficiency is still 0% for multiprocess, but it is finishing faster. Efficiency for threading is dropping off which is what you'd expect. I'm asking for something silly on a service which isn't hyper-threaded with a library that says threading but actually isn't. And if that makes sense, then nothing else will :-)

20 Threads on 20 Cores:

Process:

Number of yahtzees: 10000
Number of dice rolls: 487953
Number of abandoned sets: 1865100
Total sets: 1875100
[xxxxx@xxxxx1 ~]$ cat slurm-61106.out
Running SLURM prolog script on xxxxxxx.cluster.local
===============================================================================
Job started on Wed Sep 11 10:13:34 BST 2019
Job ID          : 61106
Job name        : processyatzee.sh
WorkDir         : /mainfs/home/xxxxxx
Command         : /mainfs/home/xxxxxx/processyatzee.sh
Partition       : scavenger
Num hosts       : 1
Num cores       : 20
Num of tasks    : 20
Hosts allocated : xxxxx
Job Output Follows ...
===============================================================================
Writing to file
==============================================================================
Running epilogue script on xxxxxxxx.

Submit time  : 2019-09-11T10:13:33
Start time   : 2019-09-11T10:13:33
End time     : 2019-09-11T10:13:47
Elapsed time : 00:00:14 (Timelimit=00:15:00)

Job Efficiency is: 56.79%

Threading:

Number of yahtzees: 10000
Number of dice rolls: 18743572
Number of abandoned sets: 10097238
Total sets: 10107238
Running SLURM prolog script on xxxxxx.cluster.local
===============================================================================
Job started on Wed Sep 11 10:11:53 BST 2019
Job ID          : 61107
Job name        : threadyahtzee.sh
WorkDir         : /mainfs/home/xxxxx
Command         : /mainfs/home/xxxxxx/threadyahtzee.sh
Partition       : scavenger
Num hosts       : 1
Num cores       : 20
Num of tasks    : 20
Hosts allocated : xxxxxx
Job Output Follows ...
===============================================================================
==============================================================================
Running epilogue script on xxxxx.
Submit time  : 2019-09-11T10:13:44
Start time   : 2019-09-11T10:13:50
End time     : 2019-09-11T10:14:57
Elapsed time : 00:01:07 (Timelimit=00:15:00)
Job Efficiency is: 7.61%

This last is very interesting. Why would multiprocess efficiency suddenly jump to 57%? Why would threading fall to 8%? I'm launching a thread per core as per the 4 on 4 test?

The conclusion is a simple one though. If you want parallel compute, don't use the threading module. Use the multiprocess module. It actually does what you want in the first place, and it's just as easy to write for.

Thanks for reading

CPU Load Tester - Yahtzee

Hi All,

Bit of a weird one this.

I wanted a CPU load tester and I didn't want to use one of the ones online. It should be fairly easy to write one that can heat up a CPU. Question is what?

Prime numbers are normally good, but that's been done. So I went with Yahtzees.

2 reasons for this:

1. The maths is pretty cool

2. I've been watching the Numberphile videos on youtube and the subject is raging over there.

Here is the code for my Yahtzee counter:

#!/usr/bin/perl
use strict;
my $randnum;
my $dice1;
my $dice2;
my $dice3;
my $dice4;
my $dice5;
my $dice6;
my $yahtzee;
my $checker;
my $rollcount;
my $checknum;
my $result;
my $yahtzeeswanted = 10;
my $dicesides = 6;

my @dice;



sub numbergen {
 my $range = 6;
 return int(rand($range)) + 1;
}

sub rollcount {
 $rollcount++;
 #print "Rollcount: $rollcount \n";
}

sub yahtzee() {
 $yahtzee++;
 @dice[$_[0]]++;
 #print "Number of yahtzees: $yahtzee \n";
}


while ($yahtzee < $yahtzeeswanted){
 $checker = 0; 
 rollcount;
 $dice1 = numbergen();
 $dice2 = numbergen();
 $dice3 = numbergen();
 $dice4 = numbergen();
 $dice5 = numbergen();
 $dice6 = numbergen();

  while ($checker <= $dicesides){
   $checker++;
   if ($dice1 == $checker && $dice2 == $checker && $dice3 == $checker && $dice4 == $checker && $dice5 == $checker && $dice6 == $checker){
    &yahtzee($checker);
   }


  }


}

print "Sided Dice: $dicesides \n";
print "Rollcount: $rollcount \n";
print "Number of yahtzees: $yahtzee \n";
print "Number of 1's: @dice[1] \nNumber of 2's: @dice[2] \nNumber of 3's: @dice[3] \nNumber of 4's: @dice[4] \nNumber of 5's: @dice[5] \nNumber of 6's: @dice[6]\n";

#print "$dice1 $dice2 $dice3 $dice4 $dice5 $dice6 \n";

If you take a look at the code you will see there are 2 declarations, one is for the number of yahtzees you want to generate, the other is for the number of sides you want your dice to have.

Have a play, and thanks for reading

VMWare - THE AUDITORS ARE COMING!

Hi All,

The Auditors are coming!

Here are 2 handy scripts for you to run against your VMWare database. Written for Oracle, but will probably work for MSSQL too.

This one, returns the OS type and name of a virtual machine, and the host it runs on:

select v.DNS_name, h.dns_name, v.guest_os

from vpx_vm v

inner join VPX_HOST h on h.id = v.host_id

where v.DNS_NAME is not null

order by h.dns_name;

 This one returns build version, name, boot-time and some blank and prefilled columns (because of the spreadsheet we had to use) You will need to replace [VCNAME] with your Virtual Center name. Funny that

select l.product_name, l.edition_name, l.product_version, '', h.boot_time, h.name, '[VCNAME]', '', '', 'Production', 'Location', '', '', '', '', h.cpu_core_count / h.cpu_count, h.cpu_count, h.product_name from vpx_lic_assets a
inner join vpx_lic_context c on a.asset_id = c.asset_id
inner join vpx_lic_licenses l on c.license_id = l.license_id

inner join vpxv_hosts h on a.name = h.dns_name;

This should help with the most basic questions. Other people want more, or less info.
Some useful tables/views:

Tables:
vpx_lic_licenses
vpx_lic_assets
vpx_lic_context
vpx_vm
vpx_host 


Views:
vpxv_hosts



Thanks for reading,