Python threading vs multiprocess

Hi all,

I've been given access to a supercomputer! That's fun!

At first I thought I was doing something clever by importing the threading module into my python script, but I quickly discovered that all was not as it seemed. I'll cut to the nub of it.

I've written some code that rolls Yahtzees (5 of a kind with 6 sided dice). It's a nice simple test that generates CPU load. In the first instance I realised that my code running on a super computer using the Threading module was no faster than my local machine. This was a problem.

After some reading I discovered that actually, although Threading allowed some form of non-serial execution, it wasn't ever designed to do what I considered to be multi-threaded computation.

So I wrote a new version of the code to leverage the multiprocess library instead. All of a sudden we're actually doing some computation. Here are the tests I performed:

4 Threads on 4 Cores:

Multiprocess:

[xxxxx@xxxxx ~]$ cat results.txt
Number of yahtzees: 10000
Number of dice rolls: 3436750
Number of abandoned sets: 1990095
Total sets: 2000095
Running SLURM prolog script on xxxxx.cluster.local
===============================================================================
Job started on Wed Sep 11 10:08:10 BST 2019
Job ID          : 61100
Job name        : processyatzee.sh
WorkDir         : /mainfs/home/xxxx
Command         : /mainfs/home/xxxxxx/processyatzee.sh
Partition       : scavenger
Num hosts       : 1
Num cores       : 4
Num of tasks    : 4
Hosts allocated : xxxxxxx
Job Output Follows ...
===============================================================================
Writing to file
==============================================================================
Running epilogue script on xxxxxxxxx.
Submit time  : 2019-09-11T10:08:06
Start time   : 2019-09-11T10:08:10
End time     : 2019-09-11T10:08:20
Elapsed time : 00:00:10 (Timelimit=00:15:00)
Job Efficiency is: 0.00%

Threading:

[xxxxx@xxxx ~]$ cat results.txt

Number of yahtzees: 10000
Number of dice rolls: 27977719
Number of abandoned sets: 10899334
Total sets: 10909334
Running SLURM prolog script on xxxxx.cluster.local
===============================================================================
Job started on Wed Sep 11 10:07:35 BST 2019
Job ID          : 61098
Job name        : threadyahtzee.sh
WorkDir         : /mainfs/home/xxxxx
Command         : /mainfs/home/xxxxx/threadyahtzee.sh
Partition       : scavenger
Num hosts       : 1
Num cores       : 4
Num of tasks    : 4
Hosts allocated :xxxxx
Job Output Follows ...
===============================================================================
==============================================================================
Running epilogue script on xxxxxx
Submit time  : 2019-09-11T10:07:34
Start time   : 2019-09-11T10:07:35
End time     : 2019-09-11T10:08:48
Elapsed time : 00:01:13 (Timelimit=00:15:00)
Job Efficiency is: 38.01%

Job efficiency is interesting here. Suggesting that Threading is more efficient even though it took 7 times longer. Something to look into.

20 Threads on 4 Cores:

Multiprocess:

Number of yahtzees: 10000
Number of dice rolls: 648418
Number of abandoned sets: 1416784
Total sets: 1426784

Running SLURM prolog script on xxxx.cluster.local
===============================================================================
Job started on Wed Sep 11 10:10:45 BST 2019
Job ID          : 61102
Job name        : processyatzee.sh
WorkDir         : /mainfs/home/xxxxxx
Command         : /mainfs/home/xxxxx/processyatzee.sh
Partition       : scavenger
Num hosts       : 1
Num cores       : 4
Num of tasks    : 4
Hosts allocated : xxxxxxxx
Job Output Follows ...
===============================================================================
Writing to file
==============================================================================
Running epilogue script on xxxxxx.

Submit time  : 2019-09-11T10:10:44
Start time   : 2019-09-11T10:10:45
End time     : 2019-09-11T10:10:55
Elapsed time : 00:00:10 (Timelimit=00:15:00)

Job Efficiency is: 0.00%

Threading: 

Number of yahtzees: 10000
Number of dice rolls: 20623770
Number of abandoned sets: 12071981
Total sets: 12081981
Running SLURM prolog script on xxxxx.cluster.local
===============================================================================
Job started on Wed Sep 11 10:11:06 BST 2019
Job ID          : 61103
Job name        : threadyahtzee.sh
WorkDir         : /mainfs/home/xxxxxxxx
Command         : /mainfs/home/xxxxxx/threadyahtzee.sh
Partition       : scavenger
Num hosts       : 1
Num cores       : 4
Num of tasks    : 4
Hosts allocated : xxxxxx
Job Output Follows ...
===============================================================================
Running epilogue script on xxxxx.

Submit time  : 2019-09-11T10:10:50
Start time   : 2019-09-11T10:11:05
End time     : 2019-09-11T10:12:09
Elapsed time : 00:01:04 (Timelimit=00:15:00)

Job Efficiency is: 38.28%

Efficiency is still 0% for multiprocess, but it is finishing faster. Efficiency for threading is dropping off which is what you'd expect. I'm asking for something silly on a service which isn't hyper-threaded with a library that says threading but actually isn't. And if that makes sense, then nothing else will :-)

20 Threads on 20 Cores:

Process:

Number of yahtzees: 10000
Number of dice rolls: 487953
Number of abandoned sets: 1865100
Total sets: 1875100
[xxxxx@xxxxx1 ~]$ cat slurm-61106.out
Running SLURM prolog script on xxxxxxx.cluster.local
===============================================================================
Job started on Wed Sep 11 10:13:34 BST 2019
Job ID          : 61106
Job name        : processyatzee.sh
WorkDir         : /mainfs/home/xxxxxx
Command         : /mainfs/home/xxxxxx/processyatzee.sh
Partition       : scavenger
Num hosts       : 1
Num cores       : 20
Num of tasks    : 20
Hosts allocated : xxxxx
Job Output Follows ...
===============================================================================
Writing to file
==============================================================================
Running epilogue script on xxxxxxxx.

Submit time  : 2019-09-11T10:13:33
Start time   : 2019-09-11T10:13:33
End time     : 2019-09-11T10:13:47
Elapsed time : 00:00:14 (Timelimit=00:15:00)

Job Efficiency is: 56.79%

Threading:

Number of yahtzees: 10000
Number of dice rolls: 18743572
Number of abandoned sets: 10097238
Total sets: 10107238
Running SLURM prolog script on xxxxxx.cluster.local
===============================================================================
Job started on Wed Sep 11 10:11:53 BST 2019
Job ID          : 61107
Job name        : threadyahtzee.sh
WorkDir         : /mainfs/home/xxxxx
Command         : /mainfs/home/xxxxxx/threadyahtzee.sh
Partition       : scavenger
Num hosts       : 1
Num cores       : 20
Num of tasks    : 20
Hosts allocated : xxxxxx
Job Output Follows ...
===============================================================================
==============================================================================
Running epilogue script on xxxxx.
Submit time  : 2019-09-11T10:13:44
Start time   : 2019-09-11T10:13:50
End time     : 2019-09-11T10:14:57
Elapsed time : 00:01:07 (Timelimit=00:15:00)
Job Efficiency is: 7.61%

This last is very interesting. Why would multiprocess efficiency suddenly jump to 57%? Why would threading fall to 8%? I'm launching a thread per core as per the 4 on 4 test?

The conclusion is a simple one though. If you want parallel compute, don't use the threading module. Use the multiprocess module. It actually does what you want in the first place, and it's just as easy to write for.

Thanks for reading

ITs journey from CAPEX to OPEX?

Hi all,

I thought I might share some thoughts about the financial aspect of the new wave of IT.
IT has always been a cost. It doesn't generate revenue, it's expensive and it's sort of invisible to the end users in the same way people on a train don't care about the rail lines.

It's taken a long time for IT to get accepted by bean counters as having a real function with definate cost benefit and now we are here we need to give them some exciting news.

We're not going to be a CAPEX for very long!

Yeah!

Why would any business who could outsource their hardware to the cloud not do so?

There could be a problem here though. When we rock up to finance and say that we need £1.2m for some tin, at least they can understand that something has been bought. It depreciates, it can be sold and it has value. The the issue now is that with the evolution of cloud offerings the only thing we could end up buying is TIME.

Time does not depreciate. Time is gone the moment it turns up. You can't (easily) sell on your time. Time has no value once its used. The day is not too far away when for some places IT will no longer be a CAPEX but a pure OPEX action.

This could mean our poor finance teams needs a heads up that IT are about to hugely shake up how things are done and may need to adjust their figures accordingly.

Food for thought certainly.

Thanks for reading,

Redefining Software Defined Anything

Hi all,

Last night at the IG16 dinner I did the social thing and asked other people what they thought of the whole SDx thing. Apparently my comment in the Q&A session had hit home with quite a few of the more technical spods and this got us talking about what we would actually describe as "Software Defined".

Fishcake: Quite nice.

First we went over the current definition of what Software Defined is and what we thought was broken with that name. Apparently my opinion was quite well received and we came to a group decision that SDx in it's current framing is actually "Profile Defined". Humans define the Profile. The Profile is pushed to a controller of some kind. The Controller pushes the Profile to the end devices.

Lamb: Really good. Succulent. Great with the ju.

What is an API anyway? Does SDx need to be API driven? We quickly came to the conclusion that the API in all the slides didn't actually have to be an API. It just has to be a method of collecting data or sending new commands. This could be as simple as WMI calls or an SSH session. As long as the controller knows what to do, and it produces an automated response it could even be as simple as dropping a file into a remote directory.

Creme Brulee: Really? Was nice but these things pop up everywhere. This one had fruit.

What should SDx be? We decided that the real innovation in SDx wasn't actually the controllers or the code or the software. The actual innovation here are Vendors providing programmatic ways of configuring devices remotely. The software element in SDx is just replacing the human pressing the buttons. But that's not what SDx should be.

We decided that in order for SDx to be a real thing it needed to do more. It needed to do more than just blindly push human written profiles to devices. It needed to do more than the devices themselves. It needs to become the central hub for the environment. It needs to react to that environment based on how the environment is behaving at the time and it needs to do this automagically. The profiles then are humans telling the controller the acceptable ranges the environment can be in and the controller makes the decisions on how that is achieved. This would be a true Software Defined environment.

Cheese: Cheese.

So what next? Well as our new definition of SDx is more automated and less human controlled it brought into the discussion of other technologies. If SDx could re-route network traffic or turn off switches to save power, for example, would we need Cisco's very clever routing protocol to find "cheaper" routes? Would we need VMWares Storage virtualisation or DRS? Were all of these things work arounds in preparation for SDx and if SDx is better, how do we wind down these technologies?

Thanks for reading,

Software Defined...anything?

Hi all,

I'm currently at the IG16 conference in Leeds, the title of which is "Software Defined...So what?". So what indeed?


A small part of the time in the talks today has been to define "Software Defined". Reading between the lines, it has almost nothing to do with "Software" defining anything and everything to do with automation and API's.

Let me go back a bit and I'll try to explain why you don't have to worry about "Software Defined...something" and why you have actually been doing this stuff for years.

SDx, the TLA being bandied around here with great abandon, is the act of abstracting the user from the API of a given vendor or set of vendors and giving them a core API that translates ubiquitous commands into Vendor specific commands.

Wow. What a sentence.

Ok. You are a Sys Admin. You have some Dell storage units and some Netapp storage units. You also have access to the API's and some skill in scripting. Now, some monster has asked you to create the same LUNs and configs on both units. So you write a script that will do that by speaking to the 2 companies API's that talks to their kit. Congratulations, you are now doing SDS.

Happy enough so far? Well it gets more interesting.

Lets say you don't actually care about storage configuration automation. You very rarely set up a new LUN, disk array, whatever. However, you've only got block storage (fibre/iscsi) and you want to automate the process of end users getting SMB or NFS shares. What you do is put a server inbetween, write some scripts that create windows/nfs shares on demand through a webpage or whatever and boom. Software Defined Storage at this layer too.

It's exactly the same deal with networking. If you write a script that interacts with some Cisco switches and/or some other kit, that is Software Defined Networking.

I raised a point in the Q&A session that this description is wrong. It isn't "Software Defined X". It's "Profile Defined X". Let's take Storage. What the SDS (Software Defined Storage) layer is actually doing is being your swiss army storage dude. It knows how to speak to all storage API's. You give the SDS a profile for it to follow, a device to poke the profile into and it'll do it configuration for you.

None of this is new. People have been writing scripts to do this for years. The only difference is now companies are getting into it and selling purpose built devices with a profile manager and a Perl script (or similar) to run against the Vendors API's.

The bonus is that because BIG IT are coming round to the idea they can monetise it. As they are trying harder to monetise it we are about to see some API standards being brought in. This means, ironically, it'll be easier for you to write code for the Vendors API's and not need their devices in the first place.

Software Defined X, again, isn't new. However the market is driving a new appproach to it. We need to be receptive to it, and actively encourage it so we can move onto more cool things. But don't forget, as it's as simple as all this, all you need is a computer and an API and you can do Software Defined anything you like.

I'm off to write a Software Defined Coffee API. Makes about as much sense as everything else.

thanks for reading,

Byte patterns - Human DNA

Hi all,

So you may have read my previous blog on counting bytes in a file. Well, I figured that as DNA is just a combination of letters (which stand for the chemicals involved) I figured I could see the ratio of the different chemicals by using my code. Would I find anything interesting?

So here they are:

X-Chromosome

Y-Chromosome

As you can see, not a lot 'interesting'. I was hoping that there would be some definite differences between the two. Saying that however, maybe the similarities and the ratio are the interesting thing about the result.

Oh, in case you are wondering. I haven't discovered a new chemical involved in DNA. That's actually line breaks!

Thanks for reading!

More Pratting Around With Colour and Data...

Hi all,

So there I was trying to work out why data is so huge, trying to work out a way of handling it better, when I had a thought. What actually is data?

Data, in simplest terms, is a string of bytes. In order for that data to be useful we put those bytes into a logical order to describe things such as text, images and the like inside a file. Normally these files have headers and other such human recognisable features to describe how the computer readable data is laid out inside the file.

I digress. Data is (currently) arranged in files for us humans to look at, they have structure, all future files will have structure, so I want to predict what those structures will be. To that end I started playing.

Plain Text file

I wrote a script in Python that can count the occurences of byte values. Its output is a png where the colours tendancy towards red shows its higher frequency compared to the total number of bytes in the file.

As you can see byte values that correspond with language based ascii characters feature heavily. Now what happens if we zip it?

Zipped Plain Text

As you can see the image is a lot flatter. This is because of the way zipping a file works. Essentially it records 1 or a number of bytes and then back references on the next occurance. I can now prove that the maths for different types of compression is different.

7z file

See? Proof positive something else is going on. Ok, ok, but it is interesting isn't it? How it's flattened out? It's something I've noticed with other file types too. Here are some below:

JPG file.

PDF file

MSI file

The smaller the file, the higher the compression, the more lossy the data, the more flat it is.

Nobbing around with data is something I've been doing for years. I am a great believer that we have already reached "L-Space". Everything ever written, ever been written, ever will be written is already out there. With the birth of huge datastores and the internet it should have transpired that every combination of bytes will have been written somewhere.

Thanks for reading,

Hybrid mind from hybrid cloud

Hi all,

The human mind is one of the greatest achievements in the natural world, it's power and utility are still not entirely known, it's limits unfathomable.
There are true geniuses, ones who can consciously direct their thoughts and imagine the universe of the very small and the very large, ones who can describe the beauty of nature in pure mathematics. There are those of us who are terrible at maths, but subconsciously your mind can estimate the speed of objects, the distance required to travel, the angle of the road, any physical limitations of movement to cross a road or catch a ball. It can also prioritise very well. Do we need to concentrate on the Lion in the bushes, or the stone in our shoe? It is a fantastic machine.

But it is not without its limitations. Yes, sometimes the mathematics ability is hidden from us. We can instinctively cross the road or catch a ball, but times tables are still a mystery. It's affected by mood, emotions, tiredness and more importantly, and what I am to focus on, distraction and procrastination.

First though, a quick leap to computing. Computers are one of the greatest inventions of the human race. The power and speed of processors has increased at a near exponential rate. The speed of storage has increased by a phenomenal amount too, since the invention of punch tape or physical switches. In the early days, we were limited to doing things one after the other, but then RAM and multi-core and multi-threaded processors came along. We could do more at once. No longer limited to sequential processing or access, things could be done at the same time and delivered faster.

And here is the rub.

In the IT sphere we have been, for decades now, trying to speed up computers. They are blisteringly fast. Any question you want answered can be retrieved very quickly. Any communication will arrive instantly and pop up on our screen. Email, Instant Messaging, Facebook, that report we wanted, that spreadsheet thats taking a while. All popping up as soon as they are ready. Each fighting for our attention. Each requiring our poor brains to switch focus again and again and again.

This is wrong.

We created computers so we could tell them what to do for us. Now they are driving. All these pop ups and alerts, each one a major distraction, each one has to be prioritised and thought about by our minds before we decide if we need to click on that report that's just come in. Do you need to open that report, or reply to Kevin who has just IM'd you? Do you need to reply to that facebook comment? Do you need to answer that email right now? Even if you don't need to talk to Kevin you still need to open the chat window to prioritise it against other things that have 'just popped up'.

My suggestion is that for our normal devices, our desktops and laptops and pads and phones and All in One's and smart watches (an ad nauseum list in the 21st century), we stop trying to make them faster. We need to make them "smarter". More like us. More for us. More like the way we think.

In order for this to happen we need to become more like the computers from yesteryear. More serial, more focussed. Like we used to be, when we used to make, build and hunt. We need to rest our conscious mind by providing things in a serial manner one job at a time. This will stop distractions. Less stress from swapping thought will also enable our subconcious mind to solve other issues, instead of trying to remember what that last email said. We don't need our computers to be fast, we need them to deliver the answer when we are ready.

At this point, we don't need our technology to be faster, but we do need to become more symbiotic with our technology. Our technology needs to be more sympathetic to our needs, our energy and our minds. Our computers should know when we are ready for another distraction; another report, another message, another picture of a cat even. Instead of the constant bonging, binging, pinging, popups and other distractions that cause our attention to be dragged away from our focus.

Thanks for reading

How to make the Internet Beautiful. The Code.

Hi All,

Ok, so hopefully you've read the other blog or seen the stuff on youtube and followed a link here or whatever. If not, go back and have a look. Also, this blog is a little bit technical. So if you're interested in the effect rather than the cause, look away now.

Images. Images everywhere. But how did I make mine?

Well at first, I decided that I would read the HTML and use that to generate image data. The smallest unit of image data is the pixel, and we need to give it at least 5 values. These are:

Red (0-255)
Green (0-255)
Blue (0-255)
X (pos)
Y (pos)

Lets ignore X,Y as they are the easiest to sort out. However, we do need to worry about RGB.

Lets go back to some HTML code...

<html><head>I love html</head><body>I really love html</body></html>

Here is some example code. Gotta love it right?

Every character there is represented by a value in something called ASCII. These are somewhere between 0 and 255. For example, a capital 'A' has a value of 65. I could have just used that value to describe either a red, green or blue value. But the thing is, reading through my code, I wanted to really test myself at 2am and write some superfluous lines of code. I change ASCII into HEX.

HEX describes numbers using the format 0-9, A-F where A is 10, B is 11 and so on up to F which is 15. 16 in HEX is 10. AA in HEX is 26. As ASCII characters are 0-255, we need 2 HEX characters for each 1 ASCII character.

My code then, has to loop from the beginning, to the end, choosing 2 characters per colour. 6 HEX characters describing all 3 colours. Red, Green and Blue.

 Once it has done that, its a simple case of changing it back down to DEC (our "normal" numbering system) so I can pass it to GD to write the pixel.

Before we get to, now I look at it, rather rushed and over complicated code, we can encode the example:

Anti-aliasing has helped a lot here. Don't forget, it takes 3 characters to make 1 pixel. There are only 16 pixels in that image. It's a 4x4 blown up really big.

Here's the code:

#!/usr/bin/perl 

#use strict;

#use warnings;

 use GD;

 use HTTP::Lite;

 use String::HexConvert ':all';

$filename=0;

while ($monkey == true){

$filename ++;

my $hexcode = "";

my $twitcode = "";

my $x=0; # x coord

my $y=0; # y coord

my $i=0; #loop var

my $row =0;

my $weight=0;

my $r =0;

my $g =0;

my $b =0;

#my $im = new GD::Image(20,10);

    $http = HTTP::Lite->new;

    #$req = $http->request("http://wemakeawesomesh.it/make")

    #$req = $http->request("http://search.twitter.com/search.json?q=%23BeliebersAreHereForJustin") 

    $req = $http->request("http://theregister.co.uk/index.html") 

    #$req = $http->request("http://feeds.bbci.co.uk/news/rss.xml")

    

        or die "Unable to get document: $!";

$twitcode = $http->body();

 $hexcode = ascii_to_hex($twitcode);

$size = sqrt((length($hexcode) / 6) );

my $im = new GD::Image($size,$size,1);

#print $hexcode;

for ($i=0;$i <= length($hexcode);$i = $i + 2){ 

$row ++;

$weight ++;

print "\n\nWEIGHT:  $weight CHARPOS:  $i \n";

if ($weight == 1){

$r = hex(substr($hexcode, $i, 2));

print "\n" . substr($hexcode, $i, 2);

}

if ($weight == 2){

$g = hex(substr($hexcode, $i, 2));

print "\n" . substr($hexcode, $i, 2);

}

if ($weight == 3){

$b = hex(substr($hexcode, $i, 2));

print "\n" . substr($hexcode, $i, 2);

print "\nCOORDS: X: $x, Y: $y----- $r,$g,$b\n";

$cursorcolour = $im->colorAllocate($r,$g,$b); 

$im->GD::Image::setPixel($x,$y,$cursorcolour);

$x++;

                $weight=0;

if ($x >= $size){

                        $x=0;

                $y ++;

}

                

}

}

binmode STDOUT;

open (MYFILE, ">frame$filename.png");

print MYFILE $im->png;

close (MYFILE);

sleep(300);

}

Wow, that's ambitious for blogspot to handle isn't it?

A few things to mention here.
1. There is a wrapper that means it will loop forever. I last used this particular version of the code to collect twitter data every 5 mins. (Which you can see with the sleep(300) above)
2. I've left some examples in there.
3. I've done some dodgy maths to get the image size.
4. I've commented out the top 2 lines.

So please, take my code, have some fun. Rewrite it (take out the rubbish stuff like the string ASCII->HEX->HEX->DEC manipulation)

If you do improve it, let me know what you did.

I'll let you work out how I did the mp3 video, and the text-> music. It won't be hard now.

Thanks for reading,

How to Make the Internet Beautiful.

Hi All,

I was sat down bored the other day. The internet is the most bland, boring and unexciting place at the moment. It's dull.

So I wondered. Can I make the internet more interesting? And the answer, surprisingly, was yes.

I turned it into pictures, like this:

Now this looks amazing. All that from some data. I've been playing with this for a while, and here are a few videos of the results of my testing:





Now this is all well and good. I can change the internet into images and set them to music. (I'll deal with the 'how' in the next blog as it's quite detailed.)

It was then that I realised I could put absolutely anything through my encoder. So I wondered...Can I encode an mp3? The answer is yes. Yes I can. What's interesting is that you can see the file header and the tail at the end too.





After that I thought a bit more....I wonder what the internet would sound like? What would happen if I used audio as the output?

Well....here it is (TURN YOUR VOLUME DOWN FIRST!):



Feel free to have a listen and a watch or 3. Make some comments, or if you have any other ideas, let me know.

Thanks for reading!

Bash script to download Infinite Monkey Cage episodes.

Hi all,

It's been a while but I thought I might share this with you.
I am a fan of a BBC Radio 4 show called "The Infinite Monkey Cage". Hosted by Prof. Brian Cox no less.

Here is an over engineered bash script to download all their episodes:

wget http://www.bbc.co.uk/podcasts/series/timc/all/ ; cat index.html | grep -Po "timc\_[0-9]*\-[0-9]*[a-z]\.mp3" | sed "s/^/wget http\:\/\/downloads.bbc.co.uk\/podcasts\/radio4\/timc\//g" | sh ; yes | rm index.html

Download and enjoy!

Thanks for Reading

How to configure Apache2.2 as a Reverse Proxy

Hi All,

The other day I was asked to create a proxy. But not just any proxy. A proxy that could handle content and that could forward on and return login requests to a 3rd party.

Ok. So that bit was new. They have just asked me to configure a tiny Content Delivery Network.

So first things first. I cracked out the 12.04 server version of Ubuntu, installed it, configured it then installed Apache.

Nice and easy.

The reason why I chose Ubuntu over the other Linux builds is that the way Ubuntu package Apache makes it really, really easy to configure. I'm not going to go into which flavour of Linux is the best, neither do I care. This is because I AM A GROWN UP.

On with the show!

So you will need some bits and pieces to go with Apache. You will need the proxy, proxy_http and headers mods. You can install these by typing the following as root (or sudo):

a2enmod proxy
a2enmod proxy_http
a2enmod headers

Again, this is really easy stuff.

Now we have our mods installed lets go and configure something. First make sure that Apache is working.

sudo service apache2 restart
then point a browser at your server. You should get an 'It's Worked!' message.

Now for the config.
Go to /etc/apache2/sites-enabled
now use a text editor to edit 000-default.

Here is my config:

ProxyPreserveHost On
ProxyVia full
<proxy>
Order deny,allow
Allow from all
</proxy>

ProxyPass / http://xxx.xxx.xxx.xxx:8888/

ProxyPassReverse / http://xxx.xxx.xxx.xxx:8888/

Header edit location 192.168.1.2 192.168.1.2:81


Let's go through this bit by bit.

ProxyPreserveHost....this preserves the host IP in the headers
ProxyVia full...Adds the Via tag to the outgoing headers so we can see where the request came from

The next bit is the permissions for the proxy. Configure the permissions in this as if it was a website. Don't dump your proxy onto the internet with the settings above. It's not secure. This is instructions on how to make a proxy, not to make a secure proxy.

No we're into the fun bit.

ProxyPass / http://whateverIPorSite.com/

This makes incoming connections proxy out to whatever the target is.

ProxyPassReverse / http://whateverIPorSite.com/

This makes returning connections proxy back correctly

That is basically a proxy right there. It'll work. Restart Apache, point your browser at it and it'll work.
Now let's have some fun.

I have an authentication server somewhere up stream. It's a 3rd party service. My clients need to authenticate with that before getting the content they want. there are 2 streams to this. The authentication stream and the content stream. to make my life easier I decided that I would use 2 ports. Port 80 would handle authentication with the 3rd party provider and 81 would serve the content. to that end I created a new site on port 81 and put the content on it. Now we need to address that Header edit location line up there.

When you authenticate your 3rd party it should return a 302, in that return code you will get a header called "Location" this is where content is due to be served from. When you have a 3rd party site handling your authentication you don't necessarily want to download your content from them. You'd rather use Akamai or something. This means rewriting the header at the proxy, before it gets back to the client.

In my case I needed to change the port number. First the command:

Header edit Location

This tells the mod I want to edit one of the returning headers called Location.
Next I have: 192.168.1.2 192.168.1.2:81

What this does is tell the mod to replace the IP of the server, with the IP and the new port of the server. When the client gets the location header, it will pop off and download it from this source rather than the 3rd party authorisation provider.

Thats about it. Questions below.

thanks for reading.

CPU Load Tester - Yahtzee

Hi All,

Bit of a weird one this.

I wanted a CPU load tester and I didn't want to use one of the ones online. It should be fairly easy to write one that can heat up a CPU. Question is what?

Prime numbers are normally good, but that's been done. So I went with Yahtzees.

2 reasons for this:

1. The maths is pretty cool

2. I've been watching the Numberphile videos on youtube and the subject is raging over there.

Here is the code for my Yahtzee counter:

#!/usr/bin/perl
use strict;
my $randnum;
my $dice1;
my $dice2;
my $dice3;
my $dice4;
my $dice5;
my $dice6;
my $yahtzee;
my $checker;
my $rollcount;
my $checknum;
my $result;
my $yahtzeeswanted = 10;
my $dicesides = 6;

my @dice;



sub numbergen {
 my $range = 6;
 return int(rand($range)) + 1;
}

sub rollcount {
 $rollcount++;
 #print "Rollcount: $rollcount \n";
}

sub yahtzee() {
 $yahtzee++;
 @dice[$_[0]]++;
 #print "Number of yahtzees: $yahtzee \n";
}


while ($yahtzee < $yahtzeeswanted){
 $checker = 0; 
 rollcount;
 $dice1 = numbergen();
 $dice2 = numbergen();
 $dice3 = numbergen();
 $dice4 = numbergen();
 $dice5 = numbergen();
 $dice6 = numbergen();

  while ($checker <= $dicesides){
   $checker++;
   if ($dice1 == $checker && $dice2 == $checker && $dice3 == $checker && $dice4 == $checker && $dice5 == $checker && $dice6 == $checker){
    &yahtzee($checker);
   }


  }


}

print "Sided Dice: $dicesides \n";
print "Rollcount: $rollcount \n";
print "Number of yahtzees: $yahtzee \n";
print "Number of 1's: @dice[1] \nNumber of 2's: @dice[2] \nNumber of 3's: @dice[3] \nNumber of 4's: @dice[4] \nNumber of 5's: @dice[5] \nNumber of 6's: @dice[6]\n";

#print "$dice1 $dice2 $dice3 $dice4 $dice5 $dice6 \n";

If you take a look at the code you will see there are 2 declarations, one is for the number of yahtzees you want to generate, the other is for the number of sides you want your dice to have.

Have a play, and thanks for reading

VMWare - THE AUDITORS ARE COMING!

Hi All,

The Auditors are coming!

Here are 2 handy scripts for you to run against your VMWare database. Written for Oracle, but will probably work for MSSQL too.

This one, returns the OS type and name of a virtual machine, and the host it runs on:

select v.DNS_name, h.dns_name, v.guest_os

from vpx_vm v

inner join VPX_HOST h on h.id = v.host_id

where v.DNS_NAME is not null

order by h.dns_name;

 This one returns build version, name, boot-time and some blank and prefilled columns (because of the spreadsheet we had to use) You will need to replace [VCNAME] with your Virtual Center name. Funny that

select l.product_name, l.edition_name, l.product_version, '', h.boot_time, h.name, '[VCNAME]', '', '', 'Production', 'Location', '', '', '', '', h.cpu_core_count / h.cpu_count, h.cpu_count, h.product_name from vpx_lic_assets a
inner join vpx_lic_context c on a.asset_id = c.asset_id
inner join vpx_lic_licenses l on c.license_id = l.license_id

inner join vpxv_hosts h on a.name = h.dns_name;

This should help with the most basic questions. Other people want more, or less info.
Some useful tables/views:

Tables:
vpx_lic_licenses
vpx_lic_assets
vpx_lic_context
vpx_vm
vpx_host 


Views:
vpxv_hosts



Thanks for reading,

Install and configuration of SRM 4.1

Hi All,

Configuration of SRM:

For the initial config of SRM, I installed it onto the same server as the Virtual Center. I am only supporting 25 VMs at these early stages. I won't cover the installation procedure here as its a piece of cake, what I will say is that you need to set up your database, login and you will also need to go into ODBC and set up your database connection before running setup.

Once this has been done, log into your virtual center server through the client.

Click on plugins, then install the vCenter Site recovery manager extension.

I'm going to take you into the GUI for this so you can see what needs configuring.

Once it has installed, you will have another tree under "Solutions and Applications"
Go into Site recovery.

• You need to provide a login for the local (primary) site and the paired site (DR)
• You also need to create a connection between the 2 sites
• You need to provide a driver and login details for your Storage device.
• Inventory mappings are source folders/datacenters and destination datacenters/resource groups
• Protection Groups are bunches of machines carved up into lumps. The lumps depend on various things like OLAs, applications, server teams and other logical groups

First things first then.

Create a new user on the SRM (live site) server.
I did this simply by creating a local user on the VC server. If you have a seperate SRM server, you'll need to create a domain account. It will need full admin priviledges on VMware though.

Create a new user on the SRM (DR/Failover) server. Same as above.

Go into the site recovery manager, click on "Site Recovery" on the tree view on the left hand side.
In the main pane, under "Protection Setup", click configure.

Follow the prompts. All you are doing is putting the user accounts in that you set up above.
Once that is done, you should notice that the local site and paired site boxes are populated. If they aren't, something has gone wrong.

Next step:

Array managers. Storage, basically.

1. Set up your LUNs and your snapmirrors/data protection transfers/remote clones, whatever your storage vendor wants to call it this week. This needs to be done first as when we get to the next bit it scans your storage to get this information. If you don't do this bit first, you'll have to run the same scan again and depending on the number of luns, you could be sat there for quite some time.
2. You will need to provide a user account for both the live and DR sides of your storage arrays. It will need to ability to snapclone a replicated lun (for testing) and to break a mirrored volume, delete a volume etc... pretty low level stuff.
3. Pop onto the vmware website and search for your storage vendors driver for SRM. Download the right ones for you (either block or CIFS or both) and copy it to your SRM server. They should come as MSI packages, or self-installing executables so run them.

Once they are installed restart the SRM services.

Back to the Vsphere client.

Click the configure button next to array managers
Click Add

fill in the IP address details and the username and password for your primary storage.
It will refresh and you should see the array ID and the device count. (device count is the number of luns or presented storage from that device.)

Add all your storage arrays like that, then hit next.

Same again, this time adding your secondary (DR) sites storage username and password

press next and this should show you a list of all replicated data stores. If you have replication switched on, you should get a list of datastores. Hit Finish.


Nearly there!

Inventory Mappings - hit configure.

Here is where you map you current live directories/networks/clusters etc... to the DR site resources. Easy enough done if you map things the same either side.

Protection Groups.
First thing you need to do, is to go onto your DR storage and present a small LUN where you can put your placeholder data files for all your protected machines. Then go back to the VClient

Click Create to create a protection group
This is volume based VM protection groups, so choose your volume
Choose where to put your placeholder data files (you're new lun would be great)
then click finish.

You're done. Now its time to create a basic plan. Plans are always created on the secondary or DR side.

• Point your VClient at your secondary VC and login.(oh quick stop here....look at all those machines!!! Don't turn them on, they are placeholders)
• Click solutions and Applications -> Site Recovery
• Click recovery plans. (we'll just do a very, very basic one. Although if your secondary and primary sites match exactly, this plan would be amazing for you)
• Click create
• Give it a name click next.
• Choose a Protection Group
• Click next - Leave the timeouts as default as we won't be doing either.

Test networks. Right, if you run the test plan, then SRM will either create "fake" vswitches, or you can make it do the "real" networking. Map the networks to the correct type here. Auto means create a non-uplinked test switch.

Suspend local VMs - if your secondary site hosts test or development machines, they rank below production machines that are going through DR process. You can configure SRM to suspend these machines, releasing those resources.

Click finish.

Want to test it?

Choose a recovery plan in the tree view on the left.
Under the summary tab you can see some buttons.
Click Test.

There you go, all done.
If you want to see what its doing, log into another client session.
On your original session drive SRM from the recovery steps tab.
Click it and click the test button (top left)
Now you can see what steps its got to. On your second session you can see what the machines are doing.

There you have it.

Basic failover in SRM.


Thanks for reading

Solaris and ZFS Including ISCSI target recovery

Hi all,

I had a rough day the other day. I was using Nexentastor to present iscsi storage to my VM Test Environment, and the repository.db file became corrupt. After a reboot the server failed to boot. At all.

I don't like Nexenta. The GUI, its main selling point, keeps crashing and I have a lack of faith in it. So I thought I'd give Solaris 11 a punt. Here is how I recovered my ZFS luns and presented them.

First, install Solaris. This is a piece of cake, just make sure you don't install solaris on disks being used by your ZFS volumes.

Once you have your OS up and running, sort your networking out. I installed the Gnome version so I did this using the GUI tool. I will cover aggregates and command line networking in another post.

If your storage is DAS based, ie its directly attached to your Solaris server, and you have more than 15 SAS disks and you can't see the others, you will need to modify one of the OS files and reboot.

The filename is /kernel/drv/sd.conf
You should see lines like this:
name="sd" class="scsi" target=16 lun=0;

Add more lines for the number of disks you have, incrementing for each line, save and reboot.

Next thing is to rescue your ZFS volumes.

Running zpool import will scan all your disks and report back on the volumes it has found. It won't import them. to actually import them run this command:

zpool import -f [poolname]

To see your imported pool type

zpool list -L

I have 3 pools, 3 volumes, 1 per pool. When I did mine, I only "rescued" one volume or pool at a time, then presented them, then did the next one. You can do yours however you want.

Next thing is to install the iscsi target software. If you are hiding behind a proxy, the easiest way of doing this is to open a terminal, then do this as root:

export http_proxy=http://username:password@yourproxy.fqdn:portnum

Then run the following:

pkg install storage-server SUNWiscsitr

Once it has installed, we can start presenting volumes. First off, we need to identify the shares. You should know what they were called. Run this command:

sbdadm import-lu /dev/zvol/rdsk/[Volumename]/[sharename]

That will import your lun. Then run:

sbdadm list-lu

This will list your luns. The number you will need is the GUID so copy it and get ready to paste.
Run this command:

stmfadm add-view [GUID]

This will share your iscsi lun with every device but also from every IP on your solaris server. This is fine for me, you may need to lock it down. I haven't investigated this as I don't need it.

Next you need to create the target process itself. Run these 2 commands to complete the storage side config:

svcadm -l target
itadm create-target

check it with:

itadm list-target

Thats you done, sort your initiators out now on your other servers. I haven't covered creating new volumes, luns or whatever. I'm hoping you find this article when you are in the shit and need some quick answers!

Thanks for reading,

Trev

Help! Some dick deleted the Author for the default sharepoint site!

Hi All,

While I was doing the below, I had a problem.
Someone had deleted the default author for a site, this stopped the stsadm tool running.

Do this nice DB stuff to get it working.
First:

Select author, siteid from webs where fullurl =
'[yoursite]'

Remember the siteid!

run this one:

Select * from userinfo where tp_siteid='[that really long code you
just remembered]'

Now choose another user who is active and remember the tp_id

Now run this one:

update webs
set author = [that tp_id you remembered for the
user]
where fullurl = '[yoursiteid]'

Thanks for reading

Trev

Migrating from Sharepoint

Hi All,

Sharepoint is a whore to get away from. But here is some useful information and how to do it, quick and dirty.

First things first, do you know where your data is? Nope! It's in the database.

This is just for interest. Run this against your sharepoint DB:

Select d.dirname, d.leafname, d.setuppathuser, a.[content] from docs d
left outer join alldocversions a on a.id = d.id
where dirname = '[your directory name in the URL]'
and a.[content] is not null

What this will do is return all the documents. The important column here is a.[content]. This is your file converted into hex. If you don't believe me copy and paste it into a hex convertor on the web or something. Paste that into notepad, save as the right file type yada yada.

Anyway, get rid of the a.[content] entries in the sql then run it again.

Acceptable loss 1. The only docs with pathsetupuser are ones with multiple versions. Single versions do not have this metadata. That is in another hex encoded column called metainfo (select metainfo from docs).

Unless you want to faff with all that, you will lose the metadata. I couldn't personally be arsed and it was an acceptable loss as we are moving to a new file storage system.

Now you need to rescue your files. You can do this by running a tool here:

c:\program files\common files\microsoft shared\web server
extensions\12\bin

This is a sample command line you will need to run:

stsadm.exe -o export -url http://[servername]/sitename]/ -versions 1 -filename d:\sharepointexport -nofilecompression

Ok, quick overview.

-o export: does what it says
-url: the base url before the /forms/allitems.aspx
-versions 1: only the latest versions of files
-filename: destination directory. This mustn't exist before you run the command.
-nofilecompression: important for us. This tool exports .dat files, with compression on we will just get one big one.

OK, run that commandline and get your files. You should have all sorts of files like:

0000000A.dat
00000004.dat
....

These are your files. You should also get an xml file called manifest. You need this as it has the metadata like the real bloody filename!

At this point, I gave up on windows. I was 3 hours in and I managed to get some help off a friend of mine.

We copied the xml file to a linux box, and used perl to parse the xml file and generate a .BAT file at the end that would rename the files and move them into the right directory. Here is the perl code:

use strict;
print "\@echo off\n";
open (my $FL, "<manifest.xml");
foreach my $line (<$FL>) {
chomp($line);
if ($line =~ /^<file url="'\">)
my $realfilename = $line;
$realfilename =~ s/.*?Name=\"(.*?)\".*/\1/;
my $directory = $line;
$directory =~ s/.*?File Url\"(.*?)\".*/\1/;
$directory =~ s/\/.*?$//;
my $currfilename = $line;
$currfilename =~ s/.*?FileValue=\"(.*?)\".*/\1/;
print "copy $currfilename \"$directory\\$realfilename\"\n";
}

on your linux box run this script and > rescuemyfiles.BAT

Right!
Copy this file to your sharepoint server, and drop it into the same directory as your rescued .dat files.
All the .dat files are, are your files. You can rename one from .dat to .jpg or whatever and they will work. All the .bat file does is grab the file info from manifest.xml and copies them into a directory for you.

So there you have it. How to resuce your files from sharepoint. Have fun!

thanks for reading,

Trev
p.s. I never said this was clean did I? :-)

 

Mythtv - Perl script to email the recordings list.

Hi All,

So, I don't want my new recordings list being available on t'interwebs, but I do have my own email server.

So I wrote some perl to parse the RSS feed and email me the results. I stuck this to a cron job and now I get a weekly email with the new recordings on.

It does need tweaking, I only want the newest recorded programs email to me. Currently this emails the lot to me. Anyway, here's the code:

#!/usr/bin/perl

use LWP::Simple;

#to read html streams

use strict;

use Mail::Sender;

my $emailmsg;

my $sender;

print "Getting Content\n";

my $content = get('http://localhost/mythweb/rss/tv/recorded') || die print "Oh fuck it";


#print $content;


$content =~ s/[^[:ascii:]]+//g; #get rid of weird chars

print 'Splitting atoms...oh LOL :-)';

print "\n";


my @lines = split(//, $content);


print "Parsing some stuff\n";

for my $line (@lines){

#print $line =~ /\(.*)\<\/title\>/;

$line =~ /\(.*)\<\/title\>/;

$emailmsg .= $1;

$line =~ /\(.*)\<\/pubDate\>/;

$emailmsg .= " - " . $1;

$emailmsg .= "\n";

#print $line =~ /\\<\!\[CDATA\[(.*)\]\]\>\<\/description\>/;

$line =~ /\\<\!\[CDATA\[(.*)\]\]\>\<\/description\>/;

$emailmsg .= $1;

$emailmsg .= "\n\n\n";

#print $emailmsg;

}



#(my $headlines) = ($content =~ /type\=\"html\"\>(.*)\<\/title\>/);

#print "$headlines\n";

$sender = new Mail::Sender {

smtp => 'IP or server address',

from => 'address@tosend.from',

auth => 'NTLM',

authid => 'username',

authpwd => 'password',

on_errors => undef,

}

or die "Can't create the Mail::Sender object: $Mail::Sender::Error\n";

$sender->Open({

to => 'who@tosendit.to',

subject => 'Mythtv recordings update'

})

or die "Can't open the message: $sender->{'error_msg'}\n";

$sender->SendLineEnc("$emailmsg");

$sender->Close()

or die "Failed to send the message: $sender->{'error_msg'}\n";


There you go. I've left my commented code in so you can see where I was going with it.

Thanks for reading,

Just upgraded your VMWare vmhost from 3.5 to 4.0 or 4.1? Read on

Hi all,

If you have recently upgraded your VM Host to ESX4.0, 4.1 or 5.0 you might want to run this code (after installing the vmware powershell extensions, this is a powershell script after all):

function reportchangetracking{

$vm_name = Get-VM -location [cluster_name] | get-view

$vmConfigSpec = New-Object VMware.Vim.VirtualMachineConfigSpec

foreach ($objitem in $vm_name) {

#Write-host $objitem.name, $objitem.config.changetrackingenabled

if($objitem.config.changetrackingenabled -neq "True"){

$vmConfigSpec.changeTrackingEnabled = $true

$vmView.ReconfigVM($vmConfigSpec)

#punch it Chewy (reloads config spec. better than having to shut the machine down)

sleep 3

Get-VM $objitem.name | New-Snapshot -Name "Temp"

sleep 5

Get-VM $objitem.name | Get-Snapshot | Where {$_.Name -eq "Temp"} | Remove-Snapshot -Confirm:$false

}

}

}

In ESX3.5, VMTools never quiesced the base disk properly when taking snapshots. This meant that when Windows Server 2008 came along with VSS, the tools didn't use it, and 3.5 never added in the setting the code does above.

When you upgraded your host, it never added this setting at the VM level, so this task needs to be done manually. If you have a lot of machines, headaches ensue. After running the script, it might be an idea to update the tools installation on your server anyway.

Note that you do not have to run this if you created a machine on ESX4.0 or greater. Its been done for you.

Use this script at your own risk. It's no fault of mine if you break something.

Thanks for reading,

Trev