SOA version mangement

I was looking for a way to not write this whole report, and fortunately, Google found this report for me. Yay! I didn't really want to write this stuff, so I'm glad I don't have to, now.

The title of "Best Practices for Artifact Versioning in Service-Oriented Systems" is pretty accurate. For the TIBCO geeks, this is required knowledge for the (deprecated) TIBCO SOA Architect qualification.

http://www.sei.cmu.edu/library/abstracts/reports/11tn009.cfm

File upload and copy pattern

I was asked a different question; but then ended up with this instead. Shame to waste it...

The pattern that I've used in the past has directories like this (under some root directory like /var/lib/uploads):

/partial
/ready
/working
/success
/error

This set of directories are required for all traffic to a given recipient. Essentially, a file makes its way through the directories, from top to bottom. All directories should be on a single file system. File uploaders/clients should be able to write to /partial and /ready. File receivers/servers should be able to write to everything, except /partial.

Step 1: A file is uploaded/copied into the /partial directory; with a (globally) unique file name. This step completes when there's sufficient confidence that the file has been copied (usually that just means that the expected number of bytes has been written without an error being thrown).
Step 2:  The file uploader/client moves the newly uploaded file into the /ready directory. DO NOT COPY THE FILE!!!! In general, moving/renaming a file within a file system is guaranteed to be an atomic operation, but copying is not. This signifies that the file is ready (from the perspective of the client).
Step 3: When the recipient application/process is ready to process a file in /ready, it should first move the file into its /working directory.
Step 4: When the recipient application has finished processing a file (due to completion, or error) it should move the file into the /success, or /error, directory.

Things to watch
- more than one file in the working directory is likely to indicate a failure.
- Any file in the error directory is likely to indicate a failure.
- "old" files in the partial directory indicate unsuccessful copies/uploads.
- "old" files in the ready directory indicate that processing has failed/slowed.
- "old" files in the working directory indicate a failure/ABEND.
- Make sure the file system doesn't fill.
- Archiving is not covered here; that's a different pattern.

Other notes;
- If there is more than one recipient (e.g.: a multi-process server) there should be multiple working directories (working01 working02, working03, etc. - one for each process.
- this is essentially a queue implementation with single phase commit transactions.
- You can implement exactly the same pattern using file renaming, rather than separate directories. I prefer directories.

Google Spanner reflections

Google Spanner (warning: PDF) has made some noise recently, and I got around to giving their paper a quick read. I was happy to discover that it describes a solution to a problem I've been thinking about for awhile. I was happy to read that my musing were heading in the right direction. The key (as the Googlers are at pains to point out, and I was also considering) is how to handle distributed co-ordination of commits (in a scalable manner).

Spanner introduces the concept of tracking time by a combination of instants, intervals, and assertions about the two - on the understanding that there's uncertainty of the "current" time. As I read through the paper, I tend to think that there's no particular need for atomic and GPS clocks (although they're definitely desirable); so long as the confidence in the current time can be established. The effect of rising uncertainty in the current time would be a reduction in performance, I imagine...

It turns out that the venerable NTP protocol is able to provide measures of uncertainty; so one could re-implement Spanner without atomic clocks (at least, initially), and could then assault the various questions of electing "leaders" and manging reads and writes in a scalable manner.

Apache Zookeeper is the first thought for the leader election, but they talk about clusters of 4 machines being a large deployment, and about "within this data centre", etc. Let's reserve Zookeeper for a use case that fits better; something without Spanner's global ambitions. Spanner's architecture means that this is a key real concern. A closer reading of the paper is necessary to figure out what's involved here; and how it intimately it's related to uncertain timing.

The distributed reads and writes problem does look a bit like Ceph...

Spanner may not be so intractable, after all.

VMware player 4.0.3 on Ubuntu 12.04

VMware player has some well-known issues with setting up the network for player under Ubuntu 12.04 (that is, where Ubuntu is the host OS).

With one small change, I followed the great instructions here: http://www.kartook.com/2012/05/vmware-virtual-network-device-unable-to-loadcompile-vm-player-4-0-2-in-ubuntu-12-04/

The change is that the script you download has the recognised version numbers hardcoded in the header.

Before you run the script (patch-modules_3.2.0.sh), open the file using your preferred text editor, and change the 4.0.2 to 4.0.3, so it looks like:

   plreqver=4.0.3

The script (and VMware player) seems to run fine for VMware Player 4.0.3 on Ubuntu 12.04. If it should make a difference, I'm running x86_64 Ubuntu 12.04.

Empty an Oracle schema, leave an empty schema

What it says on the tin. Sufficient code analysis will reveal that some bizarre schema dependencies may not be destroyed by this; but experience will show that's not actually a problem :-)

The code lives as a gist over on github. A snapshot is below, for simplicity:

set echo off
set verify off
set serveroutput on size 100000

-- Hosted at http://lastinfinitetentacle.blogspot.com/2012/03/empty-oracle-schema-leave-empty-schema.html

-- Disable all contraints
BEGIN
  FOR c IN
  (SELECT c.owner, c.table_name, c.constraint_name
   FROM user_constraints c, user_tables t
   WHERE c.table_name = t.table_name
   AND c.status = 'ENABLED'
   ORDER BY c.constraint_type DESC)
  LOOP
    dbms_utility.exec_ddl_statement('alter table ' || c.owner || '.' || c.table_name || ' disable constraint ' || c.constraint_name);
  END LOOP;
END;
/

-- remove all objects
declare

cursor dropObjectsCusor is
select 'drop ' || object_type || ' ' || object_name as sqlDropStmt
from user_objects
where object_type <> 'TABLE' and object_type <> 'INDEX'
order by object_type;

cursor dropTablesCusor is
select 'truncate table ' || object_name as sqlTruncTbl,
'drop table ' || object_name || ' cascade constraints' as sqlDropTbl
from user_objects
where object_type = 'TABLE'
order by object_type;

begin

for ob in dropTablesCusor
loop
begin
  execute immediate ob.sqlTruncTbl;
exception when others then dbms_output.put_line('Could not truncate a table.');
end;
begin
  execute immediate ob.sqlDropTbl;
exception when others then dbms_output.put_line('Could not drop a table.');
end;
end loop;

for ob in dropObjectsCusor
loop
begin
  execute immediate ob.sqlDropStmt;
exception when others then dbms_output.put_line('Could not drop some object.');
end;
end loop;


end;
/
purge recyclebin;

CSV parser in JavaScript

I wanted a JavaScript based CSV parser. I couldn't find a good one (that would allow multi-character delimiters, and text qualifiers), so I wrote it. The textToArray function should run under any js implementation; but the demonstration of it is written for a ringojs environment.

Call textToArray with a line of delimited text, and the delimiter and text qualifier you're expecting to find; it will give back an array of the elements it finds.

var textToArray = function (txtLine, del, txtQual) {
    "use strict";
    var datArr = [], newStr = "";
    while (txtLine.length > 0) {
        if (txtLine.substr(0, txtQual.length) === txtQual) {
            // get quoted block
            newStr = txtLine.substr(0, txtQual.length + txtLine.indexOf(txtQual, txtQual.length));
            datArr.push(newStr.substr(txtQual.length, newStr.length - txtQual.length * 2));
        }
        else {
            // get data block
            if (txtLine.indexOf(del) !== -1) {
                newStr = txtLine.substr(0, txtLine.indexOf(del));
            } else {
                newStr = txtLine;
            }
            datArr.push(newStr);
        }
        txtLine = txtLine.substr(newStr.length + del.length, txtLine.length);
    }
    return datArr;
};


var fs = require('fs');
var con = require('console');
var del = ";;";
var txtQual = "\"\""; // a pair of quotes.
var file = fs.open('D:/ringojs-0.8/test.txt');
var line = "";
for (line in file) {
    con.log(textToArray(line, del, txtQual).join("---"));
}

Sane coding templates for javascript

It's just craziness. Apparently it's possible to treat JavaScript like a real programming language. Very exciting stuff, given that it seems to be becoming the ethernet of programming languages - despite many and varied shortcomings it's versatility, ubiquity, and low barriers to entry are likely to ensure it sees off any competitors.

Here ( http://www.crockford.com/javascript/private.html ) is a handy clip and keep guide of how to implement common object orientation patterns. James Crockford's excellent musing continue through to inheritance too, http://javascript.crockford.com/inheritance.html

Make no mistake, I don't love Javascript, but it's got a lot of momentum. We (the technically involved population) should embrace patterns of Javascript use that will make it sustainable into the future. We need to embrace the sane subset of the language's use.

node.js is good; mongodb and couchdb are stubbornly continuing to exist, and web browsers (whether on smart phones, or desktops) aren't going anywhere. Javascript is a fact, get used to it.

Should I buy a new graphics card?

I entered into a "more detailed than necessary" examination of this. The basic questions are:

• What will it cost?
• What sort of improvement to performance can I expect (can I play Shogun 2)?
• Will any power savings pay for some new card?

Given my current rig is a pair of nVidia 8800 GT (2 x 512MB) in an SLI config; and Shogun 2 reports graphics requirements as "AMD Radeon HD 5000 and 6000 series graphics cards or equivalent" (pcgamer.com), I had no idea. I went looking for benchmarks, and couldn't find everything (my old/current  rig, and some equivalent to the Radeons) on the same page.

I compiled some stats from previous years, did some (very) rough math and came up with this worksheet:
GPU decision worksheet. It turns out that my pair of 8800GT offers identical performance to a (more) modern GeForce GTX 460 (1GB RAM), which offers sufficient performance to play Shogun 2.

The upshot of this? I can play Shogun 2 on my current rig, and it's very unlikely that I can recover the cost of the new GPU through power savings. No new kit for me :-(

How to build a (modern) Windows Server

http://jeremywaldrop.wordpress.com/2008/10/28/how-to-build-a-windows-2008-vmware-esx-vm-template/

Excellent. I think some of the decisions are little odd (turn off the firewall?!?), but this covers a lot of ground.

The cloud - public, private, and the appliances within

I talked to a sales guy the other day, one of the better ones. Not only was he forthcoming, helpful, and charming, he was knowledgable and experienced; a rare breed. I had been out of this vendor's loop for a bit and asked what was going on; he mentioned all kinds of things he thought I'd be interested in, including a big cloud push for the vendor's software. Great. I segued into software appliances and he told me they were dead. Not limping, not re-purposed - DEAD.

From a techie's point of view (which is, to be fair, not the same as a salesman's) I see these as points along a continuum, or at least pre-requisite. "The Cloud" is all about ecomomies of scale, and industrialisation of information technology. In order to achieve the desired outcomes from The Cloud, standardisation and mechanisation are pre-requisites. A software applicance is a standardised "unit" (when properly built), an abstraction of an underlying mechanisation, it's the thing you want in The Cloud.

Amazon knows this, rPath knows this, VMware knows this, and Google has been doing this implicitly (as have all big web shops) since its inception; but the commentariat seem to have completely missed it, the analysts have missed it, and most of the vendors have missed it. It's a joke.

The Cloud is only a cloud while the hard bits are hidden. When the hard bits start peeking out, The Cloud becomes The Mess. When the hard bits get hidden, you're running someone else's software, on someone else's hardware (or your own - pick a public or private cloud as suits you), and we all become a lot more happier with the result.

The Cloud == Appliances.