This is a great little blog post on how to automatically fill out templates for system configuration. What it lacks in 'completeness', it makes up for in flexibility and simplicity. Sadly, it's Ruby-based, but that's ok on my current project.
http://findingscience.com/linux/sysadmin/ruby/2010/10/27/config-template-class.html
Thursday, July 10, 2014
Tuesday, June 03, 2014
Oracle JDK on Linux, done right!
http://d.stavrovski.net/blog/post/how-to-install-and-setup-oracle-java-jdk-in-centos-6
Wooo! There's a way forward.
Wooo! There's a way forward.
Thursday, August 22, 2013
avconv doesn't love my webcam :(
I had a way to get a view of my DCS-932L, but now it's broken! Somone's gone and fixed (or introduced) a bug into the Ubuntu 12.04 build of avconv, and it's dead in the water. The IP cam didn't change, and the Ubuntu machine is updated regularly, so I'm calling it Ubuntu's change. The output from the DCS-932L is buggy over any interface, so I suspect the original fault lies there, but.. Well, I just want the picture back.
The original article is here, and I'll update it when I get to it...
Bug reporting time.
Tuesday, August 06, 2013
Clock synchronisation approaches
Further to my earlier thoughts about Spanner.
http://krzyzanowski.org/rutgers/notes/pdf/06-clocks.pdf
Saturday, July 06, 2013
Shared directories in Ubuntu using ACLs
A nifty article on how to use ACLs in Ubuntu to share directories effectively between user accounts.
http://brunogirin.blogspot.com.au/2010/03/shared-folders-in-ubuntu-with-setgid.html
I read in another blog on the way to the above link, that Ubuntu has a fairly prescriptive user permissions model, and I just wish they'd spell out what it is... Instead, users and administrators are left to pull together blogs, like this...
http://brunogirin.blogspot.com.au/2010/03/shared-folders-in-ubuntu-with-setgid.html
I read in another blog on the way to the above link, that Ubuntu has a fairly prescriptive user permissions model, and I just wish they'd spell out what it is... Instead, users and administrators are left to pull together blogs, like this...
Tuesday, July 02, 2013
Yahoo Pipe to Google maps integration
Yahoo Maps are a bit rubbish, so I looked for a way to easily pull together a map I don't dislike using.
The map's here; the original data's here; and the Yahoo pipe between is here.
This map is a handy reference to the Victorian state government's public register of convictions for food safety offences. It's also a nice example of how cloud services can be integrated and mixed, and matched - Yahoo does provide a map interface to this data, but their maps are terrible.
View Larger Map
The map's here; the original data's here; and the Yahoo pipe between is here.
This map is a handy reference to the Victorian state government's public register of convictions for food safety offences. It's also a nice example of how cloud services can be integrated and mixed, and matched - Yahoo does provide a map interface to this data, but their maps are terrible.
View Larger Map
Thursday, June 27, 2013
Viewing D-Link DCS-932L IP camera from Ubuntu
In a word (line) ...
avconv -r 15 -f mjpeg -i http://USERNAME:PASSWORD@IP-ADDRESS/video.cgi -i http://USERNAME:PASSWORD@IP-ADDRESS/audio.cgi -vcodec mpeg4 -f mpegts file:///dev/stdout | vlc file:///dev/stdin
Now, if only the audio didn't crash avconv, we'd be cooking with gas! The silent version is here, and seems stable.
avconv -r 15 -f mjpeg -i http://USERNAME:PASSWORD@IP-ADDRESS/video.cgi -vcodec mpeg4 -f mpegts file:///dev/stdout | vlc file:///dev/stdin
avconv -r 15 -f mjpeg -i http://USERNAME:PASSWORD@IP-ADDRESS/video.cgi -i http://USERNAME:PASSWORD@IP-ADDRESS/audio.cgi -vcodec mpeg4 -f mpegts file:///dev/stdout | vlc file:///dev/stdin
Now, if only the audio didn't crash avconv, we'd be cooking with gas! The silent version is here, and seems stable.
avconv -r 15 -f mjpeg -i http://USERNAME:PASSWORD@IP-ADDRESS/video.cgi -vcodec mpeg4 -f mpegts file:///dev/stdout | vlc file:///dev/stdin
Tuesday, February 26, 2013
SOA version mangement
I was looking for a way to not write this whole report, and fortunately, Google found this report for me. Yay! I didn't really want to write this stuff, so I'm glad I don't have to, now.
The title of "Best Practices for Artifact Versioning in Service-Oriented Systems" is pretty accurate. For the TIBCO geeks, this is required knowledge for the (deprecated) TIBCO SOA Architect qualification.
http://www.sei.cmu.edu/library/abstracts/reports/11tn009.cfm
The title of "Best Practices for Artifact Versioning in Service-Oriented Systems" is pretty accurate. For the TIBCO geeks, this is required knowledge for the (deprecated) TIBCO SOA Architect qualification.
http://www.sei.cmu.edu/library/abstracts/reports/11tn009.cfm
Wednesday, February 06, 2013
File upload and copy pattern
I was asked a different question; but then ended up with this instead. Shame to waste it...
The pattern that I've used in the past has directories like this (under some root directory like /var/lib/uploads):
/partial
/ready
/working
/success
/error
This set of directories are required for all traffic to a given recipient. Essentially, a file makes its way through the directories, from top to bottom. All directories should be on a single file system. File uploaders/clients should be able to write to /partial and /ready. File receivers/servers should be able to write to everything, except /partial.
Step 1: A file is uploaded/copied into the /partial directory; with a (globally) unique file name. This step completes when there's sufficient confidence that the file has been copied (usually that just means that the expected number of bytes has been written without an error being thrown).
Step 2: The file uploader/client moves the newly uploaded file into the /ready directory. DO NOT COPY THE FILE!!!! In general, moving/renaming a file within a file system is guaranteed to be an atomic operation, but copying is not. This signifies that the file is ready (from the perspective of the client).
Step 3: When the recipient application/process is ready to process a file in /ready, it should first move the file into its /working directory.
Step 4: When the recipient application has finished processing a file (due to completion, or error) it should move the file into the /success, or /error, directory.
Things to watch
- more than one file in the working directory is likely to indicate a failure.
- Any file in the error directory is likely to indicate a failure.
- "old" files in the partial directory indicate unsuccessful copies/uploads.
- "old" files in the ready directory indicate that processing has failed/slowed.
- "old" files in the working directory indicate a failure/ABEND.
- Make sure the file system doesn't fill.
- Archiving is not covered here; that's a different pattern.
Other notes;
- If there is more than one recipient (e.g.: a multi-process server) there should be multiple working directories (working01 working02, working03, etc. - one for each process.
- this is essentially a queue implementation with single phase commit transactions.
- You can implement exactly the same pattern using file renaming, rather than separate directories. I prefer directories.
The pattern that I've used in the past has directories like this (under some root directory like /var/lib/uploads):
/partial
/ready
/working
/success
/error
This set of directories are required for all traffic to a given recipient. Essentially, a file makes its way through the directories, from top to bottom. All directories should be on a single file system. File uploaders/clients should be able to write to /partial and /ready. File receivers/servers should be able to write to everything, except /partial.
Step 1: A file is uploaded/copied into the /partial directory; with a (globally) unique file name. This step completes when there's sufficient confidence that the file has been copied (usually that just means that the expected number of bytes has been written without an error being thrown).
Step 2: The file uploader/client moves the newly uploaded file into the /ready directory. DO NOT COPY THE FILE!!!! In general, moving/renaming a file within a file system is guaranteed to be an atomic operation, but copying is not. This signifies that the file is ready (from the perspective of the client).
Step 3: When the recipient application/process is ready to process a file in /ready, it should first move the file into its /working directory.
Step 4: When the recipient application has finished processing a file (due to completion, or error) it should move the file into the /success, or /error, directory.
Things to watch
- more than one file in the working directory is likely to indicate a failure.
- Any file in the error directory is likely to indicate a failure.
- "old" files in the partial directory indicate unsuccessful copies/uploads.
- "old" files in the ready directory indicate that processing has failed/slowed.
- "old" files in the working directory indicate a failure/ABEND.
- Make sure the file system doesn't fill.
- Archiving is not covered here; that's a different pattern.
Other notes;
- If there is more than one recipient (e.g.: a multi-process server) there should be multiple working directories (working01 working02, working03, etc. - one for each process.
- this is essentially a queue implementation with single phase commit transactions.
- You can implement exactly the same pattern using file renaming, rather than separate directories. I prefer directories.
Tuesday, November 27, 2012
Google Spanner reflections
Google Spanner (warning: PDF) has made some noise recently, and I got around to giving their paper a quick read. I was happy to discover that it describes a solution to a problem I've been thinking about for awhile. I was happy to read that my musing were heading in the right direction. The key (as the Googlers are at pains to point out, and I was also considering) is how to handle distributed co-ordination of commits (in a scalable manner).
Spanner introduces the concept of tracking time by a combination of instants, intervals, and assertions about the two - on the understanding that there's uncertainty of the "current" time. As I read through the paper, I tend to think that there's no particular need for atomic and GPS clocks (although they're definitely desirable); so long as the confidence in the current time can be established. The effect of rising uncertainty in the current time would be a reduction in performance, I imagine...
It turns out that the venerable NTP protocol is able to provide measures of uncertainty; so one could re-implement Spanner without atomic clocks (at least, initially), and could then assault the various questions of electing "leaders" and manging reads and writes in a scalable manner.
Apache Zookeeper is the first thought for the leader election, but they talk about clusters of 4 machines being a large deployment, and about "within this data centre", etc. Let's reserve Zookeeper for a use case that fits better; something without Spanner's global ambitions. Spanner's architecture means that this is a key real concern. A closer reading of the paper is necessary to figure out what's involved here; and how it intimately it's related to uncertain timing.
The distributed reads and writes problem does look a bit like Ceph...
Spanner may not be so intractable, after all.
Spanner introduces the concept of tracking time by a combination of instants, intervals, and assertions about the two - on the understanding that there's uncertainty of the "current" time. As I read through the paper, I tend to think that there's no particular need for atomic and GPS clocks (although they're definitely desirable); so long as the confidence in the current time can be established. The effect of rising uncertainty in the current time would be a reduction in performance, I imagine...
It turns out that the venerable NTP protocol is able to provide measures of uncertainty; so one could re-implement Spanner without atomic clocks (at least, initially), and could then assault the various questions of electing "leaders" and manging reads and writes in a scalable manner.
Apache Zookeeper is the first thought for the leader election, but they talk about clusters of 4 machines being a large deployment, and about "within this data centre", etc. Let's reserve Zookeeper for a use case that fits better; something without Spanner's global ambitions. Spanner's architecture means that this is a key real concern. A closer reading of the paper is necessary to figure out what's involved here; and how it intimately it's related to uncertain timing.
The distributed reads and writes problem does look a bit like Ceph...
Spanner may not be so intractable, after all.
Subscribe to:
Posts (Atom)