A readable, recent blog on big JVM heaps... They're bad, naughty, offer rubbish performance, but they're hard to dodge.
http://andrigoss.blogspot.com/2008/02/jvm-performance-tuning.html
Certainly, it's better than this effort from Tangosol/Coherence/Oracle ("Use our product!"). As it happens, I need to sort this for a badly built product based on Coherence... Oh! The pain!
Wednesday, October 15, 2008
Friday, June 06, 2008
Vista and VMWare network irritation
I've been working with VMWare a bit recently, on my machines. As both the machines I use are relatively new, they came with Vista. Vista has been irritating, but less so than a downgrade.
Vista wants to describe and categorize all the network interfaces. Unfortunately, that includes the virtual network adapters that VMWare sets up (and probably the same for Windows qemu users fiddling with openvpn for its TAP implementation); and it gets its knickers in a twist because they don't seem to lead anywhere. The solution for Vista (pre-SP1) was to take advantage of a bug and add bogus default gateways to the virtual interfaces. The solution for Vista SP1 is more involved, but at least seems more "proper".
Essentially, you need to hack the Windows registry and tell Vista that these interfaces aren't leading anywhere. This seems a bit odd, as there may be an extensive network infrastructure of VMs; but that's it.
Use the registry editor of your choice and follow this:
http://communities.vmware.com/message/916766#916766
Vista wants to describe and categorize all the network interfaces. Unfortunately, that includes the virtual network adapters that VMWare sets up (and probably the same for Windows qemu users fiddling with openvpn for its TAP implementation); and it gets its knickers in a twist because they don't seem to lead anywhere. The solution for Vista (pre-SP1) was to take advantage of a bug and add bogus default gateways to the virtual interfaces. The solution for Vista SP1 is more involved, but at least seems more "proper".
Essentially, you need to hack the Windows registry and tell Vista that these interfaces aren't leading anywhere. This seems a bit odd, as there may be an extensive network infrastructure of VMs; but that's it.
Use the registry editor of your choice and follow this:
http://communities.vmware.com/message/916766#916766
Wednesday, April 30, 2008
Thursday, April 24, 2008
Quick product review
I got a new laptop recently, an M1330 from Dell. I'd thought I throw out a few quick comments on the process.
Up to the minute technology says that the Penryn family of processors are excellent for laptops, with The Register suggesting that, alone, they can increase battery life significantly. In terms of graphics, the world of little laptops revolves around Intel's X3100 or nVidia's 8400M. The M1330 can have either - I went for the nVidia option as I like playing games now and again.
That left a choice of screen. I plumped for the more expensive OLED option after The Register's review. I noted that Dell seems to have responded to the review by raising the price of this option. RAM - 4GB (I could have saved £50 and blown my warranty by adding this myself, but out of £1000, why?). Hard disk - 250GB (it's enough for me).
The laptop runs nicely. Vista's a dog, but it's Vista. The graphics performance is better than I expected (TF2 runs well, except for the water effects). And the number crunching power is all I'd hoped for.
I tried Ubuntu 7.10 for a bit, and it ran well, but I hit issues with wireless networking and my home network, so restored Vista. I may try again once the new release settles back down. I have no love for Vista.
In a nutshell:
Dell M1330 8/10 - Solid.
Microsoft Vista 3/10 - Soft.
Up to the minute technology says that the Penryn family of processors are excellent for laptops, with The Register suggesting that, alone, they can increase battery life significantly. In terms of graphics, the world of little laptops revolves around Intel's X3100 or nVidia's 8400M. The M1330 can have either - I went for the nVidia option as I like playing games now and again.
That left a choice of screen. I plumped for the more expensive OLED option after The Register's review. I noted that Dell seems to have responded to the review by raising the price of this option. RAM - 4GB (I could have saved £50 and blown my warranty by adding this myself, but out of £1000, why?). Hard disk - 250GB (it's enough for me).
The laptop runs nicely. Vista's a dog, but it's Vista. The graphics performance is better than I expected (TF2 runs well, except for the water effects). And the number crunching power is all I'd hoped for.
I tried Ubuntu 7.10 for a bit, and it ran well, but I hit issues with wireless networking and my home network, so restored Vista. I may try again once the new release settles back down. I have no love for Vista.
In a nutshell:
Dell M1330 8/10 - Solid.
Microsoft Vista 3/10 - Soft.
Friday, April 18, 2008
USB dead device fix
I have a slug (NSLU2) at home, serving out a big USB hard disk. I like my slug. It's small, and it runs Debian, and so long as I remember that it's really small, it runs really well.
It looks like USB drives suffer from lots of issues with dodgey firmware and drivers. Certainly, I haven't escaped them. Every few months I lose the USB drive:
kern.log:
...
This can cause filesystem corruption as well, and is generally "A Bad Thing". Fixing this issue is (for me) a case of remounting the filesystem. I recall once I also had to cycle the power too. Fortunately, I found this page: http://www.mail-archive.com/linux-usb-users@lists.sourceforge.net/msg16510.html which has a very nifty technique for coaxing the USB stack back to life.
I've written a script to pull out the sysfs name of the USB device for a given mount point. At some point I'll add this in to my slug so that it "just works". That just leaves a log scraper to pick up each mount and get the sysfs name, wait for a failure, unmount the dead file system, rebind the USB device, check the file system for damage, and remount the filesystem. Easy!
It looks like USB drives suffer from lots of issues with dodgey firmware and drivers. Certainly, I haven't escaped them. Every few months I lose the USB drive:
kern.log:
Apr 12 20:32:54 slug kernel: scsi 1:0:0:0: rejecting I/O to dead device
...
This can cause filesystem corruption as well, and is generally "A Bad Thing". Fixing this issue is (for me) a case of remounting the filesystem. I recall once I also had to cycle the power too. Fortunately, I found this page: http://www.mail-archive.com/linux-usb-users@lists.sourceforge.net/msg16510.html which has a very nifty technique for coaxing the USB stack back to life.
I've written a script to pull out the sysfs name of the USB device for a given mount point. At some point I'll add this in to my slug so that it "just works". That just leaves a log scraper to pick up each mount and get the sysfs name, wait for a failure, unmount the dead file system, rebind the USB device, check the file system for damage, and remount the filesystem. Easy!
#!/bin/bash
# get_usb_name.sh
# e.g.: ./get_usb_name.sh /media/usb2
USB_DEV=$(mount | grep ${1} | cut -f 1 -d ' ' | sed -e 's/[0-9]$//' -e 's(^/dev/((')
USB_ADDR=$(ls -l /sys/block/${USB_DEV}/device | sed -e 's(/host.*$((' -e 's(^.*/((')
if [ -e /sys/bus/usb/drivers/usb-storage/${USB_ADDR} ]
then
echo $USB_ADDR
else
echo "could not verify usb address of ${USB_ADDR} for device ${USB_DEV}" >&2
fi
Sunday, March 09, 2008
Throw-away coding
I tinker with things a lot. There's an explosion of tiny projects that I've tried out once, and never really want to look back at, so this tip on how to have a project without saving it in VC++ is very welcome.
Coming to Eclipse? Already there? I can't find it if it is.
Personally, I always keep a scratch.txt on my desktop/home directory for keeping code snippets for re-use/reminders. Having a system that supports brief testing/fiddling is a great nod to the way (a lot of) people work.
Coming to Eclipse? Already there? I can't find it if it is.
Personally, I always keep a scratch.txt on my desktop/home directory for keeping code snippets for re-use/reminders. Having a system that supports brief testing/fiddling is a great nod to the way (a lot of) people work.
Wednesday, February 20, 2008
Devolo not-so-bad
My suspicions about the devolo kit being on the slow side have turned out to be partly true, but they were really the least of my problems. I took an hour or two to revive an ancient laptop (from when MMX instructions were something to brag about) to test different points around the network, configurations, etc. As old as the laptop is, it's quite capable of doing a few things, and it's reliable.
What I couldn't figure out was why my new PC was only able to move things over the LAN at 0.3Mb/s! It's almost modem type of speeds. Because of the way my flat is laid out it was awkward to isolate the new machine and find out the exact cause. I (literally) dusted off an ancient laptop, and pressd it into service.
After continuously rewiring most of my flat and retesting, it became clear that my new PC, all on it's lonesome, was the cause of the glacial network throughput. After some web searches, auto-scaling on vista was pointed at - no luck (didn't think so, there weren't routers in this path).
More web searches suggested NAM wasn't good. It took awhile longer to discover that NAM is Nvidia's "Network Access Manager", and it's rubbish. I uninstalled it.
Without the devolo kit or NAM in the loop, network throughput jumped up towards the expected speeds (100Mb/s). I've put the devolo kit back into it's original places, and it's not too quick - ~11Mb/s; but my best option right now (and thirty times faster than NAM was forcing me into).
During testing I found that the devolo kit, when installed in sensible places (no power strips, etc.) will get close to it's theoretical speeds. Happy days - if you have power sockets free in the right places. I'll make do.
What I couldn't figure out was why my new PC was only able to move things over the LAN at 0.3Mb/s! It's almost modem type of speeds. Because of the way my flat is laid out it was awkward to isolate the new machine and find out the exact cause. I (literally) dusted off an ancient laptop, and pressd it into service.
After continuously rewiring most of my flat and retesting, it became clear that my new PC, all on it's lonesome, was the cause of the glacial network throughput. After some web searches, auto-scaling on vista was pointed at - no luck (didn't think so, there weren't routers in this path).
More web searches suggested NAM wasn't good. It took awhile longer to discover that NAM is Nvidia's "Network Access Manager", and it's rubbish. I uninstalled it.
Without the devolo kit or NAM in the loop, network throughput jumped up towards the expected speeds (100Mb/s). I've put the devolo kit back into it's original places, and it's not too quick - ~11Mb/s; but my best option right now (and thirty times faster than NAM was forcing me into).
During testing I found that the devolo kit, when installed in sensible places (no power strips, etc.) will get close to it's theoretical speeds. Happy days - if you have power sockets free in the right places. I'll make do.
Monday, February 18, 2008
Perl threads :-(
I chucked in an entry for the Computer languages shootout game for the Perl implementation of a thread-ring. I've a few observations:
I'm going to see if libevent holds any hope for me. If not, then maybe a mass of file handles, and select statements...
- My application relies on semaphores that unblock themselves quickly. On average, there's two threads running (rather than blocking on a semaphore) at a time, but my dual core PC is only 60-80% utilised. Something in the cond_wait implementation is sleeping.
- Try as I might, semaphores were the only IPC mechanism I could get to work in a way that appeared "correct" (rather than spurious errors about shared or locked, or whatever). This may be a reflection on the workman rather than his tools.
- Getting any cross-platform, efficient, thread-safe, IPC across muliple versions of Perl, is like pulling teeth. I've got a blended semaphore-based implementation ready to go. By blended, I mean it uses (inefficient) semaphores for the tricky first lap around the ring (ensuring that all initialisation is done correctly), and then move over to "something" immediate for message passing. I need a working message passing implementation to slot in now.
- As the shootout game demonstrates, Perl threads are not even close to universally available in any sense of the term.
I'm going to see if libevent holds any hope for me. If not, then maybe a mass of file handles, and select statements...
Saturday, February 16, 2008
Devolo not-so-highspeed
Update on this item
The devolo "highspeed" homeplug gizmos are pretty good. There was a little buzz of excitement when the idea of power cables being - well, just power cables - suddenly lurched to the left by some distance and the idea of "just fat, slow cat5" snuck in next door.
The 85Mb/s claimed by Devolo isn't exactly electrifying (heh?), but that's ok, I only recently moved out my last 10Mb PC (100MBit still feels so fast). Anyway, it seems to move standard resolution movies (about 700MB or so) to the XBMC ok, and that's the one my girlfriend will be upset about. What the Devolo kit doesn't seem to is actually give me decent network throughput.
I did some testing using netcat - 6Mb/s! Ok, so the power/network isn't ideal (old wiring, too many power strips, etc.), but the "informer" app says I'm getting about 50Mb/s, so long as there's no actual traffic. If I actually ask the network to carry some traffic (as with netcat), the informer app says the devolo kit drops to 20Mb/s, while I'm only getting 6Mb useful throughput.
If I can improve the wiring setup (attach the devolo kit to it's own sockets rather than on power strips), I'll retest. So far though, in my little flat, with too few power sockets, in odd locations, the results aren't too encouraging. Do I currently feel like shelling out for the newer 200Mb/s model? Not really.
Maybe 802.11n is the next step - even with the thick walls here...
The devolo "highspeed" homeplug gizmos are pretty good. There was a little buzz of excitement when the idea of power cables being - well, just power cables - suddenly lurched to the left by some distance and the idea of "just fat, slow cat5" snuck in next door.
The 85Mb/s claimed by Devolo isn't exactly electrifying (heh?), but that's ok, I only recently moved out my last 10Mb PC (100MBit still feels so fast). Anyway, it seems to move standard resolution movies (about 700MB or so) to the XBMC ok, and that's the one my girlfriend will be upset about. What the Devolo kit doesn't seem to is actually give me decent network throughput.
I did some testing using netcat - 6Mb/s! Ok, so the power/network isn't ideal (old wiring, too many power strips, etc.), but the "informer" app says I'm getting about 50Mb/s, so long as there's no actual traffic. If I actually ask the network to carry some traffic (as with netcat), the informer app says the devolo kit drops to 20Mb/s, while I'm only getting 6Mb useful throughput.
If I can improve the wiring setup (attach the devolo kit to it's own sockets rather than on power strips), I'll retest. So far though, in my little flat, with too few power sockets, in odd locations, the results aren't too encouraging. Do I currently feel like shelling out for the newer 200Mb/s model? Not really.
Maybe 802.11n is the next step - even with the thick walls here...
Friday, February 15, 2008
SOA monitoring and discovery system
SOA is a great idea, taking chunks from EAI lessons learned, and generalising them. It occurs to me that tools for building maps of service invocations is necessary and "good". The question becomes, fairly quickly, "how?".
Asking developers and designers to be good citizens and update spreadsheets (or more sophisticated tools) doesn't work, because over a little time, the manual update system breaks down for lots of reasons (not least laziness). I suggest the only real option is to automate discovery of service invocations by watching the service interfaces themselves.
Point-to-point transports (usually TCP-based, like HTTP) make capturing traffic quite awkward, but the content can usually be extracted as it will pass through so many network elements (including the service's host machine) where a wire tap can be taken. If the communication is encrypted in-flight (as is expected for sensitive data), then an application level wiretap will be required.
BEA's AquaLogic took a very active role in this monitoring when I grappled with it; essentially as a proxy in the whole process. Maybe they made less intrusive options available in version 3... My first outing with AquaLogic didn't encourage a second try. All-in-all AquaLogic just seemed an order of magnitude too big and clunky.
I favour a distributed system of agents suitable for capturing each service invocation. Each agent could feed (largely unprocessed) data back to a cluster of analysis machines. At the end, I'd hope to see a bunch of query/display/integration tools to leverage (what would be in my experience) the only up-to-the-minute source of information about how the organisation actually hangs together.
A JMS wiretap (vendor specific, or fall-back to browsing the queues), could be implemented easily enough. For HTTP snooping, tcpdump's ok. For HTTPS, web server modules would be required (Apache, IIS?, ligHTTP?).
The analysis? Another day...
Asking developers and designers to be good citizens and update spreadsheets (or more sophisticated tools) doesn't work, because over a little time, the manual update system breaks down for lots of reasons (not least laziness). I suggest the only real option is to automate discovery of service invocations by watching the service interfaces themselves.
Point-to-point transports (usually TCP-based, like HTTP) make capturing traffic quite awkward, but the content can usually be extracted as it will pass through so many network elements (including the service's host machine) where a wire tap can be taken. If the communication is encrypted in-flight (as is expected for sensitive data), then an application level wiretap will be required.
BEA's AquaLogic took a very active role in this monitoring when I grappled with it; essentially as a proxy in the whole process. Maybe they made less intrusive options available in version 3... My first outing with AquaLogic didn't encourage a second try. All-in-all AquaLogic just seemed an order of magnitude too big and clunky.
I favour a distributed system of agents suitable for capturing each service invocation. Each agent could feed (largely unprocessed) data back to a cluster of analysis machines. At the end, I'd hope to see a bunch of query/display/integration tools to leverage (what would be in my experience) the only up-to-the-minute source of information about how the organisation actually hangs together.
A JMS wiretap (vendor specific, or fall-back to browsing the queues), could be implemented easily enough. For HTTP snooping, tcpdump's ok. For HTTPS, web server modules would be required (Apache, IIS?, ligHTTP?).
The analysis? Another day...
Subscribe to:
Posts (Atom)