A forking tiny web server

This is the uncommonly low denominator in HTTP servers. It does very little, and conforms to HTTP spec in only the most absurdly rudimentary way. Clearly, it's very insecure and poorly coded.

This toy HTTP server is supposed to be used as a tool of last resort. It's implemented in perl because perl version 5 remains ubiquitous - it comes with git on Windows!?! When you need a web server to have a particular behavior, to test a scenario, to break a client in just the right way, it's perfect - all it requires is more code. About the only thing it abstracts is raw socket handling (a bit - thanks perl), and the actual implementation of fork().

However, it's small enough to type in if you're in one of those annoyingly secure sites where you can't just download random rubbish from the Internet and execute it (where's the excitement in that?).

Features:

• Serves multiple HTTP clients simultaneously
• Will run in the most constrained environments (like ancient Unixes, or Visual Studio with git).
• Simple enough to be easily re-written for different test scenarios.
• Implements enough of the HTTP spec that curl won't complain.
• It's moderately secure (because it doesn't do much).
• No bugs (because it doesn't claim to do much)
• Is small enough to be typed in on a coffee break.
• Does some logging
• Serves up 10 GiB fast enough to flood some networks and crash some clients.
• Has enough problems that everyone can find something to fix
• Supports CPUs (more than one!)
• Has comments(? - ok, I'm stretching here)

	#!/usr/bin/env perl
	# forking tiny HTTP server
	# a toy web server
	# thanks to
	# keiya_21@yahoo.co.jp - https://gist.githubusercontent.com/keiya/2782414/raw/811565ccfa479a9e12d0653e2f5118e68e7fda37/server.pl
	use strict;
	use warnings;
	use IO::Socket::INET;
	use Socket qw(SOL_SOCKET);
	$| = 1;
	my $local_port = 9000;
	my $maximum_request_size = 65535; # any HTTP request bigger than this will be silently truncated.
	my $preferred_HTTP_body_size=1*(1000**3); # 10GiB is 10*(1024**3)
	# probably leave these alone
	my $sysread_size = 65535 * 8; # the size of the blob to try and read from the socket on each attempt. 65535 is the the socket buffer size on the machine I wrote this
	my $syswrite_size = $sysread_size; # the size to write... defaults to the same as the read size.
	sub get_more_body {
	my ($offset, $chunk_size, $preferred_total) = @_;
	if ($chunk_size > $preferred_total - $offset) {
	$chunk_size = $preferred_total - $offset
	}
	return('0' x $chunk_size);
	}
	#actual program begins
	my $sock_receive = IO::Socket::INET->new(LocalPort => $local_port, Proto => 'tcp', Listen => SOMAXCONN)
	or die "Cannot create socket: $@";
	# so we can restart our server quickly
	$sock_receive->setsockopt(SOL_SOCKET, SO_REUSEADDR, 1) or
	die "setsockopt: $!";
	print '['.$$.']: Started parent process on port '.$local_port."\n";
	my $sock_client;
	while($sock_client = $sock_receive->accept()) {
	print '['.$$.']: Connection from: '.$sock_client->peerhost().':'.$sock_client->peerport()."\n";
	if (my $pid = fork()){
	$sock_client->close();
	next;
	} else {
	print '['.$$."]: Started child process\n";
	# fiddle about with autoflush after https://perldoc.perl.org/functions/select.html
	my $old_handle = select $sock_client;
	$| = 1;
	select $old_handle;
	# read the request
	my ($this_read, $rv, $receive_buf);
	read_loop: {
	do {
	$rv = sysread($sock_client, $this_read, $sysread_size); # might overrun by 65534
	$receive_buf .= $this_read;
	if (!defined($rv)) {
	print '['.$$."]: Error reading from socket: $!\n";
	last read_loop; # it's ok to continue with a broken request - this is a toy.
	}
	# "parse" the HTTP header.
	last read_loop if $receive_buf =~ /^[A-Z]+[[:space:]].*\r\n\r\n/sm; # break read_loop
	} while ($rv && length($receive_buf) < 1 + $maximum_request_size );
	}
	# do something clever. Nope - I'll do nothing
	# send a reply
	my ($http_overhead, $send_buf);
	my $header = "HTTP/1.0 200 OK\r\nContent-Length: $preferred_HTTP_body_size\r\nConnection: Close\r\n\r\n";
	my $header_size = length($header);
	my $total_sent_size = 0;
	write_loop: {
	do {
	if ($total_sent_size < $header_size) {
	$send_buf = substr($header, $total_sent_size);
	}
	else {
	$send_buf = '';
	}
	$send_buf .= get_more_body($total_sent_size - $header_size, $syswrite_size, $preferred_HTTP_body_size);
	$rv = syswrite($sock_client, $send_buf, $syswrite_size);
	if (!defined($rv)) {
	print '['.$$."]: Error writing to socket: $!\n";
	last write_loop;
	}
	$total_sent_size += $rv;
	} while ($rv && $total_sent_size < ($header_size + $preferred_HTTP_body_size));
	}
	$sock_client->close();
	print '['.$$."]: Disconnect, exited child process.\n";
	exit;
	}
	}
	__END__

view raw forking-tiny-web-server.pl hosted with ❤ by GitHub

	#!/usr/bin/perl
	# nonforker - server who multiplexes without forking
	use strict;
	use warnings;
	use POSIX;
	use IO::Socket;
	use IO::Select;
	use Socket;
	use Fcntl;
	use Tie::RefHash;
	my $port = 1685; # change this at will
	# Listen to port.
	my $server = IO::Socket::INET->new(LocalPort => $port, Proto => 'tcp', Listen => SOMAXCONN)
	or die "Can't make server socket: $@\n";
	# begin with empty buffers
	my %inbuffer = ();
	my %outsent = ();
	my %ready = ();
	tie %ready, 'Tie::RefHash';
	my $preferred_HTTP_body_size = 10*(1024**3);
	my $send_buffer_size = 1024**3;
	my $header = "HTTP/1.0 200 OK\r\nContent-Length: $preferred_HTTP_body_size\r\nConnection: Close\r\n\r\n";
	my $header_size = length($header);
	my $total_response_size = $header_size + $preferred_HTTP_body_size;
	my $send_buf = '';
	nonblock($server);
	my $select = IO::Select->new($server);
	warn 'Started listening on port '.$port."\n";
	# Main loop: check reads/accepts, check writes, check ready to process
	while (1) {
	my ($client, $rv, $data);
	# check for new information on the connections we have
	# anything to read or accept?
	foreach $client ($select->can_read(1)) {
	if ($client == $server) {
	# accept a new connection
	$client = $server->accept();
	warn "Client $client connection from: ".$client->peerhost().':'.$client->peerport()."\n";
	$select->add($client);
	nonblock($client);
	} else {
	# read data
	$data = '';
	$rv = $client->recv($data, POSIX::BUFSIZ, 0);
	unless (defined($rv) && length $data) {
	# This would be the end of file, so close the client
	delete $inbuffer{$client};
	delete $outsent{$client};
	delete $ready{$client};
	$select->remove($client);
	close $client;
	warn "Client $client disconnected\n";
	next;
	}
	$inbuffer{$client} .= $data;
	# test whether the data in the buffer or the data we
	# just read means there is a complete request waiting
	# to be fulfilled. If there is, set $ready{$client}
	# to the requests waiting to be fulfilled.
	while ($inbuffer{$client} =~ s/^([A-Z]+[[:space:]].*\r\n\r\n).*$//sm) {
	push( @{$ready{$client}}, $1 );
	}
	}
	}
	# Any complete requests to process?
	foreach $client (keys %ready) {
	handle($client);
	}
	# Buffers to flush?
	foreach $client ($select->can_write(1)) {
	# Skip this client if we have nothing to say
	next unless exists $outsent{$client};
	if ($outsent{$client} < $header_size) {
	$send_buf = substr($header, $outsent{$client},$header_size - $outsent{$client});
	}
	else {
	$send_buf = '';
	}
	$send_buf .= get_more_body($outsent{$client}, $send_buffer_size, $total_response_size);
	$rv = $client->send($send_buf, 0);
	unless (defined $rv) {
	# Whine, but move on.
	warn "Client $client had error writing to socket: $!\n";
	next;
	}
	if ($rv <= $total_response_size || $! == POSIX::EWOULDBLOCK) {
	$outsent{$client} += $rv;
	delete $outsent{$client} if ($outsent{$client} >= $total_response_size);
	} else {
	# Couldn't write all the data, and it wasn't because
	# it would have blocked. Shutdown and move on.
	warn "Client $client disconnected. Error $!\n";
	delete $inbuffer{$client};
	delete $outsent{$client};
	delete $ready{$client};
	$select->remove($client);
	close($client);
	next;
	}
	}
	# Out of band data?
	foreach $client ($select->has_exception(0)) { # arg is timeout
	# Deal with out-of-band data here, if you want to.
	}
	}
	# handle($socket) deals with all pending requests for $client
	sub handle {
	# requests are in $ready{$client}
	# set output size to $outsent{$client}
	my $client = shift;
	my $request;
	foreach $request (@{$ready{$client}}) {
	# $request is the text of the request
	# put sent size (bytes) of reply into $outsent{$client}
	$outsent{$client} = 0;
	}
	delete $ready{$client};
	}
	# nonblock($socket) puts socket into nonblocking mode
	sub nonblock {
	my $socket = shift;
	my $flags;
	$flags = fcntl($socket, F_GETFL, 0)
	or die "Can't get flags for socket: $!\n";
	fcntl($socket, F_SETFL, $flags | O_NONBLOCK)
	or die "Can't make socket nonblocking: $!\n";
	}
	# create data chunk
	sub get_more_body {
	my ($offset, $chunk_size, $preferred_total) = @_;
	if ($chunk_size > $preferred_total - $offset) {
	$chunk_size = $preferred_total - $offset
	}
	return('0' x $chunk_size);
	}

view raw not-so-forked-web-server.pl hosted with ❤ by GitHub

VoltDB first steps - schema

The first, and most critical, step for working with VoltDB is figuring out what table(s) will be partitioned, and how. So, the schema...

The case I'm looking at is for a hypothetical ticket booking agency, for very large events (rock bands playing large stadiums, etc.). There's three basic tables:

1. seat_info
2. booking_info
3. booking_seats

seat_info is a list of all of the seats, and whatever metadata is required.
booking_info is a list of the bookings, so far, and whatever metadata is required.
booking_seats relates seats to bookings, and events.

booking_seats is the table that's interesting; it will be the largest, and hottest, by far. The schema elements for booking_seats is below:

create table booking_seats (
  booking_id varchar(10) not null,
  seat_number smallint not null,
  seat_row int not null,
  event_id int not null,
  constraint booking_seats_unique_hash unique (seat_number, seat_row, event_id)
);

partition table booking_seats on column seat_row;

create index event_booking_hash_idx on booking_seats (event_id);

booking_id is a foreign key for booking_info.
seat_number and seat_row form a composite key into the seat_info table.
event_id will be a foreign key relationship "in the future".

seat_row is used for partitioning the booking_seats table.

The VoltDB user manual has an example of a flight booking schema that's been partitioned by flight_id, analogous to event_id in my example.  The use case here is different, and so the partitioning scheme needs to be different.

• event_id would cause all of the traffic for the ticket booking system would be directed to a single partition when tickets for that event were put on sale - which sort of defeats the purpose of partitioning the data in the first place.
• booking_id could be used for partitioning, but as it's not a part of the unique constraint, the constraint would be expensive to enforce.
• seat_number isn't a good choice for partitioning because buying multiple tickets is normal, and each transaction should be kept within a single partition, if possible.
• seat_row is my choice for partitioning, because most bookings tend to be in the same row, and because it's a part of the unique constraint that prevents seats being double-booked.

In theory booking_id, or even event_id, may make a sound choice for partitioning. For this example I really would like to have aggressive partitioning of the data to see what breaks stress VoltDB's capabilities.

This is a "not bad" schema; but the query for finding available seats is a bit ugly. I added the index on event_id, to optimise the query for available seats for a given event, but testing is really essential to see if it helps or hinders performance overall.

Explorations of VoltDB

I've started on a little project to see how VoltDB lines up against its promises.

The Unique Selling Point (sort of) of VoltDB is that you can horiztonally scale an OLTP database, in a shared-nothing environment. That's quite a trick.

Horizontally-scalable OLAP systems are well-known, even passe; but getting an OLTP system that scales elegantly has been hard/impossible for a reasonable cost. Could you scale out an Oracle RAC? Sure, if you have a few million bucks for licences and hardware. Can you scale out MySQL? Sure, if you have an army of programmers willing and able to implement sharding in their applications.

The idea is that VoltDB takes the headache of managing sharding and scalable storage, from the children developers, and hands it to the crazy people DBAs.

VoltDB has a base price of Free, which I like, so I decided to give it a go. The idea is to come up with a little benchmark problem, to stress its ACIDity. My example case is for a ticketing agency, such as those types who manage major events like rock bands that will sell-out a 100,000 person stadium. At this point, I'm just mocking up a little data, and a few queries. I hope I'll get around to building a little erlvolt app to go with it.

Getting up and running was a breeze. The documentation and build is excellent in this respect, I can't fault it. I took the hardest possible route and compiled the code from scratch (on Ubuntu) without reading the documentation (which I've since read) - and it went brilliantly well, with an idiot-resistant build that coaxed me towards the correct answer when I got things wrong (generally missing dependencies which are "ant" and "g++").

For sane people, you can just register on their website, download a binary, and install it. There's an (oh-so-cool) docker image available, but I got bored of playing with that in less time than it took to download, so install it, it's easy.

I've found that by being badly behaved, I can crash the VoltDB server (and/or my browser) on my ancient little laptop. This worries me (only) a little. I have been pretty rough on the DB, throwing all kinds of broken rubbish at it, but a DB really does need to be bullet-proof. In terms of running queries, procedures, etc. voltdb just fine, and does what it says on the tin.

If anyone from VoltDB reads this, please address bug ENG-2526 - it forces developers to build misleading interfaces, and that's a really bad idea.

VoltDB looks promising, so far, and has all kinds of interesting features. I'd like to see VoltDB extend with more data types, and functions, and maybe referential integrity enforcement... and a pot of gold, and a pony...

So.... Google made a mess of your AdWords profile?

Google makes a mess of accounts, pretty regularly. With a reliability that comes with extensive automation, Google also ignores any and all appeals for intervention to fix busted accounts.

My problem: My Google account can do all kinds of stuff on Google, except manage the ads on my blog. I had to create a new account, exclusively to manage ads on my account.

Salvation came in the form of fiddling about in the AdWords/AdSense console. I poked about here, on the basis of "follow the money". While publishers provide content, which is nice; advertisers provide cash money; which is life itself, for Google. In a not-at-all-surprising twist, the console for placing ads (rather than hosting) has heaps of functions, is beautiful, and actually works! One of the AdWords functions is to allow multiple users for a given AdWords account, and additional users can be administrators! Woo!

To re-enable my account with AdSense/AdWords, I followed the following steps.:

1. If you haven't already, go make a new Google account.
2. If you haven't already, go make a blog. It may need a published post, but the content doesn't matter.
3. Login to http://adwords.google.com/ using your new account.
4. Go to the settings menu (under that little cog thing in the top-right corner of the screen).
5. Go to the "Access and Authorization" menu (listed down the left side of the screen).
6. Go to the "User Management" menu (listed down the left side of the screen).
7. Add the Email address for your broken GMail account. DO NOT check the "Administrator" checkbox, yet.
8. In your GMail account, open the email, and accept the invitation.
9. Back in the "User Management" menu (for the new Google account), you can now tick the "Administrator" checkbox new to your older, previously broken, Google account.

You'll need to have two browsers (not just windows, but separate browsers, like IE and Chrome) to make this easier, one for each Google account. You can do this with one browser, but there'll be a crazy amount of logout/login processes, and that gets boring, and can cause weird problems.

I hope this works for you. After this, I now have One Google account managing all of my blog stuff, including AdWords/AdSense; which is nice.

Adventures in Ruby and Capistrano

My my current project has seen a revisit of an old role - automating the deployment of distributed systems. Owing to the vagaries of consultancy, I found myself the developer and maintainer of a Ruby-themed deployment environment designed by someone else. I was thrown in the deep end, which is ok - the water was warm, and the current, gentle.

The (non-Ruby) components:
GitLab - centralised git repository management.
Jenkins - schedules, catalogues, and organises builds, and test runs.
Puppet - setup infrastructure and install underlying software. I'm aware that Puppet is built using Ruby, but as a user, this is well-hidden.

The Ruby bits:
rvm - The Ruby Version Manager. Installs and maintains the Ruby environment.
gem - a packaged up Ruby library. Analogous to a lightweight rpm, or (more of a stretch) msi file.
geminabox - server and caching proxy for gems built, and used.
bundle - a tool for managing gems.
rake - make in Ruby, can use gems.
Capistrano - a set of tools to do stuff (mostly rake) on other machines (via ssh).

Light touch configuration

This is a great little blog post on how to automatically fill out templates for system configuration. What it lacks in 'completeness', it makes up for in flexibility and simplicity. Sadly, it's Ruby-based, but that's ok on my current project.

http://findingscience.com/linux/sysadmin/ruby/2010/10/27/config-template-class.html

Oracle JDK on Linux, done right!

http://d.stavrovski.net/blog/post/how-to-install-and-setup-oracle-java-jdk-in-centos-6

Wooo! There's a way forward.

avconv doesn't love my webcam :(

I had a way to get a view of my DCS-932L, but now it's broken! Somone's gone and fixed (or introduced) a bug into the Ubuntu 12.04 build of avconv, and it's dead in the water. The IP cam didn't change, and the Ubuntu machine is updated regularly, so I'm calling it Ubuntu's change. The output from the DCS-932L is buggy over any interface, so I suspect the original fault lies there, but.. Well, I just want the picture back. The original article is here, and I'll update it when I get to it... Bug reporting time.

Clock synchronisation approaches

Further to my earlier thoughts about Spanner. http://krzyzanowski.org/rutgers/notes/pdf/06-clocks.pdf

Shared directories in Ubuntu using ACLs

A nifty article on how to use ACLs in Ubuntu to share directories effectively between user accounts.

http://brunogirin.blogspot.com.au/2010/03/shared-folders-in-ubuntu-with-setgid.html

I read in another blog on the way to the above link, that Ubuntu has a fairly prescriptive user permissions model, and I just wish they'd spell out what it is... Instead, users and administrators are left to pull together blogs, like this...