I was looking for a way to not write this whole report, and fortunately, Google found this report for me. Yay! I didn't really want to write this stuff, so I'm glad I don't have to, now.
The title of "Best Practices for Artifact Versioning in Service-Oriented Systems" is pretty accurate. For the TIBCO geeks, this is required knowledge for the (deprecated) TIBCO SOA Architect qualification.
http://www.sei.cmu.edu/library/abstracts/reports/11tn009.cfm
Showing posts with label EAI. Show all posts
Showing posts with label EAI. Show all posts
Tuesday, February 26, 2013
Wednesday, February 06, 2013
File upload and copy pattern
I was asked a different question; but then ended up with this instead. Shame to waste it...
The pattern that I've used in the past has directories like this (under some root directory like /var/lib/uploads):
/partial
/ready
/working
/success
/error
This set of directories are required for all traffic to a given recipient. Essentially, a file makes its way through the directories, from top to bottom. All directories should be on a single file system. File uploaders/clients should be able to write to /partial and /ready. File receivers/servers should be able to write to everything, except /partial.
Step 1: A file is uploaded/copied into the /partial directory; with a (globally) unique file name. This step completes when there's sufficient confidence that the file has been copied (usually that just means that the expected number of bytes has been written without an error being thrown).
Step 2: The file uploader/client moves the newly uploaded file into the /ready directory. DO NOT COPY THE FILE!!!! In general, moving/renaming a file within a file system is guaranteed to be an atomic operation, but copying is not. This signifies that the file is ready (from the perspective of the client).
Step 3: When the recipient application/process is ready to process a file in /ready, it should first move the file into its /working directory.
Step 4: When the recipient application has finished processing a file (due to completion, or error) it should move the file into the /success, or /error, directory.
Things to watch
- more than one file in the working directory is likely to indicate a failure.
- Any file in the error directory is likely to indicate a failure.
- "old" files in the partial directory indicate unsuccessful copies/uploads.
- "old" files in the ready directory indicate that processing has failed/slowed.
- "old" files in the working directory indicate a failure/ABEND.
- Make sure the file system doesn't fill.
- Archiving is not covered here; that's a different pattern.
Other notes;
- If there is more than one recipient (e.g.: a multi-process server) there should be multiple working directories (working01 working02, working03, etc. - one for each process.
- this is essentially a queue implementation with single phase commit transactions.
- You can implement exactly the same pattern using file renaming, rather than separate directories. I prefer directories.
The pattern that I've used in the past has directories like this (under some root directory like /var/lib/uploads):
/partial
/ready
/working
/success
/error
This set of directories are required for all traffic to a given recipient. Essentially, a file makes its way through the directories, from top to bottom. All directories should be on a single file system. File uploaders/clients should be able to write to /partial and /ready. File receivers/servers should be able to write to everything, except /partial.
Step 1: A file is uploaded/copied into the /partial directory; with a (globally) unique file name. This step completes when there's sufficient confidence that the file has been copied (usually that just means that the expected number of bytes has been written without an error being thrown).
Step 2: The file uploader/client moves the newly uploaded file into the /ready directory. DO NOT COPY THE FILE!!!! In general, moving/renaming a file within a file system is guaranteed to be an atomic operation, but copying is not. This signifies that the file is ready (from the perspective of the client).
Step 3: When the recipient application/process is ready to process a file in /ready, it should first move the file into its /working directory.
Step 4: When the recipient application has finished processing a file (due to completion, or error) it should move the file into the /success, or /error, directory.
Things to watch
- more than one file in the working directory is likely to indicate a failure.
- Any file in the error directory is likely to indicate a failure.
- "old" files in the partial directory indicate unsuccessful copies/uploads.
- "old" files in the ready directory indicate that processing has failed/slowed.
- "old" files in the working directory indicate a failure/ABEND.
- Make sure the file system doesn't fill.
- Archiving is not covered here; that's a different pattern.
Other notes;
- If there is more than one recipient (e.g.: a multi-process server) there should be multiple working directories (working01 working02, working03, etc. - one for each process.
- this is essentially a queue implementation with single phase commit transactions.
- You can implement exactly the same pattern using file renaming, rather than separate directories. I prefer directories.
Friday, February 15, 2008
SOA monitoring and discovery system
SOA is a great idea, taking chunks from EAI lessons learned, and generalising them. It occurs to me that tools for building maps of service invocations is necessary and "good". The question becomes, fairly quickly, "how?".
Asking developers and designers to be good citizens and update spreadsheets (or more sophisticated tools) doesn't work, because over a little time, the manual update system breaks down for lots of reasons (not least laziness). I suggest the only real option is to automate discovery of service invocations by watching the service interfaces themselves.
Point-to-point transports (usually TCP-based, like HTTP) make capturing traffic quite awkward, but the content can usually be extracted as it will pass through so many network elements (including the service's host machine) where a wire tap can be taken. If the communication is encrypted in-flight (as is expected for sensitive data), then an application level wiretap will be required.
BEA's AquaLogic took a very active role in this monitoring when I grappled with it; essentially as a proxy in the whole process. Maybe they made less intrusive options available in version 3... My first outing with AquaLogic didn't encourage a second try. All-in-all AquaLogic just seemed an order of magnitude too big and clunky.
I favour a distributed system of agents suitable for capturing each service invocation. Each agent could feed (largely unprocessed) data back to a cluster of analysis machines. At the end, I'd hope to see a bunch of query/display/integration tools to leverage (what would be in my experience) the only up-to-the-minute source of information about how the organisation actually hangs together.
A JMS wiretap (vendor specific, or fall-back to browsing the queues), could be implemented easily enough. For HTTP snooping, tcpdump's ok. For HTTPS, web server modules would be required (Apache, IIS?, ligHTTP?).
The analysis? Another day...
Asking developers and designers to be good citizens and update spreadsheets (or more sophisticated tools) doesn't work, because over a little time, the manual update system breaks down for lots of reasons (not least laziness). I suggest the only real option is to automate discovery of service invocations by watching the service interfaces themselves.
Point-to-point transports (usually TCP-based, like HTTP) make capturing traffic quite awkward, but the content can usually be extracted as it will pass through so many network elements (including the service's host machine) where a wire tap can be taken. If the communication is encrypted in-flight (as is expected for sensitive data), then an application level wiretap will be required.
BEA's AquaLogic took a very active role in this monitoring when I grappled with it; essentially as a proxy in the whole process. Maybe they made less intrusive options available in version 3... My first outing with AquaLogic didn't encourage a second try. All-in-all AquaLogic just seemed an order of magnitude too big and clunky.
I favour a distributed system of agents suitable for capturing each service invocation. Each agent could feed (largely unprocessed) data back to a cluster of analysis machines. At the end, I'd hope to see a bunch of query/display/integration tools to leverage (what would be in my experience) the only up-to-the-minute source of information about how the organisation actually hangs together.
A JMS wiretap (vendor specific, or fall-back to browsing the queues), could be implemented easily enough. For HTTP snooping, tcpdump's ok. For HTTPS, web server modules would be required (Apache, IIS?, ligHTTP?).
The analysis? Another day...
Subscribe to:
Posts (Atom)