Sometimes you don’t get to define the requirements, they sometimes appear to serve a higher purpose that you can’t begin to understand. All you know is that they are requirements, and there were decisions made for various reasons. Sometimes you have to play the cards that you are dealt. But it is still your choice in how to play them.

I’m talking about message formats here, In a specific transaction processing system there are two requirements that we must adhere to:

1. Accept a 8,000 – 10,000 bytes incoming fixed message format.
2. Log the Raw Message Request and Responses for all interface connections

Regarding #1 I’d prefer to see a variable message format here instead, but I understand the need of an existing system to talk in the language this it is used to. Item #2 had me very concerned when I first heard of it, with my PCI background, I was ready to put my foot down and call people crazy – (Imagining the request to log raw messages that contained track data, pin blocks, card verification numbers)  To my surprise this was not for a financial transaction processing system but for one of a different purpose.  One that exists in a highly regulated word with data retention requirements and the need integrity of the raw transaction messages for compliance and legal reasons.

The challenge I had logging the raw messages where their sheer size – 10K and when you are looking at 4-6 legs of a transaction – client request, client response, endpoint request, endpoint response, and other transaction paths that sometimes seem recursive, we have 50K of logging for a single transaction – times 3 to 5 million transactions per day – that is 150 GB to 250 GB per day of logging !

The easiest solution was to look into compression – how much time would compressing the data stream before logging it take ? Would this impact transaction processing time ? How was the raw messages used ? If we compress the message, what needs to occur on applications on the other end, what language and platform are they written in, what is a portable algorithm ?

It turns out the these messages contains many repeating unused fields with default values – these compress very well:

Enter gzip – On our platform Java’s GZipInputStream and for our clients tools the .NET GZipStream.

How did this work out ?

raw_size    comp_size   Compression %
------------------------------------------
3975        393         90.1    
10599       484         95.4

How much disk storage and SAN space and upgrades were saved Priceless.

I had the opportunity recently to visit one of our OLS.Switch customer’s retail locations. This particular customer doesn’t have a presence in the region that I work in, so I was very excited to Swipe my Card.  Probably too excited, actually, because I think I explained every line of the receipt, including the myriad of transactions that occurred, the number of message formats, transaction types, database entities, application logic, and network connections to various internal and external endpoints to my wife, who after twenty minutes didn’t share the same level of enthusiasm that I maintained throughout the conversation.

It is always fun to know what happens “inside the box” and that which seems magical to others. When I started my career at a small Third party processor this exercise was quite common, after we did system maintenance in the middle of the early morning, we would drive to the nearest corner store or gas station to test our issuing systems and connections via performing transactions on cards that we issued for.  We even got to reward ourselves with transaction amounts over $25, as we couldn’t allow Stand-In-Processing to approve these transactions, we wanted to ensure that our Issuing system authorized these transactions.

It is even more exciting to have taken part in the design, development and implementation of a given transaction. The exhilaration continues when you realize that I’m only 1 of about 5 million transactions per day that our software powers here.

Let’s do a walkthrough of the transaction that I performed:

Cashier:  Hi, Welcome. Are you enrolled in our ________ rewards program ?

Me: Yes, I don’t have my card – can you lookup by phone number ? 

Cashier:  Sure  <Enters in number that I provide>

Transaction #1 : Loyalty Card Lookup based on Phone Number also includes my point balance, level

Cashier: Can you confirm your address ?,great, that’s you. Your total is $ xx.xx , Debit or Credit ?

Me:  <Swipes my Mastercard >

Transaction #2 : Transaction Market Basket Analysis, Discount calculation, interfacing to an Offering Engine, Serializing of Coupons to print on receipt for future usage and coupon validation requests.

Transaction #3 : Credit Card Authorization

Cashier: Thank you very much for shopping at _______, By using your _________ card you saved $ x.xx, Hope you have a nice day.

Me: Thank you and running outside to share the exiting world of transaction processing to my wife.

The 20 mintues that followed including me discussing the following.

• The finer points of Loyalty Card Lookups, including how to return a list of cards, and how to address multiple requests as the cashier scrolls to fetch the next batch of card numbers for folks with common names, the challenges of either cardholders or cashiers using common phone numbers.
• Card Type and BIN Based Routing to external endpoints and message translation from one interface to another
• Sending large transaction requests with detailed shopping cart detail.
• Algorithms to generate coupon numbers that are difficult to be abused by coupon generation scripts.
• Substitution logic and the template engine to display various messages and transaction variable on to cashier receipts.
• The various batch jobs and processes that are ran to import, extract various data to support the processes.

Just another day in the life of payment application software developer.