I was having a discussion recently about payment processing in e-commerce environments, specifically Batch versus Real-Time Authorizations. Batch processing has a one-a-day connotation, or that it is simply file based.  Why would you not want to send transactions real-time was the question to me? 

My Answer:  Any time you are not face to face with a customer or if there is a delay before the customer receives the product or service, you should do Near-Time authorizations with the intent of not allowing the mechanics of the authorization impact the customer.

Let me explain:

It comes down to the customer checkout experience as well as to help prevent shopping cart abandonment the payment process should not be an impediment to the customer ordering an item. Especially if there is a glitch, communication issue, or any other outage in the authorization process including late or slow replies. You don’t want your customer to get frustrated during the order process, because they will go some where else to shop.

What is an alterative approach ?

Take the order, log it in a Database with a status of "Payment Pending" or "Authorization Pending"  Have a batch process or scheduled task or cron job that runs on a periodic basis to loop through the list of Orders with Pending Payment Statuses, Securely acquire the required authorization data elements required for the authorization request (Considerations need to be made regarding the retention of CVV2 subsequent to the authorization request, and how the authorization data is managed between the order and authorization request ) and perform Near-Time Authorizations for these orders. Approval responses update the Order Records in the Database, and "errors" and are placed to try again at the next run (Consider Reversals and Duplicate Transaction Checking here on error responses to protect your customers open-to-buy) Declines are noted as Payment Failed, and you send a payment email allowing your customer be notified that the authorization failed, and prompt them to update their payment information on a Secure Account Web Page or call you to provide it over the phone. (Note this method wont work if you are doing any 3D-Secure – Verified by Visa, MC Secure Code , stuff — Which I’ve used I think 2 times in the last 5 years.)

Our you could do it real time and put a message on your cart to your customer, "Please Try Again Later" when an issue occurs with your payment gateway or processor — They will Try Again Later, somewhere else.

In retail environments (or in e-commerce environments with large volume) where real-time is required, you need a redundant payment switch that can handle multiple outbound connections to your processors primary and secondary authorization centers (geographical diversity in their data centers) that run on separate connections (routers, outbound copper, and different telcos)  but also read this here, Andy talks about this in a little more detail, and talks about geographical diversity that runs though a common CO in an issue that a client had with an authorization provider.

I was looking at a few old pictures of my old office at a small Third Party Processor (TPP) where I was Director of Development and Technology and that I lived in for ~8 years, that had a small test lab in my office. These are probably circa 2000-2002 or so, and it required to have physical machine to run different test systems back then. All were white boxes that I put together from parts, let me take a stab at what they did from memory:

Above: G3 IMac – required to test Mac Web Browsers for early web apps, in OSX and Mac Classic Windows 2000 Workstation – Dual Processor main workstation with was dual headed CRTs Probably two Linux Workstations – probably running debian 2.2 potato or Red Hat 7.2 – Enigma or Red Hat 6.2 Zoot or Solaris 7 or 8 for x86 – one was a dedicated linux workstation, and the other I installed every new distro of linux on I think

Above: Linux box where I first played with Netsaint, now nagios, snortd, and other sysadmin/network linux tools, and testing many open-source projects. I might of ran CVS on this as well. Linux Web Server Apache Windows 2000 IIS 5.0 Web Server Mysql 3.23 Database Server Microsoft SQL Server 7.0 or 2000 Oracle 8.0 or 8i Database Server

Now I have an shuttle PC with 8 gigs of RAM and VMWare ESXi as a test server, and a Vista x64 workstation with 8gb of ram and VMWare Workstation for dev and a Black MacBook for travel with 4gb and VMWare Fusion.  I’ve consolidated a bit Oh, I also have a XEN instance of Windows 2003 on another Linux box    BTW,  I was an early user of VMware version 3.0 and used it to run Linux guests on my laptops, but that was when RAM was not so cheap, and these boxes had memory limits.

Virtualization rocks for development and testing.

On my jPOS page I added a link to a screencast that I did that shows the basic configuration and usage of the jPOS SysLogListener.

If you are not familiar with syslog, it is the logging daemon for Unix and Linux. There are implementation’s for MS Windows as well, such as Kiwi Syslogd. (and some that peel entries from the Windows/NT Event Log and forwards then to a centralized syslog server) Many alerting systems are based off of syslog events where you can define an action to call an external program/script, send an email/page/SMS notification. Or you can even use splunk as a syslog daemon and "google" your logs.

Enjoy the screencast.

If you have ever used Obopay or even social networking site Facebook, chances are that you have interacted with your mobile phone with these sites in some manner with your phone.  Obopay, is a little more obvious, but you receive text notifications when you send or received money on your mobile.  Facebook sends text messages to your registered mobile phone number for you to validate your account, Obopay also uses multi-factor authentication to validate the user of its website using a phone call and spoken code, or a text with a message and a code that need to type in a webpage. This is called Out-of-Band Authentication and your bank may have implemented something similar for its Internet banking.

Yesterday, I researched and implemented text notifications when you perform an Reload or Add Money transaction on our issuing platform to your prepaid card using an interface to a SMS Gateway. Check it out below: I’m using my Nokia E71 here.

Here is a snapshot of what my desk looks like: you can see a magtek USB card reader and a few magnetic striped cards; expired pre-paid credit, gift and merchandise return cards that used for testing purposes here.

I’ve been developing some small tools that allows for us to send transactions via a swipe in a .NET windows based application as well as in a Java Web based version to a test instance of OLS.Switch. I used to (and still do) just pipe binary message dumps over netcat pointed to our OLS.Switch’s configured server port for this specific message format.

for example:

$ cat visa_credit_sale.dump | nc 192.168.1.50 33000

where visa_credit_sale.dump would just be a binary file of the message

$ hd visa_credit.sale.dump

would look like this (intentionally blurred and is a test card number)

Here is a shot of the Virtual Point of Sale System:

and a shot of the Virtual Terminal:

Basically you can swipe a card or key-enter a card on the virtual terminal and depending on the configuration of OLS.Switch – (I’m using bin based routing here in this test setup)

Cards that start with:

• 4 – Visa
• 5 – Mastercard
• 6011 – Discover

go to our FDR North (ChasePaymentTech) Simulator and and return a simulated response.

• 3 – Amex

go to our American Express Simulator

• 7 – Stored Value

go to our Stored Value Systems Simulator

• 6 – OLS Stored Value

get switched to our own instance of OLS.Issuer – our authorization host which is not a simulator.

The vPOS and VT are sending in messages in the Visa K/Visa D or otherwise known and Visa Gen II message format (one of the incoming message formats that we support from the device side) and depending on the card type, we are building the appropriate outbound message according to the interface specs (generally an ISO8583 variant), hitting our simulators to get different responses based on amount prompting or in the case of the OLS Stored Value cards, it uses the card files, velocity and limit checking, card status and other authorization rules to authorized the card.

The neat thing? an end-to end transaction take less then 50ms on a sub $1000.00 test server on a local lan.

Here is a link to a PDF that shows the full transaction flow.

I’ll just say it; I’m proud of our release cycle for OLS.Switch.

It has been my experience (YMMV) both first hand running an authorization host/switch (issuing and acquiring) and as an IT Security Auditor and QSA – that either Core Banking applications or Payment Switches fall into one of the following when it relates to upgrades, changes, or security updates:

• "The Vendor set it up, we don’t touch it"
• "We don’t patch it because we are afraid"
• "We cringe everything we need to install a new release of the software"
• "Last time we did an upgrade, we had x amount of downtime"
• "It all goes smooth like clockwork" 

During Vulnerability Assessments and Penetration Testing on the Internal Networks that I performed– My observations from an operating system, database and application perspective – these systems are typically not keep current or run on a platform that the organization is not very familiar which and relies on outside support. The application was not cohesive to the rest of their operating environment: systems, technologies, and procedures.

Installing new releases of our software (or rather our clients installing releases of new software) is something that does not make me cringe. (and I used to not sleep very well in the past)  At least one of our clients seems to agree as well. (See Andy’s "A very simple platform to support")

We just rolled out a new release that was quite large (see Flexible Spending Accounts (New Initiatives, Part 3) and had changes that impacted pretty much every transaction path due to partial authorization and credit reversal support, and required heavy regression testing. Our agile based SDLC is a big help with this, we have very iterative development processes and frequent testing which also means less large bulking updates that break everything.

Another success factor is our simplicity of upgrading our program code and binaries. It is really as simple as:

• Stop the OLS.Switch Service or Daemon
• create a backup copy of the directory or file path where OLS.Switch is installed
• Start the OLS.Switch Service or Daemon
• Perform test transactions and monitor.
• The Back-Out plan is to stop the service and revert back to the backup copy.

Further, System Implementation Design can have a big impact on Up-time;  we run multiple independent application servers behind load balancers, that allows us to gracefully stop an application – which stops accepting new transactions when finishing to process those in its queue, and the load balancer stops routing transactions to this application server. Allowing an upgrade to be made while other application servers are still processing transactions. Uptime, not system uptime, but uptime processing transactions doesn’t have to suffer for "scheduled maintenance", or security related patches and reboots.

I think we have a low-risk upgrade/update path that our clients are very comfortable with – So in 7 months into the year – we have had a dozen releases to add additional functionality, address endpoint changes, and implement new transaction types.

I was thinking about the tools and systems and general knowledge that I use on a daily basis, and thought is would be a good exercise to document them here:

Computer Systems:

• Quad Core Q6600 Vista 64 Workstation 8GB of RAM
• Apple Macbook with 4GB RAM
• VMWare Workstation and VMWare Fusion (running various linux distros and Win XP images)
• Large Monitors 24″ +

Software:

• A good text editor – UltraEdit, TextMate or vim
• A good hex editor – UltraEdit, hexdump, hexedit or vim -b
• ssh client – PuTTY or OS native ssh clients
• svn client
• NetCat - to pipe binary message dumps to simulators, and to create listening servers to accept message dumps.
• Client and Server Simulators — write your own!
• Calculator — the DEC to HEX and HEX to DEC functions are great for header lengths – if you are a real geek you have a HP-16c
• Instant Messenging clients and Skype to communicate with your team.

Knowledge:

• Ability to read — seriously, read those specs!
• ISO8583
• TCP/IP Socket programming – both Client and Server
• Database programming experience e.g. SQL – and other O/R Mapping tools – Understanding the I/O requirements between write intensive OLTP and read intensive Data Warehouse data stores.
• Payment Processing 101 knowledge
• Data Encoding techniques and character sets and numbering systems – ACSII, EBCDIC, BCD, Binary etc.
• Basic understanding of Encryption, including symmetric, asymmetric, PIN encryption , PIN translation, and DUKPT
• Basic understanding of IT Security
• PCI and PABP/PA-DSS requirements – review the audit procedures !
• How to use Google
• Ability to read — Note: This is intentionally listed twice