Over the last few months there has been various webEx, gotoMeeting, Live Meeting, etc of product demonstrations that I’ve been a part of as a participant.

Some General Thoughts:

• If you are showing a web based application use a SSL Certificate and https:// If you are going to show a web-interface that you log in with a username and password or shows account numbers please do this- you can used a self-signed cert, but I get nervous about demo’s without this- It is just sloppy not to do.
• Mask Account Numbers when they are displayed. I get really nervous about this type of stuff and question your security posture.
• Don’t use account numbers and PIN as authentication method, (although there are certain instances where this is acceptable) don’t make this the default option.
• If you are showing a payment system – understand what PABP and PA-DSS are – and if you have customers that are "PCI Complaint" running it, this isn’t the same to me.
• Show a finished product, links that go to "Not yet completed" or pages that are not consistent in look and feel confuse me.
• When I ask how many ‘Live Customers’ use this product, I want to know about in production, not in the sales pipeline.
• If it is a MS Windows/SQL Server based product, don’t list Windows Std. Edition and MSSQL Standard Edition as required software – We need enterprise level software, there is a huge delta in TCO in licensing fees.

Things to do right:

• Simulate a live transaction against a simulator or other tool showing that it is a real system and is functional.
• Walk me through the life-cycle of certain processes that I care about.
• Be able to explain "how you would implement X" or modify Y, or how your system deals with "Z"
• I know that a product won’t solve all of my needs, so I’m looking for synergies with your team to be partners with a relationship to get your product to fit my needs.
• Be able to speak my language, and have a few competent people driving the demo.
• Show me how "someone would use this" application in the real world.

We connect and interface to about 20 different endpoints, each has its own message formats, connection methodologies, protocols, and different implementations of payment transactions sets, encryption and key management considerations, On-Line vs Offline transactions, Store and Forward (SAF), and flat file based reconciliation and clearing files among other details.

Here is a basic list of steps that we perform that assist us in getting the endpoint integration right:

1) Acquire the Host Message Format specifications and communication/protocol specifications that describe the communication and connection e.g: 2||4 byte length header, other header(s), length in network or host byte order, etc. and  read  and understand them.  Perform this same task for batch flat file based formats and file transfers.

2) Request sample dumps/files of some messages as well as for online transactions, network traces that show the communication bytes of the message, manually parse them and map them back to the spec. These should include samples of request/response exchanges.  [These should be from a test system.]  Examples often reveal nuances not documented or kept up to date in the spec.

3) Determine which message set and transactions are in play, There are many types of transactions that will be different depending on the type (e.g Debit, Credit, Gift) and characteristics of the transaction (e.g Manual, Swiped, MOTO). You need to determine which ones map to your business requirements and which ones will be used.

4) Request and/or develop certification scripts for your implementation (it helps to see what the other side thinks is important).

5) Schedule a walkthrough – This is the key event: the probability of our success is increased by scheduling and holding one or more walkthroughs.  The idea is to comb through the spec(s) and complementary material page by page, field by field and get complete alignment on the task.  It’s the absolute truth that the real nature of the interface comes out in the walkthrough. 

6) Develop and test your code to the specs and sample messages dumps. (Which is a small part of the project)

7) Use tools such as netcat  and wireshark as well as your application logs to confirm that you are sending what you think you are, and can provide detailed information to someone on the ‘other’ end of the endpoint to help troubleshoot integration. Don’t assume that you think you are sending what your think your code does, verify it locally first.

Develop a good or rather great working relationships with someone on the ‘other’ end of the endpoint, that can help assist with message format questions, perform traces, and check what you are sending to what they expect, and setup regular communication throughout the project.

Here is a list in no particular order:

Gateways:

• MerchantWarehouse MerchantWare
• Shift4 – 4go
• Element Payment Services
• Trust Commerce – TC Smart
• Network Merchants, Inc.
• eProcessing Network

(contact me and I’ll add you if you are not listed)

Most of these are ISO’s that sell you a merchant account and access to their gateway that uses "end-to-end" encryption and that it will shift the PCI and PA-DSS burden from you to them, if you are a merchant, some claim you don’t even need to go through PCI compliance because you don’t have access to the card numbers or the encryption keys to decrypt the cards (Please also see this post on this subject).  This is all really good stuff, I’ve written about End-to-End Encryption before and am a big proponent of it. This can help prevent "sniffers" and card capturing malware from capturing track data and account numbers in the clear as they traverse your internal network. Attackers would either need to install card skimmers or gain access to encryption keys, or use brute force methods against captured encrypted data to capture data at your store.

But it isn’t really End-to-End Encryption.

Let look at two examples:

1. A typical small merchant using a payment gateway
2. A large retailer or processor/gateway that uses a payment switch

A typical small merchant that uses a payment gateway:

A large retailer or processor/gateway that uses a payment switch

( uses leased lines to connect directly to a Payment Processor (FDR, Chase/PaymentTech, Fifth Third, MPS, etc ) or Interchange Network (VisaNet, BankNet, etc )

Let’s say that you are using a gateway or even a switch that supports an encrypted message format from the point-of-sale (POS). The area in RED in each diagram shows where the account number traverses the payment networks in clear text. At the small merchant example from the Gateway to the rest of the network – the account number and track data and CVV2/CVC2 data is sent in the clear. In the direct connect model with the Payment Switch (which actually just operates as a local gateway) from the payment switch to the rest of the network. So End-to-End is really not End-to-End at all. (it depends on where you define end   This should also explain why End-to-End Encryption in its current state would not of prevented the breach at Heartland Payment Systems – as a processor they need to connect and communicate over the interchange networks using TCP/IP connection and ISO-8583 messages to these endpoints.

Why is this ?  The Payment interchange networks and message formats that processors and the Interchange networks use does not support this in their current message formats (primarily ISO-8583) There is no room in the current implementations of Visa’s Base1, MasterCard’s MIP, or FDR’s message formats for example. Data Elements can be added to support this, but would require massive changes to Payment Systems infrastructures and systems.

Does any one have any solutions for this ? Please provide comments below — I’ll provide a follow-up blog post with some of my ideas.

Remember that End-to-End is really not End-to-End, it may shift or transfer some of the compliance "burden"  from the merchant to that of the processor, but still exists in clear text on private networks and at processors.  Oh, and tokenization and secure card vaults would work the same way here, the cards need to be translated to their raw value to ride the payment networks.

My Answer:  Any time you are not face to face with a customer or if there is a delay before the customer receives the product or service, you should do Near-Time authorizations with the intent of not allowing the mechanics of the authorization impact the customer.

Let me explain:

It comes down to the customer checkout experience as well as to help prevent shopping cart abandonment the payment process should not be an impediment to the customer ordering an item. Especially if there is a glitch, communication issue, or any other outage in the authorization process including late or slow replies. You don’t want your customer to get frustrated during the order process, because they will go some where else to shop.

What is an alterative approach ?

Take the order, log it in a Database with a status of "Payment Pending" or "Authorization Pending"  Have a batch process or scheduled task or cron job that runs on a periodic basis to loop through the list of Orders with Pending Payment Statuses, Securely acquire the required authorization data elements required for the authorization request (Considerations need to be made regarding the retention of CVV2 subsequent to the authorization request, and how the authorization data is managed between the order and authorization request ) and perform Near-Time Authorizations for these orders. Approval responses update the Order Records in the Database, and "errors" and are placed to try again at the next run (Consider Reversals and Duplicate Transaction Checking here on error responses to protect your customers open-to-buy) Declines are noted as Payment Failed, and you send a payment email allowing your customer be notified that the authorization failed, and prompt them to update their payment information on a Secure Account Web Page or call you to provide it over the phone. (Note this method wont work if you are doing any 3D-Secure – Verified by Visa, MC Secure Code , stuff — Which I’ve used I think 2 times in the last 5 years.)

Our you could do it real time and put a message on your cart to your customer, "Please Try Again Later" when an issue occurs with your payment gateway or processor — They will Try Again Later, somewhere else.

In retail environments (or in e-commerce environments with large volume) where real-time is required, you need a redundant payment switch that can handle multiple outbound connections to your processors primary and secondary authorization centers (geographical diversity in their data centers) that run on separate connections (routers, outbound copper, and different telcos)  but also read this here, Andy talks about this in a little more detail, and talks about geographical diversity that runs though a common CO in an issue that a client had with an authorization provider.

Yesterday, I researched and implemented text notifications when you perform an Reload or Add Money transaction on our issuing platform to your prepaid card using an interface to a SMS Gateway. Check it out below: I’m using my Nokia E71 here.

It has been my experience (YMMV) both first hand running an authorization host/switch (issuing and acquiring) and as an IT Security Auditor and QSA – that either Core Banking applications or Payment Switches fall into one of the following when it relates to upgrades, changes, or security updates:

• "The Vendor set it up, we don’t touch it"
• "We don’t patch it because we are afraid"
• "We cringe everything we need to install a new release of the software"
• "Last time we did an upgrade, we had x amount of downtime"
• "It all goes smooth like clockwork" 

During Vulnerability Assessments and Penetration Testing on the Internal Networks that I performed– My observations from an operating system, database and application perspective – these systems are typically not keep current or run on a platform that the organization is not very familiar which and relies on outside support. The application was not cohesive to the rest of their operating environment: systems, technologies, and procedures.

Installing new releases of our software (or rather our clients installing releases of new software) is something that does not make me cringe. (and I used to not sleep very well in the past)  At least one of our clients seems to agree as well. (See Andy’s "A very simple platform to support")

We just rolled out a new release that was quite large (see Flexible Spending Accounts (New Initiatives, Part 3) and had changes that impacted pretty much every transaction path due to partial authorization and credit reversal support, and required heavy regression testing. Our agile based SDLC is a big help with this, we have very iterative development processes and frequent testing which also means less large bulking updates that break everything.

Another success factor is our simplicity of upgrading our program code and binaries. It is really as simple as:

• Stop the OLS.Switch Service or Daemon
• create a backup copy of the directory or file path where OLS.Switch is installed
• Start the OLS.Switch Service or Daemon
• Perform test transactions and monitor.
• The Back-Out plan is to stop the service and revert back to the backup copy.

Further, System Implementation Design can have a big impact on Up-time;  we run multiple independent application servers behind load balancers, that allows us to gracefully stop an application – which stops accepting new transactions when finishing to process those in its queue, and the load balancer stops routing transactions to this application server. Allowing an upgrade to be made while other application servers are still processing transactions. Uptime, not system uptime, but uptime processing transactions doesn’t have to suffer for "scheduled maintenance", or security related patches and reboots.

I think we have a low-risk upgrade/update path that our clients are very comfortable with – So in 7 months into the year – we have had a dozen releases to add additional functionality, address endpoint changes, and implement new transaction types.

Stratus Technologies, famous for fault tolerant computers, the operating system VOS, and its newer ftServers line that can run Red Hat Enterprise Linux or Windows Server 2003 is giving away a book offer on Fault-Tolerance and/or Virtualization for dummies.  Get your copies here.

Stratus has two Virtualization options one for its hardware, another product is a software solution for commodity boxes.  1) VMWare, using the VMWare Infrastructure 3 on ESX,  2) Avance, which uses Citrix’s Xen and other HA components to build a pair of HA Virtual Servers.

One of the frequent concerns about deploying any payment solution is “will it be able to process my transactions in a timely manner?” This is both an easy and hard question to answer. In some instances, a bad application design can lead to poor performance. In others, it is faulty system integration of one or more of the other components causing performance bottlenecks. Generally, there are several major components in the processing of a transaction that can significantly affect throughput and response time performance.

These major components are the network, the server hardware, the encryption hardware, the system software, and the application software. How well these pieces are integrated will always have a major impact on the overall performance of any given payment solution. For example, if a throughput rate of 25 transactions per second (TPS) is required and the hardware encryption device selected is only capable of 12 encryptions or decryptions per second, then the encryption device will be the bottleneck and no amount of software tuning can or will improve the throughput above 12 TPS. Unless the throughput of the encryption device is known, then performance degradations may manifest themselves as a software performance issue rather than a system integration issue.

A payment processing software provider, unless they dictate specific requirements for some or all system components, has limited control over the effect on performance of all but the application software. Software providers will normally make recommendations for these components but cannot predicate the performance of their software products on these recommendations being followed. Most payment applications, if they measure performance, do so from time of request message arrival at the application level to time of response message departure from the application level. Some may measure overall response time only; others will measure internal processing and external “wait” time separately.

Historically, in order to have some semblance of control over as many factors as possible, payment applications were written for specific platforms. Usually these platforms were proprietary fault tolerant systems whose cost/performance ratios were degraded by the inherent overhead of providing hardware- and system software- level fault tolerance. In these instances, performance — up to a limiting point — could be bought at a premium price. Frequently these fault tolerant platforms required special software design and coding techniques to properly and fully take advantage of the fault tolerance attribute. And if other components in the path such as communications routers and connections are not redundant, the premium paid for fault tolerance is all for naught.

Generally, it pays to initially set aside specialty hardware arrangement considerations and focus first on the payment application design itself. Pursuit of high performance using a poorly written application on a premium proprietary platform will be a truly expensive undertaking. When confining the performance issue to the payment application itself, there are a few key software attributes that contribute to the tangible performance characteristics of an online transaction processing (‘OLTP’) application. These key attributes are code path length, code efficiency, database design, and encryption approach.

Code path length is relatively objective and refers to the lines of code which translate eventually into the number of machine instructions executed in the process path of a transaction. Longer paths tend to produce longer response times and lower levels of performance and, obviously, shorter paths produce the converse.

Code path efficiency is more subjective and refers to the art (or science, if that is your viewpoint) of finding the logic design that requires the least lines of code to perform a particular function and the least number of functions to complete a transaction processing flow. Generally, but not always, the more experienced designer and coder will produce better software. However, for payment processing, the addition of the experience level of understanding the nuances of OLTP (some deign to call this “real-time processing”) in general and payments OLTP in particular is another efficiency factor.

Database design and how it affects any kind of application is a well-published subject that we do not need to cover here. Suffice to say, a poor database design or a poor implementation of it will have significant impact on an OLTP application which is sweating minute changes in the milliseconds that a process path takes. Again, OLTP and payments experience goes a long way toward subjugating database design as a performance issue. Simply stated, data reads and writes must be efficient and kept to a minimum.

Encryption of in-flight or stored card data is a necessary security step in processing a payment. Without a doubt, it is an expensive process often best relegated to an off-server “single purpose computing device.” However, there are some board-level implementations that do work well if they do not steal primary server CPU cycles. In either case, once again, OLTP experience will mitigate the risk of a poorly implemented encryption design.

Payment Processing Application Scalability

Scalability is generally defined as the property of an application to improve its performance due to a change in scale of its hardware environment. Commonly, the hardware change usually involves either faster processors or more processors. It may also include chip or disk memory components with higher transfer rates. For OLTP applications, the environmental impact zone expands to include internal and external connectivity as well as database considerations. It does no good to scale the application if the encryption, database or communications processes cannot scale accordingly.

Many will look solely to an application for scalability when, in reality, it is also very much a system integration issue. Conversely, any application, including an OLTP application, can be designed to be non-scalable. One simple way is to make the application single-threaded … pretty much the “kiss of death” for an OLTP payment processing environment.

Assuming an OLTP application is multi-threaded and has an efficient code path, where can scalability go wrong? There is a multitude of ways. For example, improperly or poorly configured network routers can overload one processing path while underutilizing another path. Poorly configured servers with insufficient memory or processor power will invariably lead to poor performance. Using server clusters incorrectly can lead to load balancing and reliability issues. If they are not set up properly, database servers will severely impact an OLTP application’s ability to achieve even marginal performance.

So, where does scalability really come from or how is it best achieved? Scalability fundamentally derives from the ability of an application to take advantage of a faster server or more servers or both. That means the applications will produce improved performance via increased transaction capacity, reduce response time or both. Assuming a reasonable design and execution, most applications (our earlier single-threaded example notwithstanding) will show linear improvements in performance relative to the change in server count and/or processing power. Running on multiple servers does require the application be replicable in some means or that the servers themselves provide the replication transparently.

The multi-server approach to scalability will eventually become both complex and expensive as the number of servers increases. Increasing the processing power of a server seldom, if ever, creates any additional complexity. And, for commodity servers, Moore’s Law of computing power (2 times power increase every 18 months) comes into play and the additional expense of more processing power is not going to be significant.

Another approach is using virtualization technology to create replication. A Virtual Machine (‘VM’) will, in most cases, create a veneer of replication even when the application is not conducive to duplication. However, the VM approach possesses a fatal flaw: it creates potential single points of failure for multiple instances of the application. This weakness can be mitigated by running the VM on a proprietary fault tolerant hardware platform or in some form of clustered environment. Obviously, this two-part approach adds additional hardware costs on top of the costs for the virtualization technology. And it is a complex solution that creates a number of integration issues to be resolved.

For any payment processing application, replication creates a number of integration and configuration issues. As additional copies of the application are created, communications connections between the application and encryption devices, terminals (POS, ATM, Mobile, Kiosk, etc.) and gateways must be replicated. As connections are replicated, a decision must be made as to whether these are real or virtual connections. Juxtaposed to those choices are the decisions made on the various connection types in regards to failure points and backup paths. If a single physical communication line outage takes out three virtual connections to three separate applications instances, then all three application instances are a single point of failure by default.

When factoring the communications connections along with encryption device and database connections and communications routers into the decision cycle, the complexity of the integration and configuration process increases exponentially. And we haven’t even begun to talk about the overhead of these extra connections. Stated simply, over-replication will often create more problems than it solves.

So, it is readily apparent that scalability for an OLTP application is far more than just a matter of tossing more and faster hardware into the performance pot. For OLTP applications in general and payment processing applications specifically, scalability is a delicate balance of server power and application replication. Knowing where that balance occurs comes from years of experience designing, supporting and managing payment processing environments. In another post, I will talk about how OLS created a practical solution for the performance and scalability issues for a payments processing application.