Payment Systems / Application Demos and Presentation thoughts

Photo by The Eggplant

Over the last few months there has been various webEx, gotoMeeting, Live Meeting, etc of product demonstrations that I’ve been a part of as a participant.

Some General Thoughts:

  • If you are showing a web based application use a SSL Certificate and https:// If you are going to show a web-interface that you log in with a username and password or shows account numbers please do this- you can used a self-signed cert, but I get nervous about demo’s without this- It is just sloppy not to do.
  • Mask Account Numbers when they are displayed. I get really nervous about this type of stuff and question your security posture.
  • Don’t use account numbers and PIN as authentication method, (although there are certain instances where this is acceptable) don’t make this the default option.
  • If you are showing a payment system – understand what PABP and PA-DSS are – and if you have customers that are "PCI Complaint" running it, this isn’t the same to me.
  • Show a finished product, links that go to "Not yet completed" or pages that are not consistent in look and feel confuse me.
  • When I ask how many ‘Live Customers’ use this product, I want to know about in production, not in the sales pipeline.
  • If it is a MS Windows/SQL Server based product, don’t list Windows Std. Edition and MSSQL Standard Edition as required software – We need enterprise level software, there is a huge delta in TCO in licensing fees.

Things to do right:

  • Simulate a live transaction against a simulator or other tool showing that it is a real system and is functional.
  • Walk me through the life-cycle of certain processes that I care about.
  • Be able to explain "how you would implement X" or modify Y, or how your system deals with "Z"
  • I know that a product won’t solve all of my needs, so I’m looking for synergies with your team to be partners with a relationship to get your product to fit my needs.
  • Be able to speak my language, and have a few competent people driving the demo.
  • Show me how "someone would use this" application in the real world.

Steps to setup a successful endpoint integration


We connect and interface to about 20 different endpoints, each has its own message formats, connection methodologies, protocols, and different implementations of payment transactions sets, encryption and key management considerations, On-Line vs Offline transactions, Store and Forward (SAF), and flat file based reconciliation and clearing files among other details.

Here is a basic list of steps that we perform that assist us in getting the endpoint integration right:

1) Acquire the Host Message Format specifications and communication/protocol specifications that describe the communication and connection e.g: 2||4 byte length header, other header(s), length in network or host byte order, etc. and  read  and understand them.  Perform this same task for batch flat file based formats and file transfers.

2) Request sample dumps/files of some messages as well as for online transactions, network traces that show the communication bytes of the message, manually parse them and map them back to the spec. These should include samples of request/response exchanges.  [These should be from a test system.]  Examples often reveal nuances not documented or kept up to date in the spec.

3) Determine which message set and transactions are in play, There are many types of transactions that will be different depending on the type (e.g Debit, Credit, Gift) and characteristics of the transaction (e.g Manual, Swiped, MOTO). You need to determine which ones map to your business requirements and which ones will be used.

4) Request and/or develop certification scripts for your implementation (it helps to see what the other side thinks is important).

5) Schedule a walkthrough – This is the key event: the probability of our success is increased by scheduling and holding one or more walkthroughs.  The idea is to comb through the spec(s) and complementary material page by page, field by field and get complete alignment on the task.  It’s the absolute truth that the real nature of the interface comes out in the walkthrough. 

6) Develop and test your code to the specs and sample messages dumps. (Which is a small part of the project)

7) Use tools such as netcat  and wireshark as well as your application logs to confirm that you are sending what you think you are, and can provide detailed information to someone on the ‘other’ end of the endpoint to help troubleshoot integration. Don’t assume that you think you are sending what your think your code does, verify it locally first.

8) Develop a good or rather great working relationships with someone on the ‘other’ end of the endpoint, that can help assist with message format questions, perform traces, and check what you are sending to what they expect, and setup regular communication throughout the project.

When End-to-End Encryption is really not End-to-End.

I’m reading a lot about solutions that implement end-to-end encryption, where account numbers and track data is encrypted and can utilize a Hardware Security Module (HSM) and DUKPT or other encryption algorithms from the point-of-sale. I thought it important to share what data is actually encrypted in the payment system.


Here is a list in no particular order:


(contact me and I’ll add you if you are not listed)


Most of these are ISO’s that sell you a merchant account and access to their gateway that uses "end-to-end" encryption and that it will shift the PCI and PA-DSS burden from you to them, if you are a merchant, some claim you don’t even need to go through PCI compliance because you don’t have access to the card numbers or the encryption keys to decrypt the cards (Please also see this post on this subject).  This is all really good stuff, I’ve written about End-to-End Encryption before and am a big proponent of it. This can help prevent "sniffers" and card capturing malware from capturing track data and account numbers in the clear as they traverse your internal network. Attackers would either need to install card skimmers or gain access to encryption keys, or use brute force methods against captured encrypted data to capture data at your store.

But it isn’t really End-to-End Encryption.

Let look at two examples:

  1. A typical small merchant using a payment gateway
  2. A large retailer or processor/gateway that uses a payment switch


A typical small merchant that uses a payment gateway:



A large retailer or processor/gateway that uses a payment switch


( uses leased lines to connect directly to a Payment Processor (FDR, Chase/PaymentTech, Fifth Third, MPS, etc ) or Interchange Network (VisaNet, BankNet, etc )

Let’s say that you are using a gateway or even a switch that supports an encrypted message format from the point-of-sale (POS). The area in RED in each diagram shows where the account number traverses the payment networks in clear text. At the small merchant example from the Gateway to the rest of the network – the account number and track data and CVV2/CVC2 data is sent in the clear. In the direct connect model with the Payment Switch (which actually just operates as a local gateway) from the payment switch to the rest of the network. So End-to-End is really not End-to-End at all. (it depends on where you define end :)  This should also explain why End-to-End Encryption in its current state would not of prevented the breach at Heartland Payment Systems – as a processor they need to connect and communicate over the interchange networks using TCP/IP connection and ISO-8583 messages to these endpoints.


Why is this ?  The Payment interchange networks and message formats that processors and the Interchange networks use does not support this in their current message formats (primarily ISO-8583) There is no room in the current implementations of Visa’s Base1, MasterCard’s MIP, or FDR’s message formats for example. Data Elements can be added to support this, but would require massive changes to Payment Systems infrastructures and systems.


Does any one have any solutions for this ? Please provide comments below — I’ll provide a follow-up blog post with some of my ideas.


Remember that End-to-End is really not End-to-End, it may shift or transfer some of the compliance "burden"  from the merchant to that of the processor, but still exists in clear text on private networks and at processors.  Oh, and tokenization and secure card vaults would work the same way here, the cards need to be translated to their raw value to ride the payment networks.

Batch vs. Real-Time Processing in e-commerce environments

I was having a discussion recently about payment processing in e-commerce environments, specifically Batch versus Real-Time Authorizations. Batch processing has a one-a-day connotation, or that it is simply file based.  Why would you not want to send transactions real-time was the question to me? 

My Answer:  Any time you are not face to face with a customer or if there is a delay before the customer receives the product or service, you should do Near-Time authorizations with the intent of not allowing the mechanics of the authorization impact the customer.

Let me explain:

It comes down to the customer checkout experience as well as to help prevent shopping cart abandonment the payment process should not be an impediment to the customer ordering an item. Especially if there is a glitch, communication issue, or any other outage in the authorization process including late or slow replies. You don’t want your customer to get frustrated during the order process, because they will go some where else to shop.

What is an alterative approach ?

Take the order, log it in a Database with a status of "Payment Pending" or "Authorization Pending"  Have a batch process or scheduled task or cron job that runs on a periodic basis to loop through the list of Orders with Pending Payment Statuses, Securely acquire the required authorization data elements required for the authorization request (Considerations need to be made regarding the retention of CVV2 subsequent to the authorization request, and how the authorization data is managed between the order and authorization request ) and perform Near-Time Authorizations for these orders. Approval responses update the Order Records in the Database, and "errors" and are placed to try again at the next run (Consider Reversals and Duplicate Transaction Checking here on error responses to protect your customers open-to-buy) Declines are noted as Payment Failed, and you send a payment email allowing your customer be notified that the authorization failed, and prompt them to update their payment information on a Secure Account Web Page or call you to provide it over the phone. (Note this method wont work if you are doing any 3D-Secure – Verified by Visa, MC Secure Code , stuff — Which I’ve used I think 2 times in the last 5 years.)


Our you could do it real time and put a message on your cart to your customer, "Please Try Again Later" when an issue occurs with your payment gateway or processor — They will Try Again Later, somewhere else.


In retail environments (or in e-commerce environments with large volume) where real-time is required, you need a redundant payment switch that can handle multiple outbound connections to your processors primary and secondary authorization centers (geographical diversity in their data centers) that run on separate connections (routers, outbound copper, and different telcos)  but also read this here, Andy talks about this in a little more detail, and talks about geographical diversity that runs though a common CO in an issue that a client had with an authorization provider.

mobile commerce – sms text notifications

Picture 20If you have ever used Obopay or even social networking site Facebook, chances are that you have interacted with your mobile phone with these sites in some manner with your phone.  Obopay, is a little more obvious, but you receive text notifications when you send or received money on your mobile.  Facebook sends text messages to your registered mobile phone number for you to validate your account, Obopay also uses multi-factor authentication to validate the user of its website using a phone call and spoken code, or a text with a message and a code that need to type in a webpage. This is called Out-of-Band Authentication and your bank may have implemented something similar for its Internet banking.


Yesterday, I researched and implemented text notifications when you perform an Reload or Add Money transaction on our issuing platform to your prepaid card using an interface to a SMS Gateway. Check it out below: I’m using my Nokia E71 here.


Product Release Cycle

updateI’ll just say it; I’m proud of our release cycle for OLS.Switch.


It has been my experience (YMMV) both first hand running an authorization host/switch (issuing and acquiring) and as an IT Security Auditor and QSA – that either Core Banking applications or Payment Switches fall into one of the following when it relates to upgrades, changes, or security updates:

  • "The Vendor set it up, we don’t touch it"
  • "We don’t patch it because we are afraid"
  • "We cringe everything we need to install a new release of the software"
  • "Last time we did an upgrade, we had x amount of downtime"
  • "It all goes smooth like clockwork"  🙂


During Vulnerability Assessments and Penetration Testing on the Internal Networks that I performed– My observations from an operating system, database and application perspective – these systems are typically not keep current or run on a platform that the organization is not very familiar which and relies on outside support. The application was not cohesive to the rest of their operating environment: systems, technologies, and procedures.


Installing new releases of our software (or rather our clients installing releases of new software) is something that does not make me cringe. (and I used to not sleep very well in the past)  At least one of our clients seems to agree as well. (See Andy’s "A very simple platform to support")


We just rolled out a new release that was quite large (see Flexible Spending Accounts (New Initiatives, Part 3) and had changes that impacted pretty much every transaction path due to partial authorization and credit reversal support, and required heavy regression testing. Our agile based SDLC is a big help with this, we have very iterative development processes and frequent testing which also means less large bulking updates that break everything.


Another success factor is our simplicity of upgrading our program code and binaries. It is really as simple as:

  • Stop the OLS.Switch Service or Daemon
  • create a backup copy of the directory or file path where OLS.Switch is installed
  • Start the OLS.Switch Service or Daemon
  • Perform test transactions and monitor.
  • The Back-Out plan is to stop the service and revert back to the backup copy.


Further, System Implementation Design can have a big impact on Up-time;  we run multiple independent application servers behind load balancers, that allows us to gracefully stop an application – which stops accepting new transactions when finishing to process those in its queue, and the load balancer stops routing transactions to this application server. Allowing an upgrade to be made while other application servers are still processing transactions. Uptime, not system uptime, but uptime processing transactions doesn’t have to suffer for "scheduled maintenance", or security related patches and reboots.


I think we have a low-risk upgrade/update path that our clients are very comfortable with – So in 7 months into the year – we have had a dozen releases to add additional functionality, address endpoint changes, and implement new transaction types.

Stratus Book Offer: Fault-Tolerance for Dummies and Virtualization for Dummies

Fault-Tolerance for Dummies and Virtualization for Dummies

Stratus Technologies, famous for fault tolerant computers, the operating system VOS, and its newer ftServers line that can run Red Hat Enterprise Linux or Windows Server 2003 is giving away a book offer on Fault-Tolerance and/or Virtualization for dummies.  Get your copies here.

Stratus has two Virtualization options one for its hardware, another product is a software solution for commodity boxes.  1) VMWare, using the VMWare Infrastructure 3 on ESX,  2) Avance, which uses Citrix’s Xen and other HA components to build a pair of HA Virtual Servers.


Payment Processing Application Performance

One of the frequent concerns about deploying any payment solution is “will it be able to process my transactions in a timely manner?” This is both an easy and hard question to answer. In some instances, a bad application design can lead to poor performance. In others, it is faulty system integration of one or more of the other components causing performance bottlenecks. Generally, there are several major components in the processing of a transaction that can significantly affect throughput and response time performance.

Continue reading