10,000 TPS per second

I ran across Kiplinger’s article and picture of “The Credit or Debit Debate Visualized” It is a very nice picture of both the usage of Credit and Debit Cards over time, as well as a nice list of pros and cons and differences between each, I encourage you to check it out for a good basic summary.

In payment systems I don’t really care as much about the type of card, Credit vs Debit at a basic level to me — one has PIN’s and requires usage of HSM’s and require real-time reversals and one uses clearing files and the other reconciliation files. But I digress.

At the bottom of the picture there are some statistics, Kiplinger being a personal finance website and not to mention various “Letters“, these are mostly consumer related, but his one caught my eye:


Which makes me chuckle because lots of prospects tell us they need a system to be able to support the worlds average TPS (Transaction Per Second), or a small fraction of.

The Freeze

2866522209_02285d877c.jpgThe “Freeze” is upon us in the Payments Space. Any and all system changes should of been made and time spend making sure your payments systems and infrastructure can handle the Black Thursdays, Fridays, and Holiday Season should be complete and in monitoring mode. Rather then expand more here – Andy Orrock wrote a excellent piece on this last year: Everybody Freeze!

In the Payment Systems World, we’ve now entered “The Freeze.” This is the industry-wide term associated with the period running from approximately the Thursday before Thanksgiving (which is the last Thursday of November in the US) through to about the second full week in January. During that period, we don’t do any production releases, unless it’s a fix of a critical nature. We advocate the same practice for our clients. We also recommend that they not undertake material changes to hardware configurations, databases, scripts, or any other piece of supporting or underlying technology.

Traditionally, it’s been a good time for us to focus on big projects that have a Q1 delivery date. We can stay under the covers and make some serious progress on those bigger initiatives.

We also send out a customer letter re-emphasizing how to get a hold of us. The letter stresses the importance of vigilance and watchfulness over key production systems. It reminds our clients that we’re Always Available. Our firm was founded on the back of 24x7x365 support of mission-critical production systems. We get paid to make sure our clients can – to the fullest extent possible – enjoy the holidays with their family knowing that we’ve got their back on support.

Why all the heightened concern? It’s the nature of payment systems: there’s tremendous upsurge in volume in the freeze period. If you’ve got a latent bottleneck laying dormant and ready to strike, the unfortunate reality is that it’s going to nail you right between the eyes on a killer day like Black Friday, Cyber Monday or Christmas Eve. We service some Stored Value authorization endpoints that get massive 20x surges in volumes on December 24th. So, you’ve got to be ready.

We work the other ten-and-a-half months of the year to make this month-and-a-half as uneventful as possible.

You don’t know until you know (or go into Production)

images-2.jpgOver the last six months we have been busy building and implementing an OLS.Switch Issuer Implementation with one of our customers and their banking and payment processing partners. It has been a process of reviewing and implementing message specifications, business processing requirements, authorization rules, clearing, settlement, flat file and reporting requirements. We also filtering external messages into our IMF – Internal Message Format based on ISO8583 v2003, build an interface Card Management functions via our local API’s and message sets. Building client simulators and trying to faithfully reproduce what happens when you are connected to a real system.

Testing on test systems is the next step – replacing our client simulators with other “test” systems that are driven by simulators by the processing gateway we interfaced to. Those simulators have limitations – in their configured test suites or test scripts, some require manual entry to send original data elements for subsequent transaction types, (e.g completions and reversals). We generate clearing and settlement files and match those to on-line test transactions, and our use cases.

After on-line testing, you connect to an “Association” test environment to do “Certification” and run a week’s worth of transactions through a wider test bed. Then you are certified, your BIN goes live and then you enter a production pilot mode – where you watch everything like a hawk.

You can do all of the simulated testing for both on-line transactions and off-line clearing and settlement files that you want – when you connect to the real world and do your first pilot transaction that is where most likely you will see something that wasn’t simulated, tested, or even included in certification, it happens. You need to be proactive, set-up reviews and manual interventions, perform file generation when you have staff available to review the output before it is released for further processing.

What have we seen :

  • Test environments that are not as robust as production or not setup with up-to-date releases.
  • Certain real-world examples are hard to simulate – reversals, time-outs.
  • Thinly-trafficked transactions: (chargeback, representment)…people can’t even define these much less create them in test
  • Poor or incorrect documentation of message specifications.
  • You receive Stand-In Advices or other transactions on-line that you don’t see in testing or certification.

Production pilot is a very important phase of testing – It is where you discover and address the < 1% of issues nobody catches in prior project life-cycles. What can happen, WILL happen. What you think might be something that will occur infrequently will bite you sooner, not later.

Protect Debug Info Transaction Participant

jPOS-EE has a very handy transaction participant called “Debug” its main purpose is to dump the contents of the jPOS’s Context. While this is very helpful in test modes and during development – The context remains “un-protected” and all of the data remains in the clear. Even the ProtectedLogListener and FSDProtectedLogListener will not protect data in the context.

Enter the ProtectDebugInfo Transaction Participant a configurable implementation I wrote based on some of Alejandro’s ideas, and one that lives in most of OLS’s payment products in various specific iterations.

It’s configuration looks like:


Protecting your q2.log in this truncated example:

<entry key=‘FSDMESSAGE’>
<fsdmsg schema=‘file:cfg/fsd-base’>
account-number: ‘599999______0001’
<entry key=‘PAN’>599999______0001</entry>
<entry key=‘RESPONSE’>
<isomsg direction=“incoming”>
<field id=“0” value=“2110”/>
<field id=“2” value=“599999______0001”/>
<field id=“35” value=“599999______0001=____________________”/>
<entry key=‘REQUEST’>
<isomsg direction=“incoming”>
<field id=“0” value=“2100”/>
<field id=“2” value=“599999______0001”/>
<field id=“35” value=“599999______0001=____________________”/>

Visa’s Misuse of Authorization Fee


As of October 1st 2009 Visa has started to charge a fee of $.045 per misused authorization.

Visa defines a misused authorization as:

Authorizations that are not followed by a matching clearing transaction (or in the case of a cancelled or timed out authorization, not properly reversed)

For MOTO and e-commerce merchants who use payment gateways there may be some changes to how they perform Auth and Capture type of transactions, especially if they Authorize the transaction at order time, and later Capture or Complete the transaction when they ship a product. Or more typically, perform a $1.00 authorization first with AVS and CVV2 data, and followed by an authorization for the total amount of purchase.

Many payment gateways may or may have authorization reversals, authorization voids, or other transaction types – you will need to work with your gateway to identify what transaction types need to be performed to “reverse” an authorization for a product that you do not ship. Or if your practice is to re-auth after 10 days, and let the original expire – you will be hit with the fee.

Visa also supports a new transaction type called Account Verification – which is a $0.00 authorization – which they hope merchants will used instead of misused authorizations.

For processors and large merchants with direct connections to Visanet or Payment Gateways – These entities will want to verify that their reversal processing addresses credit reversals in the time-out scenario.

Now you’re the Switch — Successful Implementation Strategies

My colleague Andy Orrock wrote a blog post titled “Now, you’re the switch” where he summarizes challeges and witness poor implementions of some interfaces that we connect to:

But here’s the thing: once we send the transaction to you, now you’re the switch.  What I mean by that is:  your application is now beholden to the same throughput, speed, efficiency, extensibility and 24x7x365 availability concerns that define our lives.  And while there have been many that have been up to the task, there have been countless other instances where that’s not been the case.

There are are few basic models that we have seen that work well:

  1. OLS Implemented Business Logic based on customer developed prototypes.
  2. OLS Implemented Business Logic and Customer provided Database or Flat File update feeds that drive authorization decisions. 
  3. OLS Transaction Participants that call Customer Provided Software and CustomerSecretSauceManger.process() methods within our processing framework.
  4. Under our guidance, interface remotely to an local endpoint via TCP/IP Sockets or WebServices, handling concerns that we address here with tech savvy customers.

Pre-Authorization data for completions and reversals and removal of Track II Data

Late last week I received a email detailing a few message format specification changes for a processing gateway interface that OLS.Switch connects to. It discussed changes to the required data elements required for “match-up”, the data required in the original transaction that one must return in a reversal response message or in completion messages.  We leverage Host Reversals (note: these are not the same as refunds) when we don’t recieve a response from an authorizer for remove authorization, most typically on a time-out scenarios. We don’t know if the processor accepted the transaction and we just didn’t receive the response or if the processor never received the original transaction at all. In cases like these we are obligated to send reversal messages, to reverse the transaction. In a credit world where there are large open-to-buys and credit limits and expiring authorizations, this is less of a deal with debit. In the debit world mistakes here case pain for cardholders and duplicate charges occur.  We SAF (Store and Forward) our reversals and retry for a set period of time with delay intervals until we receive an acknowledgment that our reversal transaction was received. Completions can be forced sales or post authorization transactions, or in certain industries completions with updated amounts that differ then the authorization (Think restaurant tip and gas pumps transactions here.)

The changes are listed below:

Track 2 data is no longer required for completion, reversal or void transaction types. With the elimination of the Track 2 Data (Field ID 35) these transaction types will now require the Account Number (Field ID 02) and Expiration Date (Field ID 14) to be provided. All clients should make these modifications prior to your next PCI assessment.

This is great news, the is one of the last processing gateways that we interface with that required the ISO8583 Data Element 35 – Track 2 Data for reversals and completions.  If you ever noted the specific wording in the PCI DSS specification about “subsequent to the authorization” this is part of why I believe that wording was left there.  The issue with this is while SAF queues are normally located in memory – there are times when they can be configured to be persisted to disk (If they are not, think of the cardholder impact of duplicate charges) This is less data, specifically pre-authorization data that is required to be stored prior to the authorization. We have leveraged “encrypted spaces” to help protect all types of SAF Queues and are happy Track 2 Data is not required for match-ups any more – Ideally, and we hope that processors will take this a step further and remove the requirement of sending the PAN, and leverage other unique transaction identifiers or composite identifiers: Take a look at one our our “FindOriginal” Transaction Participants for a MasterCard Interface:

CriteriaImpl(TranLog:this[][date>=Thu Sep 17 09:11:41 CDT 2009, irc=1816, stan=000000087625, originalItc=100.00, acquirer=987654999, mid=123456789012345, tid=12234501, banknetReference=MQWWRJ4QW ])

No Card Number there !

PCI Council : Wireless Security Guides for Payment Cards

There are few news articles today that reference this article. – That talks about the PCI Council “Publishing” a Wireless Security Guide for Payment Cards

Update July 17/2009 – The Guide is now listed on the PCI Co Website and direct link is here.

From the Article these appears to be the relevant things from the Guidelines:

  • The guidelines requires “a firewall that demarcates the edge of the organization’s CDE – cardholder data environment
  • To combat the problem of the rogue access point, businesses will need to use a wireless analyzer or preventative measures such as a wireless intrusion detection/prevention system (IDS/IDP) regularly
  • The council is advising large organizations to set up automated scanning using a centrally managed wireless IDS/IPS system.
  • The guidelines suggest quarterly scans each year to detect rogue wireless devices that could be connected to the CDE at any location and have an incident-response plan to deal with them.
  • To isolate wireless networks that don’t transmit, store or process cardholder data, a firewall must be used, and it has to perform the functions of filtering packets based on the 802.11 protocol; performing stateful inspection of connections; and monitoring and logging traffic allowed and denied by the firewall according to PCI DSS rule 10. The firewall logs would have to be monitored daily and the firewall rules verified once every six months.
  • The wireless guideline also says “relying on a virtual LAN (VLAN) based on segmentation is not sufficient.”
  • For “in-scope wireless networks,” physical security should apply, with options that include mounting wireless access points high up on a ceiling and disabling the console interface and factory rest options by using a tamper-proof chassis.
  • Change the default settings of the access points in terms of default administrative passwords, encryption settings, reset function. Disable SNMP access to remote access points if possible. Do not advertise organization names in the SSID broadcast.
  • Use of AES encryption is recommended for WLAN networks. Specifically, information flowing through certain network segments, including secure wireless devices that connect to the private WLAN through the access points, must be encrypted.
  • Wireless usage policies should be established for “explicit management approval to use wireless networks in the CDE.” Usage policies require labeling of wireless devices with owner, contact information and purpose.

Smoking is expensive – 17 digit charge on Card for Pack of Smokes… [updated]

$23,148,855,308,184,500 for a pack of smokes,  is the title of a MSNBC.com article I read this morning.

A New Hampshire man says he swiped his debit card at a gas station to buy a pack of cigarettes and was charged over 23 quadrillion dollars.

Bank of America tells WMUR-TV only the card issuer, Visa, could answer questions. Visa, in turn, referred questions to the bank.

Our Issuing systems have parameters to Delcine transactions over a certain amount, most of our clients set those to less then 1 quadrillion dollars, I believe.   I wonder what the amount on his receipt says, I wonder if this was a POS glitch or a settlement glitch.

{UPDATE} CNN.com has some more coverage on this here

In a statement, Visa said the rogue charges affected “fewer than 13,000 prepaid transactions” and resulted from a “temporary programming error at Visa Debit Processing Services … [which] caused some transactions to be inaccurately posted to a small number of Visa prepaid accounts.”

Josh Muszynski noticed the 17-digit charge while making a routine balance inquiry.
via cnn


via StackOverFlow

Add the cents to the number and you get 2314885530818450000, which in hexadecimal is 2020 2020 2020 1250

Integrated Solutions For Retailers – PCI DSS: What Do You Know, Where Do You Stand?

The Integrated Solutions For Retailers Magazine has an articled titled PCI DSS: What Do You Know, Where Do You Stand?

For a couple of months spanning the first and second quarters of this year, Integrated Solutions For Retailers surveyed its subscribers — hundreds of retailers from many segments, ranging the gamut from small and regional chains to tier-one enterprises — on their perceptions of the PCI DSS (Payment Card Industry Data Security Standard). The survey results surprised us. Respondents exuded nearly equal parts confidence, confusion, dismay, and ignorance. Some gloated. Some swore.

Some very interesting comments here, some of my favorites:

  • From a regional grocer: “We’ve devoted no effort. PCI certification is an impossible-to-hit, moving target.”
  • Only 23.9% of retailers surveyed indicated that they’re “very familiar” with the PCI DSS.
  • 59.6% say fear of a breach is their motivation for achieving compliance.

Read it here.