Continuous integration

Testing acquirer side implementations are hard. The incoming message formats and communication protocols from the card acceptors ( Payment Terminals, Point-of-Sale Machines, Store Controllers) are known and the endpoint’s message formats and communication protocols are also known. The challenge is testing and validating the translated incoming messages to various outbound endpoints, their communication protocols and message formats. Some end-points provide simulators (very very few) others will allow test access over leased lines and communication equipment over a separate test ip/port combination. This works well for our customers to perform user acceptance and certification -to these endpoints – this isn’t viable for regression and testing during phases and development before code delivery. We have solved some of this with various custom build response simulators that have basic logic – typically transaction amount prompted to provide alternating response messages. These response messages are built from message specs or are built from captured network traffic on test systems. We can only be sure we are simulating basic transaction types and request and response channels, however. Oh, and then there is always this problem.

Issuer side implementations are easier test – you can feed into the authorization host both simulated network and local transaction sets to test implemented authorization rules and other features.

testing.jpg

In 2009 we built and launched a new Issuing Payment Switch and tested it using Continuous Integration techniques. This system has 3 primary interfaces.

  1. Network – connected to an association’s network to recieve incoming transactions based on BIN ranges.
  2. Local – Card Management style interface – Manage Cardholder, Cards, and Accounts on the system – and allow local transaction sets to be performed.
  3. Flat File generation: Authorized File, Financial File, and a Card Status and Balances File. Flat file processing – clearing/settlement/reconciliation files.

Continuous Integration as defined by Martin Fowler:

Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily – leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly. This article is a quick overview of Continuous Integration summarizing the technique and its current usage.

CI’s general steps:

  1. Maintain a code repository
  2. Automate the build
  3. Make the build self-testing
  4. Everyone commits every day
  5. Every commit (to mainline) should be built
  6. Keep the build fast
  7. Test in a clone of the production environment
  8. Make it easy to get the latest deliverables
  9. Everyone can see the results of the latest build
  10. Automate Deployment

Our CI model is based on an implementation that is scheduled multiple times a day – It checks out the code from our software repository, compiles it, builds a new database and schema and required setup data, starts our software services up – performs unit tests, shutdown the software services, and we receive an email and attachments that tell us if the code compiled and the results of the unit tests and which ones were successful and unsuccessful. The email attachments we receive contain the run log zipped of the transactions, and a unit test report.
Our Unit tests are written using the Groovy Programming Language and we leverage the TestNG testing framework. We act as a network client to our switch which was built and ran from the current source, and perform both Network and Local Side testing. The system is also setup using some of the Local Transaction sets, Here is a short list of a few of the transaction types:
Local:

  • Add Cardholder
  • Add Card
  • Add Account
  • Debit Account (Load Funds)
  • Set Cardholder/Card/Account Status (Active/Lost/Stolen/Suspended/etc)
  • Local Debit and Credits
  • Balance Inquiry
  • Expire Authorization
  • View Transaction History

Network:

  • Authorization
  • Completions
  • Chargebacks
  • Representments
  • Force Posts
  • Returns
  • Reverals

The combination of local and network transaction types are tested against various test cases.
If we setup a Cardholder with AVS information and run an AVS Authorization – do we get the expected results, and for each AVS result code ? Does an authorization on a statused card get approved ? Do transactions with amounts greater then, equal to, or less then the cardholder’s available balance get authorized or declined properly ? Authorization on a Card not Found ? You get the idea.
We build and test our issuer platform a few times a day – each developer can also run the test suite locally on their development environment, this ensures that future changes doesn’t impact existing functionality. On a test failure – relevant information in included in the autotest emails so issue discovery can be identified by our business analysts and developers without logging into test systems.
Oh, and Please don’t break the build 😉

You don’t know until you know (or go into Production)

images-2.jpgOver the last six months we have been busy building and implementing an OLS.Switch Issuer Implementation with one of our customers and their banking and payment processing partners. It has been a process of reviewing and implementing message specifications, business processing requirements, authorization rules, clearing, settlement, flat file and reporting requirements. We also filtering external messages into our IMF – Internal Message Format based on ISO8583 v2003, build an interface Card Management functions via our local API’s and message sets. Building client simulators and trying to faithfully reproduce what happens when you are connected to a real system.

Testing on test systems is the next step – replacing our client simulators with other “test” systems that are driven by simulators by the processing gateway we interfaced to. Those simulators have limitations – in their configured test suites or test scripts, some require manual entry to send original data elements for subsequent transaction types, (e.g completions and reversals). We generate clearing and settlement files and match those to on-line test transactions, and our use cases.

After on-line testing, you connect to an “Association” test environment to do “Certification” and run a week’s worth of transactions through a wider test bed. Then you are certified, your BIN goes live and then you enter a production pilot mode – where you watch everything like a hawk.

You can do all of the simulated testing for both on-line transactions and off-line clearing and settlement files that you want – when you connect to the real world and do your first pilot transaction that is where most likely you will see something that wasn’t simulated, tested, or even included in certification, it happens. You need to be proactive, set-up reviews and manual interventions, perform file generation when you have staff available to review the output before it is released for further processing.

What have we seen :

  • Test environments that are not as robust as production or not setup with up-to-date releases.
  • Certain real-world examples are hard to simulate – reversals, time-outs.
  • Thinly-trafficked transactions: (chargeback, representment)…people can’t even define these much less create them in test
  • Poor or incorrect documentation of message specifications.
  • You receive Stand-In Advices or other transactions on-line that you don’t see in testing or certification.

Production pilot is a very important phase of testing – It is where you discover and address the < 1% of issues nobody catches in prior project life-cycles. What can happen, WILL happen. What you think might be something that will occur infrequently will bite you sooner, not later.

Protect Debug Info Transaction Participant

jPOS-EE has a very handy transaction participant called “Debug” its main purpose is to dump the contents of the jPOS’s Context. While this is very helpful in test modes and during development – The context remains “un-protected” and all of the data remains in the clear. Even the ProtectedLogListener and FSDProtectedLogListener will not protect data in the context.

Enter the ProtectDebugInfo Transaction Participant a configurable implementation I wrote based on some of Alejandro’s ideas, and one that lives in most of OLS’s payment products in various specific iterations.

It’s configuration looks like:

ProtectDebugInfo.png

Protecting your q2.log in this truncated example:

<entry key=‘FSDMESSAGE’>
<fsdmsg schema=‘file:cfg/fsd-base’>
account-number: ‘599999______0001’
</fsdmsg>
</entry>
<entry key=‘PAN’>599999______0001</entry>
<entry key=‘RESPONSE’>
<isomsg direction=“incoming”>
<field id=“0” value=“2110”/>
<field id=“2” value=“599999______0001”/>
<field id=“35” value=“599999______0001=____________________”/>
</isomsg>
</entry>
<entry key=‘REQUEST’>
<isomsg direction=“incoming”>
<field id=“0” value=“2100”/>
<field id=“2” value=“599999______0001”/>
<field id=“35” value=“599999______0001=____________________”/>
</isomsg>
</entry>

Encryption options from the POS/Payment Device/Terminal

matrix-computer-screen

There are a few different ways of implementing "encryption" from the POS/Payment Device/Terminal, I though I’d look at a few in a short post:

1) Tunneling – using existing applications and their connections over an encrypted tunnel   e.g.over a VPN, SSH, stunnel, etc. This approach doesn’t require any changes to devices or message formats or the payment "server"

2) Transport level – using TLS/SSL over TCP/IP Sockets – or at a higher level (web server/web service) using HTTPS. – devices needs to support the ability and make this type of connection, message formats are not modified.

3) Data Element or Field level — if you only want to encrypt the PAN or other specific fields, and these fields are defined to support the increased length required of the encrypted payload  — this requires changes to the message formats, devices and payment "server" software. Consider truncating the Account Number/Track Data in DE 2 or DE 35 in ISO8583  for use of displaying purposes on the terminals screen or receipt and consider using another Private ISO field for the payload.

The approach will depending on what the "devices" sending the transaction can support, both from a connection perspective as well as a software perspective. I’d also recommend consider using asymmetric encryption rather than symmetric here, as then the devices would not have the ability to "decrypt" as they would not have the private/secret key, and would help with eliminating private key storage at the device level if you choose option 3. There are implementations that use HSM’s and the DUKPUT algorithm as well.

We have an implementation of #3 that I wrote about here. — relevant paragraph below:

Some of our implementations of OLS.Switch supports field or data element level encryption that is passed on from the Point of Sale system to our switch. The main thing that allow us to perform this is that:  We or our customer "own/control" the POS message format to us and can adapt and handle the programming of the POS System and POS message formats – our account number fields are not limited to 16 digits – we can handle a much large encrypted value. So over the wire – these implementations are "protected" from eavesdropping or sniffing.

I plan to write more on E2EE (End to End Encryption) in the coming weeks as well, so stay tuned !

Operations Considerations Batch Files and Extract and Import jobs

ibm_data_processing_0

I’ve been working some batch file based jobs for a project here for OLS. There are two sides of this; sending "clearing" files of transactions in a defined format to third party which I will call the extract, and receiving "refresh" or table updates from this third party, I’ll refer to this as the import. The extract file contains financial transaction records, and the import file contains entity information such as merchant information. The Extract File Layout is quite standard an looks something like:

File Header

—-Merchant Header

——–Batch Header

————Detail and Addendum Record(s)

————Detail and Addendum Record(s)

——–Batch Trailer

—-Merchant Trailer

File Trailer

 

The File Layout for Import is a list of fields per record, nothing real fancy here.

I don’t want to spend too much time about the actual mechanics of the Extract and Import jobs themselves, but rather the Operational Considerations of this and others that we have performed:

Validity — You need to decide how to handle invalid records in a file or valid records without proper supporting data (transactions for a merchant that wasn’t setup in your system) ? You can write off the bad record to an exception file, and address it later, or you can reject the full file, the approach depends by implementation and requirements. We also mark files with a .bad extension if we detect they are invalid to help prevent subsequent processing steps, like transmitting a half-baked file. We also perform duplicate file checking as well as other validation steps.

Completeness — You need to make sure that you read in the entire file or extract a complete file. Monitoring controls such as checking the number of lines and file size in an Extract file, as well as checking the last time of a file for a specific record such as a File Trailer. Reconciliation between hash totals and amounts is also a good practice. On the import side you can count the number of lines or records in a totals from a trailer record and compare that to what was imported.

Timeliness — Some extracts take minutes and others hours, scheduling and monitoring the process is essential to perform data processing on a timely basis to other parties. Monitoring "check-points" in the process as well as a % of records completed help here to detect problems proactively with monitoring. Collect job performance metrics, it is valuable to keep track of and chart the total run time of each job and compare it to its history, to detect slowdown’s or to correlate increase or decrease in process times with external events or transaction growth.

Delivery — Consideration for the delivery of the file must be made. File Transfer procedures that address file naming conventions, steps to upload a complete file (upload as a .filepart extension and the rename to the full name upon complete transfer) as well as secure delivery as well as archiving locally or remotely, compression, and any file level encryption. It is also a good practice to reconnect to the file server and perform a directory listing on the files that you uploaded to confirm that they were transferred successfully.

Security — While account numbers and such are encrypted in databases (column level, file level, internal database level) the file specifications don’t allow for encrypted card-numbers, so both file level asymmetric encryption using the public key of the recipient as well as transport level encryption to send the file (see Delivery above) need to be considered. Archival files stored on disk will also need to be stored encrypted as well.

Troubleshooting/Restart Procedures — You need to develop procedures to support the following:

· re-sending failed files

· re-running the extract or import process for a specific date

· preventing duplication or invalid files or data.

The End is Just the Beginning — Operations is just the start of a process that has no end, it requires daily care and maintenance. These processes and controls need to work in harmony on a continuous basis and be able to be enhanced based upon the results of monitoring and other operational tasks.

Steps to setup a successful endpoint integration

secret_of_my_success

We connect and interface to about 20 different endpoints, each has its own message formats, connection methodologies, protocols, and different implementations of payment transactions sets, encryption and key management considerations, On-Line vs Offline transactions, Store and Forward (SAF), and flat file based reconciliation and clearing files among other details.

Here is a basic list of steps that we perform that assist us in getting the endpoint integration right:

1) Acquire the Host Message Format specifications and communication/protocol specifications that describe the communication and connection e.g: 2||4 byte length header, other header(s), length in network or host byte order, etc. and  read  and understand them.  Perform this same task for batch flat file based formats and file transfers.

2) Request sample dumps/files of some messages as well as for online transactions, network traces that show the communication bytes of the message, manually parse them and map them back to the spec. These should include samples of request/response exchanges.  [These should be from a test system.]  Examples often reveal nuances not documented or kept up to date in the spec.

3) Determine which message set and transactions are in play, There are many types of transactions that will be different depending on the type (e.g Debit, Credit, Gift) and characteristics of the transaction (e.g Manual, Swiped, MOTO). You need to determine which ones map to your business requirements and which ones will be used.

4) Request and/or develop certification scripts for your implementation (it helps to see what the other side thinks is important).

5) Schedule a walkthrough – This is the key event: the probability of our success is increased by scheduling and holding one or more walkthroughs.  The idea is to comb through the spec(s) and complementary material page by page, field by field and get complete alignment on the task.  It’s the absolute truth that the real nature of the interface comes out in the walkthrough. 

6) Develop and test your code to the specs and sample messages dumps. (Which is a small part of the project)

7) Use tools such as netcat  and wireshark as well as your application logs to confirm that you are sending what you think you are, and can provide detailed information to someone on the ‘other’ end of the endpoint to help troubleshoot integration. Don’t assume that you think you are sending what your think your code does, verify it locally first.

8) Develop a good or rather great working relationships with someone on the ‘other’ end of the endpoint, that can help assist with message format questions, perform traces, and check what you are sending to what they expect, and setup regular communication throughout the project.

Payment Systems Tools – Netcat

Netcat is described as the Swiss Army Knife for TCP/IP. I use it in two different ways:

1) Build a small server that dumps the contents it receives to a file. (Helpful for reverse engineering and debugging purposes)

$ nc -l -p 10000 > out.txt

This will set netcat to listen on port 10000 and redirect the output to the file out.txt

For Instance, I configured a client program that sends messages to a server and connected to my localhost:10000 instead:

$hd out.txt

00000000 00 4d 30 31 30 30 70 2000 00 00 c0 00 00 31 36  |.M0100p ……16|

00000010 34 31 31 31 31 31 31 31 31 31 31 31 31 31 31 31 |4111111111111111|

00000020 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 31 |0000000000000001|

00000030 30 30 31 32 33 34 35 36 31 32 33 34 35 36 30 31 |0012345612345601|

00000040 31 32 33 34 35 36 30 31 30 31 30 31 30 31 20      |12345601010101 |

This shows me that there is a 2 byte header (00 4d) of 77 bytes in front of this ISO8583 0100 (30 31 30 30) message.

2) Use it as a client to send message dumps. (Helpful for a quick and dirty method to send data over TCP/IP)

$ cat out.txt | nc localhost 10000

This will send the contents of out.txt and ‘pipe’ it to netcat that will attempt to connect to localhost on port 10000 and send the contents of out.txt.

When End-to-End Encryption is really not End-to-End.

I’m reading a lot about solutions that implement end-to-end encryption, where account numbers and track data is encrypted and can utilize a Hardware Security Module (HSM) and DUKPT or other encryption algorithms from the point-of-sale. I thought it important to share what data is actually encrypted in the payment system.

 

Here is a list in no particular order:

Gateways:

(contact me and I’ll add you if you are not listed)

 

Most of these are ISO’s that sell you a merchant account and access to their gateway that uses "end-to-end" encryption and that it will shift the PCI and PA-DSS burden from you to them, if you are a merchant, some claim you don’t even need to go through PCI compliance because you don’t have access to the card numbers or the encryption keys to decrypt the cards (Please also see this post on this subject).  This is all really good stuff, I’ve written about End-to-End Encryption before and am a big proponent of it. This can help prevent "sniffers" and card capturing malware from capturing track data and account numbers in the clear as they traverse your internal network. Attackers would either need to install card skimmers or gain access to encryption keys, or use brute force methods against captured encrypted data to capture data at your store.

But it isn’t really End-to-End Encryption.

Let look at two examples:

  1. A typical small merchant using a payment gateway
  2. A large retailer or processor/gateway that uses a payment switch

 

A typical small merchant that uses a payment gateway:

Slide2

 

A large retailer or processor/gateway that uses a payment switch

Slide1

( uses leased lines to connect directly to a Payment Processor (FDR, Chase/PaymentTech, Fifth Third, MPS, etc ) or Interchange Network (VisaNet, BankNet, etc )

Let’s say that you are using a gateway or even a switch that supports an encrypted message format from the point-of-sale (POS). The area in RED in each diagram shows where the account number traverses the payment networks in clear text. At the small merchant example from the Gateway to the rest of the network – the account number and track data and CVV2/CVC2 data is sent in the clear. In the direct connect model with the Payment Switch (which actually just operates as a local gateway) from the payment switch to the rest of the network. So End-to-End is really not End-to-End at all. (it depends on where you define end :)  This should also explain why End-to-End Encryption in its current state would not of prevented the breach at Heartland Payment Systems – as a processor they need to connect and communicate over the interchange networks using TCP/IP connection and ISO-8583 messages to these endpoints.

 

Why is this ?  The Payment interchange networks and message formats that processors and the Interchange networks use does not support this in their current message formats (primarily ISO-8583) There is no room in the current implementations of Visa’s Base1, MasterCard’s MIP, or FDR’s message formats for example. Data Elements can be added to support this, but would require massive changes to Payment Systems infrastructures and systems.

 

Does any one have any solutions for this ? Please provide comments below — I’ll provide a follow-up blog post with some of my ideas.

 

Remember that End-to-End is really not End-to-End, it may shift or transfer some of the compliance "burden"  from the merchant to that of the processor, but still exists in clear text on private networks and at processors.  Oh, and tokenization and secure card vaults would work the same way here, the cards need to be translated to their raw value to ride the payment networks.

SPSP Webinar: Secure Hashing and Scope Reduction Methods

Hashing is one of the most basic constructs of Payment Systems development, as it relates to both secure storage of passwords as well as a method of rendering card numbers unreadable (PCI 3.4), you can also use the hash as a lookup or index in a database if you are performing column level encryption especially asymmetric encryption. I hope the presentation includes the difference between "secure hashing" and "naked hashing"

I encourage my readers to attend the following webcast by the Security of Payment Security Professionals, of which I am a member.

 

Wednesday, January 21th, 2009 the Society of Payment Security Professionals is hosting a webinar on “Secure Hashing and Scope Reduction Methods”.  You can register for this event online.

 

I plan to do a follow-up post on this and perhaps with a few code examples 🙂

jPOS Twitter Integration

For fun I decided to develop a TwitterLogListener class for the jPOS Framework. It leverages JTwitter – the Java library for the Twitter API from Daniel Winterstein. With this, you can send alerts and/or messages to a twitter account for monitoring from a jPOS application.

You need to configure a TwitterLogListener in your logger:

<log-listener class="org.jpos.util.TwitterLogListener">
 <property name="twitterUsername" value="paymentsystems" />
 <property name="twitterPassword" value="SuperSecretPassword" />
 <property name="tags" value="twitter" />
 <property name="prefix" value="[jPOS]"/>
 </log-listener>

And you need to Log an event with a ‘twitter’ tag:

<script name="test" logger="Q2">
     import org.jpos.util.*;
         for (int i=0; qbean.running(); i++) {
              LogEvent evt = log.createLogEvent ("twitter");
              evt.addMessage ("This is a Twitter message test from jPOS! " +   i);
              Logger.log (evt);
	           Thread.sleep (10000L);
       }
</script>

Look at the messages in a twitter account: http://twitter.com/paymentsystems 🙂

For more details on this see this post: http://is.gd/eDbI