Apple's Enhanced Format and our awesome workaround!

The problems with Apple's new Enhanced notification format:

1. Apple never sends a response for messages that succeeded, so we can't block on reading the stream since it may never have any data available to read!
2. An error response may not immediately be returned before you have a chance to send more notifications to the stream, but before Apple closes the connection. These notifications that might still ‘make it through’ are left in limbo. They are never acknowledged or delivered, and never have error responses returned for them.

Workarounds

I’ve looked at many libraries, written in many different languages, to see how they worked around this problem. In about 99% of the cases I’ve observed, they all use the same, sadly inefficient approach: Waiting.

So we have a connection to Apple’s APNS server, and we want to send notifications over that connection repeatedly, and as fast as possible. Apple never sends us a response if a notification was sent successfully, but if one failed, they will send us an error response and close our connection.
The problem is, if we keep sending notifications over and over again, we might send a second, or third notification before Apple ever sends us an error response for the first one that failed. If this happens, the second and third notifications are never delivered, and are lost forever. The easiest way to solve this is to asynchronously read from the connection stream, waiting for an error response. In doing so, however, this means you must also wait a little while after you write each notification to the stream to see if your asynchronous read ever receives anything. You can’t just do a synchronous blocking read on the stream since you’d be waiting indefinitely if the notification succeeded (since Apple sends no response in this case). To make matters worse, Apple doesn’t guarantee how quickly an error response will be sent to us. I’ve seen libraries wait for an error response from anywhere between 100 to 500 milliseconds.

It should be painfully obvious to you by now why this approach is flawed. If you have to wait even 100 milliseconds after sending every notification, that would take you almost 28 hours to send 1,000,000 notifications over a single connection!

Most libraries employ the use of multiple connections to circumvent this new issue they’ve created for themselves by waiting between each notification. If you use 10 connections to apple’s servers, that cuts your time down to 2.8 hours. This is better, but why should it take 10 connections 2.8 hours to deliver a theoretical maximum of only 300MB of data (1,000,000 notifications * 301 bytes maximum size per notification)? This is asinine!

PushSharp's Awesomesauce Workaround

I just couldn’t stand the thought of wasting 100-500 milliseconds per notification sent. I figured there had to be a better way, and I think I’ve found it!

PushSharp employs a technique that is fairly easy in theory, and was a bit more difficult to implement in code. Each time a notification is written to the connection stream, it is then added to a ‘Sent’ queue. If an error response is received, the corresponding notification is located in the ‘Sent’ queue (by its identifier). Anything before the error-causing notification in the queue is removed and assumed to be successfully sent. Anything after the error-causing notification is assumed to be lost, and re-queued to the ‘To Send’ queue to be tried again.

There is also a cleanup thread running that constantly checks the oldest notification in the ‘Sent’ queue to see if it’s older than a few seconds and if so, it is assumed to have been successfully sent and is removed from the ‘Sent’ queue. This effectively moves the waiting period for an error response outside of the scope of the connection to the APNS servers.