Sunday, July 8, 2012

Lessons Learnt - Building an SDP

I was recently asked what lessons had I learnt in regards to building an SDP and I thought it was an interesting question in many aspects and a question I would attempt explore in the form of a post on this blog.
Now to prefix this post I am obviously only a cog in the many wheels it takes to build a product like a Single Dealer Platform and these opinions are solely mine and many members of the team, past and present, will obviously ave differing opinions. The Alpha platform has had its ups and downs, as had the whole market, however I am pleased to say it is a successful and profitable multi asset platform and I am proud to have played a part in its success. That being said there are things that could have smoothed the sometime bumpy ride. However I dont feel it is fair to throw stones as many of these problems are industry wide and we have interesting issue of having two industries to deal with, finance and technology and to ease the thought process in this post I will take the question and rephrase to if you had to build an SDP what would focus on? 

Form Key Relationships

Business Buy-In

The platform should be supporting the business and not conflicting with exiting goals and other teams strategies. If you have sales teams or team members that feel the product is competition you need to get them on board or you risk having to work against their negative energy. Sales and traders can have significant sway in the bank so obviously having them as allies make for a much smoother project. The opposition to new products is not isolated to these profiles. Anyone who has developed software internal in a corporation will have come up against this, so it is a common problem we must over come.  Given the platform should enhance their daily dealings by being able to provide up and cross selling facilities as well as a portal-like experience into research and post trade facilities, the product should actually be a good thing for the sales desks.  

Backend Systems

Be prepared  - Front end systems like SDPs do not work without good service  from the various providers required to price and book trades. There is more to an SDP than this, however these systems are key. Work closely with these teams - the are your life blood. When delivery times are tight you may have to call in favours from these teams. At a business level these teams goals must have some alignment with that of the SDP or their work slate will continually be serving other priorities. To have a beautiful fornt end that is not integrated is an effort in futility and will quickly raise concerns from the bill payers.

Clear Vision

Firstly know your strengths and play to them. Every bank cant not be the best in FX. Although most SDPs will have a strong FX and Fixed income feature set, trying to have the best prices and lowest latency is a costly venture. If you are going to try to take on the big boys you had better have a strong financial backer. If you are not one of those players, acknowledge it and find where your strengths are and use them. A strong product owner will be key here. His role will be to clearly understand the direction of the bank and to know what will drive the platform. He should have a clear understanding of our income sources and our user base. Defining core users is also key here. This will provide direction in the type of platform we will technically deliver : Desktop, RIA, Web, mobile or a combination should be a business decision based on the users of the platform, not based on the latest technical craze? Lastly we must have clear success criteria for each deliverable and the platform as a whole. When this is clear the team can easily all move in the same direction.

Deliver the Goods

In my opinion: Delivery is king. Continuous delivery is a logical progression of a well oiled development team. To achieve continuous delivery you should address some key, common concerns early in the project. Authentication and authorisation, handling of entitlements, security concerns, auditing, logging and other such cross cutting concerns are all usually defined by the bank at a department or group level. IMO these should be addressed in the first iteration while other team members could possibly be working on PoCs realting to functional deliverables. This is also a good time to investigate some standard deployment concerns such as the handling of independent and aggregated modules, dependency management, versioning, forward and backward compatible migrations. From a higher level architectural stance the team will also need to define how they could deal with future scalability, reliability and regional requirements. Do we have the infrastructure to provide five 9s to a global audience with latency under 10ms? If not now is a good time to let the business know so they can lead the charge on making the necessary changes if required or so they can have their expectations realigned appropriately. The reason I prefer to address this early is because the cost is so much higher when done later in the project and delivery becomes an expectation and a core team value. It also raises the notion of non functional cost to the business; fives 9's up time is not free, nor are follow the sun deployments, the sooner the business are aware of this the better for everyone.

Justify your Costs

By understanding what brings value to the business we can more appropriately prioritise our work. One major issue that I have encountered on many projects is the confusion of the prioritisation of features that generating income versus those that prevent losses or reduce the cost of doing business. Both are of value.
Features that I can see SDPs typically spending too much on are technical "play things" (new frameworks etc), performance and low value "nice to haves". A good Product owner that has a good relation with his technical leads and architects should be able to address these issues easily.

Typical neglected features are resiliency, monitoring and work related to minimising maintenance cost. No one wants to pay for any of these but we often dont realise we want the consequence of not having them even less. The unfortunate thing is we often discover this far too late. IMO the key ways of reducing maintenance cost is clean well tested code (ie employing BDD and TDD).
Another key issue in resource allocation. It is easy to think due to the large mount of money involved in building these platforms that a large number of people should be involved. For any one who has read Fred Brooks' Mythical Man-Month or has managed very large software teams will know that the communication overhead of throwing more resources at the problem will most likely only exacerbate the problem. In addition to this hiring the wrong resources can also be catastrophic. Because of this I prefer, like many agile teams, to have small self managing teams comprised of exceptional people that are completely accountable for the delivery of their feature. The cost IMO of having unaccountable people or inappropriately skilled people on the delivery team is often just too high.


The person that originally asked the question would possibly be surprised that my answer wasn't more technical as I think he wanted to hear "Use product X". I believe the tools used should ideally be representative of the way the team works and the product that needs to be delivered.  Now this is not always feasible as there will be company wide mandates in place, however how that team works within those confines can often be quite impressive.

Like any project I believe a team led well, with a common clear direction and with members driven to deliver success will find a way.

A couple of books to me really resonate in this area, the Brooks classic as mentioned earlier and surprisingly to some The Art of Scalability. I originally bought this while working on the streaming price distribution components for Alpha but to be honest go more benefit in the non technical sections. The authors strongly believe a scalable system comes for a well oiled team that allows for scaling. If you are looking at starting a large scale system like an SDP would recommend this book for more than just the excellent technical read it is.

Back to the code

The last couple of years have been a whirlwind and in my time running a relatively large part of a relatively large project I was often unable to spend time keeping up to date in any technology other than that at hand at the present time.  I have finally been able to put my head up for some air so to speak and get my head back in the tech world that I feel I have turned my back on in the last 18 months. I may post more about this soon however for now it is updates on stuff I have been playing with lately.

Im quite keen to dive back into JavaScript, which would have sounded weird to me 10 years ago as it was a language that at the time I thought was just a hack fest. Having grown up somewhat and having more experience with the language it is obvious that it deserves a bit more respect than that. With HTML 5 and Windows 8 fully harnessing javascript and the acknowledgement of node.js as a legitimate backend technology, JavaScript is here to stay. As I started diving back into the JS world I realised that my knowledge of JS fundamentals was woefully lacking.

My first step back into the JS journey is getting my head around what JS actually is (vs what I thought it was). Being a prototype based language and not a class based language, it offers different ways of solving similar solutions to languages I'm more used to. This does not mean you cant use conventional OO based techniques, we know you can, but it means you may have to rethink how you could best solve those problems. Most of the change comes into the definition of the objects, not so much the use of them... well... kinda. For more good reading on JS for people with OO background have a look into this nice blog series on the topic.

One of the major steps forward in JS, as the language has not really changed, are the new APIs available.  I have the fortune of having some friends that are in web based start ups that I can tap into as a source of up-to-date trends and whats hot in the "valley". A few of those things that I have picked up lately are UnderScore.js, Backbone.js and in the testing realm Jasmine and Sinon for isolation/mocking.

UnderScore is a handy utility library that provides a bunch of assistance functions especially helpfully, amongst other things, in the handling of collections. As it is a utility library its super helpful and at 1000 lines in a single file, if you really want to know how it all works, you are only an hour or two away from figuring it all out. I would certainly read through the website as it quickly and succinctly describes all its functions. Being a .Net fan-boi, Linq is dear to my heart, so I was please to see that method chaining was supported albeit not as cleanly as something like linq.js, but hey you cant have everything :).

Backbone to me is much more interesting. I have many times bumbled my way through DOM manipulation handling various client side events using JS.  Backbone comes to the rescue by providing a Model-View binding approach that I am much more comfortable with (at least conceptually) and leads to clean SoC and a testable model, which is all good stuff.
Learning the basics of Backbone is not super obvious and the docs IMO are not super helpful either (I have been spoilt lately learning rails with a tonne of good docs). Fortunately the PeepCode series has been a fantastic resource with 3 video tutorials and the accompanying source code so you can follow along.

Now there is no point in saying Backbone is super testable without testing it and Jasmine with Sinon fit in nicely here. Jasmine and Sinon are covered in the PeepCode series and also in this nice blog series.
Backbone goes a long way in addressing my issues that I had with my previous "DOM manipulation" heavy approach, and Jasmine fits hand and hand with an improve testing approach. If you are used to BDD style frameworks like Cucumber or SpecFlow then Jasmine maybe for you, however if you prefer standard xUnit style testing there is always QUnit.

Sinon is also a nice fit with testing as Backbone has some convention based features that allow you to hit backend servers for model updates and persistence; Sinon steps in and saves you some pretty serious pain if you were serious testing your models over the wire.

Hopefully I will give some more feedback on these various APIs and approaches as I get a bit more experience in them, however for now it is just a link fest. Depending on how well my ventures into Backbone go I may or may not use Reactive Extensions for JS. Reactive Extension (RX) is something that is pervasive on the code base I currently work on so it is an obvious choice for me to experiment with. RX is a brilliant API for event based data streams and is something that I think will become common place in .Net in the very near future. For more info check out Intro To Rx

See y'all soon.