Modelling cost of delay of technical debt

Don Reinertsen and others propose Cost of Delay as a metric for prioritising work in product delivery teams (read: agile). This can be applied building new features and to repaying technical debt.

There are multiple types of technical debt:

• Issues which are the root cause of failures affecting customers and therefore affecting profits, like a database query that doesn’t scale and makes your site unusably slow at busy times
• Issues which might cause a failure in the future, like a single point of failure
• System design issues which make the software harder to maintain and affect cycle time, like lack of test coverage or god objects

Could we build a model for each type of debt? This would allow us to capture a few inputs for each new issue and calculate the cost of delay quickly. We could also update our cost of delay calculations when we gain new information, either by plugging in new inputs or by changing the assumptions in the model.

Issues affecting product availability

Cost of delay per week = probability of failure occuring each week * mean time
to recovery in hours * cost of hour of downtime.

We can create a general model for technical debt which has caused failures in the past or which might cause failures in the future. For past failures, we can calculate the probability of failure each week from availability metrics or incident post-mortems. For things that might happen in the future, we must estimate that probability, but the input is the same. You can use base rates from other sources, e.g. if your point of failure is a single server, you could use industry server failure rate data to guide your estimation.

Likewise, the mean time to recovery, how long it takes to get everything working normally again, can be calculated from records or estimated based on similar failures.

The cost of an hour of downtime depends on the type of failure. It might be any revenue you make for an hour if your whole site or app is down. If non-transactional features are not working, the cost might be more to your reputation, which knocks on to your revenue/profit in ways which are hard to predict. The trick is to come up with a consistent model for different issues, based on your company goals. You may not know the monetary cost of a flakey application, but if one organisation and your customers value reliability, then you should weight it high in your prioritisation.

To keep things simple, we could count the cost of an hour of downtime as the lifetime value of the customers we would have acquired in that hour, and assume that our returning customers are perfectly loyal - that we will not churn a single one. obviously that’s naive, but the extent to which that introduces error into the model varies from business to business.

Issues that increase cycle time

Cost of delay per week = total dev hours lost per week

A developer hour is worth (average annual salary / working hours in a year) * employee cost multipler. An employee costs not just their salary. They receive other benefits, need space to work, a desk, a computer, etc. We represent this as the employee cost multiplier.

Increased cycle time has a large effect on which opportunities you pursue. What new features would we build and launch if our cycle times were 10 times shorter? What profits would they generate? This is impossible to estimate. The simplest approach from the perspective of cost of delay is not to try.

Culture and Policy

As a regulated financial services company, Wonga has a number of clearly-defined policies covering aspects of its operations from marketing to customer services to software engineering. The goal of these is to ensure proper care is taken and that mistakes are avoided.

Other organisations, notably Facebook, rely heavily on culture to avoid issues. Anyone can directly access their servers and make changes, but their culture is such that no one ever does.

Culture is flexible, robust to change and everybody is empowered to make the decisions, but it’s tough to build and maintain. Policies are centralised and inflexible, however they may be easier to scale.

Comparison

Go Faster With Angular

Go faster with Angular – Ben Godfrey at The Lead Developer 2015 from White October on Vimeo.

Generate JSON Index Of Images With awk and identify

I maintain a manifest file for a Trigger.io mobile app which includes a set of images indexed by size. I’ve taken to generating the JSON with awk and Imagemagick’s identify command:

identify returns a wealth of information about an image file:

$ identify Default~iphone.png
Default~iphone.png[6] PNG 320x480 320x480+0+0 8-bit sRGB 11.5KB 0.000u 0:00.000

I want to transform that into some JSON. awk is designed for exactly this kind of text transformation problem.

$ identify *.png | awk '{gsub("\.png.*$", ".png", $1); printf "\"%s\": \"%s\",\n", $3, $1}' | sort
"1024x748": "Default-Portrait@2x~ipad.png",
"1024x768": "Default-Landscape~ipad.png",
"2048x1496": "Default-Landscape@2x~ipad.png",
"320x480": "Default~iphone.png",
"640x1136": "Default-568h@2x~iphone.png",
"640x960": "Default@2x~iphone.png",
"768x1004": "Default-Portrait~ipad.png",

Copy and paste into manifest, remove trailing comma, make cup of tea.

EDIT: Achieve the same effect with the -format parameter for identify (-format docs here):

identify -format '"%wx%h": "%f",\n' *.png

Type-check Your JavaScript With TypeScript

TypeScript is a superset of JavaScript with optional static typing. The DefinitelyTyped project contains type definitions for major JavaScript libraries contributed by the community. You can use these definitions and TypeScript’s compiler to check your code for errors, just by adding references.

We have a large AngularJS project. I’d like to type-check all our code and maybe find a few bugs.

Install TypeScript

Clone DefinitelyTyped to references

The DefinitelyTyped project provides type definitions for AngularJS, Jasmine and about 100 other common JavaScript libraries.

To keep it simple, I’m just going to clone the whole lot into my project, where I can reference anything I need.

Rename All .js Files To .ts

By renaming all our files, we let tsc and other tools know the code should be treated as TypeScript.

Let’s give our unix skills a workout and do this with a find one-liner:

Create References Includes For The Project

TypeScript code needs to be annotated with comments to link to the type definitions we downloaded from DefinitelyTyped. E.g. for a Jasmine spec which tests an Angular controller, we might need the following block at the top of the file:

You’ll need a reference for each library you’re using. To simplify things, let’s create a couple of reference files which define the shared references for our module definitions and test specs and include those in our code.

Note that test.ts references module.ts.

Add References To All .ts Files

Now, let’s add the references to every TypeScript file with a couple more find one-liners (requires GNU sed, if you’re on OS X, install with brew install gnu-sed --default-names).

Phew!

The path attribute needs to be right. Luckily, all of the code in our project is 5 levels deep in the directory structure. If your code doesn’t have such a regular structure, you’ll need to do something more clever.

Compile Everything With tsc

Finally, let’s run everything through the TypeScript compiler and see the issues we have to resolve:

Adding Definitions For Missing Modules

Adding type definitions for your own modules or 3rd-party modules missing from DefinitelyTyped is straightforward. A few examples:

“Bubble” Lines Up And Down Like Sublime Text In Vim

NB: You can see this hack in my .vim repo on GitHub.

Sublime Text allows you to use ^⌘↑ and ^⌘↓ to “bubble” a selected block of text up and down in a file.

For a single line, we could just use the dd, j, p and P motion and editing commands to move the current line around:

However, this doesn’t work for multiline selections and it overwrites the default register, wiping anything you’ve previously deleted to there.

A better solution is to use the :move command. We can safely use it in both normal and visual mode because it works with multiline blocks:

We pass locations relative to the selection to :move, '<-2 is 2 lines before the first line of the selection, '>+1 is one line after the last line.

In the visual mode mappings, we use gv to return to visual mode with the previous selection highlighted after the :move is complete.

By default, MacVim maps ^⌘↑ and ^⌘↓ to move around the quickfix list. We can disable that by setting the menu option’s key to <nop>. Add this to your .gvimrc:

If you hit ^⌘↑ or ^⌘↓ with a character-wise visual selection, the whole line will still be moved.

You can map different keys for different platforms.

We can also extend this to fix indentation of the block as you move it:

This is a bit unpredictable in my experience. YMMV.

Vim Auto-closers Compared

I type {, Vim types } for me. Vim has a number of plugins to do this and each does the job slightly differently.

1. delimitMate by Raimondi (484 GitHub stars)
2. Auto-Pairs by Miao Jiang (181)
3. smartinput by Kana Natsuno (239)
4. AutoClose by Karl Guertin (44)
5. AutoClose by Townk (212)

My Requirements

1. Don’t break HTML templates that use double braces, e.g. {{name}}
2. Create a new block easily, i.e. type { and the matching } is inserted and the cursor is positioned on a blank line between the two, correctly indented.
3. Handle escaped string terminators
4. Don’t break other plugins
5. Understand that " is the comment character in Vim
6. BONUS! Press return within a string splits line, adds delimiters and a string concatenation operator.

AutoClose and AutoClose

The AutoClose plugins are much simpler than the 3 tested above. If you’re using one and happy, great. If you’re not using anything, try delimitMate, Auto-Pairs or smartinput.

The Winner: Auto-Pairs

I’m going with Auto-Pairs unless it breaks or I find something better.

I’d be interested to investigate smartinput a bit more, but Auto-Pairs gets the job done with minimal effort.

What About String Splitting?

None of the tested plugins wrap strings. One way to implement this would be to use formatexpr and formatprg, which tells Vim to run lines through a formatter.

Should You Move Away From Heroku To Protect Your Users?

Heroku provides an SSL add-on for encrypting traffic between browsers and the Heroku network.

However, SSL is terminated at the Heroku routing layer. Traffic is then passed unecrypted through the network to your application. That network is AWS.

A Heroku support agent confirmed this by email:

“I have confirmed with our routing team that HTTPS traffic will be terminated at the SSL Endpoint, and passed over to the runtime (where your application resides) in plaintext.”

If your customers are submitting sensitive data to your app, like payment card details or personally-identifiable information, then that data is transmitted between nodes in Heroku’s network without encryption.

Threat?

Does this mean your traffic can be intercepted by other users? I’m not a security expert but I believe it’s a risk you might not want to take with customer data.

Apps running on Heroku do not have root access, but it’s possible to sniff network traffic even without root access, e.g. Sniffing with Wireshark as a Non-Root User.

Heroku’s Security policy claims that packet sniffing is not possible:

“Managed firewalls prevent IP, MAC, and ARP spoofing on the network and between virtual hosts to ensure spoofing is not possible. Packet sniffing is prevented by infrastructure including the hypervisor which will not deliver traffic to an interface which it is not addressed to. Heroku utilizes application isolation, operating system restrictions, and encrypted connections to further ensure risk is mitigated at all levels.”

About a month ago, Heroku revised their networking model to improve isolation between dynos:

“Previously, network interfaces were shared between multiple dynos. […] The shared network interface […] resulted in a low grade information leak where one dyno could obtain some information about connections made by other dynos. This information did not include any customer data or other customer identifying information. But it broke the core principle of tenant isolation. With the new networking model, dynos now have fully isolated network configurations.”

PCI

The PCI standard requires that sensitive data be encrypted at rest and when transmitted over public networks, but not internal networks. Heroku’s network is surely public, despite their considerable pains to make it look and feel like an internal network.

Alternatives

AWS allows traffic to be encrypted between an Elastic Load Balancer and EC2 instances via a feature called backend server authentication.