How to measure the progress of your Lean Startup

This and following articles will give a detailed and practical example of how you can measure a lean startup, with one of the modern tools for Intelligence and analytics, TrueSight Intelligence. This and the next articles will be written as I build the solution and company. I will also focus on my lessons learned along the way.

My company

First a short introduction to the company.

I am building my company SalesValue. The company develops and delivers a Software as a Service (SaaS), solution for  Sales Enablement. The ambition is to help sales teams in B2B organisations help their customers. Help the customers make the right choices and get value from the solutions they buy.

The emphasis is on helping the customers. We believe, that most customers like to be helped and guided, but do not like to get pushed into making a decision that might not be in their best interest. We also believe, that the right sales people is motivated by helping their customers - not by a large cash bonus. We want to help these sales people help their customers.

We help the sales team by helping them find the right tools at the right time. SalesValue is the link between your CRM solution and your file storages. By connecting the 2, we help you sales team find the best guides, whitepapers, presentations, proposals, etc. that helps their customers make the right choice.

We are inspired by CEB's Challenger Sale and Challenger Customer, and help our customers implement the learnings from these books in their organisation.

A Lean startup

We build the company, SalesValue, as a Lean Startup. A key element of this is the Validated Learning, which can be described as the following process:

1. Isolate critical assumptions for testing
2. Draft your hypothesis to be tested
3. Build an experiment
4. Measure the results
5. Collect the data and learning in a systematic way

One of the first hypothesis that we want to test is:

• We believe sales people at least twice times weekly will search for content using categories and tags like customers, products, etc.  to find knowledge to help their customers

We use TrueSight Intelligence to measure the usage of our solution and collect the data for the experiments that we build.

Using TrueSight Intelligence

TrueSight Intelligence (TSI) is a great tool to easily collect data directly as the users are interacting with the solution. TSI provides a nice interface for importing metrics and events, and gives us a fast way for setting up KPI's and reports that we can use to measure the results of our hypothesis.

TSI gives us 2 ways to collect data:

1. Metrics
   • Time series data like how the number of users, files, products, etc. develop over time
   • 
2. Events
   • One time events like logins, views, likes, etc.
   • 

Metrics

We want to measure the following metrics, that helps us evaluate the current state of the solution:

• Users
• Files
• Customers
• Products
• Projects and opportunities
• Comments

TSI also makes it very easy to collect information of system health, like availability and response time. This is helpful to correlate system performance with user actions.

In TSI we first need to configure the metrics that we want to measure. From the setup menu select metrics, and add the metrics. We don't measure second-by-second, so we have changed the default resolution to 1 hour (3600000 ms).

We can then add metrics from SalesValue on a regular basis through a scheduled task. We use the following command in my Ruby on Rails application to create the metrics in TSI.

task :truesight_intelligence_metrics => :environment do
  # Basic parameters
  uri = URI("https://api.truesight.bmc.com/v1/measurements")
  req = Net::HTTP::Post.new(uri, 'Content-Type' => 'application/json')
  req.basic_auth ENV['truesight_account_name'], ENV['truesight_account_password']

  # Users
  req.body = {
      "source": "IT_Value",
      "metric": "USERS",
      "measure": User.count,
      "metadata": {
          "app_id": "IT_Value"
      }
  }.to_json
  res = Net::HTTP.start(uri.hostname, uri.port, :use_ssl => true) do |http|
    http.request(req)
  end
  puts "TSI user response - #{res.body}"
...

In TSI we can then create a dashboard for our app, that gives an overview of the current status and development of these key metrics.

Events

We also want track the user behavior on my solution, to get a feeling of how they use the solution. We use TrueSight Intelligence event to track this. We track the following behaviors initially:

• View file
• Download file
• Like file
• Mark file as best practice file

We use the following code to add events:

module ClassMethods
  def create_tsi_event(event_class, user_name, org_name, title, message)
    # Basic parameters
    uri = URI("https://api.truesight.bmc.com/v1/events")
    req = Net::HTTP::Post.new(uri, 'Content-Type' => 'application/json')
    req.basic_auth ENV['truesight_account_name'], ENV['truesight_account_password']

    # Event
    req.body =  {
        "fingerprintFields": [
            "my_id"
        ],
        "eventClass": event_class,
        "source": {
            "ref": "SalesValue",
            "type": "host"
        },
        "message": message,
        "title": title,
        "properties": {
            "user_name": user_name,
            "org_name": org_name,
            "my_id": Device.friendly_token.first(20)
        },
        "severity": "INFO",
        "status": "CLOSED",
        "tags": [
            event_class,
            "app_id:SalesValue"
        ]
    }.to_json


    res = Net::HTTP.start(uri.hostname, uri.port, :use_ssl => true) do |http|
      http.request(req)
    end
    puts "TSI #{event_class} response - #{res.body}"
  end
end

A small trick we needed help with was setting up the events so that they showed up as individual events and not get grouped together. The trick was to ensure that the fingerprint field is unique, but the unique value needs to go into the properties (my_id) and then reference this property from fingerprint field. We use the Device friendly token to generate a unique key.

We can then create a similar dashboard of the events over time, which gives a very useful overview of how the users use the solution. When we deploy updates to the solution we can quickly measure the changes in user behavior.

SalesValue will be available to the public by April 2018, but you can sign up for early beta access here.