Agile Prioritization and Finishing Projects & Features

On every team, you have a number of projects that are not-quite completed.  Agile encourages it – by frequently re-prioritizing based on value, almost by definition, you end up postponing the least important parts of a project or epic.

I am not talking about things that can never be truly “done,” and become expensive to improve.  For example, you may be able to increase conversion from 3% to 3.5%, but to get to 4% become exponentially harder.

I am talking about things that do have a bounded end – projects, deployments, or specific new features.

To give some examples:

  • We changed the way we initialize tests a functional few months ago. We time-boxed the exercise and as part of it the developer created a new API and migrated some of the tests to use the new API.  However, there were many tests that did not get migrated.  So we had two ways to initialize data in our tests, and of course, some developers continued using the old way as a result (since when they consulted some samples, that’s what they found).
  • An infrastructure project that completed the main change, but then the team was refocused so it wasn’t documented. As a result, another person may not be able to understand the current setup, or even the same people, trying to make another change in a few months, will forget how it was done.
  • A new feature was rolled out and it was working fine.  However, there were other parts of the system that looked similar to the user, but behaved differently.  You have more and more users confused and they trust your system less.

I am not advocating to adopt an extreme definition of done, i.e. don’t release until a feature is fully integrated / finished.  I think even if functionality is not fully consistent within the system, much of the time, releasing it sooner allows for faster learning to occur (maybe the feature is a bad idea).  But I am advocating for making sure you pay attention to what is left over.

Those last few items out of a large list should be given more weight.  If possible, they should to be prioritized soon after the bulk of the project, even if you have higher value items to do.  When you have too much of these loose ends everywhere, you end up having a lot of problems, some of which can be very difficult to diagnose and fix.  They carry little value in themselves, but the value is magnified by the fact that you can cross one more large project fully off the list.   You will no longer need to have the cognitive overhead of remembering that this thing is a vestige of something bigger that’s almost gone.

Here is to fully finishing more projects in 2015!

Happy New Year!

Architecture Seams

We have been planning a large scale refactoring project, and as part of the initial planning I’ve done an analysis of what I call architecture seams in our software.  I found it useful so wanted to share what I’ve done.  I plan to do this analysis at regular intervals, or rather update it now that the ground work is done.

An Architecture Seam is my expansion on Michael Feathers’ word Seam from book Working Effectively with Legacy Code.  Michael Feather’s definition of a seam:

A seam is a place where you can alter behavior in your program without editing in that place.  Every seam has an enabling point, a place where you can make the decision to use one behavior or another.

Feathers is using it on the code level; I am using it on architecture level – be it an application module, layer, deployment, or some other point of change in your system.

What is an Architecture Seam

I use seams for leverage points in architecture or deployment – places where you have existing or easy to get decoupling. It could even be a logical decoupling – i.e. the code is actually intertwined, but you feel like if you were building the system again you would work to keep it separate.   Once a seam is identified, you can do an assessment of whether the seam is already in place, or something that is potentially useful, and also its costs and benefits.  Some seams add overhead and take active work to maintain.

Here are three examples:

  1. We had a part in our architecture where we synchronized data  between two applications in two directions through an HTTP API.   It allowed us to deploy our system with multiple independent clusters.  However, we realized this was only used in one specific place, was expensive to maintain and was a source of multiple problems in development, operations, and testing.  It was a very costly seam to maintain.
  2. Another seam was an internal HTTP API we maintained between our backend and our UI (the UI is written using JSF, so there was an alternative of having it in process).  We wanted to have strict separation in this place with intention of releasing a public RESTful API in the future.  Also, since the API was one-way request/response, the overhead was much less than in previous example.
  3. Another seam we had gave us ability to separate our job server and API backend separately.  The advantage would be separating our UI users and jobs, so a catastrophic failure in the job failure (such as memory leak) would not affect business users using the application. This is an example of unexploited seam.  Our technology was designed to support it, but we weren’t taking advantage of it.   We made a plan to test and use the seam and currently have it deployed in this way.

Why Manage Seams

Any good technical leader or architect already has an intuitive understanding of the architecture break points.  However, it’s easy to forget some aspect of the system, so a systematic analysis is useful.  Seams can happen on multiple levels, from software architecture, to functional areas, to deployment, to infrastructure architecture.  I found formalizing the seams and listing them out allowed us to think a bit deeper and better evaluate impact of decisions.  We added new seams in areas where we need more flexibility and and worked to remove existing seams that were too costly to maintain.

This becomes especially important as your application complexity or data volume grows substantially.   As complexity grows, certain things that used to be easy become hard at higher volumes.  This is when exploiting seams starts becoming especially useful.   Sometimes it becomes obvious that something that you’ve been living for a while needs to find another life, like a separate application or a different technical approach.

Big Technical Projects

Like I said above, we did this analysis as part of the planning phase for a large scale refactoring project (topic for a later post).   In the refactoring, we are exploiting some architecture seams we have in place, and introducing some new ones to give us more flexibility in other areas.  However, the main benefit is from removing a seam that we no longer need that I described above – a place where we need to synchronize data between what effectively are separate applications.   It introduces enough development overhead and problems that removing it will significantly simplify the system, speed up development and reduce operational costs.   Win on many levels.

Revisiting your seams analysis helps you make sure your architecture supports the business needs, and identify potential problems.  It allows you to better manage large technical projects by keeping the different moving parts that could be affected organized.  In a future post, I’ll talk about the different types of seams, and perhaps got into a bit more detail about the mechanics of the analysis.    If you found this useful, let me know.

Thanks to Josh Diehl for his feedback on an early version of this post.

Coaching for Performance

LifeLabs workshop on Coaching at CTO School

LifeLabs workshop on Coaching at CTO School

Last Monday, July 8th,  LifeLabsNewYork gave a workshop on coaching at NY CTO School Meetup.   LeeAnn Renninger, PhD, the founder of LifeLabs, went through some research and best practices around how the best managers approach coaching and growing their people.

The first part was about how to coach an employee through a specific problem, and the second part was about goal setting.  Since there is a lot of stuff out there on goal setting, this post will focus on coaching through a specific problem.

The key to being a good coach is asking questions.   Research shows great managers ask many more questions than average ones.  They guide their employees through to help them discover the answers for themselves, so they can grow to trust themselves to solve problems.
Importantly, helping people discover things for themselves results in much better learning.   Research shows that when students in an academic setting are encouraged to figure out things for themselves, they are able to integrate skills at a MUCH higher rate (15% vs 75%).  So by coaching and letting people learn how to think through the problems for themselves helps them learn how to tackle similar problems for the future.

Asking Questions and the Questions Funnel

Asking questions is a skill in itself, and it’s important to be authentically curious about the answers to your questions.  The key is to try to understand how your report thinks and where they are coming from.  It is also very important to ask open ended questions to encourage talking and thinking through options.  Open ended questions are ones that have an answer more than a Yes / No, they ask to explain about something or to talk about alternatives.

There are three types of questions / phases you should go through during a coaching session – Clarifiers, Probes, and Deepeners.  You can visualize it as a funnel:

Copyright LifeLabsNewYork

Copyright LifeLabsNewYork

Clarifiers.  First, you want to clarify that you understand the person’s perspective and fully understand the situation.    Some examples:

  • Who, what, when, where, how?
  • Tell me more about this problem.
  • Such as? Can you give an example?
  • Can you walk me through your thinking process?
Probes.  Then you want to probe for alternatives and potential obstacles.  Here you try to help your report explore the problem space.  A lot of times your report already had the solutions they need, and just need to think through them and get confidence or decide which one to choose.  This is also a good chance to share some of your experiences and perspectives, but try to keep the focus on the employee and his / her ideas.  They may have their own way of accomplishing the same outcome, and it’s the outcomes that are important.
  • What have you tried so far?
  • What are some of the potential challenges you see?
  • What is stopping you?
  • If you could do it over again, what you would you do differently?
  • How else could a person handle this?
Deepeners.   These questions help move the conversation to what actions the report will take, or what can be learned for the future.
  • What action will you take?
  • What are your next steps?
  • Are there advantages to starting sooner?
  • What’s the backup plan?

During the coaching session, it is important to reflect back to the person you are talking to what you heard.   These are called playbacks – you are playing back what you heard.  This both makes sure that you truly understood the other person, and also helps the person to feel understood.   This is very important, especially during clarifier question phase.  Use something like “I just want to make sure I got it right, let me paraphrase what I just heard…”

To summarize, coaching is a key technique of a manager to help grow your team members over time.  You are investing in the long-term success of your employees and are helping them learn how to solve problems for themselves.  Your role here is less of a mentor (somebody who guides based on their experience), but more of a collaborator.  People often already know how to go about to solve a problem and simply need a structured way to think through the problem and get confidence around the best way to proceed.  As a coach, you guide their thinking and support them in their action plan.

We concluded with reminding that as a manager, you are not ALWAYS a coach, you wear many hats.   Sometimes you have to tell people what to do, sometimes you have to share your past experience to mentor them, and sometimes you have to play psychologist (though try to avoid that one).
LeeAnn recommended the following books to learn more

BTW, I’ve taken at least five different LifeLabs (www.lifelabsnewyork.com) courses over the last year.  They have some great public classes (and private corporate training) for a lot of soft skills.  This workshop was  based on some corporate training workshops they have given to companies such as Etsy, Google, Victoria’s Secret, Whole Foods, and others.

What is CTO School?

CTO School is an organization and a meetup I founded and was co-organizing over the last two years, which has been a very fun project for me.  This post summarizes how it came about, and its mission.

Introduction – Challenges of a Startup CTO

The role of a technical leader is a fundamentally challenging one, but fundamentally, his or her primary job is to make sure the company’s technology strategy serves its business strategy.    This involves a complex set of skills that span deep technology knowledge, process management skills, and managerial, leadership and executive skills.

For a technical leader at a startup, a key challenge is that in most cases they are unprepared for the role, since most of the time the CTO is somebody who only has some of the skills they need to excel at the role.    When one finds him or herself in the role, it’s often difficult to recognize that there are whole areas of your job that are important, and you can find that out through making mistakes.

When somebody is promoted to technical lead position at a larger company he or she has other people around him, managers and mentors who have done this job before.  They are typically still part of the technology hierarchy, which provides somewhat of a shielding from other aspects of the business.    Startup CTO does not have this built-in support network.  Additionally, the startup CTO role has more variety in it, since you are more likely to be involved in business strategy decisions, and are in many cases part of the executive team.   On top of this, if the startup is having any success, both technology, team and process need to scale rapidly, so the role is constantly changing…

While technology is a strength CTOs typically start with, technology landscape is vast and is rapidly changing, so one must continuously keep up in their specific chosen platform, as well as round out their knowledge over time.

CTO School is a forum that aims to help startup CTOs and Technical Leaders become better in their complex roles.  It aims to bring together to share ideas, mistakes, and best practices, and help them form that network of peers who they can learn from and share experience with.

A Brief History

CTO School started originally in the fall of 2010 as a series of seminars.   The spark to start this endeavor was provided by an April 2010 blog post by Charlie O’Donnel, and we collaborated with him and NextNY group (http://www.nextny.org) to run a five-session series.

This five-session series turned out to be very popular and fun for all involved.   AOL Ventures provided space at AOL headquarters, and we had over a hundred attendees for most sessions.   It was clear that this type of group met a real need.

We had several types of talks, from one-person for the whole evening, to two unrelated topics, to panel discussions.   After the talks, we also went out for a beer and some food.    After the first session, another four-session series was run in 2010 in Skylabs downtown in the financial district, and this is when Peter Bell of PowWow, Kurt Schrader of Intent Media, Liz Crawford of BirchBox and me, Jean Barmash of EnergyScoreCards decided to keep this going as more of an ongoing meetup.  The fifth session of the fall program was also our first meeting as a meetup was in December 2011, and we now have over 300 members in the meetup.

Some 2010 Sessions were video taped, and some people took notes for 2011 sessions – links below.

Goals for CTO School

CTO School is a meetup for CTOs, VP Eng,  or Tech Leads that are interested in improving their skills and learning from each other.  We are more focused on startups in terms of topic selection, but welcome all technologists.   The goal is to have an ongoing forum for senior technical people (esp. in startups) to discuss their issues, meet each other, and share expertise.   Secondary goals is to help grow the New York technology and startup ecosystem by building better technical leadership capacity.

The focus is both on people who are already in CTO / VP Eng roles AND people who are on their way there.  One of the “use cases” is to have CTO school be an organization that a CTO encourages his lieutenants to join CTO school.  If they attend meetings regularly, over time they will get exposed to the different areas of technical leadership, which will prepare them for more senior roles.

Who can join?

The ideal member is a CTO, VP. Eng, Architect, Technical Lead, or somebody with a few years of experience that’s on the cusp of becoming the technical lead.   Membership is by approval; we do not let in non-technical people, and try to gently discourage people who are just starting out their careers (though we sill let them in if they feel strongly about joining).     This is one of the ways we try to keep this group free of non-relevant content, including trying to keep recruiters and business people looking for technical co-founders away from this group.    We want this to be a group of peers who can really relate to each other.   Please join at http://www.ctoschool.org.

In New York, we meet on the second Monday of every month and have a speaker on relevant topics, from general topics, like how to create great engineering culture, to technology areas – e.g. comparison of cloud platforms.

Core Skill Areas for CTOs – Technology, Process and Management

We view CTO role as combining three broad skills areas:

  • Technology Skills.  This one is the easiest to define, they need to be a great technologist.    This involves knowing one or more technology platforms, software libraries, components, and of course knowing how to program.
  • Process Skills.  Here we are talking about Software Development Life Cycle and all related processes such as Quality Assurance, requirements gathering, Product Management, User Experience and Design. etc.    CTOs with some experience typically have been part of teams and have seen some of these processes at work, though it’s a much different task to participate in a part of the  process vs understanding the different tradeoffs behind defining a process.    Scrum / Agile comes under this area.
  • Executive / Managerial Skills.  These are more general skills.  They relate to recruiting, motivating, and managing people, with interacting with business stakeholders.   Skills like public speaking, evangelizing your product, helping out with sales, networking, etc.  This is the area that most CTOs probably have the least experience with.

When thinking about the key skill sets that either a CTO, VP of Engineering, or Technical lead needs to have to manage a small to moderately sized team, there are three skills areas that seem to be the most important. This is complicated by the fact that the role’s requirements are very specific to a particular company and depends both on the purpose of the company and the products it sells.  CTO School’s goal is to strike the right balance between the three core skill areas.

When at one of the meetings we asked for feedback on these three areas, we overall got positive feedback.   One thing that people made clear is that what they found especially valuable is the real-life experience.   There are books on technology management, books on specific technology areas, but having somebody who has worked on something or stuggled with something and overcame it is very valuable to see in-person.

Note:  There is some great work done on the role of CTOs available on the Internet.  One particularly interesting framework is Tom Berray’s, which talks about four quadrants of the CTO role, which is also a useful way to look at the role (summary by CTO of Amazon).   They seem to focus to larger companies more than smaller teams, and many attributes can be mapped to our skill areas.    It’s useful to study those frameworks to understand the role more fully.

Conclusion.  

This is the current vision and focus.  We have many ideas for this to grow in the future, but the initial goal is to continue having great content and to have this become a very valuable resource to members and the community.

References & Resources:

More Great Articles on Role of CTO & Startup CTO

More interesting links:

CTO School Organizers – Jean, Peter, Kurt & Liz

LUXr – Lean User Experience Residency

My company EnergyScoreCards has been taking part in the LUXr program in the fall of 2011.  LUXr stands for Lean User Experience Residency, and it’s a program that helps companies to think through their user experience and product issues in  semi-structured format.

The program is 10 sessions over 10 weeks.  Ours was on a Friday from 10 to 4.  Each session starts with an external speaker talk for an hour about their area of expertise, whether user testing, or agile product management, metrics, or other topics relevant to lean startups.   The speaker often stays for a few hours or even the whole day, which allows the team to talk to the expert about their specific issues.

After the speaker we do a quick standup meeting about progress made since last week and what the team is planning to do for the day.  After that it’s whatever your team needs to work on.  Josh Seiden, the program director, is around to help you work through blocks, or to help you frame a particular problem you are working on.  He has great experience both on the UX side and product side, so he is very helpful.  Our team has worked on big changes in navigation to our product, new alerting infrastructure, some smaller page redesigns.  Most importantly, the program helps the team build up capacity and gives us confidence to continue doing a better job both managing the product and thinking of way to improve user experience.

LUXr program itself has many branches, including in San Francisco.  Their mission is to help startups become better at producing usable products, and in addition to the residency programs, they have other workshops, and planning things like shorter video classes and other things.

Lastly, I already mentioned that Josh has been amazing as director of the program.  Seems like he intents to stay involved with LUXr, but he just announced a new venture, Proof, a new product development studio, which sounds amazing (his two partners, Giff and Jeff, were speakers at LUXr, so we got to know them a bit – they are awesome).   They have the dream team of New York UX / Product People working with companies to help them design better products and user experience.

I really enjoyed the LUXr program and would recommend it to any company that feels they need to improve their UX and / or product processes, no matter what stage they are in. Of the four companies in this batch, two companies were pretty late stage (we are two and a half years in with a reasonably mature product), and two were just starting up.

How to Add Security to Grails Links / Override Existing Taglib

I wanted to add a parameter to tag so I can have it optionally have it render depending on a user’s security. I wanted to say something like

I thought that since my site is already sprinkled with g:link tags, the easiest way would be to override the original taglib. However, since the TagLibs are defined as closures, I couldn’t just subclass the original ApplicationTagLib.

I have already been able to override existing tags from before by bringing the implementation into the new Tag – grails gives you a message when it sees two tags with the same name defined, but picks the one from your application, correctly assuming that you are overriding the original implementation.

I found from this thread that to get a reference to the original taglib, I had to go through Spring applicationContext. In my initial implementation, I got a reference to the original taglib, and tried to call the closure. However, that caused a StackOverFlowException.

Thanks to the Grails User list, and more precisely Ian Roberts (post here), I got help, so here is my final code (as you can see, I am using Shiro).

import org.codehaus.groovy.grails.plugins.web.taglib.*
import org.apache.shiro.SecurityUtils

class ModifiedLinkTagLib {

static namespace = "g"

def link = { attrs, body ->
   if (attrs.role=="edit" && (!SecurityUtils.subject.hasRole(EDIT_ROLE))) {
     return
   }
   def applicationTagLib = grailsApplication.mainContext.getBean('org.codehaus.groovy.grails.plugins.web.taglib.ApplicationTagLib')
   applicationTagLib.link.call(attrs, body)
  }
}

Meeting Ticker

This is brilliant – a ticker that shows you how much the meeting costs.

Via Matt Raible’s post on Passionate Programmer book.

New Focus

I am excited to announce that after spending years reading and learning about how to start companies, I am finally doing something about it. I recently left Alfresco and am in the process of starting a new service that will create tools for measuring energy efficiency of buildings.

A bit of a change in focus, but as the technical part of the founding team, I hope that my experience will be useful as I am now building out a prototype version of the service.

I won’t promise that I will blog more, though I certainly intend to, including more about the service itself and what I am working on. One of the things I’ve been thinking about recently is the transition from an employee, which I have been all my professional life, to an entrepreneur and how the two are different.

StackOverflow Rocks!

StackOverflow is site created by Jeff Atwood of the excellent blog CodingHorror.com together with Joel Spolsky of Joel On Software that allows programmers to ask each other questions in an open forum. 

Very quickly, it’s become one of the best technical resources on the web, now whenever I have a generic question about programming, I go there and do a quick search (you can also map it as OpenSearch in your browser).  Of the past 10 such situations, there was already a thread there on the topic, with majority of responses very well reasoned.  A lot of great developers hang out there and there is a ton of useful information.  

In addition to frequenting the site, I also listen to the StackOverflow Podcast – the Signal to Noise ratio isn’t amazing, but it’s a good background conversation to listen to while doing something else.

NY Architecture Conference Presentation

Last week I spoke at the NY IT Architecture Conference.  My main topic was on Content-Oriented Architectures (copy of presentation here).  It addressed how the Content Repository is a new abstraction better suited for content applications, one that combines advantages of the database (i.e. transactions, rich content modeling, data integrity), and file systems (hierarchy, ability to store large files), while adding some new constructs not easily present in either, i.e. granular item-level security, workflow, content transformations, etc.    There are a few good resources I found in preparing for the talk. One that stood out was a description of what content repositories offer you at Gadgetopia here.

In addition to the main topic, I also participated in Rich Internet Applications Panel, talking about interesting technologies such as Adobe Flex, Ajax and its cousin Comet.

As to the conference overall, I have to say that since my primary interest is software architecture and technology, I didn’t always find interesting topics that were immediately relevant (a few of the ones i really wanted to attend overlapped with my talks :-().  However, this did allow me to branch out and hear about topics I am not as directly involved with, such as proving value of architecture, and establishing Architecture as a profession within our industry.

Upcoming Speaking Engagement June 12th – Enterprise 2.0 Conference

By the way, I am also speaking at the Enterprise 2.0 Conference.   The topic is Programmable Web: Consequences for the Enterprise.  I am on at 9:30 am on Thursday, June 12th.   Here is the Description:

The internet is becoming programmable. Many sites are providing data access APIs as the Software As as A Service paradigm shift is taking place. Mashups have been around for years, but recently social networking sites have also joined the fray by opening up their own APIs. Facebook is one of the leaders of the movement. Having released their API in May of 2007, there are many thousands of Facebook applications in use today. Google countered with OpenSocial Project. We will discuss the internet as an application development platform in general, and look at how some of the leading social networking APIs work. We will then discuss how these concepts can be applied in the enterprise to enable better information sharing and collaboration.