🧐 agile monocle S01E04: Is data science not agile enough?

Silos, silos everywhere.

Every time a new capability starts to get traction, we automatically fall into the specialization trap: we need teams of specialists and then we don’t know how to integrate these specialists into our agile teams and projects, and then “data science is not agile”.

As a former “data-something guy” I find everything about the integration of data science at the organizational level very, very interesting.

This is a longer episode (and I have a draft for a sequel too) because it’s a subject I’ve been researching and working on for a while, and I’m interested on working on it in the future, so I did some homework.

Data is the new fire, not the new oil

I think what makes data science slightly different from previous capabilities we had to integrate in our products and projects is that we live in an era where we are witnessing the maximum gap between the amount of data available, both publicly and privately, and the amount of people who understand the basics of what data is.

So we have lots of data, and an increasing amount of it, and a few people really able to understand it — and no, data is by no means the “new oil”, a more apt metaphor is “data is like fire”.

So, like fire, data can be used for good and for bad, it can be harmful and it can be helpful, but above everything else, data must be understood by far more people, so before delving into tips and tricks of how to integrate data science into our products and project, it’s worth thinking about the subject of data literacy.

Drawing from this interesting fireside chat about data literacy from the Data Visualization Society, I’ll try to summarize some key concept about what data literacy means and how you should act about it:

“91% on Americans feel like they lost control of their data, they don’t know what they can do to get it back […] for most people data are impenetrable and they are only for people with ‘rarified skills’, we [as data practitioners] have expected for people to come at our table, but it’s incumbent upon us [as data practitioners] to build communities around data literacy”

Mary Aviles
Freelance multi-sector human experience strategist and researcher

“With ‘data empathy’ you can understand the flow of data in the organization but most importantly you can understand the audience so you can understand that everyone doesn’t use data in the same way […] You don’t need to be a data scientist in order to be data literate […]”

Allen Hillery
Columbia University, Adjunct Faculty

“[…] people and context are important […] how data can be used in their day-to-day life? […] realize that every single situation or most situations and decisions that you make are based on data and if you can find a way to read and interpret that data for yourself you’ll probably find that you are making better decisions in the long run.”

— Rama Raphalalani
Monitoring, Evaluation, Research & Learning Officer, Data Innovator

🛠 Now, try this: Based of that conversation, I found this worksheet. There is a great companion article called “The UX of Data” explaining how and why you should reduce the gap of confusion regarding data in your organization.

🛠 Or try this: It might be useful to apply an analytics maturity model (there, I said it…) to brainstorm about what’s the level of understanding about data in your organization.

It’s data, not pins

As I was mentioning at the start, with new needs come new capabilities, with new capabilities we automatically switch to “experts” in a certain field, we create new job titles, and then everything gets reduced to “we need a data [something] person” or “we need a team of data [something]”.

This is what I would call, based on the previous topic of data literacy, an “outsourcing of data literacy”: unless we take a step to improve our overall understanding of data in our context, we’ll be always running circles around how to integrate data in our work, projects and products.

There are ways out of this vicious cycle: first of all, let’s stop considering the data-something person as a absolute specialist, removed by any context. The data-something person lives and breathes inside the context of your industry, your market, your product, your team.

This is important for two aspects:

  • On the level of the individual, we might want to strive for more generalists, end-to-end data-something people. This is hard to do because we live in a world that still values specialists over generalists, but “Working end-to-end provides increased context. While specialized roles can increase efficiency, it reduces context (for the data scientist) and leads to suboptimal solutions.”;

  • On the team level, whether you are an internal team in a bigger company or a data-something agency, you should avoid to become a “data science pin factory”. Again, we fall trap of old, industrial models in order to rationalize and optimize the workflow in terms efficiency in spite of the efficacy. “The goal of assembly lines is execution. […] But the goal of data science is not to execute. Rather, the goal is to learn and develop profound new business capabilities. […] With data science, you learn as you go, not before you go.”

How to break these data science silos for people, teams and organizations?

Flex your skills, not your job titles

We used to think that specialization was a thing for bigger companies: bigger companies usually have the luxury to hire highly specialized people and then put hordes of managers on top of them in order to handle to communication overhead.

And hence we usually think that smaller companies, not having that luxury, are inherently more fluid and generalists in the way they operate.

That is not always the case, as we have:

  • Smaller companies or teams that are quite exactly a replica of the above mentioned “data science pin factory”;

  • Bigger companies that are realizing that they can’t afford to handle the communication overhead anymore.

So, if Netflix is realizing they can’t put people and teams in silos, why should you?

“Below are three broadly defined personas to help illustrate some of the different backgrounds, motivations, and activities of individuals in the analytics role at Netflix. […] Ultimately, these skills are all on a continuum, some broad and some deep, and these are just a few examples of such expertise. So if you find yourself connecting with any part of these descriptions, the analytics role could be for you.”

I’ll take the “skills on a continuum” from Netflix to introduce the concept of “flexing”:

“As a team goes about their work, the need to flex may arise to address disruptions, bottlenecks, and issues with flow through the system. Flexing in an agile context is when a team member (or members) temporarily contribute to a role or activity that is not their primary function.”

🛠 Now, try this: map out you process, line up all the people that contribute, pinpoint where, how much and with what level of expertise. See the gaps, see the opportunities, start to tackle how can you break down the sources of bottlenecks, waiting times, incomprehension, dependencies or simple lack of skills. For maximum impact and fun you could also do this exercise between multiple teams.

Is data science agile?

I’ll be brief: ultimately, agile values and principles are based off empiricism, so if you do data science you are inherently approaching your work with the scientific method so there is simply no place for the statement “data science is not agile”.

What you actually mean with “data science is not agile” is that you can’t fit the result of a (probably disconnected) data science team or vendor to the results of another (probably dysfunctional) Scrum team.

I hope that this issue of agile monocle helped you to take a few step backs from your sprints and see a bigger picture. I really should write a sequel because, you guessed it, there is a bit more to say about it.

What do you see?

This newsletter is about a monocle, not about a monologue. I want to hear from you so if you want to ask about anything or suggest a topic, you can reply to the newsletter, comment here below, or send a request here. I’d like to make it as much interactive as possible. Don’t make me open a Slack channel.

See you next week, agile monocle episode 5 will be out on Monday, October 12th.

(header image: Waldemar Brandt on Unsplash)