DevOps for Data Science – DevOps Maturity


In this series on DevOps for Data Science, I explained what DevOps is, and given you lots of resources to go learn more about it. Now we can get to the details of implementing DevOps in your Data Science Projects.

Consider that the standard Software Development Lifecycle (SDLC) with Data Science algorithms or API’s added in looks something like this:

In traditional DevOps, each phase of the SDLC is aware of the “downstream” phases – that’s the People (discussions and meetings), Process (procedures and frameworks), and Products (Tools and software) part of DevOps. Also note that Monitoring is not just monitoring the performance of the end-released applications, it’s also now monitoring the SDLC itself. All of this awareness is the “Shift-Left” mentality I mentioned earlier as essential to DevOps for Data Science.

But let’s dig into that a bit. The “Configure” phase is easy enough to think about for the Data Science team. They want to be involved in the design of the solution anyway, so that they ensure the right questions are being asked, the right data is available (or can be obtained), and all the rest of the project parameters. And of course the Data Science team is part of the “Code” phase, so that’s just built-in.

But “build”? What does that even mean to the Data Scientist? Testing? How does one build a unit test, functional test, security test, and other tests, into a Data Science algorithm? And how do  you use Continuous Integration (CI) and Continuous Delivery (CD) into a Data Science deployment? Packaging, Deployment – these are all things that a Data Scientist doesn’t normally have to consider in their day-to-day work.

Not to worry – we’ll get to all these phases. In fact, many shops still struggle with DevOps, even where there are no Data Science aspects. What I have found that helps the most is not to try and do everything at once. Develop a “Maturity Model” that you can aspire to, and take it one step at a time. Let that settle into your regular SDLC, and then move on.

For Data Science, I find this progression works best – taking these one step at a time, and building on the previous step – the entire series is here:

  1. Infrastructure as Code (IaC)
  2. Continuous Integration (CI) and Automated Testing
  3. Continuous Delivery (CD)
  4. Release Management (RM)
  5. Application Performance Monitoring
  6. Load Testing and Auto-Scale

In the articles in this series that follows, I’ll help you implement each of these in turn.

 

(If you’d like to implement DevOps, Microsoft has a site to assist. You can even get a free offering for Open-Source and other projects: https://azure.microsoft.com/en-us/pricing/details/devops/azure-devops-services/)

Advertisements

Published by: BuckWoody

Buck Woody works on the Microsoft Cloud and AI Team, and uses data and technology to solve business and science problems. With over 35 years of professional and practical experience in computer technology, he is also a popular speaker at conferences around the world; author of over 700 articles and seven books (databases, machine learning, and R) sits on various Data Science Boards at two US Universities, and specializes in advanced data analysis techniques. He is passionate about mentoring and growing the next generation of data professionals. Specialties: Data, Data Science, Databases, Communication, Teaching, Speaking, Writing, Cloud Computing, Security Clifton's Strengths: Individualization, Learner, Connectedness, Positivity, Achiever, Ideation

Categories Data Science, Artificial Intelligence and Advanced Analytics Project Management, DevOps, Learning Data ScienceTags5 Comments

5 thoughts on “DevOps for Data Science – DevOps Maturity”

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.