diff --git a/content/posts/2024-05-27-plumbing-decisions-and-automation-de-hyping-data-and-ai/amateur-versus-professional-data-and-ai-otter.webp b/content/posts/2024-05-27-plumbing-decisions-and-automation-de-hyping-data-and-ai/amateur-versus-professional-data-and-ai-otter.webp new file mode 100644 index 000000000..1c77b2660 Binary files /dev/null and b/content/posts/2024-05-27-plumbing-decisions-and-automation-de-hyping-data-and-ai/amateur-versus-professional-data-and-ai-otter.webp differ diff --git a/content/posts/2024-05-27-plumbing-decisions-and-automation-de-hyping-data-and-ai/index.md b/content/posts/2024-05-27-plumbing-decisions-and-automation-de-hyping-data-and-ai/index.md new file mode 100644 index 000000000..561ef3205 --- /dev/null +++ b/content/posts/2024-05-27-plumbing-decisions-and-automation-de-hyping-data-and-ai/index.md @@ -0,0 +1,64 @@ +--- +title: "Plumbing, Decisions, and Automation: De-hyping Data & AI" +author: Yanir Seroussi +type: post +date: 2024-05-27T02:00:00+00:00 +url: /2024/05/27/plumbing-decisions-and-automation-de-hyping-data-and-ai/ +cover: + relative: true + image: amateur-versus-professional-data-and-ai-otter.webp + alt: contrasting an amateur and a professional otter; the amateur asks about tools, the professional asks about plumbing, decisions, and automation +summary: Three essential questions to understand where an organisation stands when it comes to Data & AI (with zero hype). +tags: + - artificial intelligence + - business + - career + - data engineering + - data science + - data strategy + - startups +--- +Data & AI health is hard to define. Recently, it occurred to me that its essence can be distilled with three questions: +1. **Plumbing:** What's the state of your data engineering lifecycles? +2. **Decisions:** How do you use descriptive, predictive, and causal modelling to support decisions? +3. **Automation:** How do you use AI to automate processes? + +These questions help identify gaps and opportunities. While each question focuses on the present state, it's natural to follow up with plans for a brighter future. + +In practice, you would go deep on each area. Each question is a door that leads to a corridor with many more doors. + +## Amateurs versus professionals + +If you've ever worked with data, you'd have a sense of what amateur and professional answers to the above questions may look like. In practice, answers are multifaceted and fall on a continuum. But here are some simplified examples from each end of the continuum: + +| | Amateur | Professional | +|----------------|-----------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------| +| **Plumbing** | Rudimentary pipelines, manually-populated spreadsheets | All necessary data is trustworthy and available on tap | +| **Decisions** | Relying on one-off charts and models, along with the intuition of HiPPOs (highest-paid persons' opinions) | Relying on relevant data and modelling efforts that are proportional to the gravity of each decision | +| **Automation** | Superficial use of off-the-shelf tools | Deep, mindful integration of tech to replace manual work where it delivers the most value | + +## Going down the rabbit hole + +The three areas pretty much define my career, but there is always much more to learn. The main message of this post is that little has changed since [Harrington Emerson uttered these words in 1911](https://yanirseroussi.com/2017/10/15/advice-for-aspiring-data-scientists-and-other-faqs/): + +> As to methods, there may be a million and then some, but principles are few. The person who grasps principles can successfully select their own methods. The person who tries methods, ignoring principles, is sure to have trouble. + +(OK, one thing did change – Emerson used _man_ rather than _person_, but I fixed it for him.) + +You can explore further with these posts: +1. **Plumbing:** Fully understanding [the data engineering lifecycle](https://yanirseroussi.com/til/2024/04/05/the-data-engineering-lifecycle-is-not-going-anywhere/) is more important than mastering a single tool. +2. **Decisions:** According to [my 2018 definition](https://yanirseroussi.com/2018/07/22/defining-data-science-in-2018/), this is what data science is all about. There's endless depth to building descriptive, predictive, and causal models. But the key to rising above tool hype is understanding [the _why_ of data science](https://yanirseroussi.com/2016/09/19/ask-why-finding-motives-causes-and-purpose-in-data-science/), which is to support decisions. +3. **Automation:** The term _AI_ is around peak hype right now. This makes it easy for cynics to dismiss the over-excited claims of AI proponents. Avoid cynicism – [simply think of AI as automation](https://yanirseroussi.com/til/2023/10/06/artificial-intelligence-was-a-marketing-term-all-along-just-call-it-automation/) and [understand that relentless but mindful automation is key to success in our world](https://yanirseroussi.com/til/2024/05/25/adapting-to-the-economy-of-algorithms/). + +## More questions to probe the Data-to-AI health of startups + +This post is a slight detour from the series on [my Data-to-AI Health Check for Startups](https://yanirseroussi.com/data-to-ai-health-check/). I figured it's a valuable detour since I now see the triad of Plumbing, Decisions, and Automation as the essence of Data & AI health for any organisation. + +Previous posts in the series: + +* [Assessing a startup's data-to-AI health: Overview and motivation](https://yanirseroussi.com/2024/04/22/assessing-a-startups-data-to-ai-health/) +* [Business questions to ask before taking a startup data role](https://yanirseroussi.com/2024/05/06/business-questions-to-ask-before-taking-a-startup-data-role/) +* [Probing the People aspects of an early-stage startup](https://yanirseroussi.com/2024/05/13/probing-the-people-aspects-of-an-early-stage-startup/) +* [Question startup culture before accepting a data-to-AI role](https://yanirseroussi.com/2024/05/20/question-startup-culture-before-accepting-a-data-to-ai-role/) + +[You can download a guide containing all the questions as a PDF](https://yanirseroussi.com/data-to-ai-health-check/). I'm still planning to cover Processes & Project Management next – hopefully I won't get detoured again. Feedback is always welcome!